cURL not workig as expected on php shell_exec() - php
I'm trying to download a file from an external server, on which I submit a form and returns a download file (PDF).
Copying the request as cURL from the Network tab in Chrome works fine in the Terminal (it downloads the PDF) but not in shell_exec() (I get the submit form page as output).
Here are the verbose output from both curls.
This one belows works fine:
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 107.180.12.118...
* TCP_NODELAY set
* Connected to operaciones.ahmex.com.mx (107.180.12.118) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
} [5 bytes data]
* (304) (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* (304) (IN), TLS handshake, Server hello (2):
{ [102 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [2881 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: OU=Domain Control Validated; CN=operaciones.ahmex.com.mx
* start date: May 25 17:30:17 2019 GMT
* expire date: Jul 24 17:04:13 2020 GMT
* subjectAltName: host "operaciones.ahmex.com.mx" matched cert's "operaciones.ahmex.com.mx"
* issuer: C=US; ST=Arizona; L=Scottsdale; O=GoDaddy.com, Inc.; OU=http://certs.godaddy.com/repository/; CN=Go Daddy Secure Certificate Authority - G2
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x55f8419d44c0)
} [5 bytes data]
> POST /potentials/generar_certificado HTTP/2
> Host: operaciones.ahmex.com.mx
> authority: operaciones.ahmex.com.mx
> cache-control: max-age=0
> origin: https://operaciones.ahmex.com.mx
> upgrade-insecure-requests: 1
> content-type: multipart/form-data; boundary=----WebKitFormBoundary1IwLan7m4erUfeoh
> user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.87 Safari/537.36
> sec-fetch-mode: navigate
> sec-fetch-user: ?1
> accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3
> sec-fetch-site: same-origin
> referer: https://operaciones.ahmex.com.mx/potentials/
> accept-encoding: gzip, deflate, br
> accept-language: en-US,en;q=0.9
> cookie: ci_session=(I removed this)
> Content-Length: 971
>
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
} [5 bytes data]
* We are completely uploaded and fine
{ [5 bytes data]
100 971 0 0 100 971 0 701 0:00:01 0:00:01 --:--:-- 700
100 971 0 0 100 971 0 407 0:00:02 0:00:02 --:--:-- 406< HTTP/2 200
< date: Tue, 13 Aug 2019 14:57:41 GMT
< server: Apache
< x-powered-by: PHP/5.6.40
< content-disposition: attachment; filename="CER-0196084.pdf"
< cache-control: private, max-age=0, must-revalidate
< pragma: public
< vary: Accept-Encoding,User-Agent
< content-encoding: gzip
< content-type: application/x-download
<
{ [5 bytes data]
100 12590 0 11619 100 971 3823 319 0:00:03 0:00:03 --:--:-- 4141
100 65749 0 64778 100 971 15423 231 0:00:04 0:00:04 --:--:-- 15650
100 65749 0 64778 100 971 12452 186 0:00:05 0:00:05 --:--:-- 12710
100 65749 0 64778 100 971 10443 156 0:00:06 0:00:06 --:--:-- 13442
100 65749 0 64778 100 971 8990 134 0:00:07 0:00:07 --:--:-- 13439
100 65749 0 64778 100 971 7893 118 0:00:08 0:00:08 --:--:-- 10288
100 78659 0 77688 100 971 8609 107 0:00:09 0:00:09 --:--:-- 2675
100 104k 0 103k 100 971 11625 106 0:00:09 0:00:09 --:--:-- 10514
* Connection #0 to host operaciones.ahmex.com.mx left intact
This one not:
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 107.180.12.118...
* TCP_NODELAY set
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to operaciones.ahmex.com.mx (107.180.12.118) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
} [5 bytes data]
* (304) (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* (304) (IN), TLS handshake, Server hello (2):
{ [102 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [2881 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: OU=Domain Control Validated; CN=operaciones.ahmex.com.mx
* start date: May 25 17:30:17 2019 GMT
* expire date: Jul 24 17:04:13 2020 GMT
* subjectAltName: host "operaciones.ahmex.com.mx" matched cert's "operaciones.ahmex.com.mx"
* issuer: C=US; ST=Arizona; L=Scottsdale; O=GoDaddy.com, Inc.; OU=http://certs.godaddy.com/repository/; CN=Go Daddy Secure Certificate Authority - G2
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x55a28da8e4c0)
} [5 bytes data]
> POST /potentials/generar_certificado HTTP/2
> Host: operaciones.ahmex.com.mx
> authority: operaciones.ahmex.com.mx
> cache-control: max-age=0
> origin: https://operaciones.ahmex.com.mx
> upgrade-insecure-requests: 1
> content-type: multipart/form-data; boundary=----WebKitFormBoundaryWbbB3yby9oTZCuNV
> user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.87 Safari/537.36
> sec-fetch-mode: navigate
> sec-fetch-user: ?1
> accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3
> sec-fetch-site: same-origin
> referer: https://operaciones.ahmex.com.mx/potentials/
> accept-encoding: gzip, deflate, br
> accept-language: en-US,en;q=0.9
> cookie: ci_session=(I removed this)
> Content-Length: 972
>
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
} [5 bytes data]
* We are completely uploaded and fine
{ [5 bytes data]
< HTTP/2 200
< date: Tue, 13 Aug 2019 15:01:02 GMT
< server: Apache
< x-powered-by: PHP/5.6.40
< vary: Accept-Encoding,User-Agent
< content-encoding: gzip
< content-length: 3120
< content-type: text/html; charset=UTF-8
<
{ [5 bytes data]
100 4092 100 3120 100 972 3312 1031 --:--:-- --:--:-- --:--:-- 4343
* Connection #0 to host operaciones.ahmex.com.mx left intact
Here's the cURL, just removed the cookie session
curl 'https://operaciones.ahmex.com.mx/potentials/generar_certificado' -H 'authority: operaciones.ahmex.com.mx' -H 'cache-control: max-age=0' -H 'origin: https://operaciones.ahmex.com.mx' -H 'upgrade-insecure-requests: 1' -H 'content-type: multipart/form-data; boundary=----WebKitFormBoundaryWbbB3yby9oTZCuNV' -H 'user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.87 Safari/537.36' -H 'sec-fetch-mode: navigate' -H 'sec-fetch-user: ?1' -H 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3' -H 'sec-fetch-site: same-origin' -H 'referer: https://operaciones.ahmex.com.mx/potentials/' -H 'accept-encoding: gzip, deflate, br' -H 'accept-language: en-US,en;q=0.9' --data-binary $'------WebKitFormBoundaryWbbB3yby9oTZCuNV\r\nContent-Disposition: form-data; name="nombre2"\r\n\r\nSergio\r\n------WebKitFormBoundaryWbbB3yby9oTZCuNV\r\nContent-Disposition: form-data; name="apellido_paterno"\r\n\r\nMendoza\r\n------WebKitFormBoundaryWbbB3yby9oTZCuNV\r\nContent-Disposition: form-data; name="apellido_materno"\r\n\r\nNegrete\r\n------WebKitFormBoundaryWbbB3yby9oTZCuNV\r\nContent-Disposition: form-data; name="rfc"\r\n\r\nMENS8804144J4\r\n------WebKitFormBoundaryWbbB3yby9oTZCuNV\r\nContent-Disposition: form-data; name="importe"\r\n\r\n1500000\r\n------WebKitFormBoundaryWbbB3yby9oTZCuNV\r\nContent-Disposition: form-data; name="banco"\r\n\r\nSantander\r\n------WebKitFormBoundaryWbbB3yby9oTZCuNV\r\nContent-Disposition: form-data; name="estado"\r\n\r\nJalisco\r\n------WebKitFormBoundaryWbbB3yby9oTZCuNV\r\nContent-Disposition: form-data; name="municipio"\r\n\r\nZapopan\r\n------WebKitFormBoundaryWbbB3yby9oTZCuNV\r\nContent-Disposition: form-data; name="action"\r\n\r\nEnviar\r\n------WebKitFormBoundaryWbbB3yby9oTZCuNV--\r\n' --compressed -o cert.pdf
if you are using php why not try readfile method?
If you want to use console wget is better for download instead cURL
Related
Error when filling database with false data with faker in laravel 8
I started a project with laravel 8, I was adjusting the seeders to generate fake data with faker in the database, I have a table with images which I generate random images with faker, my ImageFactory.php file is like this: namespace Database\Factories; use App\Models\Image; use Illuminate\Database\Eloquent\Factories\Factory; class ImageFactory extends Factory { /** * The name of the factory's corresponding model. * * #var string */ protected $model = Image::class; /** * Define the model's default state. * * #return array */ public function definition() { return [ 'url' => 'posts/' . $this->faker->image('./public/storage/posts', 640, 480, null, false) ]; } } when executing the command: php artisan migrate:fresh --seed generates this error in the console: copy(https://via.placeholder.com/640x480.png/001144?text=corrupti): Failed to open stream: Connection timed out at vendor/fakerphp/faker/src/Faker/Provider/Image.php:121 Looking for a solution to this I found that in the Image.php file that generates error, after the line where CURLOPT_FILE is placed these two lines and the problem would be solved, so that it would look like this: curl_setopt($ch, CURLOPT_FILE, $fp); curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false); //aggregated line curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); //aggregated line $success = curl_exec($ch) && curl_getinfo($ch, CURLINFO_HTTP_CODE) === 200; existente However this did not solve my problem and the error still persists, I am using laravel 8 and php 8. Update I'm using Ubuntu 18.04, running the command curl -vo /dev/null "via.placeholder.com/640x480.png/001144?text=corrupti" I got this: % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 2600:3c00::f03c:91ff:fe60:d792... * TCP_NODELAY set * Trying 45.33.24.119... * TCP_NODELAY set 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to via.placeholder.com (45.33.24.119) port 80 (#0) > GET /640x480.png/001144?text=corrupti HTTP/1.1 > Host: via.placeholder.com > User-Agent: curl/7.58.0 > Accept: */* > < HTTP/1.1 200 OK < Server: nginx/1.6.2 < Date: Sat, 13 Feb 2021 14:36:37 GMT < Content-Type: image/png < Content-Length: 1558 < Last-Modified: Sat, 09 Jan 2021 14:00:02 GMT < Connection: keep-alive < ETag: "5ff9b6e2-616" < Expires: Sat, 20 Feb 2021 14:36:37 GMT < Cache-Control: max-age=604800 < X-Cache: L1 < Accept-Ranges: bytes < { [1126 bytes data] 100 1558 100 1558 0 0 3386 0 --:--:-- --:--:-- --:--:-- 3386 * Connection #0 to host via.placeholder.com left intact
encoding language fails
PHP code below fails to retrieve correct characters when used : echo $html = file_get_contents("http://www.tsetmc.com/tsev2/data/instinfofast.aspx?i=65883838195688438&c=34+"); the result is : ���\�%PKJDA��ۈ�0�o'�z��W�"�7o�E��J:�%�+�=o�h#Ĥ�T�Jv�L�$��IT��1҈IY �B L�g�Mt����� �S]>>�����������j#�Tu97������#"jD��C�3x0�����I"("D�W��Bd��9������J�^ȑ���T��[e��K����r�ZB����r�Z#�w��4G� � �C�b�%8��PR�/���ع���a=�o��s���H�G�
This is because the output is 'gzip'ed, you need to 'unzip' it (see 'Content-Encoding'): D:\Temp>curl -v "http://www.tsetmc.com/tsev2/data/instinfofast.aspx?i=65883838195688438&c=34+" -o output.data % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 79.175.151.173... * TCP_NODELAY set * Connected to www.tsetmc.com (79.175.151.173) port 80 (#0) > GET /tsev2/data/instinfofast.aspx?i=65883838195688438&c=34+ HTTP/1.1 > Host: www.tsetmc.com > User-Agent: curl/7.55.1 > Accept: */* > < HTTP/1.1 200 OK < Cache-Control: public, max-age=1 < Content-Type: text/html; charset=utf-8 < Content-Encoding: gzip < Expires: Sat, 21 Dec 2019 09:43:48 GMT < Last-Modified: Sat, 21 Dec 2019 09:43:47 GMT < Vary: * < Server: Microsoft-IIS/10.0 < X-Powered-By: ASP.NET < X-Powered-By: ARR/3.0 < X-Powered-By: ASP.NET < Date: Sat, 21 Dec 2019 09:42:59 GMT < Content-Length: 155 < { [155 bytes data] 100 155 100 155 0 0 155 0 0:00:01 --:--:-- 0:00:01 662 * Connection #0 to host www.tsetmc.com left intact D:\Temp> unzipping (on Windows): D:\Temp>"c:\Program Files\7-Zip\7z.exe" x output.data output 7-Zip 18.05 (x64) : Copyright (c) 1999-2018 Igor Pavlov : 2018-04-30 Scanning the drive for archives: 1 file, 155 bytes (1 KiB) Extracting archive: output.data -- Path = output.data Type = gzip Headers Size = 10 Everything is Ok Size: 239 Compressed: 155 D:\Temp>type output 12:29:59,A ,9055,9098,9131,9072,9217,9000,3582,17432646,158598409673,0,20191221,122959;;2#100400#9055#9055#20091#1,2#60000#9050#9058#554#1,1#1000#9040#9059#993#2,;66660,417193,674167;13450748,3981898,0,13913408,3519238,1255,9,0,899,11;;;1; D:\Temp>
Varnish seems not to be caching pages at all
I have a PHP Symfony application which is served by nginx. * << BeReq >> 492062 - Begin bereq 492061 fetch - Timestamp Start: 1572337898.474535 0.000000 0.000000 - BereqMethod GET - BereqURL / - BereqProtocol HTTP/1.0 - BereqHeader Host: xxx - BereqHeader X-Forwarded-Host: xxx - BereqHeader X-Real-IP: xxx - BereqHeader X-Forwarded-Proto: https - BereqHeader HTTPS: on - BereqHeader User-Agent: Wget/1.19.4 (linux-gnu) - BereqHeader Accept: */* - BereqHeader X-Forwarded-For: 127.0.0.1 - BereqProtocol HTTP/1.1 - BereqHeader Accept-Encoding: gzip - BereqHeader X-Varnish: 492062 - VCL_call BACKEND_FETCH - VCL_return fetch - BackendOpen 26 boot.default 127.0.0.1 8080 127.0.0.1 43676 - BackendStart 127.0.0.1 8080 - Timestamp Bereq: 1572337898.474685 0.000150 0.000150 - Timestamp Beresp: 1572337903.642006 5.167471 5.167321 - BerespProtocol HTTP/1.1 - BerespStatus 200 - BerespReason OK - BerespHeader Server: nginx/1.14.0 (Ubuntu) - BerespHeader Content-Type: text/html; charset=UTF-8 - BerespHeader Transfer-Encoding: chunked - BerespHeader Connection: keep-alive - BerespHeader Set-Cookie: PHPSESSID=slaurqvo3msh9uklerbht0nd2h; path=/; domain=.xxx; HttpOnly - BerespHeader Cache-Control: max-age=3600, public - BerespHeader Date: Tue, 29 Oct 2019 08:31:39 GMT - BerespHeader Age: 20 - BerespHeader Content-Encoding: gzip - TTL RFC 3600 10 0 1572337904 1572337884 1572337899 0 3600 - VCL_call BACKEND_RESPONSE - TTL VCL 86420 10 0 1572337884 - TTL VCL 86420 3600 0 1572337884 - TTL VCL 140 3600 0 1572337884 - VCL_return deliver - BerespHeader Vary: Accept-Encoding - Storage malloc Transient - ObjProtocol HTTP/1.1 - ObjStatus 200 - ObjReason OK - ObjHeader Server: nginx/1.14.0 (Ubuntu) - ObjHeader Content-Type: text/html; charset=UTF-8 - ObjHeader Set-Cookie: PHPSESSID=slaurqvo3msh9uklerbht0nd2h; path=/; domain=.xxx; HttpOnly - ObjHeader Cache-Control: max-age=3600, public - ObjHeader Date: Tue, 29 Oct 2019 08:31:39 GMT - ObjHeader Content-Encoding: gzip - ObjHeader Vary: Accept-Encoding - Fetch_Body 2 chunked stream - Gzip u F - 24261 118266 80 80 194017 - BackendReuse 26 boot.default - Timestamp BerespBody: 1572337903.644744 5.170209 0.002738 - Length 24261 - BereqAcct 275 0 275 342 24261 24603 - End * << Request >> 492061 - Begin req 492060 rxreq - Timestamp Start: 1572337898.474380 0.000000 0.000000 - Timestamp Req: 1572337898.474380 0.000000 0.000000 - ReqStart 127.0.0.1 57354 - ReqMethod GET - ReqURL / - ReqProtocol HTTP/1.0 - ReqHeader Host: xxx - ReqHeader X-Forwarded-Host: xxx - ReqHeader X-Real-IP: xxx - ReqHeader X-Forwarded-For: xxx - ReqHeader X-Forwarded-Proto: https - ReqHeader HTTPS: on - ReqHeader Cache-Control: max-age=15000 - ReqHeader Connection: close - ReqHeader User-Agent: Wget/1.19.4 (linux-gnu) - ReqHeader Accept: */* - ReqHeader Accept-Encoding: identity - ReqUnset X-Forwarded-For: xxx - ReqHeader X-Forwarded-For: xxx, 127.0.0.1 - VCL_call RECV - ReqUnset X-Forwarded-For: xxx, 127.0.0.1 - ReqHeader X-Forwarded-For: 127.0.0.1 - VCL_return hash - ReqUnset Accept-Encoding: identity - VCL_call HASH - VCL_return lookup - HitMiss 492059 104.991348 - VCL_call MISS - VCL_return fetch - Link bereq 492062 fetch - Timestamp Fetch: 1572337903.643494 5.169113 5.169113 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespHeader Server: nginx/1.14.0 (Ubuntu) - RespHeader Content-Type: text/html; charset=UTF-8 - RespHeader Set-Cookie: PHPSESSID=slaurqvo3msh9uklerbht0nd2h; path=/; domain=.xxx; HttpOnly - RespHeader Cache-Control: max-age=3600, public - RespHeader Date: Tue, 29 Oct 2019 08:31:39 GMT - RespHeader Content-Encoding: gzip - RespHeader Vary: Accept-Encoding - RespHeader X-Varnish: 492061 - RespHeader Age: 20 - RespHeader Via: 1.1 varnish (Varnish/5.2) - VCL_call DELIVER - VCL_return deliver - Timestamp Process: 1572337903.643520 5.169140 0.000026 - RespUnset Content-Encoding: gzip - RespHeader Accept-Ranges: bytes - RespHeader Connection: close - Gzip U D - 24261 118266 80 80 194017 - Timestamp Resp: 1572337903.645192 5.170812 0.001672 - ReqAcct 313 0 313 381 118266 118647 - End * << Session >> 492060 - Begin sess 0 HTTP/1 - SessOpen 127.0.0.1 57354 a0 127.0.0.1 80 1572337898.474305 24 - Link req 492061 rxreq - SessClose TX_EOF 5.171 - End Somehow, Varnish seems to save the website in cache: - VCL_call DELIVER - VCL_return deliver VCL Configuration: vcl 4.0; # Default backend definition. Set this to point to your content server. backend default { .host = "127.0.0.1"; .port = "8080"; } sub vcl_recv { // Remove all cookies except the session ID. if (req.http.Cookie) { set req.http.Cookie = ";" + req.http.Cookie; set req.http.Cookie = regsuball(req.http.Cookie, "; +", ";"); set req.http.Cookie = regsuball(req.http.Cookie, ";(PHPSESSID)=", "; \1="); set req.http.Cookie = regsuball(req.http.Cookie, ";[^ ][^;]*", ""); set req.http.Cookie = regsuball(req.http.Cookie, "^[; ]+|[; ]+$", ""); if (req.http.Cookie == "") { // If there are no more cookies, remove the header to get page cached. unset req.http.Cookie; } } } sub vcl_backend_response { # Happens after we have read the response headers from the backend. # # Here you clean the response headers, removing silly Set-Cookie headers # and other mistakes your backend does. if (beresp.http.Surrogate-Control ~ "ESI/1.0") { unset beresp.http.Surrogate-Control; set beresp.do_esi = true; } } sub vcl_deliver { # Insert Diagnostic header to show Hit or Miss if (obj.hits > 0) { set resp.http.X-Cache = "HIT"; set resp.http.X-Cache-Hits = obj.hits; } else { set resp.http.X-Cache = "MISS"; } } What is wrong there?
You shall not cache pages with a Set-Cookie Header! also you are quoting the wrong lines to determine whether it was cached or not: VCL_call MISS
cURL HTTP request from WikiMedia API not working
I've read tons of cURL tutorials (I'm using PHP) and there's always the same basic code, which doesn't work for me! No specific errors, just no result. I want to make a HTTP request from Wikipedia and get the result in JSON format. Here's the code : $handle = curl_init(); $url = "http://fr.wikipedia.org/w/api.php?action=query&titles=Albert%20Einstein&prop=info&format=json"; curl_setopt_array($handle, array( CURLOPT_URL => $url, CURLOPT_RETURNTRANSFER => true ) ); $output = curl_exec($handle); if (!$output) { exit('cURL Error: '.curl_error($handle)); } $result= json_decode($output,true); print_r($result); curl_close($handle); Would like to know what I'm doing wrong.
Your code is correct but it seems Wikipedia doesn't send back the data when using PHP curl (maybe some headers or other parameters must be set for it to work). If all you need is to retrieve some data though, you can simply use file_get_contents which works fine: $output = file_get_contents("http://fr.wikipedia.org/w/api.php?action=query&titles=Albert%20Einstein&prop=info&format=json"); echo $output; Edit: Just for information, I found what the issue is. When running curl -v on that URL, the following comes up: * Trying 91.198.174.192... * Connected to fr.wikipedia.org (91.198.174.192) port 80 (#0) > GET /w/api.php?action=query&titles=Albert%20Einstein&prop=info&format=json HTTP/1.1 > Host: fr.wikipedia.org > User-Agent: curl/7.47.0 > Accept: */* > < HTTP/1.1 301 Moved Permanently < Date: Wed, 17 May 2017 13:54:31 GMT < Server: Varnish < X-Varnish: 852298595 < X-Cache: cp3031 int < X-Cache-Status: int < Set-Cookie: WMF-Last-Access=17-May-2017;Path=/;HttpOnly;secure;Expires=Sun, 18 Jun 2017 12:00:00 GMT < Set-Cookie: WMF-Last-Access-Global=17-May-2017;Path=/;Domain=.wikipedia.org;HttpOnly;secure;Expires=Sun, 18 Jun 2017 12:00:00 GMT < X-Client-IP: 86.214.172.57 < Location: https://fr.wikipedia.org/w/api.php?action=query&titles=Albert%20Einstein&prop=info&format=json < Content-Length: 0 < Connection: keep-alive < * Connection #0 to host fr.wikipedia.org left intact So what's happening is that the actual content is on the https url, not http, so by requesting https://fr.wikipedia.org/w/api.php?action=query&titles=Albert%20Einstein&prop=info&format=json it should work directly. The reason it works with file_get_contents is because in this case the redirection is done automatically.
Why php5-fpm post requests are slow, while same php-cli code/console curl works extremely fast?
I am performing POST request to local api web service (via LAN) with dummy method, that works very fast itself (less then a second). Problem is that if i use php5-fpm it takes extremely long time to perform POST request (curl post, streams). If i use php-cli script with same code or console curl command - it works very fast, like a charm. Strange thing: if i use system to execute console curl command from php5-fpm it tooks a lot of time to perform the request. So, no way to cheat :( I am using direct ipv4 address to eliminate DNS problems (i tried to define CURLOPT_IPRESOLVE option, but perfomance was same). If i omit curl CURLOPT_POSTFIELDS option, request is very fast in php5-fpm too. I am using debian jessie with official php 5.6.9 package. So, why this problem is happening with php5-fpm? My curl php code: $data = json_encode([ 'id' => 1, 'method' => 'test', 'sid' => session_id(), ]); $curl = curl_init(); curl_setopt($curl, CURLOPT_URL, 'http://192.168.182.22'); curl_setopt($curl, CURLOPT_RETURNTRANSFER, true); curl_setopt($curl, CURLOPT_POST, true); curl_setopt($curl, CURLOPT_HTTPHEADER, [ 'Content-type: application/json', 'Content-length: ' . strlen($data) ]); curl_setopt($curl, CURLOPT_POSTFIELDS, $data); $out = curl_exec($curl); curl_close($curl); Curl get info output (shows that starttransfer_time is very slow): array(26) { ["url"]=> string(22) "http://192.168.182.22/" ["content_type"]=> string(24) "text/html; charset=UTF-8" ["http_code"]=> int(200) ["header_size"]=> int(532) ["request_size"]=> int(188) ["filetime"]=> int(-1) ["ssl_verify_result"]=> int(0) ["redirect_count"]=> int(0) ["total_time"]=> float(10.285093) ["namelookup_time"]=> float(3.2E-5) ["connect_time"]=> float(0.000156) ["pretransfer_time"]=> float(0.000181) ["size_upload"]=> float(82) ["size_download"]=> float(99) ["speed_download"]=> float(9) ["speed_upload"]=> float(7) ["download_content_length"]=> float(-1) ["upload_content_length"]=> float(82) ["starttransfer_time"]=> float(10.285052) ["redirect_time"]=> float(0) ["redirect_url"]=> string(0) "" ["primary_ip"]=> string(14) "192.168.182.22" ["certinfo"]=> array(0) { } ["primary_port"]=> int(80) ["local_ip"]=> string(14) "192.168.182.20" ["local_port"]=> int(49286) } Another try with streams. It also works slow (a bit faster then curl, but anyway i can get 40-50 seconds delay): $url = 'http://192.168.182.22/'; $data = json_encode([ 'id' => 1, 'method' => 'test', 'sid' => session_id(), ]); $headers = [ 'Content-type: application/json', 'Connection: close', 'Content-Length: ' . strlen($data) ]; $options = array( 'http' => array( 'header' => join("\r\n", $headers) . "\r\n", 'method' => 'POST', 'content' => $data, ), ); $context = stream_context_create($options); $result = file_get_contents($url, false, $context); There is no significant difference in diff /etc/php5/fpm/php.ini /etc/php5/cli/php.ini: 303c303 < disable_functions = pcntl_alarm,pcntl_fork,pcntl_waitpid,pcntl_wait,pcntl_wifexited,pcntl_wifstopped,pcntl_wifsignaled,pcntl_wexitstatus,pcntl_wtermsig,pcntl_wstopsig,pcntl_signal,pcntl_signal_dispatch,pcntl_get_last_error,pcntl_strerror,pcntl_sigprocmask,pcntl_sigwaitinfo,pcntl_sigtimedwait,pcntl_exec,pcntl_getpriority,pcntl_setpriority, --- > disable_functions = 363c363 < expose_php = Off --- > expose_php = On 393c393 < memory_limit = 256M --- > memory_limit = -1 660c660 < post_max_size = 200M --- > post_max_size = 8M 810c810 < upload_max_filesize = 200M --- > upload_max_filesize = 2M Main thing in tcpdump i can see differ - php5-fpm requests includes ARP requests. I can't see such in console curl/php-cli. Perhaps problem happens when remote api server wants to return response, but can't find a way how to perform this? (#Mircea adviced to add static arp routes, we added them, ARP requests disappeared from tcpdump, but pauses are still present). tcpdump output for php5-fpm: 00:55:22.347822 IP 192.168.182.20.52659 > 192.168.182.22.80: Flags [S], seq 4210728690, win 29200, options [mss 1460,sackOK,TS val 301284147 ecr 0,nop,wscale 7], length 0 E..<..#.#..~...........P..........r............ ..;3........ 00:55:22.347898 IP 192.168.182.22.80 > 192.168.182.20.52659: Flags [S.], seq 2702388053, ack 4210728691, win 28960, options [mss 1460,sackOK,TS val 280740546 ecr 301284147,nop,wscale 7], length 0 E..<..#.#.M#.........P....+U......q .R......... ......;3.... 00:55:22.347906 IP 192.168.182.20.52659 > 192.168.182.22.80: Flags [.], ack 1, win 229, options [nop,nop,TS val 301284147 ecr 280740546], length 0 E..4..#.#..............P......+V........... ..;3.... 00:55:22.347942 IP 192.168.182.20.52659 > 192.168.182.22.80: Flags [P.], seq 1:206, ack 1, win 229, options [nop,nop,TS val 301284147 ecr 280740546], length 205 E.....#.#..............P......+V.....o..... ..;3....POST / HTTP/1.1 Host: 192.168.182.22 Content-Type: application/x-www-form-urlencoded Content-Length: 82 Accept: */* {"id":1,"method":"station.sync","params":[1],"sid":"1gn8f4mhctui9ki91bla6osnf4"} 00:55:22.347997 IP 192.168.182.22.80 > 192.168.182.20.52659: Flags [.], ack 206, win 235, options [nop,nop,TS val 280740547 ecr 301284147], length 0 E..4..#.#.c..........P....+V........2...... ......;3 00:55:27.356575 ARP, Request who-has 192.168.182.22 tell 192.168.182.20, length 28 ..........).4............... 00:55:27.356777 ARP, Reply 192.168.182.22 is-at 00:0c:29:46:99:bc, length 46 ..........)F........).4....................... 00:55:36.742347 IP 192.168.182.22.80 > 192.168.182.20.52659: Flags [P.], seq 1:558, ack 206, win 235, options [nop,nop,TS val 280744145 ecr 301284147], length 557 E..a..#.#.a..........P....+V........a...... ......;3HTTP/1.1 200 OK Server: nginx/1.6.2 Date: Wed, 01 Jul 2015 21:55:36 GMT Content-Type: application/json Transfer-Encoding: chunked Connection: keep-alive Expires: Thu, 19 Nov 1981 08:52:00 GMT Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Pragma: no-cache Access-Control-Allow-Credentials: true Access-Control-Allow-Headers: Content-Type, X-Requested-With Access-Control-Allow-Methods: GET, POST, OPTIONS p3p: CP=IDC DSP COR ADM DEVi TAIi PSA PSD IVAi IVDi CONi HIS OUR IND CNT 16 {"id":1,"result":true} 0 00:55:36.742374 IP 192.168.182.20.52659 > 192.168.182.22.80: Flags [.], ack 558, win 237, options [nop,nop,TS val 301287746 ecr 280744145], length 0 E..4..#.#..............P......-............ ..IB.... 00:55:36.742486 IP 192.168.182.20.52659 > 192.168.182.22.80: Flags [F.], seq 206, ack 558, win 237, options [nop,nop,TS val 301287746 ecr 280744145], length 0 E..4..#.#..............P......-............ ..IB.... 00:55:36.742559 IP 192.168.182.22.80 > 192.168.182.20.52659: Flags [F.], seq 558, ack 207, win 235, options [nop,nop,TS val 280744145 ecr 301287746], length 0 E..4..#.#.c..........P....-..........:..... ......IB 00:55:36.742566 IP 192.168.182.20.52659 > 192.168.182.22.80: Flags [.], ack 559, win 237, options [nop,nop,TS val 301287746 ecr 280744145], length 0 E..4..#.#..............P......-............ ..IB.... 00:55:41.744102 ARP, Request who-has 192.168.182.20 tell 192.168.182.22, length 46 ..........)F.................................. 00:55:41.744121 ARP, Reply 192.168.182.20 is-at 00:0c:29:04:34:c9, length 28 ..........).4.......)F...... tcpdump output for fast (console/php-cli) requests: 0:43:46.854062 IP 192.168.182.20.52527 > 192.168.182.22.80: Flags [S], seq 560731240, win 29200, options [mss 1460,sackOK,TS val 301110274 ecr 0,nop,wscale 7], length 0 E..<.j#.#............/.P!l.h......r............ ............ 00:43:46.854198 IP 192.168.182.22.80 > 192.168.182.20.52527: Flags [S.], seq 1849811122, ack 560731241, win 28960, options [mss 1460,sackOK,TS val 280566673 ecr 301110274,nop,wscale 7], length 0 E..<..#.#.M#.........P./nA..!l.i..q ........... ............ 00:43:46.854216 IP 192.168.182.20.52527 > 192.168.182.22.80: Flags [.], ack 1, win 229, options [nop,nop,TS val 301110274 ecr 280566673], length 0 E..4.k#.#............/.P!l.inA............. ........ 00:43:46.854265 IP 192.168.182.20.52527 > 192.168.182.22.80: Flags [P.], seq 1:233, ack 1, win 229, options [nop,nop,TS val 301110274 ecr 280566673], length 232 E....l#.#............/.P!l.inA............. ........POST / HTTP/1.1 User-Agent: curl/7.38.0 Host: 192.168.182.22 Accept: */* Content-Length: 84 Content-Type: application/x-www-form-urlencoded {"id":1, "sid": "1gn8f4mhctui9ki91bla6osnf4","method":"station.sync", "params":[10]} 00:43:46.854317 IP 192.168.182.22.80 > 192.168.182.20.52527: Flags [.], ack 233, win 235, options [nop,nop,TS val 280566673 ecr 301110274], length 0 E..4."#.#.R%.........P./nA..!l.Q....b...... ........ 00:43:46.891248 IP 192.168.182.22.80 > 192.168.182.20.52527: Flags [P.], seq 1:615, ack 233, win 235, options [nop,nop,TS val 280566682 ecr 301110274], length 614 E....##.#.O..........P./nA..!l.Q........... ........HTTP/1.1 200 OK Server: nginx/1.6.2 Date: Wed, 01 Jul 2015 21:43:46 GMT Content-Type: application/json Transfer-Encoding: chunked Connection: keep-alive Expires: Thu, 19 Nov 1981 08:52:00 GMT Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Pragma: no-cache Access-Control-Allow-Credentials: true Access-Control-Allow-Headers: Content-Type, X-Requested-With Access-Control-Allow-Methods: GET, POST, OPTIONS p3p: CP=IDC DSP COR ADM DEVi TAIi PSA PSD IVAi IVDi CONi HIS OUR IND CNT 4f {"id":1,"error":{"code":"SERVER_ERROR","message":"Station #10 is not running"}} 0 00:43:46.891273 IP 192.168.182.20.52527 > 192.168.182.22.80: Flags [.], ack 615, win 238, options [nop,nop,TS val 301110283 ecr 280566682], length 0 E..4.m#.#............/.P!l.QnA............. ........ 00:43:46.891932 IP 192.168.182.20.52527 > 192.168.182.22.80: Flags [F.], seq 233, ack 615, win 238, options [nop,nop,TS val 301110283 ecr 280566682], length 0 E..4.n#.#............/.P!l.QnA............. ........ 00:43:46.896944 IP 192.168.182.22.80 > 192.168.182.20.52527: Flags [F.], seq 615, ack 234, win 235, options [nop,nop,TS val 280566684 ecr 301110283], length 0 E..4.$#.#.R#.........P./nA..!l.R....`i..... ........ 00:43:46.896954 IP 192.168.182.20.52527 > 192.168.182.22.80: Flags [.], ack 616, win 238, options [nop,nop,TS val 301110285 ecr 280566684], length 0 E..4.o#.#............/.P!l.RnA............. ........
One thing I've noticed on my own instance of FPM (using Nginx) is that I will not get interim output. I've never bothered to investigate this as it has never mattered for my own usage needs (small units of work with little output). In your case, perhaps the response is being sent quickly but the connection is kept alive -- causing the recipient to wait around for more data that never arrives. You may be able to check on this by investigating server side logging to check response times from the server's point of view.
I just track down a similar issue of my own. I think it might be the same reason. I encounter the similar issue like avasin. I found that php-fpm curl to localhost is slow, but php-cli curl to localhost is extremely fast. Plus, debugging curl timing shows high starttransfer_time in php-fpm but rather low in php process. I found the root cause yesterday, the issue is the php-fpm www.conf config. Becuase php-fpm has low start_server, php-fpm actually has to process curl with limited number of php-fpm processes. The solution is to increase the number of start_server, this will affect the number of concurrency which your php-fpm could handle. The performance will decrease drastically when the concurrency exceeds the number of current php-fpm processes. Hence, making everything feels slow. The php-cli does not have the same issue since it could launch many php processes as long as you have enough memory, so it is always fast. I did a lot of experiments here php curl localhost is slow when making concurrent requests. Take a look if you're interested.