php - cURL response header different than browser response header - php

I want to send a request using cURL and retrieve the response header.
Using a browser the response header is as follow:
HTTP/1.0 302 Moved Temporarily
Content-Type: text/html; charset=utf-8
Cache-Control: no-cache
Location: "Correct URL"
Expires: Fri, 01 Jan 1990 00:00:00 GMT
Date: Tue, 30 Oct 2012 08:32:24 GMT
Server: Google Frontend
Content-Length: 0
But when I send the request using cURL the response header is as follow:
HTTP/1.1 302 Found
Content-Type: text/html; charset=utf-8
Cache-Control: no-cache
Location: "Wrong URL"
Expires: Fri, 01 Jan 1990 00:00:00 GMT
Date: Tue, 30 Oct 2012 09:12:14 GMT
Server: Google Frontend
Content-Length: 0
I want to know what is causing the response to return different URLs. This is a small php sample out of many samples and things I tried with no avail.
<?php
$url = "url";
$ch = curl_init( $url );
curl_setopt( $ch, CURLOPT_FOLLOWLOCATION, true );
curl_setopt( $ch, CURLOPT_HEADER, true );
curl_setopt( $ch, CURLOPT_RETURNTRANSFER, true );
curl_setopt( $ch, CURLOPT_COOKIEJAR, "cookie.txt" );
curl_setopt( $ch, CURLOPT_COOKIEFILE, "cookie.txt" );
curl_setopt( $ch, CURLOPT_USERAGENT, "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.4) Gecko/20091030 Gentoo Firefox/3.5.4" );
list( $header, $contents ) = preg_split( '/([\r\n][\r\n])\\1/', curl_exec( $ch ), 2 );
curl_close( $ch );
$header_text = preg_split( '/[\r\n]+/', $header );
foreach ( $header_text as $headers ) {
echo $headers . "</br>";
}
?>

There is some difference between the requests sent through the browser and through curl (almost certainly in the HTTP headers) that causes the difference in the responses.
You should capture the request from the browser (perhaps using an HTTP proxy like Fiddler for convenience) and compare its headers to those from your curl request. One (or more) of the differences you will find is the reason for what you are seeing.

Related

PHP curl to display output for URL like 'HTTP/1.1 200 OK'

I am trying to get the response/status code including the header for multiple URL's. My first try I was successful in getting the status code i.e. 200 or 301 or 302. But I want the output to be like HTTP/1.1 200 OK or HTTP/1.1 302 Found etc.
Below is my code in which I get just the response code i.e. 200 or 301.
<?php
$line = "https://www.pnc.com";
$ch = curl_init($line);
curl_setopt($ch, CURLOPT_URL, $line);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_HEADER,true);
curl_setopt($ch, CURLOPT_NOBODY, true);
curl_setopt($ch, CURLOPT_TIMEOUT,10);
$out = curl_exec($ch);
$ret = true;
if ($out !== false) {
$statuscode = curl_getinfo($ch,CURLINFO_HTTP_CODE);
echo "Response Code:" .$statuscode. "\r";
}
curl_close($ch);
?>
Actual Ouput
Response Code:HTTP/1.1 200 OK
ETag: "846caf551746b12e876c0bad5a50830d:1475788950"
Last-Modified: Thu, 06 Oct 2016 21:22:30 GMT
Accept-Ranges: bytes
Content-Length: 17609
Content-Type: text/html
Expires: Fri, 21 Sep 2018 15:35:01 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Fri, 21 Sep 2018 15:35:01 GMT
Connection: keep-alive
Expected Output
HTTP/1.1 200 OK.
Can someone help me with this?
This can be done using strtok
Example:
$statuscode = strtok( $statuscode, "\n" );
-or-
Using explode is pretty standard:
$statuscode = explode( "\n", $statuscode )[0];
There are definitely other methods as well
However probably the cleanest way might be to use cURL itself:
$response = curl_getinfo( $ch, CURLINFO_HTTP_CODE );

Logging into a website with CURL PHP not working

I have a problem logging in to a website with CURL and PHP.
I test with the Firefox add-on HttpRequester and this worked.
Result login:
POST https://www.balatarin.com/sessions
Content-Type: application/x-www-form-urlencoded
session[login]=testeruni&session[password]=123456789&session[remember_me]=1&commit=%D9%88%D8%B1%D9%88%D8%AF&utf8=%E2%9C%93&authenticity_token[![httprequester][1]][1]
-- response --
200 OK
Server: shield
Date: Thu, 19 Jan 2017 13:51:54 GMT
Content-Type: text/html; charset=utf-8
status: 200 OK
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
x-ua-compatible: IE=Edge,chrome=1
Etag: W/"7418542e936fbdfe20002faf11876845"
Cache-Control: must-revalidate, private, max-age=0
Set-Cookie: _balat_session_new=BAh7C0kiDHVzZXJfaWQGOgZFRmkD964BSSIPc2Vzc2lvbl9pZAY7AEZJIiUzZGUxMzIyN2ZhZDVmMDUzOGE3OGY0YTRhZDkzNmUyMQY7AFRJIhZpbnB1dF9kZXZpY2VfdHlwZQY7AEZJIgpNT1VTRQY7AEZJIhRob3Zlcl9zdXBwb3J0ZWQGOwBGVEkiCmZsYXNoBjsARm86JUFjdGlvbkRpc3BhdGNoOjpGbGFzaDo6Rmxhc2hIYXNoCToKQHVzZWRvOghTZXQGOgpAaGFzaHsGOgtub3RpY2VUOgxAY2xvc2VkRjoNQGZsYXNoZXN7BjsKSSI22YbYrtiz2Kog2YXYtNiq2LHaqSDahtmG2K8g2KjYp9mE2KfahtmHINi02YjbjNivLgY7AFQ6CUBub3cwSSIQX2NzcmZfdG9rZW4GOwBGSSIxT3krNk5nM1NTM2IreXc4SUtxbW9yN2NmMXQrdUNLWWdubFRRYmpidmtNTT0GOwBG--2c2a72f8ec27564250ba084d97998aefba4af11a; path=/; secure; HttpOnly geo=0
X-Request-Id: 521288561d7cfff0ef8fe8d72080760c
X-Runtime: 0.188862
X-Rack-Cache: miss
Content-Encoding: gzip
Via: 1.1 google
Alt-Svc: clear
Expires: Thu, 19 Jan 2017 13:51:54 GMT
X-Firefox-Spdy: h2
but it does not login with curl in PHP. I tested all headers in my CURL but it does not login, only works with HttpRequester.
public function actionLoggin()
{
$url = 'https://www.balatarin.com/sessions';
$headers[] = 'Content-Type: application/x-www-form-urlencoded';
$headers[] = 'Host: www.balatarin.com';
$headers[] = 'User-Agent: Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:50.0) Gecko/20100101 Firefox/50.0';
$headers[] = 'Referer: https://www.balatarin.com/login';
$params = array(
'session[login]' => 'testeruni',
'session[password]' => '123456789',
'session[remember_me]' => '0',
'commit' => 'ورود',
'utf8' => '✓',
'authenticity_token' => '',
);
//open connection
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query($params));
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt($ch, CURLOPT_COOKIEJAR, 'bala_cookie.txt');
curl_setopt($ch, CURLOPT_COOKIEFILE, 'bala_cookie.txt');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$result = curl_exec($ch);
curl_close($ch);
echo $result;
}
Here is my cookie file:
# Netscape HTTP Cookie File
# https://curl.haxx.se/docs/http-cookies.html
# This file was generated by libcurl! Edit at your own risk.
www.balatarin.com FALSE / FALSE 0 logged_in 1
#HttpOnly_www.balatarin.com FALSE / TRUE 0 _balat_session_new BAh7CToOcmV0dXJuX3RvMDoMdXNlcl9pZGkDj60BOhJsb2dpbl9yZXRyaWVzMEkiD3Nlc3Npb25faWQGOgZFRkkiJTgwN2ZmMDRjMGUzMzkyMDIyZWY5YzBmZTQxN2FmZWMzBjsIVA%3D%3D--d47dd61bc9900449cca69ebd727041c3946a13ba
www.balatarin.com FALSE / FALSE 0 geo 0
www.balatarin.com FALSE / FALSE 1516368886 corr b8ed93fa279a469a637b

$_SESSION variable empty on multiple cURL requests

In my first cURL request i upload a file and i set a $_SESSION variable with the name, extension etc. In my second cURL request i want to move the uploaded file from tmp folder to user folder but baddly the $_SESSION variable is empty. why?
first request code code looks like this:
$upload = curl_init();
curl_setopt($upload, CURLOPT_URL, "http://localhost/upload/" );
curl_setopt($upload, CURLOPT_POST, true );
curl_setopt($upload, CURLOPT_RETURNTRANSFER, true );
curl_setopt($upload, CURLOPT_USERAGENT, $_SERVER["HTTP_USER_AGENT"] );
curl_setopt($upload, CURLOPT_HTTPHEADER, array("Set-Cookie: data=" . urldecode($cookie) ));
curl_custom_postfields($upload, $fields, $files);
$res = curl_exec($upload);
curl_close($upload);
and the second request code: following the first request:
$submit = curl_init();
curl_setopt($submit, CURLOPT_URL, "http://localhost/" );
curl_setopt($submit, CURLOPT_RETURNTRANSFER, true );
curl_setopt($submit, CURLOPT_USERAGENT, $_SERVER["HTTP_USER_AGENT"] );
curl_setopt($submit, CURLOPT_POST, count($fields));
curl_setopt($submit, CURLOPT_POSTFIELDS, $fields_string );
curl_setopt($submit, CURLOPT_HEADER, true);
curl_setopt($submit, CURLOPT_HTTPHEADER, array("Cookie: data=" . urldecode($cookie) ));
$res = curl_exec($submit);
curl_close($submit);
is there any option to keep session alive? is the same problem i meet on AJAX requests when i start using javascript with AJAX i think.
my response header:
HTTP/1.1 200 OK
Date: Wed, 04 Mar 2015 02:24:22 GMT
Server: Apache/2.2.12 (Win32) DAV/2 mod_ssl/2.2.12 OpenSSL/0.9.8k mod_autoindex_color PHP/5.3.0 mod_perl/2.0.4 Perl/v5.10.0
X-Powered-By: PHP/5.3.0
Set-Cookie: PHPSESSID=g7hc328ij8lr63mps6ub44gat2; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Content-Length: 22
Content-Type: text/html
I think you need a cookie jar to keep track of your session:
curl_setopt($ch, CURLOPT_COOKIEJAR, "/tmp/cookies");

cURL not sending POST data in some requests

I've been stuck on this problem for several hours: I'm using PHP and cURL to write a sort of PHP proxy. Almost everything works fine, setting cookies, handling redirects, and submitting forms using POST.
Basically, I'm trying to mirror a remote website with my local proxy. To do so, I redirect every request to http://localhost/resource to http://localhost/proxy.php?url=http://remotesite.com/resource that will fetch the resource on the remote website. The redirect is handled by a 404 error page on .htaccess but I guess that using mod_rewrite would not change things.
I'm testing my proxy on a complex application (the latest version of WordPress) deployed on a remote server. The WordPress login works fine and uses POST. However, I found a page where updating a form does not work, and for which all the POST data is not sent at all to the server.
Here is what I see with wireshark listening on the loopback interface:
POST /proxy/wp-admin/media.php?attachment_id=691&action=edit HTTP/1.1
Host: localhost
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.23) Gecko/20110921 Ubuntu/10.04 (lucid) Firefox/3.6.23
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 115
Connection: keep-alive
Referer: http://localhost/proxy/wp-admin/media.php?attachment_id=691&action=edit
Cookie: [snip]
Cache-Control: max-age=0
Content-Type: application/x-www-form-urlencoded
Content-Length: 501
attachments%5B691%5D%5Bmenu_order%5D=0&attachments%5B691%5D%5Bpost_title%5D=fb&attachments%5B691%5D%5Bimage_alt%5D=&attachments%5B691%5D%5Bpost_excerpt%5D=&attachments%5B691%5D%5Bpost_content%5D=foobar&attachments%5B691%5D%5Burl%5D=http%3A%2F%2Flocalhost%2Fproxy%2Fwp-content%2Fuploads%2F2009%2F04%2Ffb.gif&save=Aggiorna+media&post_id=&attachment_id=691&action=editattachment&_wp_original_http_referer=&_wpnonce=02caf30462&_wp_http_referer=%2Fwp-admin%2Fmedia.php%3Fattachment_id%3D691%26action%3Dedit
HTTP/1.1 200 OK
Date: Wed, 19 Oct 2011 16:18:56 GMT
Server: Apache/2.2.14 (Ubuntu)
X-Powered-By: PHP/5.3.2-1ubuntu4.10
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 5441
Keep-Alive: timeout=15, max=100
Connection: Keep-Alive
Content-Type: text/html; charset=UTF-8
[content]
While, what I see if listening on the interface connected to the internet is:
POST /wp-admin/media.php?attachment_id=691&action=edit HTTP/1.1
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.23) Gecko/20110921 Ubuntu/10.04 (lucid) Firefox/3.6.23
Host: www.remotesite.com
Accept: */*
Referer: http://www.remotesite.com/wp-admin/media.php?attachment_id=691&action=edit
Cookie: [snip]
X-Forwarded-For: 127.0.0.1
Content-Length: 0
Content-Type: application/x-www-form-urlencoded
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Date: Wed, 19 Oct 2011 16:25:13 GMT
Server: LiteSpeed
Connection: close
X-Powered-By: PHP/5.2.17
Expires: Wed, 11 Jan 1984 05:00:00 GMT
Last-Modified: Wed, 19 Oct 2011 16:25:13 GMT
Cache-Control: no-cache, must-revalidate, max-age=0
Pragma: no-cache
X-Frame-Options: SAMEORIGIN
Content-Type: text/html; charset=UTF-8
[content]
As you see my proxy is not transmitting the post data to the remote server.
I expect the problem to be related to the encoding of the POSTDATA, as POST variables in this case are in an array (attachments[691][menu_order]=0; attachments[691][post_content]=foobar and so on...).
I tried several changes as suggested by similar posts but haven't managed to change the behavior of the script at all. All this because apparently the first (local) POST sends the data to localhost, but cURL is unable to fetch the POST data (indeed, file_get_contents("php://input") in the code below reads 0 bytes).
I paste here part of my code hoping somebody can help me:
$ch = curl_init( $url );
$headers = array();
if ( isset($_SERVER['CONTENT_TYPE']) ) {
// commenting this out or changing to multipart/form-data does not change anything
array_push($headers, "Content-Type: ".$_SERVER['CONTENT_TYPE'] );
}
if ( count($headers) > 0 ) {
curl_setopt( $ch, CURLOPT_HTTPHEADER, $headers );
}
$postdata = file_get_contents("php://input"); //this turns out to be empty - and so is $_POST
//REQUEST METHOD: since pages are redirected from a 404 error page, we have to handle
//a redirect, so the real method is specified in REDIRECT_REQUEST_METHOD
if ( isset($_SERVER['REDIRECT_REQUEST_METHOD']) && isset($postdata) ){
if ($_SERVER['REDIRECT_REQUEST_METHOD'] == 'POST'){
curl_setopt( $ch, CURLOPT_POST, true );
}
}
else{
if (isset($_SERVER['REQUEST_METHOD']) && $_SERVER['REQUEST_METHOD'] == 'POST' ){
curl_setopt( $ch, CURLOPT_POST, true );
}
}
if ( isset($_SERVER['CONTENT_LENGTH'] ) && $_SERVER['CONTENT_LENGTH'] > 0 ) {
curl_setopt( $ch, CURLOPT_POSTFIELDS, $postdata );
}
//set cookies
curl_setopt($ch, CURLOPT_COOKIEJAR, '/tmp/cookietofwd');
curl_setopt($ch, CURLOPT_COOKIEFILE,'/tmp/cookietofwd');
curl_setopt( $ch, CURLOPT_FOLLOWLOCATION, true );
curl_setopt( $ch, CURLOPT_HEADER, true );
curl_setopt( $ch, CURLOPT_RETURNTRANSFER, true );
$out=curl_exec( $ch );
[...]
Why don't you get your POST data from $_POST? The application/x-www-form-urlencoded header appears to be an AJAX request instead of a regular POST, which I am not sure how it is handled by php://input.
You could so something like:
curl_setopt( $ch, CURLOPT_POSTFIELDS, http_build_query($_POST) );
Is there a specific reason why you want this in PHP? Why don't you just use nginx? (http://nginx.org/en/)
It will probably do a better job (and faster).

How to remove HTTP headers from CURL response?

I have a php script that returns just plain text without any html. Now I want to make a cURL request to that script and I get the following response:
HTTP/1.1 200 OK
Date: Mon, 28 Feb 2011 14:21:51 GMT
Server: Apache/2.2.14 (Ubuntu)
X-Powered-By: PHP/5.2.12-nmm2
Vary: Accept-Encoding
Content-Length: 6
Content-Type: text/html
6.8320
The actuall response is just 6.8320 as text without any html. I want to retrieve it from the response above by just removing the header information.
I already minified the script a bit:
$url = $_GET['url'];
if ( !$url ) {
// Passed url not specified.
$contents = 'ERROR: url not specified';
$status = array( 'http_code' => 'ERROR' );
} else if ( !preg_match( $valid_url_regex, $url ) ) {
// Passed url doesn't match $valid_url_regex.
$contents = 'ERROR: invalid url';
$status = array( 'http_code' => 'ERROR' );
} else {
$ch = curl_init( $url );
if ( strtolower($_SERVER['REQUEST_METHOD']) == 'post' ) {
curl_setopt( $ch, CURLOPT_POST, true );
curl_setopt( $ch, CURLOPT_POSTFIELDS, $_POST );
}
curl_setopt( $ch, CURLOPT_FOLLOWLOCATION, true );
curl_setopt( $ch, CURLOPT_HEADER, true );
curl_setopt( $ch, CURLOPT_RETURNTRANSFER, true );
curl_setopt( $ch, CURLOPT_USERAGENT, $_GET['user_agent'] ? $_GET['user_agent'] : $_SERVER['HTTP_USER_AGENT'] );
list( $header, $contents ) = preg_split( '/([\r\n][\r\n])\\1/', curl_exec( $ch ), 2 );
$status = curl_getinfo( $ch );
curl_close( $ch );
}
// Split header text into an array.
$header_text = preg_split( '/[\r\n]+/', $header );
if ( true ) {
if ( !$enable_native ) {
$contents = 'ERROR: invalid mode';
$status = array( 'http_code' => 'ERROR' );
}
// Propagate headers to response.
foreach ( $header_text as $header ) {
if ( preg_match( '/^(?:Content-Type|Content-Language|Set-Cookie):/i', $header ) ) {
header( $header );
}
}
print $contents;
}
Any idea what I need to change to remove the header information from the response?
Just set CURLOPT_HEADER to false.
Make sure you put set the header flag:
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HEADER, true );
curl_setopt($ch, CURLOPT_TIMEOUT, Constants::HTTP_TIMEOUT);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, Constants::HTTP_TIMEOUT);
$response = curl_exec($ch);
Do this after your curl call:
$header_size = curl_getinfo($ch, CURLINFO_HEADER_SIZE);
$headerstring = substr($response, 0, $header_size);
$body = substr($response, $header_size);
EDIT:
If you'd like to have header in assoc array, add something like this:
$headerArr = explode(PHP_EOL, $headerstring);
foreach ($headerArr as $headerRow) {
preg_match('/([a-zA-Z\-]+):\s(.+)$/',$headerRow, $matches);
if (!isset($matches[0])) {
continue;
}
$header[$matches[1]] = $matches[2];
}
Result print_r($header):
(
[content-type] => application/json
[content-length] => 2848
[date] => Tue, 06 Oct 2020 10:29:33 GMT
[last-modified] => Tue, 06 Oct 2020 10:17:17 GMT
)
Don't forget to close connection curl_close($ch);
Update the value of CURLOPT_HEADER to 0 for false
curl_setopt($ch, CURLOPT_HEADER, 0);
Just for a later use if anyone else needs. I was into same situation, but just need to remove header text, not content. The response i was getting in the header was (including white space):
HTTP/1.1 200 OK
Cache-Control: private, no-cache, no-store, must-revalidate
Content-Language: en
Content-Type: text/html
Date: Tue, 25 Feb 2014 20:59:29 GMT
Expires: Sat, 01 Jan 2000 00:00:00 GMT
Pragma: no-cache
Server: nginx
Vary: Cookie, Accept-Language, Accept-Encoding
transfer-encoding: chunked
Connection: keep-alive
I wanted to remove starting from HTTP till keep-alive with white space:
$contents = preg_replace('/HTTP(.*)alive/s',"",$contents);
that did for me.
If you are using nuSoap, you can access data without headers with $nsoap->responseData or $nsoap->response, if you want the full headers.
Just in case someone needs that.
If for some reason you have to curl_setopt($ch, CURLOPT_HEADER, 1); to get cookies for example, the following worked for me. Not sure if it's 100% reliable but worth a try
$foo = preg_replace('/HTTP(.*)html/s',"",$curlresult);
$content = null;
$ch = curl_init();
$rs = curl_exec($ch);
if (CURLE_OK == curl_errno($ch)) {
$content = substr($rs, curl_getinfo($ch, CURLINFO_HEADER_SIZE));
}
curl_close($ch);
echo $content;
If someone already saved the curl response to a file (like me) and therefore don't know how big the header was to use substr, try:
$file = '/path/to/file/with/headers';
file_put_contents($file, preg_replace('~.*\r\n\r\n~s', '', file_get_contents($file)));
Just do not set the curl_header in the curl request or set it to z or false
like this
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_HEADER, false);
Just don't set CURLOPT_HEADER!

Categories