I am using my webserver for testing, and I am not too sure about how to use the headers of the request right.
It contains of some values that can be set with the curl_setopt() function like the CURLOPT_REFERER, and on the other hand I can simply set up all the header myself using CURLOPT_HTTPHEADER
This is how I am doing this now:
curl_setopt($curl, CURLINFO_HEADER_OUT, true);
$headers=array("Host: localhost/mySite/index.php","Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8","Accept-Language: en-us,en;q=0.5","Accept-Encoding: gzip, deflate","Connection: keep-alive","Referer: https://localhost/mySite/index.php");
curl_setopt($curl, CURLOPT_HTTPHEADER, $headers);
But some issues rise when I set the cookie for example.
I see in my webrowser that I have on the very first request a cookie value:
Cookie: ASP.NET_SessionId=rrnrwk453nraysm4u1adlrjk;
ASPSESSIONIDQCDSDBDD=DLHLOOIAJEMBHJODKPGMNEED; ASPSESSIONIDQADQDACD=EMNBELFBMNKGBFKIOOHFPPGI
and for the much that I know the cookie is a response from the server, so this doesn't quite makes sense for me.
My question is, what should I do if I wanted to imitiate my webrowser, how should I go about this?
How do I build the REQUEST HEADER? is it simply copying the contents of HTTPLiveHeaders ?
But what about the cookies ?
Am I doing things right here ?
Related
I`m using cuRL to get some data from remote server... The response is in JSON format..
This is my code:
$ch = curl_init();
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER , 1);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)");
curl_setopt($ch, CURLOPT_URL, 'http://www.myaddress.com/mypage.php');
curl_setopt($ch, CURLOPT_POSTFIELDS, array("id" => $id));
$return = curl_exec($ch);
curl_close($ch);
If I access the link in the browser the page load OK, but if I access through the cuRL return a 404 error...
I can guess a few things that it can be checked from the server side, to show the error.
1) As it is stated in other answers, be sure to set all the necessary headers, you can check them e.g. by firebug, as it is shown in here,
or you can get the headers by php get_headers function.
to set it use
curl_setopt($ch, CURLOPT_HTTPHEADER, array("HeaderName: HeaderValue"));
2) When you open a page in the browser(excluding form submit with post method) it makes a get request, instead of post, so if in the server side it is checked $_GET, then your post request will not be considered.
3) If you sure that it should be a post request(say, it is a form submit), then the following can be a problem: some forms can have hidden fields, that again are being checked in the server, and if they are not set, error can be returned. So, you should look at the source code of the form and add them(if there are any) to your post parameters.
4) if you are submitting a form, be sure to set the submit button with its name and value as well, because similar to hidden fields, this can be checked as well.
5) Cookies can be a problem as well, because by default browser has it , and curl does not. To to able to set and read cookies use this code
// set cookie
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie_file);
// use cookie
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie_file);
here, $cookie_file path to the cookies file. Do not know in linux or mac, but in windows be sure to use absolute path to the cookie file.
6) Also, you can set the referer by
curl_setopt($ch, CURLOPT_REFERER, 'http://www.myaddress.com/mypage.php');
EDIT: In case of ajax request you might want to add a header X-Requested-With with value as XMLHttpRequest
It's possible the server check the HTTP Header, it's the case in the majority of case.
So add the same HTTP Header of your browser, verify with Firebug :
curl_setopt($ch, CURLOPT_HTTPHEADER, array('SomeName: SomeValue'));
Probably there is something else the browser is sending your cURL code is not. You can use any of the tools other folks have suggested, Firebug, Wireshark, Fiddler, etc, etc.
What you need to do is add missing pieces to your request to match the browser as closely as possible in the cURL request until the remote page responds with a 200.
I notice you're doing a POST. In many cases what happens with your browser is you visit a page with a GET request. A session is initialized on the remote site and a cookie is saved in your browser with the session id.
This cookie then needs to be supplied along with subsequent POST requests. PHP cURL has many options to support cookies. There may be other requirements such as CSRF tokens and so forth.
Again, reverse-engineering is the key.
I have, of course, read several questions with exactly this asked, but I have to say it didn't work for me at all. What I am about to accomplish is
sending 'X-Requested-With: XMLHttpRequest' header via PHP and curl
sending other http request headers via PHP and curl
provided solutions didn't work for me.
How do I know I'm not sending right http request headers?
Simply by
(1)comparing real headers generated by XMLHttpRequest(triggering JQuery click) and those simulated by PHP and curl in Firefox add-on Live HTTP headers
(2)Print_r() -ing $_SERVER variable in target script
What do I get that is incorrect/below my expectations?
First and most important:
Firefox Live HTTP headers does not capture my headers (just like they don't exists).
Second, by print_r($_SERVER):
if I get anything of simulated headers at all, I get [HTTP_X_REQUESTED_WITH] => XMLHttpRequest - not the: [X_REQUESTED_WITH] => XMLHttpRequest.
That problem persists almost for any header I send via curl_setopt($ch, CURLOPT_HTTPHEADER, $curl_header) - any of these is being prefixed with 'HTTP' ('Header1: value1' - I get 'HTTP_HEADER1').
I'm using XAMPP with PHP version 5.4.7, CURL 7.24.0 .
Before I ask if what I'm trying to accomplish is possible or maybe not and say thanks in advance for responses, it's not bad idea to provide my code - one of many code solutions that I've tried.
$curl_header = array('X-Requested-With: XMLHttpRequest');
$data = "name=miloshio"; // just to be sure I'm doing the POST request
$ch = curl_init('http://example.com/test.php');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, $curl_header);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $data);
curl_setopt($ch, CURLOPT_VERBOSE, true);
$result = curl_exec($ch);
echo $result;
Sum of my questions:
Is it possible to send exactly 'X-Requested-With: XMLHttpRequest'
header via PHP and curl?
Is it possible to avoid attaching 'HTTP_' prefix to custom headers
send by PHP and curl?
Are there well-known limitations in matter of using PHP and curl?
Firefox Live HTTP headers won't show your headers as they're sent by the server to another server and not to the client(browser).
Curl send the headers correctly, using CURLOPT_PROXY You can try to put curl traffic through a debuging proxy like Fiddler if You're using windows for development, I'm sure there are linux alternatives
If you try to get the headers from $SERVER variable, they will be prefixed with HTTP, you can use apache_request_headers to get the headers without HTTP_ prefix.
I want to set a request header for a url xyz.com
is it the right way to set it in php?
header('Authorization: AuthSub token="xxxxxx"');
header('location:https://www.google.com/accounts/AuthSubRevokeToken');
I am trying to set the header for this URL for a call.But the Authorization: AuthSub header doesnt shows up in the request headers section of the FireFox NET panel.Which is used to show the requests.
Any idea about it?
Thanx.
I was using curl previously,But it didnt seemed to issue any request as i cant see it in the NET panel of FireFox.
Code is as follows:
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL,"https://www.google.com/accounts/AuthSubRevokeToken");
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl, CURLOPT_FAILONERROR, true);
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($curl, CURLOPT_HTTPHEADER, array('Authorization: AuthSub token="1/xxx"'
));
$result = curl_exec($curl);
curl_close($curl);
echo 'hererer'.$result;exit;
header sets response headers, not request headers. (If you were trying to send a HTTP request elsewhere, it would have no effect.)
Please also note what the manual says about Remember that header() must be called before any actual output is sent, ....
And turn on error_reporting(E_ALL); before using header() to see if that is the issue for you.
Header names and values need to be separated by one colon plus a space, so the location "header" is just wrong, it should be:
header('Location: https://www.google.com/accounts/AuthSubRevokeToken');
(It's common to write the case this way, too, but not a need)
Next to that the header function is setting response headers, not request headers. So you're basically using the wrong tool.
In PHP you can not set request headers, that's part of the client (e.g. browser), not the server. So header just looks wrong here. Which HTTP client are you using?
A call, as in using CURL to request another page? The header() function applies only for web-browser<->server communications. It cannot affect any requests your server-side script does to other webservers. For that, you need to modify the particular method you're using, e.g. curl or streams.
For curl, see CURLOPT_HTTPHEADER here: http://php.net/curl_setopt
I'm using a web-service from a provider who is being a little too helpful in anticipating my needs. They have given me a HTML snippet to paste on my website, for users to click on to trigger their services. I'd prefer to script this process, so I've got a php script which posts a cURL request to the same url, as appropriate. However, this provider is keeping tabs on my session, and interprets each new request as an update of the first one, rather than each being a unique request.
I've contacted the provider regarding my issue, and they've gone so far as to inform me that their system is working as intended, and that it's impossible for me to avoid using the same ASP.NET session for each subsequent cURL request. While my favored option would be to switch to a different vendor, that doesn't appear to be an option right now. Is there a reliable way to get a new ASP.NET session with each cURL request?
I've tried the following set of CURLOPT's, to no avail:
//initialize curl
$ch = curl_init($url);
//build a string out of the post_vars
$post_str = http_build_query($post_vars);
//set the necessary curl options
curl_setopt($ch, CURLOPT_TIMEOUT, 30);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FAILONERROR, 1);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $post_str);
curl_setopt($ch, CURLOPT_FAILONERROR, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_COOKIESESSION, 1);
curl_setopt($ch, CURLOPT_FRESH_CONNECT, 1);
curl_setopt($ch, CURLOPT_FORBID_REUSE, 1);
curl_setopt($ch, CURLOPT_USERAGENT, "UZ_".uniqid());
curl_setopt($ch, CURLOPT_REFERER, CURRENT_SITE_URL."index.php?newsession=".uniqid());
curl_setopt($ch, CURLOPT_HTTPHEADER, array("Pragma: no-cache", "Cache-Control: no-cache"));
//execute the call to the backend script, retrieve the results
$xmlstr = curl_exec($ch);
If cURL isn't helping much, why not try other methods to call the services from your script, like php's file() function, or file_get_contents().
If you see do not see any difference at all, then the service provider might be using your ip to track your requests. Try using some proxy for a test.
Normal Asp.net session is tracked by a cookie called ASP.NET_SessionId. This cookie is sent within the response to your first request. So as long as your curl requests don't send back this asp.net cookie, each of your requests will have no connection to each other. Use the curl -c option to see what cookies are flying in-between you and them. Overriding this cookie with a cookie file should work if you confirm that it is normal asp.net session being used here.
It is quite poor for a service to use session (http has much cleaner ways of maintaining state which ReST exploits) so I wouldn't completely rule out the vendor switch option.
Well given the options you are using, it seems you have covered your basics. Can you find out how their sessions are setup?
If you know how they setup a session, IE what they use (if it is IP or what not) and then you can figure out a work around. Another option is trying to set the cookies in a different cookie file:
CURLOPT_COOKIEFILE - The name of the file containing the cookie data. The cookie file can be in Netscape format, or just plain HTTP-style headers dumped into a file.
But if all they do is check cookies your current code should work. If you can figure out what the cookie's name is, you can pass a custom cookie that is blank with the request to see if that works. But if you can get information out of them on how their session's work, that would be best.
use these two line to handle the session:
curl_setopt($ch, CURLOPT_COOKIEJAR, "path/to/cookies.txt"); // cookies.txt should be writable
curl_setopt($ch, CURLOPT_COOKIEFILE, "path/to/cookies.txt");
i have a hosted script somewhere that only accept POST request.
example, some.hosted/script.php
how can i setup another simple php that can accept GET request and then POST it to the hosted script.
so that i can put up a link like this: other.site/post2hostedscript.php?postthis=data
and then it POST postthis=data to the hosted script.
tnx
edit:
post2hostedscript.php do not give any result.
the result will go directly to some.hosted/script.php
just as if the user POST directly at the hosted script.
Your post2hostedscript.php will have to :
Fetch all parameters received as GET
Construct a POST query
Send it
And, probably, return the result of that POST request.
This can probably be done using curl, for instance ; something like this should get you started :
$queryString = $_SERVER['QUERY_STRING'];
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "http://www.othersite.com/post2hostedscript.php");
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, false);
curl_setopt($ch, CURLOPT_POSTFIELDS, $queryString);
curl_exec($ch);
curl_close($ch);
For a list of options that can be used with curl, you can take a look at the page of curl_setopt.
Here, you'll have to use, at least :
CURLOPT_POST : as you want to send a POST request, and not a GET
CURLOPT_RETURNTRANSFER : depending on whether you want curl_exec to return the result of the request, or to just output it.
CURLOPT_POSTFIELDS : The data that will be posted -- i.e. what you have in the query string of your incoming request.
And note that the response from the POST request might include some interesting HTTP header -- if needed, you'll have to fetch them (see the CURLOPT_HEADER option), and re-send the interesting ones in your own response (see the header function).
Take a look at the "curl" functions, they provide everything you need.
You might consider replacing all instances of $_POST in the old script to $_REQUEST, which will result in it accepting both GET and POST alike.