I can't figure out how to redirect after CURL executing. I found something like
curl_setopt($ch, CURLOPT_AUTOREFERER, TRUE);
But I need something like
curl_setopt($ch, CURLOPT_AFTER_SUCCESS_GO_TO, "http://anotherpage.com");
Use CURLOPT_FOLLOWLOCATION if you want curl to automatically follow a "redirect" (which is a 3XX response and a Location: response header).
If you just want to fetch another URL after the first request succeeds, then just issue another one...
Related
I want to use curl to scrape multiple pages of an online shop. The problem that i have is that the urls are seo friendly - or something like that - and they look like this:
https://shopname.com/product-id-title-of-a-product.html
If i use the entire url it works and i'm able to get the data that i'm looking for but the only variable in that title that i know is the ID:
https://shopname.com/product-294
Is there a way to scrape that url in this case?
The url that only has the ID in it does REDIRECT to the full url.
And this is the code that i'm using:
$curl = curl_init();
$url = 'https://shopname.com/product-294';
curl_setopt($curl, CURLOPT_URL, $url);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
$result = curl_exec($curl);
Curl provides the option CURLOPT_FOLLOWLOCATION.
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);
The documentation states:
TRUE to follow any "Location: " header that the server sends as part of the HTTP header (note this is recursive, PHP will follow as many "Location: " headers that it is sent, unless CURLOPT_MAXREDIRS is set).
Therefore it would be advisable to set CURLOPT_MAXREDIRS aswell, for example to limit the execution to 1 redirection:
curl_setopt($curl, CURLOPT_MAXREDIRS, 1);
Like this you should be automatically be redirected to the original url without any further programming.
I think you need to capture the response headers in the curl object, that should contain the redirect url within them, and then you can parse that out and do a second curl request to get the url you are after.
Try using an app like postman or insomnia to assist you in this process.
In my PHP software I have an self-update feature which sends a HTTP request with the POST method to a certain URL (to a PHP script). Now this URL has changed (I moved the script to another directory), but to stay backward compatible I want to use the script at the old URL to redirect the POST request to the new location. I tried to use the HTTP 307 status code but the second time PHP makes the request, it changes the method from POST to GET, although it must not do this (at least I though this is what the 307 code is for). I use PHP 5.4.29 on Windows 7 as Apache (2.2.27) module and I sniffed the traffic to make sure that HTTP 1.1 is used in the request and the response.
This is how I make a POST request:
<?php
$requestData = http_build_query(
array(
"param1" => "value1",
// and so on...
)
);
$requestOptions = array("http"=>
array
(
"protocol_version"=>"1.1",
"method"=>"POST",
"header"=>array(
"Content-type: application/x-www-form-urlencoded",
"Connection: close",
),
"content"=>$requestData,
)
);
$requestContext = stream_context_create($requestOptions);
$serverResponse = #file_get_contents("http://localhost/old/long/path/update.php", false, $requestContext);
?>
I tried to redirect manually and automatically by PHP:
<?php
// Redirect manually
header("HTTP/1.1 307 Temporary Redirect");
header("Location: http://localhost/update.php");
// or redirect automatically
header("Location: http://localhost/update.php", true, 307);
?>
According to the sniffed data, everything looks normal. HTTP 1.1 is used in request and response and code 307 is used. But the second time PHP sends the request (to the new location, still with HTTP 1.1, ..) it simply changes the method to GET and my POST payload is lost.
Again: This is not a user / browser redirection - I redirect PHP. I make my request myself and manually though my software and I want to redirect it to a new location. This has nothing to do with a security related topic.
It looks like file_get_contents does not repost the data, possibly for the reason highlighted by #daiscog.
Curl however will repost to the redirected url:
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'http://localhost/old/long/path/update.php');
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_FRESH_CONNECT, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $requestData);
$serverResponse = curl_exec($ch);
However it would make more sense to either handle this at the server level (eg an Apache url rewrite) or to simply include the new file in the old one:
//old.php
include('path/to/new.php');
The W3C HTTP/1.1 specification for 307 states:
If the 307 status code is received in response to a request other than
GET or HEAD, the user agent MUST NOT automatically redirect the
request unless it can be confirmed by the user, since this might
change the conditions under which the request was issued.
The point behind this is that it is unsafe for a server to tell a client to POST to a different URL as that may be a malicious attempt to get the client to send data somewhere the user did not intend, so POST redirections are not possible.
I want to set a request header for a url xyz.com
is it the right way to set it in php?
header('Authorization: AuthSub token="xxxxxx"');
header('location:https://www.google.com/accounts/AuthSubRevokeToken');
I am trying to set the header for this URL for a call.But the Authorization: AuthSub header doesnt shows up in the request headers section of the FireFox NET panel.Which is used to show the requests.
Any idea about it?
Thanx.
I was using curl previously,But it didnt seemed to issue any request as i cant see it in the NET panel of FireFox.
Code is as follows:
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL,"https://www.google.com/accounts/AuthSubRevokeToken");
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl, CURLOPT_FAILONERROR, true);
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($curl, CURLOPT_HTTPHEADER, array('Authorization: AuthSub token="1/xxx"'
));
$result = curl_exec($curl);
curl_close($curl);
echo 'hererer'.$result;exit;
header sets response headers, not request headers. (If you were trying to send a HTTP request elsewhere, it would have no effect.)
Please also note what the manual says about Remember that header() must be called before any actual output is sent, ....
And turn on error_reporting(E_ALL); before using header() to see if that is the issue for you.
Header names and values need to be separated by one colon plus a space, so the location "header" is just wrong, it should be:
header('Location: https://www.google.com/accounts/AuthSubRevokeToken');
(It's common to write the case this way, too, but not a need)
Next to that the header function is setting response headers, not request headers. So you're basically using the wrong tool.
In PHP you can not set request headers, that's part of the client (e.g. browser), not the server. So header just looks wrong here. Which HTTP client are you using?
A call, as in using CURL to request another page? The header() function applies only for web-browser<->server communications. It cannot affect any requests your server-side script does to other webservers. For that, you need to modify the particular method you're using, e.g. curl or streams.
For curl, see CURLOPT_HTTPHEADER here: http://php.net/curl_setopt
I'm trying to perform a redirect using cURL. I can load the page fine, that's not a problem, but if I load say google.com non of the images load and the site does not work (obviously because its just printing the HTML and not actually doing a redirect).
Is there any way to perform a redirect using cURL? Sort of similar to how ...
header("Location: http://google.com");
... works?
Any help would be much appreciated.
Well, from my understading, it seems like OP want's to redirect the user to the search results URL.
Using the GoogleAPI would be a first choice and to achieve something like that, I would do this:
<?php
$query = "firefox";
$apiUrl = "http://ajax.googleapis.com/ajax/services/search/web?v=1.0&q=".urlencode($query);
$ch = curl_init();
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_URL, $apiUrl);
$content = curl_exec($ch);
$content = json_decode($content);
$luckyUrl = $content->responseData->results[0]->unescapedUrl;
header("Location: ".$luckyUrl);
?>
The code above works like 'I feel lucky'....
Use curl with -L
-L/--location
(HTTP/HTTPS) If the server reports that the requested page has
moved to a different location (indicated with a Location: header
and a 3XX response code), this option will make curl redo the
request on the new place. If used together with -i/--include or
-I/--head, headers from all requested pages will be shown. When
authentication is used, curl only sends its credentials to the
initial host. If a redirect takes curl to a different host, it
won't be able to intercept the user+password. See also --loca‐
tion-trusted on how to change this. You can limit the amount of
redirects to follow by using the --max-redirs option.
When curl follows a redirect and the request is not a plain GET
(for example POST or PUT), it will do the following request with
a GET if the HTTP response was 301, 302, or 303. If the response
code was any other 3xx code, curl will re-send the following
request using the same unmodified method.
So when using cURL
add
curl_setopt($process, CURLOPT_FOLLOWLOCATION, 1);
I'm afraid it is impossible to force the client's browser to send certain POST values and refers, you can only force it to go somewhere, hence header().
Does this answer your question?
It's should to work.pls try this: header( 'Location: http://www.google.com' ).Use the (')single cote instead of "(double)
i have a hosted script somewhere that only accept POST request.
example, some.hosted/script.php
how can i setup another simple php that can accept GET request and then POST it to the hosted script.
so that i can put up a link like this: other.site/post2hostedscript.php?postthis=data
and then it POST postthis=data to the hosted script.
tnx
edit:
post2hostedscript.php do not give any result.
the result will go directly to some.hosted/script.php
just as if the user POST directly at the hosted script.
Your post2hostedscript.php will have to :
Fetch all parameters received as GET
Construct a POST query
Send it
And, probably, return the result of that POST request.
This can probably be done using curl, for instance ; something like this should get you started :
$queryString = $_SERVER['QUERY_STRING'];
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "http://www.othersite.com/post2hostedscript.php");
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, false);
curl_setopt($ch, CURLOPT_POSTFIELDS, $queryString);
curl_exec($ch);
curl_close($ch);
For a list of options that can be used with curl, you can take a look at the page of curl_setopt.
Here, you'll have to use, at least :
CURLOPT_POST : as you want to send a POST request, and not a GET
CURLOPT_RETURNTRANSFER : depending on whether you want curl_exec to return the result of the request, or to just output it.
CURLOPT_POSTFIELDS : The data that will be posted -- i.e. what you have in the query string of your incoming request.
And note that the response from the POST request might include some interesting HTTP header -- if needed, you'll have to fetch them (see the CURLOPT_HEADER option), and re-send the interesting ones in your own response (see the header function).
Take a look at the "curl" functions, they provide everything you need.
You might consider replacing all instances of $_POST in the old script to $_REQUEST, which will result in it accepting both GET and POST alike.