currently im using this code to get the headers:
get_headers( $url, 1 );
wondering how i can do with curl (more faster); this is my curl code:
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, $url);
curl_exec($curl);
$info= curl_getinfo($curl);
curl_close($curl);
but i got "another" headers, i need the 'Location' header to see where go a bit.ly url or another redirect service.
Thanks in Advance.
If you want to extract the Location header, use this regex...
preg_match_all('/^Location: (?P<location>.*?)$/m', $headers, $matches);
var_dump($matches['location'][0]);
Ideone.
If you want to stop cURL from following the Location header, check out Charles' answer.
You want to turn the CURLOPT_FOLLOWLOCATION option is set to true.
edit:
Whoops, looks like you also need CURLOPT_RETURNTRANSFER to get the content back from curl_exec and CURLOPT_HEADER to make sure headers are included in that.
Related
I am trying to verify my recapture with google, but I am getting a response of null
I copy and paste the information to Postman and sent the request and I received a positive response.
I copied the link in my browser as a GET request and I also got a response.
I am not sure what causing this, as all information is correct.
here is my code.
// set API URL
$url = 'https://www.google.com/recaptcha/api/siteverify';
// Collection object
$data = [
'secret' => 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx', //<--- my reCaptcha secret key
'response' => $_POST['recaptcha']
];
// Initializes a new cURL session
$curl = curl_init($url);
// Set the CURLOPT_RETURNTRANSFER option to true
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
// Set the CURLOPT_POST option to true for POST request
curl_setopt($curl, CURLOPT_POST, true);
// Set the request data as JSON using json_encode function
curl_setopt($curl, CURLOPT_POSTFIELDS, json_encode($data));
// Execute cURL request with all previous settings
$response = curl_exec($curl);
// Close cURL session
curl_close($curl);
echo 'the response was ' . $response . PHP_EOL;
I saw this but didn't help me. PHP cURL not return a response, POSTMAN returns response
Because it is an HTTPS url you may need to add:
curl_setopt($curl,CURLOPT_SSL_VERIFYPEER, false);
It's a good idea to use CURLOPT_FOLLOWLOCATION.
I always use it. It is in my standard options.
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);
While Google does not return a 302 redirect HTTP status code, they do something funny with the initial request. I made an HTML form and submitted it and a json response was returned but looking at the Browser's headers my initial request disappeared.
I do not think Google wants the post data as JSON.
In the API Request documentation is says METHOD:POST.
It says noting about making the request with JSON.
Google is expecting an array.
Try removing the json_encode().
curl_setopt($curl, CURLOPT_POSTFIELDS, $data);
If Google (unlikely) wants a JSON request, you need to add
Content-Type: application/json to the HTTP header.
By adding this header curl will put the "post data" in the body and will not use the
default POST header: application/x-www-form-urlencoded
$request = array();
$request = 'Content-Type: application/json';
curl_setopt($curl, CURLOPT_HTTPHEADER, $request);
I do not understand why there was no response. I would have at least expected:
{
"success": false,
"error-codes": [
"missing-input-secret"
]
}
You may want to add this code after your curl_exec($curl)
This should give you all the details of you request and Google's response.
$response = curl_exec($curl);
$info = rawurldecode(var_export(curl_getinfo($curl),true));
echo "<pre>\n$info<br>\n</pre>";
If you want to see your outgoing request header (recommended) add this option and the header will be in the curl_getinfo:
curl_setopt($curl, CURLINFO_HEADER_OUT, true);
I'm a bit concerned about a NULL being returned. Was the word NULL returned? Or did you see nothing for $response in your string?
curl does not return a NULL. So maybe it was a false. Meaning there is likely a typo. echo does not show boolean false if $response was false you would not see it. Maybe add a
if($response == false){echo "curl failed<br>";}
Even if it failed, all the above is still true.
I looked over your code and I see nothing that would cause curl to fail. Even with the issues, there still should have been some sort of response. And there may have been an HTTP status code that would not show in your $response. It would be in the curl_getinfo
Are you using your local machine in linux? If that so, it is probably because you don't have curl installed in your system. So do
sudo apt install php{version}-curl
I can't figure out how to redirect after CURL executing. I found something like
curl_setopt($ch, CURLOPT_AUTOREFERER, TRUE);
But I need something like
curl_setopt($ch, CURLOPT_AFTER_SUCCESS_GO_TO, "http://anotherpage.com");
Use CURLOPT_FOLLOWLOCATION if you want curl to automatically follow a "redirect" (which is a 3XX response and a Location: response header).
If you just want to fetch another URL after the first request succeeds, then just issue another one...
I want to use curl to scrape multiple pages of an online shop. The problem that i have is that the urls are seo friendly - or something like that - and they look like this:
https://shopname.com/product-id-title-of-a-product.html
If i use the entire url it works and i'm able to get the data that i'm looking for but the only variable in that title that i know is the ID:
https://shopname.com/product-294
Is there a way to scrape that url in this case?
The url that only has the ID in it does REDIRECT to the full url.
And this is the code that i'm using:
$curl = curl_init();
$url = 'https://shopname.com/product-294';
curl_setopt($curl, CURLOPT_URL, $url);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
$result = curl_exec($curl);
Curl provides the option CURLOPT_FOLLOWLOCATION.
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);
The documentation states:
TRUE to follow any "Location: " header that the server sends as part of the HTTP header (note this is recursive, PHP will follow as many "Location: " headers that it is sent, unless CURLOPT_MAXREDIRS is set).
Therefore it would be advisable to set CURLOPT_MAXREDIRS aswell, for example to limit the execution to 1 redirection:
curl_setopt($curl, CURLOPT_MAXREDIRS, 1);
Like this you should be automatically be redirected to the original url without any further programming.
I think you need to capture the response headers in the curl object, that should contain the redirect url within them, and then you can parse that out and do a second curl request to get the url you are after.
Try using an app like postman or insomnia to assist you in this process.
I need to get the location header. From what I've read, this should be as simple as
curl_setopt($c, CURLOPT_HEADER, true);
curl_setopt($c, CURLOPT_FOLLOWLOCATION, false);
If I don't include those two options, the curl request works fine, but I'm not able to get the location header.
If I do include those two options, then I get a 401 error.
The Location that is being returned should be a URL that does require an additional login. What am I doing wrong?
If it makes a difference, I'm doing a PUT.
Update:
Turns out I was looking too much at the trees and not enough at the forest. When generated the information needed for a response to this question I realized the issue was actually with the call prior to this call.
Before the PUT, I need to get a session token. Since I wasn't parsing out the headers when getting the session token, I was getting a blank session token which was resulting in the 401 for the PUT.
HTTP response headers are included to the result of curl_exec when you set CURLOPT_HEADER option. So if you want to get response headers you have to extract it from the result using substr() and curl_getinfo(CURLINFO_HEADER_SIZE).
Here is a sample:
$c = curl_init();
// setting options including CURLOPT_HEADER
// ...
$result = curl_exec($c);
$info = curl_getinfo($c);
curl_close($c);
// extracting headers from result:
$n = $info['header_size'];
$headers = rtrim(substr($result, 0, $n));
$content = substr($result, $n);
i use curl to get the data (http request) from a website through php. and some of the information of the data is stored inside the header information. i can simply fetch the data using curl_exec, but if i try to fetch the header information using curl_getinfo, the information is missing. it supposed to be worked just like ajax. can somebody help me with this?
curl_getinfo doesn't give you the headers, it just gives meta information about the last request. The CURLOPT_HEADER options makes sure the headers are included in the output:
...
curl_setopt($c, CURLOPT_HEADER, true);
$data = curl_exec($c);
list($headers, $body) = explode("\n\n", $data, 2);