I have been messing with CURL for the past day and I cannot seem to figure out how to parse out the return data. I know I could write a REGEX to extract data from the response but it seems like there is some function I am probably missing. Here is what I would like to try and do.
I want to make POST to a different domain and get back 3 things
1. the response headers
2. the response data
3. a session cookie
Is there a way I can get those 3 things back separately? right now I just get back a plain text response with the response header and the response data. I would like to be able to do something like
$Response = curl_exec($Curl_Connection);
$ResponseData = $Response['Data'];
$ResponseHeader = $Response['Header'];
ResponseCookie = $Response['Cookie'];
Does curl provide anything like this?
As I answered:
no post data returned when requesting headers CURLOPT_HEADER. PHP CURL
Add:
curl_setopt($Curl_Connection, CURLOPT_HEADER, TRUE);
$Response = curl_exec($Curl_Connection);
$curlHeaderSize=curl_getinfo($ch,CURLINFO_HEADER_SIZE);
$ResponseData = mb_substr($result, $curlHeaderSize);
$ResponseHeader = mb_substr($result, 0, $curlHeaderSize);
preg_match_all('|Set-Cookie: (.*);|U', $ResponseHeader, $content);
$ResponseCookie = implode(';', $content[1]);
According to the curl docs:
/* TRUE to include the header in the output. */
curl_setopt($Curl_Connection, CURLOPT_HEADER, TRUE);
So the header is added to the output of curl_exec command. I've been spitting this out and I don't see any way to retrieve the header separately from the response body. It gets worse when you start reading compressed output (zipped, inflated)
On top of that, it's a one big string, not an array, so in case you you want the header in a format like for example curl_getinfo brings back an array.
The easiest is to do this:
$backend_output = curl_exec($ch);
list( $backend_response_headers, $backend_response_body)
= explode("\r\n\r\n", $backend_output, 2);
That will split those 2 up, but you end up with a string for the response header, not an array which would be so much more helpful. Now making an arrray from that isn't so evident, even with a regex as you can't split on for example something simple like /(\w)\s:(\w)/ as ':' can occur in certain fields. It would be very cool if curl would offer the headers seperately but so far as I go through the docs, it doesn't seem to be there.
Concerning your session cookies, I believe you need to use CURLOPT_COOKIESESSION = true option for that, but I have less experience on cookies as I hardly ever have the professional need to use them. Good luck
update: The headers you can parse with http://php.net/manual/en/function.http-parse-headers.php or a custom function from the user comment section if you lack pecl.
Related
I tried to find any widget to show Tinkoof's bank currency rate, because it changes every 60sec., but nothing.
Finally I found this API, but there is no any documentation for it.
I tried to find any articles about parsing, but I guess there's no use in that because of absence of any tag.
I need to show the currency rate on my website via this API. Any idea?
Big thanks!
You just need to fetch the content You can use cURL or file_get_contents()
cURL version:
<?php
$url = "https://www.tinkoff.ru/api/v1/currency_rates";
$curl = curl_init($url);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, TRUE);
$r = curl_exec($curl);
curl_close($curl);
$array = json_decode($r, true);
echo "<pre>";
print_r($array);
echo "</pre>";
?>
file_get_contents version:
<?php
$r = file_get_contents('https://www.tinkoff.ru/api/v1/currency_rates');
echo "<pre>";
echo print_r(json_decode($r, true));
echo "</pre>";
?>
Both of them will work unless the remote website requires you to be human (has extra verifications to stop robot requests). cURL would be a better way if that were the case because you can fake a user agent using a header array.
Once you have the array build it's just a matter of accessing the required data. using $r as an array result of the remote json structure.
It looks pretty straightforward to me. For someone with a decent knowledge of PHP will do this, provided the output is:
Now with the above information, I would:
Get the result to PHP using file_get_contents().
Parse the result as an array using json_decode($contents, $true).
Using the above result, I would get display the value using: $output["payload"]["rates"][0]["buy"] or something similar.
At this time of writing, the above will get me 58:
Okay, I haven't been able to find a solution to this as of yet, and I need to start asking questions on SO so I can get my reputation up and hopefully help out others.
I am making a wordpress plugin that retrieves a json list of items from a remote site. Recently, the site added a redirecting check for a cookie.
Upon first request without the cookie, 302 headers are provided, pointing to a second page which also returns a 302 redirect pointing to the homepage. On this second page, however, the set-cookie headers are also provided, which prevents the homepage from redirecting yet again.
When I make a cURL request to a url on the site, however, it fails in a redirect loop.
Now, obviously the easiest solution would be to fix this on the remote server. It should not be implementing that redirect for api routes. But that at the moment is not an option for me.
I have found how to retrieve the set-cookie header value from a 2** code response, however I cannot seem to figure out how to access that value when 302 headers are provided, and cURL returns nothing but an error.
Is there a way to access the headers even when it reaches the maximum (20) redirects?
Is it possible to stop the execution after a set number of redirects?
How can I get this cookie's value so I can provide it in a final request?
If you use the cURL option CURLOPT_HEADER the data you get back from curl_exec will include the headers from each response, including the 302.
If you enable cookie handling in cURL, it should pick up the cookie set by the 302 response just fine unless you prefer to handle it manually.
I often do something like this when there could be multiple redirects:
$ch = curl_init($some_url_that_302_redirects);
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_COOKIEFILE, ''); // enable curl cookie handling
$result = curl_exec($ch);
// $result contains the headers from each response, plus the body of the last response
$info = curl_getinfo($ch); // info will tell us how many redirects were followed
for ($i = 0; $i < intval($info['redirect_count']); ++$i) {
// get headers from each response
list($headers, $response) = explode("\r\n\r\n", $response, 2);
// DO SOMETHING WITH $headers HERE
// If there was a redirect, headers will be all headers from that response,
// including Set-Cookie headers
}
list($headers, $body) = explode("\r\n\r\n", $response, 2);
// Now $headers are the headers from the final response
// $body is the content from the final response
You already had problems before you started trying to add cookies into the mix. Doing a single redirect is bad for performance. Using a 302 response as a means of dissociating data presentation from data retrieval under HTTP/1,1 or later is bad (it works, but is a violation of the protocol - you should be using a 303 if you really must redirect).
Trying to set a cookie in a 3xx response will not work consistently across browsers. Setting a cookie in an Ajax response will not work consistently across browsers.
It should not be implementing that redirect for api routes
Maybe the people at the remote site are trying to prevent you leeching their content?
Fetch the homepage first in an iframe to populate the cookie and record a flag in your domain on the browser.
I actually found another SO question, of course after I posted, that lead me in the right direction to make this possible, HERE
I used the WebGet class to make the curl request. It has not been maintained for three years, but it still works fine.
It has a function that makes the curl request without following through on the redirect loop.
There are a lot of curl options set in that function, and curl is not returning an error in it, so I'm sure the exact solution could be simpler. HERE is a list of curl options for anyone who would like to delve deeper.
Here is how I handle each of the responses to get the final response
$w = new WebGet();
$cookie_file = 'cookie.txt';
if (!file_exists($cookie_file)) {
$cookie_file_inter = fopen($cookie_file, "w");
fclose($cookie_file_inter);
}
$w->cookieFile = $cookie_file; // must exist and be writable
$w->requestContent($url);
$headers = $w->responseHeaders;
if ($w->responseStatusCode == 302 && isset($headers['LOCATION'])) {
$w->requestContent($headers['LOCATION']);
}
if ($w->responseStatusCode == 302 && isset($headers['LOCATION'])) {
$w->requestContent($headers['LOCATION']);
}
$response = $w->cachedContent;
Of course, this is all extremely bad practice, and has severe performance implications, but there may be some rare use cases that find themselves needing to do this.
I've got a cURL request that works from the command line, but I can't figure out how to translate it into the PHP implementation of cURL. I believe that my issue is with the formatting of the data that's being sent, but I'm not 100% sure that is the case. This is the curl command I want to send:
# curl -X PUT -d '{"shared_link": {"access":"open"}}' \
-H "Authorization: Bearer ACCESSCODE" https://api.example.com/files/12345
I know that command works! So here's how I'm trying to reproduce it in PHP (where I have a variety of other curl commands working, but none quite like this).
$url = 'https://api.example.com/files/1234';
$header = array('Authorization: Bearer ACCESSCODE');
$data = array('shared_link'=>array('access'=>'open'));
$ch = curl_init();
curl_setopt($ch,CURLOPT_URL,$url);
curl_setopt($ch,CURLOPT_CUSTOMREQUEST,'PUT');
curl_setopt($ch,CURLOPT_RETURNTRANSFER,true);
curl_setopt($ch,CURLOPT_POSTFIELDS,$data);
curl_setopt($ch,CURLOPT_HTTPHEADER,$header);
$output = curl_exec($ch);
curl_close($ch);
return $output;
What I expect to get (and what I do get, when running the command from the command line) is a JSON response from the API server containing all the info about the file, including a change to the 'shared_link' value. What I get instead is a JSON response containing all the info about the file, with the 'shared_link' value unchanged. This is identical to if I were sending a GET request, or a PUT request where the format of the data was valid, but which didn't match any of the file's fields that were possible to update.
So I don't know if the problem with my request is the format of the data (this is what I think is the most likely) or whether I'm correctly configuring curl to do a PUT. I believe I'm doing that the way I've seen it described in a number of other examples, but it still seems a bit strange to me, so I can't be totally sure I'm doing it correctly. In addition, there could be some other area where I'm making a mistake that I don't recognize!
I've tried a number of different ways of formatting the data, none of which worked for me, including:
// This fails with 400 bad request
$data = http_build_query(array('shared_link'=>array('access'=>'open')));
// So does this
$data = urlencode("shared_link[access]=open");
// This fails because the JSON gets converted into objects,
// which POSTFIELDS won't accept
$data = json_decode('{"shared_link": {"access":"open"}}');
I'm running out of things to try. Can someone help me figure out what I'm doing wrong? And if there's any relevant information that I've left out, just let me know and I'll be happy to provide it.
Thanks in advance.
EDIT:
So the answer, it turns out, was so obvious that I overlooked it!
All I had to do was:
$data = '{"shared_link": {"access": "open"}}'
So yeah, question answered. Thanks CBroe!
In my (sort of) defense, the documentation for CURLOPT_POSTFIELDS (http://php.net/manual/en/function.curl-setopt.php) says that the value for that option has to be either an array or a urlencoded string, which is what I was going on. So I was working under that assumption, which was clearly mistaken, since what worked is definitely neither an array nor a urlencoded string.
You can try this
$post_body = http_build_query(
array(
'shared_link[access]' => 'open'
)
);
After that, pass curl opt as curl_setopt($curl, CURLOPT_POSTFIELDS, urldecode($post_body));
I'm using a file_get_contents to interact with an api for simple GET requests... however sometimes it throws headers signifying there's been an error. How can I get these headers and determine if there's a problem?
Php will set $http_response_header after file_get_contents which contains the response headers as an array of header lines/strings. Its not necessary to use curl if all you want is the headers responses (and probably shouldn't, some LAMP stacks still don't have cURL).
Doc on $http_response_header: http://php.net/manual/en/reserved.variables.httpresponseheader.php
Example:
file_get_contents('http://stacksocks.com');
foreach ($http_response_header as $header)
{
echo $header . "<br>\n";
}
Tips taken from post in comments:
1) The value changes with each request
made.
2) When used in methods/functions, the
current value must be passed to the
method/function. Using
$http_response_header directly in the
method/function without being assigned
a value by a function/method parameter
will result in the error message:
Notice: Undefined variable:
http_response_header
3) The array length and value
locations in the array may change
depending on the server being queried
and the response received. I'm not
sure if there are any 'absolute' value
positions in the array.
4) $http_response_header ONLY gets
populated using file_get_contents()
when using a URL and NOT a local file.
This is stated in the description when
it mentions the HTTP_wrapper.
Use curl instead of file_get_contents.
See: http://www.php.net/manual/en/curl.examples-basic.php
I imagine if your communicating with a REST Api then your actaully wanting the Http Status code returned. In which case you could do something like this:
<?php
$ch = curl_init("http://www.example.com/api/users/1");
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_exec($ch);
if(curl_getinfo($ch, CURLINFO_HTTP_CODE) == 501) {
echo 'Ops it not implemented';
}
fclose($fp);
?>
file_get_contents('http://example.com');
var_dump($http_response_header);
I've got a simple php script to ping some of my domains using file_get_contents(), however I have checked my logs and they are not recording any get requests.
I have
$result = file_get_contents($url);
echo $url. ' pinged ok\n';
where $url for each of the domains is just a simple string of the form http://mydomain.com/, echo verifies this. Manual requests made by myself are showing.
Why would the get requests not be showing in my logs?
Actually I've got it to register the hit when I send $result to the browser. I guess this means the webserver only records browser requests? Is there any way to mimic such in php?
ok tried curl php:
// create curl resource
$ch = curl_init();
// set url
curl_setopt($ch, CURLOPT_URL, "getcorporate.co.nr");
//return the transfer as a string
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
// $output contains the output string
$output = curl_exec($ch);
// close curl resource to free up system resources
curl_close($ch);
same effect though - no hit registered in logs. So far it only registers when I feed the http response back from my script to the browser. Obviously this will only work for a single request and not a bunch as is the purpose of my script.
If something else is going wrong, what debugging output can I look at?
Edit: D'oh! See comments below accepted answer for explanation of my erroneous thinking.
If the request is actually being made, it would be in the logs.
Your example code could be failing silently.
What happens if you do:
<?PHP
if ($result = file_get_contents($url)){
echo "Success";
}else{
echo "Epic Fail!";
}
If that's failing, you'll want to turn on some error reporting or logging and try to figure out why.
Note: if you're in safe mode, or otherwise have fopen url wrappers disabled, file_get_contents() will not grab a remote page. This is the most likely reason things would be failing (assuming there's not a typo in the contents of $url).
Use curl instead?
That's odd. Maybe there is some caching afoot? Have you tried changing the URL dynamically ($url = $url."?timestamp=".time() for example)?