Is it possible to get partial response using PHP cURL? - php

Here is my code
$url = "partial_response.php";
$sac_curl = curl_init();
curl_setopt($sac_curl, CURLOPT_HTTPGET, true);
curl_setopt($sac_curl, CURLOPT_URL, $url);
curl_setopt($sac_curl, CURLOPT_RETURNTRANSFER, true);
curl_setopt($sac_curl, CURLOPT_HEADER, false);
curl_setopt($sac_curl, CURLOPT_TIMEOUT, 11);
$resp = curl_exec($sac_curl);
curl_close($sac_curl);
echo $resp;
Partial_response.php
header( 'Content-type: text/html; charset=utf-8' );
echo 'Job waiting ...<br />';
for( $i = 0 ; $i < 10 ; $i++ )
{
echo $i . '<br/>';
flush();
ob_flush();
sleep(1);
}
echo 'End ...<br/>';
From the about code am trying to get a partial response from partial_response.php. what I want is, I need curl to return me "Job waiting.." alone instead of waiting for the partial_response.php to complete the loop and return the entire data. so when I reduce CURLOPT_TIMEOUT below 11 i dont get any response at all. Kindly clarify my doubt.
Thanks in advance.

I later realized that cURL could not do what I want, I used stream_context_get_options
to achieve what I wanted. Here it is, http://www.php.net/manual/en/function.stream-context-get-options.php.

No, I'm afraid not. At least not that I know of any, this is simply because PHP is a synchronous language, meaning you cannot "skip" tasks. (I.e. curl_exec() will always - no matter what - be executed until the request is completed)

I'm not sure abut the timeout, but you can get partial response using cURL by using the CURLOPT_WRITEFUNCTION flag:
curl_setopt($ch, CURLOPT_WRITEFUNCTION, $callback);
Where $ch is the Curl handler, and $callback is the callback function name. This command will stream response data from remote site. The callback function can look something like:
$result = '';
$callback = function ($ch, $str) {
global $result;
//$str has the chunks of data streamed back.
$result .= $str;
// here you can mess with the stream data either with $result or $str.
// i.e. look for the "Job waiting" string and terminate the response.
return strlen($str);//don't touch this
};
If not interrupted at the end $result will contain all the response from remote site.
So combining everything will look something like:
$result = '';
$callback = function ($ch, $str) {
global $result;
//$str has the chunks of data streamed back.
$result .= $str;
// here you can mess with the stream data either with $result or $str.
// i.e. look for the "Job waiting" string and terminate the response.
return strlen($str);//don't touch this
};
$url = "partial_response.php";
$sac_curl = curl_init();
curl_setopt($sac_curl, CURLOPT_HTTPGET, true);
curl_setopt($sac_curl, CURLOPT_URL, $url);
curl_setopt($sac_curl, CURLOPT_RETURNTRANSFER, true);
curl_setopt($sac_curl, CURLOPT_HEADER, false);
curl_setopt($sac_curl, CURLOPT_TIMEOUT, 11);
curl_setopt($ch, CURLOPT_WRITEFUNCTION, $callback);
curl_exec($sac_curl); // the response is now in $result.
curl_close($sac_curl);
echo $result;

Related

Sending array using curl

I want to send data from server 1 to server 2, first I select necessary data from the database, but how to send data with curl? I understand that I cannot send $result parameter just like in my code, but how should I do this?
My Code server 1:
public function setDivisions(){
$result = $this->conn->query("SELECT *FROM data_divisions");
$ch = curl_init('https://example.com/api.php?siteid='.$this->site_key.'');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $result);
curl_setopt($ch, CURLOPT_POST, 1);
$response = curl_exec($ch);
print_r($response);
}
Code on server 2:
$array = $_POST['result'];
//loop throw array and insert data into database
you can use it that way.
$ch = curl_init('https://upxxx.cod3fus1ontm.com/curl/json');
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode((object)["records" => json_encode($result)]));
$response = curl_exec($ch);
var_dump($response);
on receipt, like this!
$json = file_get_contents("php://input");
$content = json_decode($json, true);
$records = json_decode($content['records'], true);
foreach($records as $record) {
echo $record['id'] . " - " . $record['text'] . "<br/>";
}
remember, that as you did to encode, you will have to do to decode
Come on, php://input returns all raw data after the request's HTTP headers, regardless of content type.
When we do this with file_get_contents (which reads the contents of a file and puts it into a variable of type string), we can read the content that was sent by curl.
Reviewing the code, you can optimize as follows, when sending to the server you placed, I suggested:
curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode((object)["records" => json_encode($result)]));
you can replace it with:
curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($result));
it's much simpler, but it's very important to put the query result inside a json_encode
finally, you can access the entire body of the message, through file_get_contents ("php://input") and put it inside a variable, which is nothing more than a JSON of the your query.
for you to understand how the parameters were passed, it is interesting to do the following:
$json = file_get_contents("php: // input");
var_dump($json); <- Here you see the thing as it is.
$records = json_decode($json, true); <- Here you generate an object with the content of json
var_dump($records);
With that, I think that solves the situation.
on server 1
$result = "result=".base64_encode($result)
curl_setopt($ch, CURLOPT_POSTFIELDS, $result);
...
on server 2
$array = base64_decode($_POST['result']);

Get response code, headers and contents from curl_exec

I can't seem to figure out a way to get response code, headers and contents from my curl_exec.
Here's my last attempt:
$url = '...';
$curl = curl_init($url);
curl_setopt($curl, CURLOPT_CAINFO, 'PATH_TO_.cer');
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
//curl_setopt($curl, CURLOPT_HEADER, true); //ATTEMPT1
/*curl_setopt($curl, CURLOPT_HEADERFUNCTION, function($curl, $header) use (&$headers){ //ATTEMPT2
$len = strlen($header);
$header = explode(':', $header, 2);
if (count($header) < 2){ // ignore invalid headers
return $len;
}
$headers[strtolower(trim($header[0]))][] = trim($header[1]);
print_r($headers);
return $len;
});*/
$contents = curl_exec($curl);
curl_close($curl);
I tried ATTEMPT1 and ATTEMPT2
In the headers of my requests, there are plenty of fields such as rate limits that I need to retrieve every time, but I also need the response code and the content of the request.
EDIT: So far I found that I'd need to use ATTEMPT1 over ATTEMPT2 because the last one is just an handcrafted version I can't seem to understand that is getting rid of the response code in the header.
You're looking for curl_getinfo()
For instance, to get the response code:
$http_code = curl_getinfo($curl, CURLINFO_HTTP_CODE);

How to obtain curl exec response

I am developing a PHP script that:
Gets 10 rows from DB (works well)
Sends addresses from these rows to Map API
Obtains data
Save results in DB (works well)
I don't know how to obtain the final response of curl_exec. curl_exec
in $response = call('GET', $query);
- even when hasn't completed returns something in response.
These my loop that is waiting for the response of the call function. But it doesn't work json_decode($stringJSON) is invoked earlier than it is obtained the final response
$requestHandled = false;
$response = "";
$response = call('GET', $query);
while(!$requestHandled) {
if(strlen($response) > 5){
$response = mb_convert_encoding($response, "UTF-8");
$stringJSON = get_magic_quotes_gpc() ? stripslashes($response) : $response;
echo $stringJSON;
$jsonObject = "+";
echo $jsonObject;
$jsonObject = json_decode($stringJSON);
echo $jsonObject;
$requestHandled = true;
}
}
This is my curl call function
function call($method, $url, $data = false) {
$ch = curl_init();
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, $method);
curl_setopt($ch, CURLOPT_URL, $url);
if ($data) {
curl_setopt($ch, CURLOPT_POSTFIELDS, $data);
$headers = array();
$headers[] = 'Content-Type: application/json';
$headers[] = 'Content-Length: ' . strlen($data);
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
}
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
return curl_exec($ch);
Please help. Has spent a half of the day solving it
So the result of the var_dump( $response ) is an empty string?
In that case, it simply means the response from the server is empty, because I just tested your function call() and it seems to work just fine. In other words, make sure the URL (with the method GET) you are trying to call actually returns data to begin with.
I wouldn't be surprised if you simply have a typographical error somewhere in the URL.
Also, for debugging purposes, temporarily replace
return curl_exec($ch);
with
$result = curl_exec($ch);
var_dump(curl_getinfo($ch));
return $result;
and investigate the info in the var_dump() result, to make sure the response code is 200 (or any other that indicates success) and not in the 4xx or 5xx ranges (respectively indicating either a client or server error), for instance.
See curl_getinfo() for more information about what useful information will be returned about your last curl transfer.
To address OPs comment(s):
Please try this complete script (nothing else: no while loop, no stripslashes(), no json_decode(), etc):
<?php
function call($method, $url, $data = false) {
$ch = curl_init();
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, $method);
curl_setopt($ch, CURLOPT_URL, $url);
if ($data) {
curl_setopt($ch, CURLOPT_POSTFIELDS, $data);
$headers = array();
$headers[] = 'Content-Type: application/json';
$headers[] = 'Content-Length: ' . strlen($data);
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
}
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$result = curl_exec($ch);
var_dump(curl_getinfo($ch)); // check this result and make sure the response code is 200
return $result;
}
$query = 'fill in the correct URL! (with http(s)://)';
$response = call('GET', $query);
var_dump( $response ); // check this result and see if it's an empty string
die;
If the var_dump( $response ); returns an empty string, it means your script works fine but the URL you are calling simply returned an empty response.
If $response is not empty and actually contains JSON data, replace
var_dump( $response );
with
$stringJSON = mb_convert_encoding( $response, "UTF-8" );
echo $stringJSON; // make sure this contains valid JSON data
// stripslashes() should not be needed
var_dump( json_decode( $stringJSON ) ); // if JSON data was valid, you should get a valid PHP data structure

Get Final URL From Double Shortened URL (t.co -> bit.ly -> final)

I couldn't convert a double shortened URL to expanded URL successfully using the below function I got from here:
function doShortURLDecode($url) {
$ch = #curl_init($url);
#curl_setopt($ch, CURLOPT_HEADER, TRUE);
#curl_setopt($ch, CURLOPT_NOBODY, TRUE);
#curl_setopt($ch, CURLOPT_FOLLOWLOCATION, FALSE);
#curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
$response = #curl_exec($ch);
preg_match('/Location: (.*)\n/', $response, $a);
if (!isset($a[1])) return $url;
return $a[1];
}
I got into trouble when the expanded URL I got was again a shortened URL, which has its expanded URL.
How do I get final expanded URL after it has run through both URL shortening services?
Since t.co uses HTML redirection through the use of JavaScript and/or a <meta> redirect we need to grab it's contents first. Then extract the bit.ly URL from it to perform a HTTP header request to get the final location. This method does not rely on cURL to be enabled on server and uses all native PHP5 functions:
Tested and working!
function large_url($url)
{
$data = file_get_contents($url); // t.co uses HTML redirection
$url = strtok(strstr($data, 'http://bit.ly/'), '"'); // grab bit.ly URL
stream_context_set_default(array('http' => array('method' => 'HEAD')));
$headers = get_headers($url, 1); // get HTTP headers
return (isset($headers['Location'])) // check if Location header set
? $headers['Location'] // return Location header value
: $url; // return bit.ly URL instead
}
// DEMO
$url = 'http://t.co/dd4b3kOz';
echo large_url($url);
Finally found a way to get the final url of a double shortened url. The best way is to use longurl api for it.
I am not sure if it is the correct way, but i am at last getting the output as the final url needed :)
Here's what i did:
<?php
function TextAfterTag($input, $tag)
{
$result = '';
$tagPos = strpos($input, $tag);
if (!($tagPos === false))
{
$length = strlen($input);
$substrLength = $length - $tagPos + 1;
$result = substr($input, $tagPos + 1, $substrLength);
}
return trim($result);
}
function expandUrlLongApi($url)
{
$format = 'json';
$api_query = "http://api.longurl.org/v2/expand?" .
"url={$url}&response-code=1&format={$format}";
$ch = curl_init();
curl_setopt ($ch, CURLOPT_URL, $api_query );
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt ($ch, CURLOPT_CONNECTTIMEOUT, 0);
curl_setopt($ch, CURLOPT_HEADER, false);
$fileContents = curl_exec($ch);
curl_close($ch);
$s1=str_replace("{"," ","$fileContents");
$s2=str_replace("}"," ","$s1");
$s2=trim($s2);
$s3=array();
$s3=explode(",",$s2);
$s4=TextAfterTag($s3[0],(':'));
$s4=stripslashes($s4);
return $s4;
}
echo expandUrlLongApi('http://t.co/dd4b3kOz');
?>
The output i get is:
"http://changeordie.therepublik.net/?p=371#proliferation"
The above code works.
The code that #cryptic shared is also correct ,but i could not get the result on my server (maybe because of some configuration issue).
If anyone thinks that it could be done by some other way, please feel free to share it.
Perhaps you should just use CURLOPT_FOLLOWLOCATION = true and then determine the final URL you were directed to.
In case the problem is not a Javascript redirect as in t.co or a <META http-equiv="refresh"..., this is reslolving stackexchange URLs like https://stackoverflow.com/q/62317 fine:
public function doShortURLDecode($url) {
$ch = #curl_init($url);
#curl_setopt($ch, CURLOPT_HEADER, TRUE);
#curl_setopt($ch, CURLOPT_NOBODY, TRUE);
#curl_setopt($ch, CURLOPT_FOLLOWLOCATION, FALSE);
#curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
$response = #curl_exec($ch);
$cleanresponse= preg_replace('/[^A-Za-z0-9\- _,.:\n\/]/', '', $response);
preg_match('/Location: (.*)[\n\r]/', $cleanresponse, $a);
if (!isset($a[1])) return $url;
return parse_url($url, PHP_URL_SCHEME).'://'.parse_url($url, PHP_URL_HOST).$a[1];
}
It cleans the response of any special characters, that can occur in the curl output before cuttoing out the result URL (I ran into this problem on a php7.3 server)

Reading POST data in PHP from cUrl

I am using cUrl in PHP to request from some external service.
Interesting enough, the server is responding with raw "multipart/form-data" instead of binary file data.
My website is using a shared hosting, therefore PECL HTTP is not an option.
Is there a way to parse this data with PHP?
Sample code:
$response = curl_exec($cUrl);
/* $response is raw "multipart/form-data" string
--MIMEBoundaryurn_uuid_DDF2A2C71485B8C94C135176149950475371
Content-Type: application/xop+xml; charset=utf-8; type="text/xml"
Content-Transfer-Encoding: binary
(xml data goes here)
--MIMEBoundaryurn_uuid_DDF2A2C71485B8C94C135176149950475371
Content-Type: application/zip
Content-Transfer-Encoding: binary
(binary file data goes here)
*/
EDIT: I tried piping the response to a localhost HTTP request, but the respond data is likely to exceed the allowed memory size in PHP process. Expending mem limit is not very practical, this action also dramatically reduces the server performance dramatically.
If there is no alternatives to the original question, you may suggest a way to handle very large POST requests, along with XML parsing, in terms of streams in PHP.
I know this would be hard, please comment. I am open for discussions.
if you need the zip file from the response I guess you could just write a tmp file to save the curl response to, and stream that as a workaround:
Never tried that with multipart curls, but I guess it should work.
$fh = fopen('/tmp/foo', 'w');
$cUrl = curl_init('http://example.com/foo');
curl_setopt($cUrl, CURLOPT_FILE, $fh); // redirect output to filehandle
curl_exec($cUrl);
curl_close($cUrl);
fclose($fh); // close filehandle or the file will be corrupted
if you do NOT need anything but the xml part of the response you might want to disable headers
curl_setopt($cUrl, CURLOPT_HEADER, FALSE);
and add option to only accept xml as a response
curl_setopt($cUrl, CURLOPT_HTTPHEADER, array('Accept: application/xml'));
//That's a workaround since there is no available curl option to do so but http allows that
[EDIT]
A Shot in the dark...
can you test with these curlopt settings to see if modifiying these help anything
$headers = array (
'Content-Type: multipart/form-data; boundary=' . $boundary,
'Content-Length: ' . strlen($requestBody),
'X-EBAY-API-COMPATIBILITY-LEVEL: ' . $compatLevel, // API version
'X-EBAY-API-DEV-NAME: ' . $devID,
'X-EBAY-API-APP-NAME: ' . $appID,
'X-EBAY-API-CERT-NAME: ' . $certID,
'X-EBAY-API-CALL-NAME: ' . $verb,
'X-EBAY-API-SITEID: ' . $siteID,
);
$cUrl = curl_init();
curl_setopt($cUrl, CURLOPT_URL, $serverUrl);
curl_setopt($cUrl, CURLOPT_TIMEOUT, 30 );
curl_setopt($cUrl, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($cUrl, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($cUrl, CURLOPT_HTTPHEADER, $headers);
curl_setopt($cUrl, CURLOPT_POST, 1);
curl_setopt($cUrl, CURLOPT_POSTFIELDS, $requestBody);
curl_setopt($cUrl, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($cUrl, CURLOPT_FAILONERROR, 0 );
curl_setopt($cUrl, CURLOPT_FOLLOWLOCATION, 1 );
curl_setopt($cUrl, CURLOPT_HEADER, 0 );
curl_setopt($cUrl, CURLOPT_USERAGENT, 'ebatns;xmlstyle;1.0' );
curl_setopt($cUrl, CURLOPT_HTTP_VERSION, 1 ); // HTTP version must be 1.0
$response = curl_exec($cUrl);
if ( !$response ) {
print "curl error " . curl_errno($cUrl ) . PHP_EOL;
}
curl_close($cUrl);
[EDIT II]
This is just a try, as mentioned I cannot get my curled pages to respond with a multipart form data. So be gentle with me here ;)
$content_type = ""; //use last know content-type as a trigger
$tmp_cnt_file = "tmp/tmpfile";
$xml_response = ""; // this will hold the "usable" curl response
$hidx = 0; //header index.. counting the number of different headers received
function read_header($cUrl, $string)// this will be called once for every line of each header received
{
global $content_type, $hidx;
$length = strlen($string);
if (preg_match('/Content-Type:(.*)/', $string, $match))
{
$content_type = $match[1];
$hidx++;
}
/*
should set $content_type to 'application/xop+xml; charset=utf-8; type="text/xml"' for the first
and to 'application/zip' for the second response body
echo "Header: $string<br />\n";
*/
return $length;
}
function read_body($cUrl, $string)
{
global $content_header, $xml_response, $tmp_cnt_file, $hidx;
$length = strlen($string);
if(stripos ( $content_type , "xml") !== false)
$xml_response .= $string;
elseif(stripos ($content_type, "zip") !== false)
{
$handle = fopen($tmp_cnt_file."-".$hidx.".zip", "a");
fwrite($handle, $string);
fclose($handle);
}
/*
elseif {...} else{...}
depending on your needs
echo "Received $length bytes<br />\n";
*/
return $length;
}
and of course set the proper curlopts
// Set callback function for header
curl_setopt($cUrl, CURLOPT_HEADERFUNCTION, 'read_header');
// Set callback function for body
curl_setopt($cUrl, CURLOPT_WRITEFUNCTION, 'read_body');
don't forget to NOT save the curl response to a variable because of the memory issues,
hopefully all you need will be in the $xml_response above anyways.
//$response = curl_exec($cUrl);
curl_exec($cUrl);
And for parsing your code you can refer to $xml_response and the temp files you created starting with tmp/tmpfile-2 in this scenario. Again, I have not been able to test the code above in any way. So this might not work (but it should imho ;))
[EDIT III]
Say we want curl to write all incoming data directly to another (outgoing) stream, in this case a socket connection
I'm not sure if it is as easy as this:
$fs = fsockopen($host, $port, $errno, $errstr);
$cUrl = curl_init('http://example.com/foo');
curl_setopt($cUrl, CURLOPT_FILE, $fs); // redirect output to sockethandle
curl_exec($cUrl);
curl_close($cUrl);
fclose($fs); // close handle
else we will have to use our known write and header functions with just a little trick
//first open the socket (before initiating curl)
$fs = fsockopen($host, $port, $errno, $errstr);
// now for the new callback function
function socket_pipe($cUrl, $string)
{
global $fs;
$length = strlen($string);
fputs($fs, $string); // add NOTHING to the received line just send it to $fs; that was easy wasn't it?
return $length;
}
// and of course for the CURLOPT part
// Set callback function for header
curl_setopt($cUrl, CURLOPT_HEADERFUNCTION, 'socket_pipe');
// Set the same callback function for body
curl_setopt($cUrl, CURLOPT_WRITEFUNCTION, 'socket_pipe');
// do not forget to
fclose($fs); //when we're done
The thing is, not editing the result and simply piping it to $fs will make it necessary that apache is listening on a certain port which you then assign your script to.
Or you will need to add ONE header line directly after fsockopen
fputs($fp, "POST $path HTTP/1.0\n"); //where path is your script of course
I'm sorry i can't help much because you did not put much code but i remember i was having a similar issue when i was playing with curl_setopt options.
Did you use CURLOPT_BINARYTRANSFER?
From php documentation -> CURLOPT_BINARYTRANSFER-> TRUE to return the raw output when CURLOPT_RETURNTRANSFER is used.
just set CURLOPT_RETURNTRANSFER CURLOPT_POST
$c = curl_init($url);
curl_setopt($c, CURLOPT_RETURNTRANSFER, true);
curl_setopt($c, CURLOPT_CONNECTTIMEOUT, 1);
curl_setopt($c, CURLOPT_TIMEOUT, 1);
curl_setopt($c, CURLOPT_POST, 1);
curl_setopt($c, CURLOPT_POSTFIELDS,
array());
$rst_str = curl_exec($c);
curl_close($c);
You can re-assemble you Binary data doing something like this, I hope it helps.
$file_array = explode("\n\r", $file, 2);
$header_array = explode("\n", $file_array[0]);
foreach($header_array as $header_value) {
$header_pieces = explode(':', $header_value);
if(count($header_pieces) == 2) {
$headers[$header_pieces[0]] = trim($header_pieces[1]);
}
}
header('Content-type: ' . $headers['Content-Type']);
header('Content-Disposition: ' . $headers['Content-Disposition']);
echo substr($file_array[1], 1);
If you don't need binary data, have you tried below?
curl_setopt($c, CURLOPT_NOBODY, true);

Categories