Curl request not flushing data periodically - php

I am making a curl request to a function and in that function the data is being flushed periodically. So I want to display the data as soon as it is flushed in my calling function. But my response is displayed collectively after the request is over. I want to display response side by side.
Code
require_once 'mystream.php';
stream_wrapper_register("var", "mystream") or die("Failed to register protocol");
$myVar = '';
// Open the "file"
$fp = fopen("var://myVar", "r+");
// Configuration of curl
$ch = curl_init();
$output = ' ';
$url = '';
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $output);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, 0 );
//curl_setopt($ch, CURLOPT_BUFFERSIZE, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FILE, $fp); // Data will be sent to our stream ;-)
curl_exec($ch);
curl_close($ch);
// Don't forget to close the "file" / stream
fclose($fp);
mystream.php code
<?php
class mystream{
protected $buffer = '';
function stream_open($path, $mode, $options, &$opened_path) {
return true;
}
public function stream_write($data) {
ob_start();
echo $data;
ob_end_flush();
ob_flush();
flush();
// Extract the lines ; on y tests, data was 8192 bytes long ; never more
$lines = explode("\n", $data);
// The buffer contains the end of the last line from previous time
// => Is goes at the beginning of the first line we are getting this time
$lines[0] = $this->buffer . $lines[0];
// And the last line os only partial
// => save it for next time, and remove it from the list this time
$nb_lines = count($lines);
$this->buffer = $lines[$nb_lines-1];
unset($lines[$nb_lines-1]);
// Here, do your work with the lines you have in the buffer
//var_dump($lines);
//echo '<hr />';
return strlen($data);
}
}
Any leads would be highly appreciated.

Related

Calculate MD5 of file being downloaded with PHP and CURL

I have some cURL call that download a large file.
I'm wondering if it is possible to calculate hash when the file is still downloading?
I think the progress callback function is the right place for accomplish that..
function get($urlget, $filename) {
//Init Stuff[...]
$this->fp = fopen($filename, "w+");
$ch = curl_init();
//[...] irrelevant curlopt stuff
curl_setopt($ch, CURLOPT_FILE, $this->fp);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_NOPROGRESS, 0);
curl_setopt($ch, CURLOPT_PROGRESSFUNCTION, array($this,'curl_progress_cb'));
$httpCode = curl_getinfo($ch, CURLINFO_HTTP_CODE);
$ret = curl_exec($ch);
if( curl_errno($ch) ){
$ret = FALSE;
}
curl_close($ch);
fclose($this->fp);
return $ret;
}
function curl_progress_cb($dltotal, $dlnow, $ultotal, $ulnow ){
//... Calculate MD5 of file here with $this->fp
}
Its possible to calculate md5 hash of partially downloaded file, but it does not make too much sense. Every downloaded byte will change your hash diametrally, what is the reason behind going with this kind solution?
If you need to have md5 hash for entire file than the answer is NO. Your program has to first download the file and then generate the hash.
I just do it:
in a file wget-md5.php, add the below code:
<?php
function writeCallback($resource, $data)
{
global $handle;
global $handle_md5_val;
global $handle_md5_ctx;
$len = fwrite($handle,$data);
hash_update($handle_md5_ctx,$data);
return $len;
}
$handle=FALSE;
$handle_md5_val=FALSE;
$handle_md5_ctx=FALSE;
function wget_with_curl_and_md5_hashing($url,$uri)
{
global $handle;
global $handle_md5_val;
global $handle_md5_ctx;
$handle_md5_val=FALSE;
$handle_md5_ctx=hash_init('md5');
$handle = fopen($uri,'w');
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, $url);
curl_setopt($curl, CURLOPT_BUFFERSIZE, 64000);
curl_setopt($curl, CURLOPT_WRITEFUNCTION, 'writeCallback');
echo "wget_with_curl_and_md5_hashing[".$url."]=downloading\n";
curl_exec($curl);
curl_close($curl);
fclose($handle);
$handle_md5_val = hash_final($handle_md5_ctx);
$handle_md5_ctx=FALSE;
echo "wget_with_curl_and_md5_hashing[".$url."]=downloaded,md5=".$handle_md5_val."\n";
}
wget_with_curl_and_md5_hashing("http://archlinux.polymorf.fr/core/os/x86_64/core.files.tar.gz","core.files.tar.gz");
?>
and run:
$ php -f wget-md5.php
wget_with_curl_and_md5_hashing[http://archlinux.polymorf.fr/core/os/x86_64/core.files.tar.gz]=downloading
wget_with_curl_and_md5_hashing[http://archlinux.polymorf.fr/core/os/x86_64/core.files.tar.gz]=downloaded,md5=5bc1ac3bc8961cfbe78077e1ebcf7cbe
$ md5sum core.files.tar.gz
5bc1ac3bc8961cfbe78077e1ebcf7cbe core.files.tar.gz

How to get the image extention from function file_get_content()

I have a url something like this
$url ="www.domain.com/image.php?id=123&idlocation=987&number=01";
Previously i was getting the extension using following code
$img_details= pathinfo($url);
But this won't work any more since the url has other variables also . So in this case how to get the Image name and extension .
I know i should first download the file using
$contenido = file_get_contents($url);
But don't know how to get the files name/extention from this
Thanks in advance
$ch = curl_init();
$url = 'http://www.domain.com/image.php?id=123&idlocation=987&number=01';
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_NOBODY, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$results = split("\n", trim(curl_exec($ch)));
foreach($results as $line) {
if (strtok($line, ':') == 'Content-Type') {
$parts = explode(":", $line);
echo trim($parts[1]);
}
}
Return: image/png
Already answered in: Get mime type of external file using cURL and php
The above answers focus both on the mime-type and they will work in most cases they both need additional resource usage - either a second network call to get the mime type or more disk reads/writes to save the file to disk and use the exif-imagetype function. And both will not return the file name which was a part of the question. Here is a download function using curl that will return a downloaded file as an array with name,mime type and content. Additionally it will try to get the name from the URL if possible.
Sample usage
$file=downloadfile($url);
echo "Name: ".$file["name"]."<br>";
echo "Type: ".$file["mimetype"]."<br>";
Code
function downloadfile($url){
global $headers;
$headers=array();
$file=array("content"=>"","mimetype"=>"","name"=>"");
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_HEADERFUNCTION, 'readHeaders');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$file["content"]=curl_exec($ch);
if(sizeof($headers)){
foreach($headers as $header){
if(!strncasecmp($header,'content-type:',13)){
$file["mimetype"]=trim(substr($header,13));
}
elseif(!strncasecmp($header,'content-disposition:',20)){
$file["name"]=trim(substr(strstr($header,'filename='),9));
}
}
}
if(!$file["name"]){
$query=strpos("?",$url);
$file["name"]=basename(substr($url,0,($query?$query:strlen($url))));
}
unset($headers);
return $file;
}
function readHeaders($ch, $header) {
global $headers;
array_push($headers, $header);
return strlen($header);
}

Is it possible to get partial response using PHP cURL?

Here is my code
$url = "partial_response.php";
$sac_curl = curl_init();
curl_setopt($sac_curl, CURLOPT_HTTPGET, true);
curl_setopt($sac_curl, CURLOPT_URL, $url);
curl_setopt($sac_curl, CURLOPT_RETURNTRANSFER, true);
curl_setopt($sac_curl, CURLOPT_HEADER, false);
curl_setopt($sac_curl, CURLOPT_TIMEOUT, 11);
$resp = curl_exec($sac_curl);
curl_close($sac_curl);
echo $resp;
Partial_response.php
header( 'Content-type: text/html; charset=utf-8' );
echo 'Job waiting ...<br />';
for( $i = 0 ; $i < 10 ; $i++ )
{
echo $i . '<br/>';
flush();
ob_flush();
sleep(1);
}
echo 'End ...<br/>';
From the about code am trying to get a partial response from partial_response.php. what I want is, I need curl to return me "Job waiting.." alone instead of waiting for the partial_response.php to complete the loop and return the entire data. so when I reduce CURLOPT_TIMEOUT below 11 i dont get any response at all. Kindly clarify my doubt.
Thanks in advance.
I later realized that cURL could not do what I want, I used stream_context_get_options
to achieve what I wanted. Here it is, http://www.php.net/manual/en/function.stream-context-get-options.php.
No, I'm afraid not. At least not that I know of any, this is simply because PHP is a synchronous language, meaning you cannot "skip" tasks. (I.e. curl_exec() will always - no matter what - be executed until the request is completed)
I'm not sure abut the timeout, but you can get partial response using cURL by using the CURLOPT_WRITEFUNCTION flag:
curl_setopt($ch, CURLOPT_WRITEFUNCTION, $callback);
Where $ch is the Curl handler, and $callback is the callback function name. This command will stream response data from remote site. The callback function can look something like:
$result = '';
$callback = function ($ch, $str) {
global $result;
//$str has the chunks of data streamed back.
$result .= $str;
// here you can mess with the stream data either with $result or $str.
// i.e. look for the "Job waiting" string and terminate the response.
return strlen($str);//don't touch this
};
If not interrupted at the end $result will contain all the response from remote site.
So combining everything will look something like:
$result = '';
$callback = function ($ch, $str) {
global $result;
//$str has the chunks of data streamed back.
$result .= $str;
// here you can mess with the stream data either with $result or $str.
// i.e. look for the "Job waiting" string and terminate the response.
return strlen($str);//don't touch this
};
$url = "partial_response.php";
$sac_curl = curl_init();
curl_setopt($sac_curl, CURLOPT_HTTPGET, true);
curl_setopt($sac_curl, CURLOPT_URL, $url);
curl_setopt($sac_curl, CURLOPT_RETURNTRANSFER, true);
curl_setopt($sac_curl, CURLOPT_HEADER, false);
curl_setopt($sac_curl, CURLOPT_TIMEOUT, 11);
curl_setopt($ch, CURLOPT_WRITEFUNCTION, $callback);
curl_exec($sac_curl); // the response is now in $result.
curl_close($sac_curl);
echo $result;

Reading POST data in PHP from cUrl

I am using cUrl in PHP to request from some external service.
Interesting enough, the server is responding with raw "multipart/form-data" instead of binary file data.
My website is using a shared hosting, therefore PECL HTTP is not an option.
Is there a way to parse this data with PHP?
Sample code:
$response = curl_exec($cUrl);
/* $response is raw "multipart/form-data" string
--MIMEBoundaryurn_uuid_DDF2A2C71485B8C94C135176149950475371
Content-Type: application/xop+xml; charset=utf-8; type="text/xml"
Content-Transfer-Encoding: binary
(xml data goes here)
--MIMEBoundaryurn_uuid_DDF2A2C71485B8C94C135176149950475371
Content-Type: application/zip
Content-Transfer-Encoding: binary
(binary file data goes here)
*/
EDIT: I tried piping the response to a localhost HTTP request, but the respond data is likely to exceed the allowed memory size in PHP process. Expending mem limit is not very practical, this action also dramatically reduces the server performance dramatically.
If there is no alternatives to the original question, you may suggest a way to handle very large POST requests, along with XML parsing, in terms of streams in PHP.
I know this would be hard, please comment. I am open for discussions.
if you need the zip file from the response I guess you could just write a tmp file to save the curl response to, and stream that as a workaround:
Never tried that with multipart curls, but I guess it should work.
$fh = fopen('/tmp/foo', 'w');
$cUrl = curl_init('http://example.com/foo');
curl_setopt($cUrl, CURLOPT_FILE, $fh); // redirect output to filehandle
curl_exec($cUrl);
curl_close($cUrl);
fclose($fh); // close filehandle or the file will be corrupted
if you do NOT need anything but the xml part of the response you might want to disable headers
curl_setopt($cUrl, CURLOPT_HEADER, FALSE);
and add option to only accept xml as a response
curl_setopt($cUrl, CURLOPT_HTTPHEADER, array('Accept: application/xml'));
//That's a workaround since there is no available curl option to do so but http allows that
[EDIT]
A Shot in the dark...
can you test with these curlopt settings to see if modifiying these help anything
$headers = array (
'Content-Type: multipart/form-data; boundary=' . $boundary,
'Content-Length: ' . strlen($requestBody),
'X-EBAY-API-COMPATIBILITY-LEVEL: ' . $compatLevel, // API version
'X-EBAY-API-DEV-NAME: ' . $devID,
'X-EBAY-API-APP-NAME: ' . $appID,
'X-EBAY-API-CERT-NAME: ' . $certID,
'X-EBAY-API-CALL-NAME: ' . $verb,
'X-EBAY-API-SITEID: ' . $siteID,
);
$cUrl = curl_init();
curl_setopt($cUrl, CURLOPT_URL, $serverUrl);
curl_setopt($cUrl, CURLOPT_TIMEOUT, 30 );
curl_setopt($cUrl, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($cUrl, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($cUrl, CURLOPT_HTTPHEADER, $headers);
curl_setopt($cUrl, CURLOPT_POST, 1);
curl_setopt($cUrl, CURLOPT_POSTFIELDS, $requestBody);
curl_setopt($cUrl, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($cUrl, CURLOPT_FAILONERROR, 0 );
curl_setopt($cUrl, CURLOPT_FOLLOWLOCATION, 1 );
curl_setopt($cUrl, CURLOPT_HEADER, 0 );
curl_setopt($cUrl, CURLOPT_USERAGENT, 'ebatns;xmlstyle;1.0' );
curl_setopt($cUrl, CURLOPT_HTTP_VERSION, 1 ); // HTTP version must be 1.0
$response = curl_exec($cUrl);
if ( !$response ) {
print "curl error " . curl_errno($cUrl ) . PHP_EOL;
}
curl_close($cUrl);
[EDIT II]
This is just a try, as mentioned I cannot get my curled pages to respond with a multipart form data. So be gentle with me here ;)
$content_type = ""; //use last know content-type as a trigger
$tmp_cnt_file = "tmp/tmpfile";
$xml_response = ""; // this will hold the "usable" curl response
$hidx = 0; //header index.. counting the number of different headers received
function read_header($cUrl, $string)// this will be called once for every line of each header received
{
global $content_type, $hidx;
$length = strlen($string);
if (preg_match('/Content-Type:(.*)/', $string, $match))
{
$content_type = $match[1];
$hidx++;
}
/*
should set $content_type to 'application/xop+xml; charset=utf-8; type="text/xml"' for the first
and to 'application/zip' for the second response body
echo "Header: $string<br />\n";
*/
return $length;
}
function read_body($cUrl, $string)
{
global $content_header, $xml_response, $tmp_cnt_file, $hidx;
$length = strlen($string);
if(stripos ( $content_type , "xml") !== false)
$xml_response .= $string;
elseif(stripos ($content_type, "zip") !== false)
{
$handle = fopen($tmp_cnt_file."-".$hidx.".zip", "a");
fwrite($handle, $string);
fclose($handle);
}
/*
elseif {...} else{...}
depending on your needs
echo "Received $length bytes<br />\n";
*/
return $length;
}
and of course set the proper curlopts
// Set callback function for header
curl_setopt($cUrl, CURLOPT_HEADERFUNCTION, 'read_header');
// Set callback function for body
curl_setopt($cUrl, CURLOPT_WRITEFUNCTION, 'read_body');
don't forget to NOT save the curl response to a variable because of the memory issues,
hopefully all you need will be in the $xml_response above anyways.
//$response = curl_exec($cUrl);
curl_exec($cUrl);
And for parsing your code you can refer to $xml_response and the temp files you created starting with tmp/tmpfile-2 in this scenario. Again, I have not been able to test the code above in any way. So this might not work (but it should imho ;))
[EDIT III]
Say we want curl to write all incoming data directly to another (outgoing) stream, in this case a socket connection
I'm not sure if it is as easy as this:
$fs = fsockopen($host, $port, $errno, $errstr);
$cUrl = curl_init('http://example.com/foo');
curl_setopt($cUrl, CURLOPT_FILE, $fs); // redirect output to sockethandle
curl_exec($cUrl);
curl_close($cUrl);
fclose($fs); // close handle
else we will have to use our known write and header functions with just a little trick
//first open the socket (before initiating curl)
$fs = fsockopen($host, $port, $errno, $errstr);
// now for the new callback function
function socket_pipe($cUrl, $string)
{
global $fs;
$length = strlen($string);
fputs($fs, $string); // add NOTHING to the received line just send it to $fs; that was easy wasn't it?
return $length;
}
// and of course for the CURLOPT part
// Set callback function for header
curl_setopt($cUrl, CURLOPT_HEADERFUNCTION, 'socket_pipe');
// Set the same callback function for body
curl_setopt($cUrl, CURLOPT_WRITEFUNCTION, 'socket_pipe');
// do not forget to
fclose($fs); //when we're done
The thing is, not editing the result and simply piping it to $fs will make it necessary that apache is listening on a certain port which you then assign your script to.
Or you will need to add ONE header line directly after fsockopen
fputs($fp, "POST $path HTTP/1.0\n"); //where path is your script of course
I'm sorry i can't help much because you did not put much code but i remember i was having a similar issue when i was playing with curl_setopt options.
Did you use CURLOPT_BINARYTRANSFER?
From php documentation -> CURLOPT_BINARYTRANSFER-> TRUE to return the raw output when CURLOPT_RETURNTRANSFER is used.
just set CURLOPT_RETURNTRANSFER CURLOPT_POST
$c = curl_init($url);
curl_setopt($c, CURLOPT_RETURNTRANSFER, true);
curl_setopt($c, CURLOPT_CONNECTTIMEOUT, 1);
curl_setopt($c, CURLOPT_TIMEOUT, 1);
curl_setopt($c, CURLOPT_POST, 1);
curl_setopt($c, CURLOPT_POSTFIELDS,
array());
$rst_str = curl_exec($c);
curl_close($c);
You can re-assemble you Binary data doing something like this, I hope it helps.
$file_array = explode("\n\r", $file, 2);
$header_array = explode("\n", $file_array[0]);
foreach($header_array as $header_value) {
$header_pieces = explode(':', $header_value);
if(count($header_pieces) == 2) {
$headers[$header_pieces[0]] = trim($header_pieces[1]);
}
}
header('Content-type: ' . $headers['Content-Type']);
header('Content-Disposition: ' . $headers['Content-Disposition']);
echo substr($file_array[1], 1);
If you don't need binary data, have you tried below?
curl_setopt($c, CURLOPT_NOBODY, true);

How can I get file name from header using curl in php?

how can i determine the file name in header when i get with php. lock at this:
<?php
/*
This is usefull when you are downloading big files, as it
will prevent time out of the script :
*/
set_time_limit(0);
ini_set('display_errors',true);//Just in case we get some errors, let us know....
$fp = fopen (dirname(__FILE__) . '/tempfile', 'w+');//This is the file where we save the information
$ch = curl_init('http://www.example.com/getfile.php?id=4456'); // Here is the file we are downloading
/*
the server get me an header 'Content-Disposition: attachment; filename="myfile.pdf"'
i want get 'myfile.pdf' from headers. how can i get it ?
*/
$fileNameFromHeader = '?????????????';
curl_setopt($ch, CURLOPT_TIMEOUT, 50);
curl_setopt($ch, CURLOPT_FILE, $fp);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_exec($ch);
curl_close($ch);
fclose($fp);
// rename file
rename(dirname(__FILE__) . '/tempfile', dirname(__FILE__) . $fileNameFromHeader);
Create a callback that reads headers, and parse them yourself. Something like:
function readHeader($ch, $header)
{
global $responseHeaders;
$url = curl_getinfo($ch, CURLINFO_EFFECTIVE_URL);
$responseHeaders[$url][] = $header;
return strlen($header);
}
... curl stuff ...
// if you need to call a class method use this:
// curl_setopt($ch, CURLOPT_HEADERFUNCTION, array(&$this,'readHeader'));
// for a non class method use this
curl_setopt($ch, CURLOPT_HEADERFUNCTION, 'readHeader');
$result = curl_exec($ch);
// all headers are now in $responseHeaders
I would use the function Byron mentioned. However, if you just want the fileNameFromHeader, I would include this in the readHeader function:
function readHeader($ch, $header)
{
global $responseHeaders;
$url = curl_getinfo($ch, CURLINFO_EFFECTIVE_URL);
$responseHeaders[$url][] = $header;
// $url = 'http://stackoverflow.com/questions/4091203/how-can-i-get-file-name-from-header-using-curl-in-php';
$params = explode('/', $url);
$fileNameFromHeader = $params[count($params) - 1];
//return strlen($header);
return $fileNameFromHeader;
}

Categories