How to check whether file completely downloaded using php CURL - php
I am downloading a recording from an external url and saving it using CURL as follows:
$ch = curl_init($Recording);
$fp = fopen($recording_file_loc, 'wb');
curl_setopt($ch, CURLOPT_FILE, $fp);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_exec($ch);
curl_close($ch);
fclose($fp);
I need to change the file permissions once file is completely downloaded as follows.
chmod($recording_file_loc , 0640);
How can i check and ensure that file is completely downloaded before executing chmod??
updated:
I updated my code as follows:
$statusCode = curl_getinfo($ch, CURLINFO_HTTP_CODE);
and
if($statusCode == 200){
chmod($recording_file_loc , 0640);
}
else{
echo $statusCode;
}
You need to put to check if the download process is complete.
curl_setopt($ch, CURLOPT_PROGRESSFUNCTION, 'progress'); // call progress function
curl_setopt($ch, CURLOPT_NOPROGRESS, false);
Then you need to define a function which checks the download progress
// progress function definition
function progress($resource,$download_size, $downloaded, $upload_size, $uploaded)
{
// Progress
if($download_size > 0)
echo $downloaded / $download_size * 100;
if($downloaded / $download_size == 1){
// chmod code here
}
}
Check this link cURL download progress in PHP
on transfers where curl detected any errors, curl_errno($ch) should no longer return 0, so if(curl_errno($ch)!==0), something bad probably happened to your download.
another thing, as pointed out by #Pamela in a comment, if the response code is not 2XX (like HTTP 200 OK or HTTP 204 No Content), that's another sign something probably went wrong, which can be detected by doing if(((string)curl_getinfo($ch,CURLINFO_RESPONSE_CODE))[0]!=='2')
so..
if(curl_errno($ch)!==0 || ((string)curl_getinfo($ch,CURLINFO_RESPONSE_CODE))[0]!=='2'){
// the download probably failed.
}
generally speaking, this may be impossible to detect on servers that doesn't implement "Content-Length" headers, if you're downloading from a server that doesn't support Content-Length, then there may be no standardized way to detect the broken download at all.. (you may have to inspect what you've downloaded to make sure it's what you expect or something, idk)
for example, on transfers where the body length doesn't match the "Content-Length" header, curl_errno($ch) returns int(56) (instead of the usual int(0)), and curl_exec($ch) returns bool(false) (PS! if you used CURLOPT_RETURNTRANSFER, then it may contain a string instead of bool)
here's a little HTTP server sending "Content-Length: 3", then cutting the connection after just sending 2 (of allegedly 3) bytes of the body:
<?php
$port=1234;
$srv=socket_create_listen($port);
while(($conn=socket_accept($srv))){
$headers=implode("\r\n",array(
"HTTP/1.1 200 OK",
"Content-Type: text/plain",
"Content-Length: 3",
"Connection: close",
"","",
));
// i lied! i said 3 bytes body, but only send 2 bytes body
$body="ab";
$response=$headers.$body;
var_dump(strlen($response),socket_write($conn,$response));
socket_close($conn);
}
and an accompanying test script:
<?php
$ch=curl_init("http://127.0.0.1:1234");
var_dump(curl_exec($ch));
var_dump(curl_errno($ch),curl_error($ch));
printing:
abbool(false)
int(56)
string(38) "Recv failure: Connection reset by peer"
Get the info like this:
$ch = curl_init($Recording);
$fp = fopen($recording_file_loc, 'wb');
curl_setopt($ch, CURLOPT_FILE, $fp);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_exec($ch);
$info = curl_getinfo($ch);
curl_close($ch);
fclose($fp);
Then check the "download_content_length" against "size_download" like this:
if($info["download_content_length"]==$info["size_download"])
{
//Download complete!
}
else
{
//Error
}
Note that it works only if server sends the Content-Length header in advance.
Related
php://input empty when sending long JSON via raw POST
I am sending raw JSON POST data with cURL via PHP, and the server receives it correctly when the JSON is not too long. However, it seems to receive an empty body when it's bigger. This is how I get the body from the server: $dataReceived = file_get_contents("php://input"); When I log $dataReceived on the server side, the expected content is in the log when not big, e.g. this: {"products":[],"allProductIds":["898","920","937","947","979","1007","1044","1064","1124","1146","1162","1178","1194","1215","1239","1291","1346","1395","1444","1629","1673","1695","1757","1868","1929","1992","2018","2050","2096","2112","2136","2178","2225","2280","2303","2318","2340","2394","2447","2459","2476","2486","2510","2520","2558","2591","2626","2655","2689","2727","2975","3175","3224","3283","3311","3333","3342","3353","3376","3389","3400","3419","3469","3506","3535","3610","3722","3765","3782","3802","4002","4020","4039","4060","4094","4130","4171","4213","4246","4281","4317","4348","4426","4473","4502","4545","4558","4563","4663","4688","4713","4733","4757","4875","4924","4984","5028","5057","5088","5102","5117","5145","5174","5197","5216","5236","5255","5275","5310","5336","5369","5422","5433","5464","5477","5505","5516","5541","5563","5587","5616","5627","5673","5703","5744","5773","5822","5839","5903","5999","6075","6196","6326","6483","6794","8360","8429","8599","8724","8798","8953","9150","9198","9247","9295","9347","9415","9463","9636","9991","10105","10219","10337","10451","10566","10681","10796","10912","11134","11244","11354","11465","11576","11687","11798","11874","11924","11973","12025","12074","12125","12175","12640","12699","12838","12844","12848","12852","12862","12866","12870","12874","12880","12884","12888","12894","12935","12943","12947","12952","12958","12962","12966","12970","12974","12979","12988","12992","12997","13001","13010","13012","13016","13020","13026","13030","13034","13039","13049","13059","13093","13128","13144","13148","13154","13158","13162","13172","13182","13217","13223","13368","2851","3828","4804"],"brandlyProductIds":["229082","226046","223246","216800","216750","215456","213613","213157","143436","143406","143362","143312","134066","134022","221020","242298","144021","143921","242348","333387","333424","333443","333517","333624","333681","334021","334043","334086","334135","334159","334184","334220","334261","334318","334351","334376","334398","334448","334505","334530","334552","334581","334606","334639","334672","334701","334734","334763","334792","334825","334962","335099","335325","335382","335439","335464","335482","335491","335500","335521","335532","335553","335575","335680","335770","335795","335877","335991","336032","336049","336066","336088","336276","336293","336310","336329","336370","336411","336452","336493","336530","336561","336598","336631","336705","336751","336782","336819","336850","336859","336977","337002","337033","337058","337083","337133","337207","337256","337366","337403","337428","337456","337473","337490","337515","337540","337565","337582","337599","337616","337637","337682","337710","337738","337791","337800","337831","337881","337906","337928","337953","337981","338006","338031","338056","338099","338135","338175","338201","338244","338269","338340","338422","187730","338815","338968","339121","339427","340235","340308","340461","340614","340687","340840","340987","341060","341133","341206","341279","341352","341425","341675","341981","342072","342254","342350","342441","342532","342714","342805","342896","343042","343115","343188","343261","343407","343480","343553","343626","343690","343754","343818","343882","343946","344010","225836","225770","344210","344214","344216","344218","303935","303610","303601","344223","344234","344236","344238","344241","204360","344254","344256","344262","344264","344268","344270","344272","344274","344277","344279","344281","344283","344285","344287","344288","344290","344292","344295","344299","344301","344303","344308","344313","344330","344333","344335","344337","344340","344342","344344","344349","344354","344371","150836","344434"]} But i get an empty $dataReceived for example when the sent JSON is the one in this pastebin https://pastebin.com/p5ZLrxig This is the code i have to send the request, with verbose log protected static function _apiCall(string $url, array $additionalHeaders = [], string $method="GET", string $data=''){ if($method=="POST"){ $curlLog = __DIR__ . '/curl-log.txt'; $f = fopen($curlLog, 'w'); $ch = curl_init($url); curl_setopt($ch, CURLOPT_VERBOSE, true); curl_setopt($ch, CURLOPT_STDERR, $f); curl_setopt($ch, CURLOPT_STDOUT, $f); curl_setopt($ch, CURLOPT_POST, true); curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); $hdr = [ 'Content-Type: application/json', 'Content-Length: ' . strlen($data) ]; foreach($additionalHeaders as $h) $hdr[] = $h; curl_setopt($ch, CURLOPT_HTTPHEADER, $hdr); curl_setopt($ch, CURLOPT_POSTFIELDS, $data ); $info = curl_getinfo($ch); $result = curl_exec($ch); curl_close($ch); fclose($f); mail("myemail#email.com","$url curl info",print_r($info,true) . ", result:$result, curl log: " . file_get_contents("file://$curlLog") . "\r\n, data:$data"); } else { // ... } } This is the log i get in the email for the big JSON: https://pastebin.com/0pYHTk4h Now, can it be an issue with POST size limit, either in PHP or Apache settings? But i doubt so, since the data is just 33Kb Content-Length: 34581 Also, if it was the case, shouldn't we expect the server (Apache + PHP 7.4.x) to return an HTTP error code like HTTP 500? Instead, we have HTTP 100 and HTTP 200. Seems not related to JSON formatting too, again because we'd expect the server to return an error code. One aspect that may be relevant: the called URL (from which i am logging php://input on server side) actually is a custom Wordpress REST API endpoint (that i have created by calling register_rest_route inside a rest_api_init action handler). I am not currently aware of POST size limits or related issues with WP API, i have no idea if the issue can be with WP. Any hints are appreciated!
How to download client-site a video with cURL / PHP?
If the following PHP is runnig, I would like to download the file with `cURL on the client-site. It means, if one of my website visitor run's an action, which starts this PHP file, it should download the file on this PC. I tried with different locations, but without any success. If I run my code, it does always download it on my WebSever, which is not this, what I want. <?php //The resource that we want to download. $fileUrl = 'https://www.example.com/this-is-a-example-video'; //The path & filename to save to. $saveTo = 'test.mp4'; //Open file handler. $fp = fopen($saveTo, 'w+'); //If $fp is FALSE, something went wrong. if($fp === false){ throw new Exception('Could not open: ' . $saveTo); } //Create a cURL handle. $ch = curl_init($fileUrl); //Pass our file handle to cURL. curl_setopt($ch, CURLOPT_FILE, $fp); //Timeout if the file doesn't download after 20 seconds. curl_setopt($ch, CURLOPT_TIMEOUT, 20); //Execute the request. curl_exec($ch); //If there was an error, throw an Exception if(curl_errno($ch)){ throw new Exception(curl_error($ch)); } //Get the HTTP status code. $statusCode = curl_getinfo($ch, CURLINFO_HTTP_CODE); //Close the cURL handler. curl_close($ch); if($statusCode == 200){ echo 'Downloaded!'; } else{ echo "Status Code: " . $statusCode; } ?> How can I change the cURL downloading process to the client-site?
PHP cannot run client-side. You could use cURL to download data to the server (without saving it to a file) and then output that data to the client. Don't do this: //Open file handler. $fp = fopen($saveTo, 'w+'); or this: //Pass our file handle to cURL. curl_setopt($ch, CURLOPT_FILE, $fp); Then capture the output: //Execute the request. curl_exec($ch); Should be: //Execute the request. $output = curl_exec($ch); Then you can: echo $output; … but make sure you set the Content-Type and consider setting the Content-Length response headers. You might also want Content-Disposition. Under most circumstances, it would probably be better to simply send the browser to fetch the file directly instead of proxying it through the server. $fileUrl = 'https://www.example.com/this-is-a-example-video'; header("Location: $fileUrl");
Save copy of a photo from Google Places API Place Photos using curl
I'm trying to grab a photo from Google Place Photos using curl and save it on my server. The request format as per the Google API documentation is like this: https://maps.googleapis.com/maps/api/place/photo?maxwidth=400&photoreference=CoQBegAAAFg5U0y-iQEtUVMfqw4KpXYe60QwJC-wl59NZlcaxSQZNgAhGrjmUKD2NkXatfQF1QRap-PQCx3kMfsKQCcxtkZqQ&sensor=true&key=AddYourOwnKeyHere So I tried this function: function download_image1($image_url, $image_file){ $fp = fopen ($image_file, 'w+'); $ch = curl_init($image_url); // curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); // enable if you want curl_setopt($ch, CURLOPT_FILE, $fp); // output to file curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1); curl_setopt($ch, CURLOPT_TIMEOUT, 1000); // some large value to allow curl to run for a long time curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0'); // curl_setopt($ch, CURLOPT_VERBOSE, true); // Enable this line to see debug prints curl_exec($ch); curl_close($ch); // closing curl handle fclose($fp); // closing file handle } download_image1($photo, "test.jpg"); ..where $photo holds the request url. This is not working, it saves an empty image with header errors, it probably is because the request is not the actual url of the photo. Also, in the request url, it's not possible to know which image extension I'm going to get (jpg, png, gif, etc) so that's another problem. Any help on how to save the photos appreciated. EDIT: I get the header errors "Can't read file header" in my image viewer software when I try to open the image. The script itself doesn't show any errors.
I found a solution here: http://kyleyu.com/?q=node/356 It gives a very useful function to return the actual URL after redirection: function get_furl($url) { $furl = false; // First check response headers $headers = get_headers($url); // Test for 301 or 302 if(preg_match('/^HTTP\/\d\.\d\s+(301|302)/',$headers[0])) { foreach($headers as $value) { if(substr(strtolower($value), 0, 9) == "location:") { $furl = trim(substr($value, 9, strlen($value))); } } } // Set final URL $furl = ($furl) ? $furl : $url; return $furl; } So you pass the Google Place Photo request uRL to this function and it returns the actual URL of the photo after the redirection which then can be used with CURL. It also explains that sometimes, the curl option curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1); doesn't always work.
https://stackoverflow.com/a/23540352/2979237 We can minimize the above code to change as adding 1 as the second parameter of get_header() function like $headers = get_headers($url, 1);. that will return the associate values.
How will get content-size if I open remote file via HTTP using fopen
So. $f = fopen('http_url_site_com_my_file_ext'); $info = stream_get_meta_data($f); http://us1.php.net/manual/ru/function.stream-get-meta-data.php as we read in docs, I can find content-lenth in echo $info['wrapper-data'][#] ; // --> Content-length: 438; also echo $info['wrapper_type']; //--> http In my case, I see cURL instead http and $info['wrapper-data']['headers']; //empty array So I cann't to get length of responce. =========== info http://www.php.net/manual/en/reserved.variables.httpresponseheader.php we can get the response headers such $f = fopen('http_url_site_com_my_file_ext'); var_dump($http_response_header); // --> null. $data = fread($f, 100); var_dump($http_response_header); // --> array of all response headers. But it is very very bad for my code. we open a lot of files at the start(if one of them is fault - die()) and then, we read from opened files. ============ QUESTION 1)if I will compile php without "--with-curlwrappers", why this experimental feature is present??? ---php 5.4.21 (php-fpm) ---in phpinfo, I see no build option as '--with-curlwrappers' 2) or how can I get response headers without reading stream ??? please, help me.
Is it mandatory to use fopen instead of cURL? This example would work with cURL $ch = curl_init(); curl_setopt($ch, CURLOPT_URL, $http_url_site_com_my_file_ext); curl_setopt($ch, CURLOPT_HEADER, true); curl_setopt($ch, CURLOPT_NOBODY, true); curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); $r = curl_exec($ch); print_r($r); Also you can use http_head if HTTP extension is enabled.
Redirect page after process complete in PHP
I have some process in page. It's downloading a file. <?php // getting file with CURL $ch = curl_init ('http://adress.com/file.csv'); curl_setopt($ch, CURLOPT_HEADER, 0); curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); curl_setopt($ch, CURLOPT_FOLLOWLOCATION, false); curl_setopt($ch, CURLOPT_BINARYTRANSFER,1); curl_setopt($ch, CURLOPT_TIMEOUT,7500); $rawdata=curl_exec($ch); curl_close ($ch); // End Curl Process // SAVE FILE $name = Rand(100000,1000000); $fp = fopen('files/'.$name.'.csv','w'); fwrite($fp, $rawdata); fclose($fp); // END SAVE PROCESS // REDIRECT OTHER PAGE **header('Location: x.php');** // END OF PAGE ?> This is allright. But there is a problem. It redirects without complete process. I want to redirect after all processes are done. The file is empty when i redirect with that method. What must I do? How can i redirect page after all processes are done? php or javascript/jquery.
I think set a callback when the header is complete is the best solution. $ch = curl_init(); curl_setopt($ch, CURLOPT_URL, 'http://www.php.net/'); // here is the hack - Set callback function for headers curl_setopt($ch, CURLOPT_HEADERFUNCTION, 'read_header'); curl_exec($ch); curl_close($ch); //Callback function for header function read_header($ch, $string) { if (trim($string) == 'HTTP/1.1 200 OK') header("location: http://www.php.net"); }
Your code should be fine, and should complete the process before redirecting. PHP runs in a single thread, so the next statement won't be executed until the previous one completes. This means that your header() line won't execute until fwrite() has finished writing the file. In fact, setting a Location header doesn't even mean the PHP script stops. You can test this by creating a dummy file and running unlink() on it after issuing the Location header. The file will still be deleted. All that Location does is sends a new location to your browser. Your browser then retrieves the new page before the old one is done running.
You can set sleep(seconds) and check if it's enough for complete process. Also you can put a simple check: $start_size = 0; while (1) { $new_file_size = filesize($filename); if ($start_size != $new_file_size) { $start_size = $new_file_size; sleep(30) } else { break; } }
If you aren't returning the file to the client, you can do the following: ignore_user_abort(); header("Content-Length: 0", true, HTTP_RESPONSE_204_NO_CONTENT); header("Connection: close", true); flush(); This will return control back to the client and close the browser connection, but the server-side process will continue to work. However, I agree with #AgentConundrum that the header() line won't be executed until the other processes are complete.