I am trying to find a way to stop an active PHP Curl download. I am downloading large files from a remote server and sometimes I would like to cancel the download after it has started. I have tried returning false within CURLOPT_PROGRESSFUNCTION, however that did not work. I also tried deleting the file that was being downloaded, and that did not work either (web stats showed the download was continuing).
The below code is triggered via a quick ajax call:
$ch = curl_init( $file->url );
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_NOPROGRESS, false );
curl_setopt($ch, CURLOPT_FILE, $targetFile); //save the file to here
curl_setopt( $ch, CURLOPT_PROGRESSFUNCTION, function($resource, $download_size, $downloaded_size, $upload_size, $uploaded_size) use ($download_id) {
if ( $download_size == 0 ) {
$progress = 0;
} else {
$progress = round( $downloaded_size * 100 / $download_size );
}
// if download complete trigger completed function
if($progress == 100) {
self::DownloadCompleted($download_id);
}
});
$curl = curl_exec($ch);
Solution was to return a non-zero value in the CURLOPT_PROGRESSFUNCTION function, as per drew010 in the comment.
To get this done I added a check within the function to see if a file exists, if it does the function returns 1 and aborts. I just create a file in the directory with the same name as the download ID when I want to cancel the download. It works well for me.
$ch = curl_init( $file->url );
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_NOPROGRESS, false );
curl_setopt($ch, CURLOPT_FILE, $targetFile); //save the file to here
curl_setopt( $ch, CURLOPT_PROGRESSFUNCTION, function($resource, $download_size, $downloaded_size, $upload_size, $uploaded_size) use ($download_id) {
//if the file exists, the download is aborted
if(file_exists('path/to/directory/cancel.'.$download_id)) {
Self::CleanupCancelledDownload; //function to clean up the partially downloaded file, etc.
return 1; //returning a non-zero value cancels the CURL download.
}
});
$curl = curl_exec($ch);
Related
I can't manage to get a file to upload the DailyMotion API.
I'm following these steps : https://developer.dailymotion.com/video-upload/upload-new-video-api/
The first API call runs fine and returns me the "upload_url" that I feed into the method you'll see below.
It fails on the 2nd step and the response error is :
{"error":"missing content size","seal": "[some string]"}
How am I supposed to set the content size ?
the code for the 2nd call :
<?php
namespace PierreMiniggio\YoutubeChannelCloner\Dailymotion\API;
class DailymotionFileUploader
{
public function upload(string $uploadUrl, string $filePath): ?string
{
$formattedFile = function_exists('curl_file_create')
? curl_file_create(str_replace('\\', '/', $filePath))
: sprintf("#%s", $filePath)
;
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $uploadUrl);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt(
$ch,
CURLOPT_POSTFIELDS,
['file' => $formattedFile]
);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$response = curl_exec($ch);
//var_dump($response); die(); // the output I printed above
curl_close($ch);
if (empty($response)) {
return null;
}
$jsonResponse = json_decode($response, true);
if (empty($jsonResponse) || ! isset($jsonResponse['url'])) {
return null;
}
return $jsonResponse['url'];
}
}
OS : W10
I made sure the file path and the upload URL are correct.
I tried using the dailymotion/sdk lib and use the function that uploads a file instead of using my curl requests, but I get the exact same error.
Problem Solved, It was issue with curl/PHP.
I was running the script in PHP 7.4.3-dev, updated to 7.4.11 and it solved the issue.
More Info about the bug here : https://bugs.php.net/bug.php?id=79013
I have some cURL call that download a large file.
I'm wondering if it is possible to calculate hash when the file is still downloading?
I think the progress callback function is the right place for accomplish that..
function get($urlget, $filename) {
//Init Stuff[...]
$this->fp = fopen($filename, "w+");
$ch = curl_init();
//[...] irrelevant curlopt stuff
curl_setopt($ch, CURLOPT_FILE, $this->fp);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_NOPROGRESS, 0);
curl_setopt($ch, CURLOPT_PROGRESSFUNCTION, array($this,'curl_progress_cb'));
$httpCode = curl_getinfo($ch, CURLINFO_HTTP_CODE);
$ret = curl_exec($ch);
if( curl_errno($ch) ){
$ret = FALSE;
}
curl_close($ch);
fclose($this->fp);
return $ret;
}
function curl_progress_cb($dltotal, $dlnow, $ultotal, $ulnow ){
//... Calculate MD5 of file here with $this->fp
}
Its possible to calculate md5 hash of partially downloaded file, but it does not make too much sense. Every downloaded byte will change your hash diametrally, what is the reason behind going with this kind solution?
If you need to have md5 hash for entire file than the answer is NO. Your program has to first download the file and then generate the hash.
I just do it:
in a file wget-md5.php, add the below code:
<?php
function writeCallback($resource, $data)
{
global $handle;
global $handle_md5_val;
global $handle_md5_ctx;
$len = fwrite($handle,$data);
hash_update($handle_md5_ctx,$data);
return $len;
}
$handle=FALSE;
$handle_md5_val=FALSE;
$handle_md5_ctx=FALSE;
function wget_with_curl_and_md5_hashing($url,$uri)
{
global $handle;
global $handle_md5_val;
global $handle_md5_ctx;
$handle_md5_val=FALSE;
$handle_md5_ctx=hash_init('md5');
$handle = fopen($uri,'w');
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, $url);
curl_setopt($curl, CURLOPT_BUFFERSIZE, 64000);
curl_setopt($curl, CURLOPT_WRITEFUNCTION, 'writeCallback');
echo "wget_with_curl_and_md5_hashing[".$url."]=downloading\n";
curl_exec($curl);
curl_close($curl);
fclose($handle);
$handle_md5_val = hash_final($handle_md5_ctx);
$handle_md5_ctx=FALSE;
echo "wget_with_curl_and_md5_hashing[".$url."]=downloaded,md5=".$handle_md5_val."\n";
}
wget_with_curl_and_md5_hashing("http://archlinux.polymorf.fr/core/os/x86_64/core.files.tar.gz","core.files.tar.gz");
?>
and run:
$ php -f wget-md5.php
wget_with_curl_and_md5_hashing[http://archlinux.polymorf.fr/core/os/x86_64/core.files.tar.gz]=downloading
wget_with_curl_and_md5_hashing[http://archlinux.polymorf.fr/core/os/x86_64/core.files.tar.gz]=downloaded,md5=5bc1ac3bc8961cfbe78077e1ebcf7cbe
$ md5sum core.files.tar.gz
5bc1ac3bc8961cfbe78077e1ebcf7cbe core.files.tar.gz
I am offering my users to use remote-upload to download the content directly on my server from (for example) their own server instead of local uploads. For that I'm using cURL. And now I want to get the remaining time cURL needs to complete the download (bitrate would be fine too).
Is there a way I can return the remaining time cURL needs to finish the download via the PHP curl module or do I need to run the command-line interface and somehow put the output in a file and then read from there (since PHP blocks execution when using shell_exec() or exec())?
I'm already getting the bytes expected to download and how many curl already downloaded. This is the associated code as far:
function write_progress($ch, $original_size, $current_size, $os_up, $cs_up) {
global $anitube, $cache;
$cache->write("webupload-progress-".$anitube->input['inputname'], serialize(array("total" => $original_size, "loaded" => $current_size)));
}
ini_set('max_execution_time', '0');
$handle_file = "/tmp/file-".$anitube->generate(20).".tmp";
if(DIRECTORY_SEPARATOR == "\\") {
$handle_file = IN_DIR.$handle_file;
}
$file = fopen($handle_file, "w+");
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, urldecode(trim($anitube->input['filename'])));
curl_setopt($ch, CURLOPT_FILE, $file);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 5);
curl_setopt($ch, CURLOPT_USERAGENT, $_SERVER['HTTP_USER_AGENT']);
curl_setopt($ch, CURLOPT_NOPROGRESS, false);
curl_setopt($ch, CURLOPT_PROGRESSFUNCTION, "write_progress");
if($anitube->users['ip'] == "127.0.0.1") {
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
}
$data = curl_exec($ch);
if(curl_errno($ch) > 0) {
file_put_contents(IN_DIR."/logs/curl_errors.txt", date("d.m.Y H:i:s")." - Errno: ".curl_errno($ch)." - Error: ".curl_error($ch)." - cURL 4: ".print_r(curl_getinfo($ch), true)."\n", FILE_APPEND);
die(json_encode(array("success" => 0, "response" => $language->word('remote_file_not_available'))));
} elseif(curl_getinfo($ch, CURLINFO_HTTP_CODE) != 200) {
file_put_contents(IN_DIR."/logs/curl_errors.txt", date("d.m.Y H:i:s")." - Error: Connection denied - HTTP-Response-Code: ".curl_getinfo($ch, CURLINFO_HTTP_CODE)." - cURL 4: ".print_r(curl_getinfo($ch), true)."\n", FILE_APPEND);
die(json_encode(array("success" => 0, "response" => $language->word('remote_file_not_available'))));
}
curl_close($ch);
fclose($file);
PHP's cURL library does not appear to provide the estimated time remaining, but it is fairly trivial to calculate this from PHP using the CURLOPT_PROGRESSFUNCTION callback function. I've created a working example below. Note that if GZIP compression is enabled, the output will probably be delayed until the entire request is complete.
Example:
<?php
header( 'Content-Type: text/plain' );
//A helper function to flush output buffers.
function flush_all() {
while ( ob_get_level() ) {
ob_end_flush();
}
flush();
}
$download_start_time = null;
function write_progress( $ch, $original_size, $current_size, $os_up, $cs_up ) {
global $download_start_time;
//Get the current time.
$now = microtime( true );
//Remember the start time.
if ( ! $download_start_time ) {
$download_start_time = $now;
}
//Check if the download size is available yet.
if ( $original_size ) {
//Compute time spent transfering.
$transfer_time = $now - $download_start_time;
//Compute percent already downloaded.
$transfer_percentage = $current_size / $original_size;
//Compute estimated transfer time.
$estimated_tranfer_time = $transfer_time / $transfer_percentage;
//Compute estimated time remaining.
$estimated_time_remaining = $estimated_tranfer_time - $transfer_time;
//Output the remaining time.
var_dump( $estimated_time_remaining );
flush_all();
}
}
//Example usage.
$file = fopen( 'tmp.bin', "w+");
$ch = curl_init();
curl_setopt( $ch, CURLOPT_URL, 'https://ftp.mozilla.org/pub/mozilla.org/firefox/releases/35.0.1/win32/en-US/Firefox%20Setup%2035.0.1.exe' );
curl_setopt( $ch, CURLOPT_FILE, $file );
curl_setopt( $ch, CURLOPT_NOPROGRESS, false );
curl_setopt( $ch, CURLOPT_PROGRESSFUNCTION, 'write_progress' );
$data = curl_exec( $ch );
I am developing a script for a music company in PHP that has different servers so they need to display a file if it exists or not on the external server
like they have 3 versions of each music file mp3 mp4 etc ..... and they are accessing the files (each version ) from there specific external server . i have made three solutions for it all of them worked like charm but they are making the server slow .
First Method :
$handle = curl_init($url);
curl_setopt($handle, CURLOPT_RETURNTRANSFER, TRUE);
/* Get the HTML or whatever is linked in $url. */
$response = curl_exec($handle);
/* Check for 404 (file not found). */
$httpCode = curl_getinfo($handle, CURLINFO_HTTP_CODE);
if($httpCode == 404) {
/* Handle 404 here. */
}
curl_close($handle);
/* Handle $response here. */
Second Method : Using NuSOAP i made an api which checks internally the file and returns yes/no
Third Method:
function checkurl($url)
{
return true;
$file_headers = #get_headers($url);
//var_dump($file_headers);
if($file_headers[0] == 'HTTP/1.1 302 Moved Temporarily' || $file_headers[0] =='HTTP/1.1 302 Found') {
$exists = false;
}
else {
$exists = true;
}
return $exists;
}
So i need a solution that doesn't makes the server slow any suggestions
Be sure to issue a HEAD request, not GET, since you don't want to get the file contents. And maybe you need to follow redirects, or not...
Example with curl (thanks to this blog post):
<?php
$url = 'http://localhost/c.txt';
echo "\n checking: $url";
$c = curl_init();
curl_setopt( $c, CURLOPT_RETURNTRANSFER, true );
curl_setopt( $c, CURLOPT_FOLLOWLOCATION, true );
curl_setopt( $c, CURLOPT_MAXREDIRS, 5 );
curl_setopt( $c, CURLOPT_CUSTOMREQUEST, 'HEAD' );
curl_setopt( $c, CURLOPT_HEADER, 1 );
curl_setopt( $c, CURLOPT_NOBODY, true );
curl_setopt( $c, CURLOPT_URL, $url );
$res = curl_exec( $c );
echo "\n\ncurl:\n";
var_dump($res);
echo "\nis 200: ";
var_dump(false !== strpos($res, 'HTTP/1.1 200 OK'));
SOAP or other web service implementation can be an option if the file is not available by HTTP.
If you want to use get_headers(), please note that by default it's slow because it issues a GET request. To use HEAD request, you should change the default stream context (please check get_headers() on php manual):
stream_context_set_default(
array(
'http' => array(
'method' => 'HEAD'
)
)
);
I thought it works with above answers but it wasnt working where there were too many requests so i finally try again and again and found this solution its working perfectly actually the problem was redirects too many of them so i set time_out 15 in curl and it worked
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_NOBODY, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_TIMEOUT, 15);
$r = curl_exec($ch);
$r = split("\n", $r);
var_dump($r);
If I get title of the page, I can tell the download link is active or dead.
For example: "Free online storage" is title of dead link and "[file name]" is the title of active link (mediafire). But my page takes too long to respond, so is there any other way to check if a download link is active or dead?
That is what i have done:
<?php
function getTitle($Url){
$str = file_get_contents($Url);
if(strlen($str)>0){
preg_match("/\<title\>(.*)\<\/title\>/",$str,$title);
return $title[1];
}
}
?>
Do not perform a GET request, which downloads the whole page/file, but HEAD request, which gets only the HTTP headers, and check if the status is 200, and the content-type is not text/html
Something like this...
function url_validate($link)
{
#[url]http://www.example.com/determining-if-a-url-exists-with-curl/[/url]
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $link);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 10);
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_NOBODY, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_MAXREDIRS, 10); //follow up to 10 redirections - avoids loops
$data = curl_exec($ch);
curl_close($ch);
preg_match_all("/HTTP\/1\.[1|0]\s(\d{3})/",$data,$matches);
$code = end($matches[1]);
if(!$data)
{
return(false);
}
else
{
if($code==200)
{
return(true);
}
elseif($code==404)
{
return(false);
}
}
}
You can safely use any cURL library function. It is legitimate and thus would not regarded as a hacking attempt. The only requirement is that your web hosting company has cURL extension installed, which is very likely.
cURL should do the job. You can check the headers returned and the text content as well if you want.