I want to download file with CURLOPT_FILE option.
$file_name = 'm1.dat';
$file = fopen($file_name, 'w');
$ch = curl_init($download_url);
curl_setopt($ch, CURLOPT_FILE, $file);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_HEADER, false);
$res = curl_exec($ch);
curl_close($ch);
fclose($file);
I expect, that file would be saved in $file, but $file is empty, and curl_exec returns content of
the file in $res. So, i must use file_put_contents($file_name, $res) to save the file.
But I want to download BIG files (1Gb and more), and, as I read, downloading whith CURLOPT_FILE is more memory efficient. With file_put_contents my script can run out of memory.
So, why curl not saving file to $file? I use PHP 7.3.9 and I have such problem both on Windows and Linux OS.
From the notes of curl_setopt (https://www.php.net/manual/en/function.curl-setopt.php#99082) it looks as though you should have the CURLOPT_RETURNTRANSFER before the CURLOPT_FILE option...
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FILE, $file);
Related
I am trying to download a ZIP file using cURL, from a given URL.
I received an URL from a supplier where I should download a ZIP file. But everytime I try to download the ZIP file I get the page that says that I am not logged in.
The url where I should get the file from looks like this:
https://www.tyre24.com/nl/nl/user/login/userid/USERID/password/PASSWORD/page/L2V4cG9ydC9kb3dubG9hZC90L01nPT0vYy9NVFE9Lw==
Here you see that the USERID, and PASSWORD are variables that are filled in with the correct data. The strange thing is that if I enter the URL in my browser it seems to work, the zip file is getting downloaded.
But everytime I call that URL with cURL, I seem to get a incorrect login page. Could someone tell me what I am doing wrong?
It seems like that there is a redirect behind the given URL, that is why I have putted in the cURL call: curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
Here is my code:
set_time_limit(0);
//File to save the contents to
$fp = fopen ('result.zip', 'w+');
$url = "https://www.tyre24.com/nl/nl/user/login/userid/118151/password/5431tyre24/page/L2V4cG9ydC9kb3dubG9hZC90L01nPT0vYy9NVFE9Lw==";
//Here is the file we are downloading, replace spaces with %20
$ch = curl_init(str_replace(" ","%20",$url));
curl_setopt($ch, CURLOPT_TIMEOUT, 50);
//give curl the file pointer so that it can write to it
curl_setopt($ch, CURLOPT_FILE, $fp);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
$data = curl_exec($ch);//get curl response
//done
curl_close($ch);
Am I doing something wrong?
To download a zip file from the external source via CURL use one of the following approaches:
First approach:
function downloadZipFile($url, $filepath){
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_BINARYTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 0);
$raw_file_data = curl_exec($ch);
if(curl_errno($ch)){
echo 'error:' . curl_error($ch);
}
curl_close($ch);
file_put_contents($filepath, $raw_file_data);
return (filesize($filepath) > 0)? true : false;
}
downloadZipFile("http://www.colorado.edu/conflict/peace/download/peace_essay.ZIP", "result.zip");
A few comments:
to get data back from the remote source you have to set
CURLOPT_RETURNTRANSFER option
instead of consequent calls of fopen ... fwite functions you can
use file_put_contents which is more handy
And here is screenshot with result.zip which was downloaded a few minutes earlier using the above approach:
Second approach:
function downloadZipFile($url, $filepath){
$fp = fopen($filepath, 'w+');
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, false);
curl_setopt($ch, CURLOPT_BINARYTRANSFER, true);
//curl_setopt( $ch, CURLOPT_SSL_VERIFYPEER, false );
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 10);
curl_setopt($ch, CURLOPT_FILE, $fp);
curl_exec($ch);
curl_close($ch);
fclose($fp);
return (filesize($filepath) > 0)? true : false;
}
Include following lines of code after curl_init() .i think this will work.
CURLOPT_RETURNTRANSFER :::
TRUE to return the transfer as a string of the return value of
curl_exec() instead of outputting it out directly.
CURLOPT_USERAGENT::The contents of the "User-Agent: " header to be used in a HTTP request.
Read more about curl_setopt here.
$ch = curl_init(str_replace(" ","%20",$url));
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt ($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6");
I am struck with this. I want to download a csv file from url using curl. I have referred all the answer in stackoverflow and tried all. But not getting what i am expected. i have the following code.
define("COOKIE_FILE", "cookie.txt");
$path = "settlement_file/test.csv";
set_time_limit(0);
$fp = fopen ($path, 'w+');//This is the file where we save the information
$ch = curl_init(str_replace(" ","%20",$url));//Here is the file we are downloading, replace spaces with %20
curl_setopt($ch, CURLOPT_TIMEOUT, 50);
curl_setopt($ch, CURLOPT_FILE, $fp); // write curl response to file
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER,false);
curl_setopt ($ch, CURLOPT_COOKIEFILE, COOKIE_FILE);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_SSLVERSION,3);
curl_exec($ch); // get curl response
curl_close($ch);
fclose($fp);
You have to remove the curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); statement. It makes curl_exec return the data instead of writing it to a file. Since it comes after the curl_setopt($ch, CURLOPT_FILE, $fp); it overrides that, so just remove the former line.
I have a file to download via curl in PHP about 16Mb, is a zip file,m I want to download and when is finished to download extract it, after extract parse every file inside it.
This is my code:
$ch2=curl_init();
curl_setopt($ch2, CURLOPT_URL, $this->URL);
curl_setopt($ch2, CURLOPT_TIMEOUT, 5040);
curl_setopt($ch2, CURLOPT_HEADER, 0);
curl_setopt($ch2, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch2, CURLOPT_POST, 1);
curl_setopt($ch2, CURLOPT_POSTFIELDS,$this->XMLRequest);
curl_setopt($ch2, CURLOPT_SSL_VERIFYHOST, 1);
curl_setopt($ch2, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($ch2, CURLOPT_SSLVERSION, 3);
curl_setopt($ch2, CURLOPT_FOLLOWLOCATION, true);
$httpHeader2 = array(
"Content-Type: text/xml; charset=UTF-8",
"Content-Encoding: UTF-8"
);
// Execute request, store response and HTTP response code
$xml = curl_exec($ch2);
$this->errno=curl_getinfo( $ch2, CURLINFO_HTTP_CODE );
curl_close($ch2);
$file2 = fopen('json_upload/item.zip','w+');
fwrite($file2, $xml);
fclose($file2);
$zip = new ZipArchive;
echo'<p>json_upload/item.zip</p>';
$zip->open('json_upload/item.zip', ZIPARCHIVE::CREATE | ZIPARCHIVE::OVERWRITE);
$zip->extractTo('json_upload/item/');
$zip->close();
In this mode the file downloaded is 1,6Mb not all and if I try to extract manually return me an error like is damaged.
If I comment the extract code of the zip the file is downloaded completely and if I extract it manually it works fine.
How can I execute the extract of the zip only when the download is complete?
And after how can I execute the parse of each file inside only when the extract command finished?
Thanks
Instead of using fwrite, use file_put_contents()
Replace
$file2 = fopen('json_upload/item.zip','w+');
fwrite($file2, $xml);
fclose($file2);
With
file_put_contents('json_upload/item.zip', $xml);
I have the following code for downloading images from remote URL's - the code seems to work ok except where there is a space in the URL - i've put in the str_replace to try to fix this but the problem persists - can anyone suggest what i'm doing wrong when they download they appear as 0kb files.
$image_url = str_replace(" ", "%20", $image_url);
$fp = fopen($source_image, "wb");
$ch = curl_init($image_url);
curl_setopt($ch, CURLOPT_FILE, $fp);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_exec($ch);
curl_close($ch);
fclose($fp);
Ok sussed it - the site uses https on the images (god knows why) the following worked..
$fp = fopen($source_image, "wb");
$ch = curl_init($image_url);
curl_setopt($ch, CURLOPT_FILE, $fp);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
I have the following php code which I found here:
function download_xml()
{
$url = 'http://tv.sygko.net/tv.xml';
$ch = curl_init($url);
$timeout = 5;
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
$data = curl_exec($ch);
echo("curl_exec was succesful"); //This never gets called
curl_close($ch);
return $data;
}
$my_file = 'tvdata.xml';
$handle = fopen($my_file, 'w');
$data = download_xml();
fwrite($handle, $data);
What I'm trying to do is to download the xml at the specified url and save it to the disk. However, it stops once about 80% finished and never reaches the echo call after the curl_exec call. I'm not sure why, but I believe this is because it runs out of memory. Therefore I would like to ask if it is possible to make curl write the data to the file every time it has downloaded say 4kb. If this is not possible, do anybody know a way to get the xml file stored at the url downloaded and stored on my disk using php?
Thank you very much,
BEN.
EDIT:
This is the code now, it doesnt work. It writes the data to the file but still only about 80% of the document. Maybe it isn't because it exceeds memory but some other reason? I really can't believe it is this hard to copy a file from a URL to the disc...
<?
$url = 'http://tv.sygko.net/tv.xml';
$my_file = fopen('tvdata.xml', 'w');
$ch = curl_init($url);
$timeout = 300;
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FILE, $my_file);
curl_setopt($ch, CURLOPT_FAILONERROR, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
curl_setopt($ch, CURLOPT_BUFFERSIZE, 4096);
curl_exec($ch) OR die("Error in curl_exec()");
echo("got to after curl exec");
fclose($my_file);
curl_close($ch);
?>
Here comes a fully working example:
public function saveFile($url, $dest) {
if (!file_exists($dest))
touch($dest);
$file = fopen($dest, 'w');
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_PROGRESSFUNCTION, 'progressCallback');
curl_setopt($ch, CURLOPT_BUFFERSIZE, (1024*1024*512));
curl_setopt($ch, CURLOPT_NOPROGRESS, FALSE);
curl_setopt($ch, CURLOPT_FAILONERROR, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_TIMEOUT, 15);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13');
curl_setopt($ch, CURLOPT_FILE, $file);
curl_exec($ch);
curl_close($ch);
fclose($file);
}
?>
The secret lies withing setting CURLOPT_NOPROGRESS to FALSE, and then, CURLOPT_BUFFERSIZE will make the callback report for every CURLOPT_BUFFERSIZE bytes reached. The smaller value, the more frequently it will report. This also depends on your download speed, etc, so don't count on it to report every X seconds, since it will report for every X bytes received/transferred.
Your timeout is set to 5 seconds which might be too short depending on the file size of the document. Try increasing it to 10-15 just to make sure it has enough time to complete the transfer.
There's an option called CURELOPT_FILE that allows you to specify a file handler that curl should write to. I'm pretty sure it will do "right" thing and "write" as it reads, avoiding your memory problem
$file = fopen('test.txt', 'w'); //<--------- file handler
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,'http://example.com');
curl_setopt($ch, CURLOPT_FAILONERROR,1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION,1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
curl_setopt($ch, CURLOPT_TIMEOUT, 15);
curl_setopt($ch, CURLOPT_FILE, $file); //<------- this is your magic line
curl_exec($ch);
curl_close($ch);
fclose($file);
curl_setopt the CURLOPT_FILE - The file that the transfer should be written to. The default is STDOUT (the browser window)
http://us2.php.net/manual/en/function.curl-setopt.php