this is very weird, php curl download stops at 95% all the time. its driving me crazy.
here is the code that i'm using nothing fancy
$fp = fopen($file, 'w');
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "http://www.domain.com/");
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch,CURLOPT_HTTPHEADER,array("ETag: $rddash"));
curl_setopt($ch, CURLOPT_FILE, $fp);
curl_exec($ch);
curl_close($ch);
fclose($fp);
Something i noticed, the remote website is using Etag, so i used it but still not working.
what could be the reason the download stops before it completes??
Maybe a timeout issue in your php.ini settings. Use set_time_limit(0); in your code.
See the manual for more details.
Also check the PHP error log.
Related
I am trying to download an image from the internet to my local folder:
$loc = "downloaded images/".basename($_POST["url"]);
if(file_put_contents($loc,file_get_contents($_POST["url"])));
echo "Downloaded successfully";
else
echo error_get_last()['message'];
The code works only in my local server. When I ran the code in my live server, this huge warning keeps popping up in my console:
Warning: file_get_contents(https://neindiabroadcast.com/wp-content/uploads/2022/11/20221122_071752.jpg): Failed to open stream: Connection timed out in
I have also tried using cURL:
set_time_limit(0); // unlimited max execution time
$loc = "downloaded images/".basename($_POST["url"]);
$host = $_POST["url"];
$ch = curl_init();
curl_setopt($ch, CURLOPT_TIMEOUT, 28800);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_URL, $host);
curl_setopt($ch, CURLOPT_REFERER, $host);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
$result = curl_exec($ch);
curl_close($ch);
$fp = fopen($loc, 'wb');
fwrite($fp, $result);
fclose($fp);
When I try to open the file, it says the file is corrupted. What's surprising is that, both the codes work perfectly only when I try them in my local server.
My max_execution_time is set to unlimited, so the time required to execute shouldn't be the issue here. Then why is this issue occurring? Please help me in resolving this issue
After hours of struggling, I finally came up with a solution. Every website has a no CORS policy which is kind of a security feature, where the server won't allow you to read the data of that website. To bypass this, all you have to do is to use a proxy url:
var proxy_url = 'https://api.codetabs.com/v1/proxy?quest=';
Prepend this url to the url of the website whose data you want to fetch, and you are done :)
I have the following PHP script that works perfectly 99% of the time. But it will not download an image from this one server which I think is running Varnish Cache.
<?php
$imglink = 'http://www.dirtrider.com/wp-content/uploads/2014/10/WD-10_1_14-001.jpg';
$ch = curl_init($imglink);
$fp = fopen('/home/path/www/tmp/test.jpg', "w");
curl_setopt($ch, CURLOPT_FILE, $fp);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_TIMEOUT, 5);
curl_exec($ch);
fclose($fp);
You get a 403 Forbidden error if you use CURL to load that image. You can work around this error very easily. Just add an alternate user agent for your CURL request:
curl_setopt($ch, CURLOPT_USERAGENT, $_SERVER['HTTP_USER_AGENT']);
And et voila! It works like a charm. Seems like Varnishe Cache blocks CURL requests which use a CURL default user agent.
my curl code show me a blank page
<?
$ch = curl_init();
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($ch, CURLOPT_URL,"http://mysite/scripts/showsomething.php");
curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
$result=curl_exec ($ch);
curl_close ($ch);
$core = explode('<!--- BEGIN -->', $result);
$main = explode('<!--- END -->', $core[1]);
echo $main[0];
?>
this code works fine on localhost, but not on server...
There can be several reasons behind your problem.
1) Change <? into <?php and see whether it works or not.
2) For a short test, run this code from your server and check whether it shows you the output or not.
<?php
echo "sabuj";
?>
3) Some site seek for useragent string on their website request. When they found no useragent, they use to return blank page. You can overcome this with setting an useragent like below.
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0');
4) Another thing you can do is, access the server with ssh client(if you have any) and run the url with wget tool and see whether you can download your page or not.
wget "http://yoursite/page/blabla/...php"
5) Finally, run your curl code with verbose mode enabled. It will help you to debug your curl requests.
curl_setopt($ch, CURLOPT_VERBOSE,true);
Working Code...
$ch = curl_init();
$timeout = 10;
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
$data = curl_exec($ch);
curl_close($ch);
var_dump($data);
Hope it helps...
Turn your error reporting all the way to 11.
error_reporting(E_ALL);
This will tell you a little more about the issue you are facing. The most common culprit when working with curl on a dev environment and moving to a server is a missing curl package.
You can check to see if you have curl installed by doing the following:
if(!function_exists('curl_version')) {
throw new Exception('Curl package missing');
}
Thus "PHP Fatal error: Call to undefined function curl_init() in..." is a very common error that is thrown.
Some additional debugging tips are to..
print_r the response of curl_exec($handle);
print_r curl_error($handle) this will give you a curl error code.
setup a curl proxy using set_opt and CURLOPT_PROXY, set the value to your ip:port, open the port 8888 on your router and install charles proxy. This will show you an exact printout of the request and response.
I've read the previous answers and it seems that your code is fine but problem is connectivity to remote server, please try this:
<?
//debug
error_reporting(E_ALL);
ini_set('display_errors', 1);
//debug end
$ch = curl_init();
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($ch, CURLOPT_URL,"http://mysite/scripts/showsomething.php");
curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
$result=curl_exec ($ch);
curl_close ($ch);
echo result;
?>
what does it output ?
Notes:
Do you administer the other server?
if not, is there a possibility that your ip was blocked by the remote server?
Does script http://mysite/scripts/showsomething.php contains any errors ?
If you're able to edit it, please enable php errors on showsomething.php by adding the following code at the top of it :
error_reporting(E_ALL);
ini_set('display_errors', 1);
I've solved the problem by adding the following code.
The problem is SSL.
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
I hope it helps.
I'm trying something with curl in php.
I read about the use of CURLOPT_VERBOSE to help debugging.
But it gives no output. I checked my phpinfo() and under cURL stands:
debug => no
I guess that is the problem here. How can I set this to yes?
I looked in php.ini and could find it.
Also no luck on google.
also no luck here: How to enable features for php-curl
I hope someone can help me!
Having a debug build of cURL isn't required for the CURLOPT_VERBOSE setting to work, but by default CURLOPT_VERBOSE information gets output to STDERR which is only visible on the console. So if you are running PHP from a browser, the verbose output isn't sent anywhere you can see.
You can set the option CURLOPT_STDERR to a file handle and the verbose output should be written there.
You should be able to do this:
$stdout = fopen('php://output', 'w+');
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'http://www.example.com');
curl_setopt($ch, CURLOPT_VERBOSE, 1);
curl_setopt($ch, CURLOPT_STDERR, $stdout);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$data = curl_exec($ch);
I'm not sure if I'm encountering a PHP bug at the moment (running PHP 5.4.5 via FastCGI) because I don't see the verbose output in the browser, but if I run it from the command line I do. If I fwrite to $stdout I do see that output in the browser but still nothing from cURL so I know the handle is valid.
If you experience the same issue, here is a workaround:
$tempout = fopen('php://temp', 'w+');
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'http://www.example.com');
curl_setopt($ch, CURLOPT_VERBOSE, 1);
curl_setopt($ch, CURLOPT_STDERR, $tempout);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$data = curl_exec($ch);
rewind($tempout);
$debug = stream_get_contents($tempout);
echo $debug; // the data from CURLOPT_VERBOSE
Hope that helps.
I have sites where stored some xml files, and I want to download to our server, we don't have ftp connection, so we can download through http. I always used file(url) is there any better way to download files through php
If you can access them via http, file() (which reads the file into an array) and file_get_contents() (which reads content into a string) are perfectly fine provided that the wrappers are enabled.
Using CURL could also be a nice option:
// create a new CURL resource
$ch = curl_init();
// set URL and other appropriate options
curl_setopt($ch, CURLOPT_URL, "http://www.server.com/file.zip");
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_BINARYTRANSFER, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
set_time_limit(300); # 5 minutes for PHP
curl_setopt($ch, CURLOPT_TIMEOUT, 300); # and also for CURL
$outfile = fopen('/mysite/file.zip', 'wb');
curl_setopt($ch, CURLOPT_FILE, $outfile);
// grab file from URL
curl_exec($ch);
fclose($outfile);
// close CURL resource, and free up system resources
curl_close($ch);