I am trying to download an image from the internet to my local folder:
$loc = "downloaded images/".basename($_POST["url"]);
if(file_put_contents($loc,file_get_contents($_POST["url"])));
echo "Downloaded successfully";
else
echo error_get_last()['message'];
The code works only in my local server. When I ran the code in my live server, this huge warning keeps popping up in my console:
Warning: file_get_contents(https://neindiabroadcast.com/wp-content/uploads/2022/11/20221122_071752.jpg): Failed to open stream: Connection timed out in
I have also tried using cURL:
set_time_limit(0); // unlimited max execution time
$loc = "downloaded images/".basename($_POST["url"]);
$host = $_POST["url"];
$ch = curl_init();
curl_setopt($ch, CURLOPT_TIMEOUT, 28800);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_URL, $host);
curl_setopt($ch, CURLOPT_REFERER, $host);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
$result = curl_exec($ch);
curl_close($ch);
$fp = fopen($loc, 'wb');
fwrite($fp, $result);
fclose($fp);
When I try to open the file, it says the file is corrupted. What's surprising is that, both the codes work perfectly only when I try them in my local server.
My max_execution_time is set to unlimited, so the time required to execute shouldn't be the issue here. Then why is this issue occurring? Please help me in resolving this issue
After hours of struggling, I finally came up with a solution. Every website has a no CORS policy which is kind of a security feature, where the server won't allow you to read the data of that website. To bypass this, all you have to do is to use a proxy url:
var proxy_url = 'https://api.codetabs.com/v1/proxy?quest=';
Prepend this url to the url of the website whose data you want to fetch, and you are done :)
Related
A developer did a custom PHP script for me for importing posts into Wordpress from CSV file. The script worked fine on a staging site, which was on a different server, but when we moved it to my server, it can't download the CSV file and even if I manually import the file to the folder, it won't import it. It doesn't show any errors, just a blank page.
It's a shared hosting, so the provider has set the max_execution_time to 120, which will be enough for the script to run, but it times out on 30 seconds.
The script is using curl_setopt to get the file. The PHP version is 5.5
$userAgent = 'FreeRock.Eu/2.0 (http://www.freerock.eu/share.php)';
$ch = curl_init();
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_USERAGENT, $userAgent);
curl_setopt($ch, CURLOPT_URL,$address);
curl_setopt($ch, CURLOPT_FAILONERROR, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_AUTOREFERER, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER,true);
curl_setopt($ch, CURLOPT_TIMEOUT, 400);
$html = curl_exec($ch);
if (!$html) {
echo "<br /> error number:" .curl_errno($ch);
echo "<br /> error:" . curl_error($ch);
exit;
}
return $html;
}
Then I have:
$z_html = fake_user_agent_http_get('https://www.myfilelocation.com');
$myfile = fopen("promotions.csv", "w") or die("Unable to open file!");
fwrite($myfile, $z_html);
fclose($myfile);
Would appreciate any help here.
Thanks
Alexis
There can be numerous reasons for this. Lets try a few things:
1) Add this in the top of the script so maybe you can see the error:
ini_set('display_errors',true);
error_reporting(e_all);
2) Try add the following in the top of the code to avoid the 30 seconds max execution time:
set_time_limit(0);
3) The hosting provider probably have php error logs available. Try to download them to figure out what the error is.
Update - the timeout was caused by an nginx settings, but because it's a shared hosting, the 30 seconds timeout cannot be changed. The solution is to run the script via SSH.
I have the following PHP script that works perfectly 99% of the time. But it will not download an image from this one server which I think is running Varnish Cache.
<?php
$imglink = 'http://www.dirtrider.com/wp-content/uploads/2014/10/WD-10_1_14-001.jpg';
$ch = curl_init($imglink);
$fp = fopen('/home/path/www/tmp/test.jpg', "w");
curl_setopt($ch, CURLOPT_FILE, $fp);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_TIMEOUT, 5);
curl_exec($ch);
fclose($fp);
You get a 403 Forbidden error if you use CURL to load that image. You can work around this error very easily. Just add an alternate user agent for your CURL request:
curl_setopt($ch, CURLOPT_USERAGENT, $_SERVER['HTTP_USER_AGENT']);
And et voila! It works like a charm. Seems like Varnishe Cache blocks CURL requests which use a CURL default user agent.
my curl code show me a blank page
<?
$ch = curl_init();
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($ch, CURLOPT_URL,"http://mysite/scripts/showsomething.php");
curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
$result=curl_exec ($ch);
curl_close ($ch);
$core = explode('<!--- BEGIN -->', $result);
$main = explode('<!--- END -->', $core[1]);
echo $main[0];
?>
this code works fine on localhost, but not on server...
There can be several reasons behind your problem.
1) Change <? into <?php and see whether it works or not.
2) For a short test, run this code from your server and check whether it shows you the output or not.
<?php
echo "sabuj";
?>
3) Some site seek for useragent string on their website request. When they found no useragent, they use to return blank page. You can overcome this with setting an useragent like below.
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0');
4) Another thing you can do is, access the server with ssh client(if you have any) and run the url with wget tool and see whether you can download your page or not.
wget "http://yoursite/page/blabla/...php"
5) Finally, run your curl code with verbose mode enabled. It will help you to debug your curl requests.
curl_setopt($ch, CURLOPT_VERBOSE,true);
Working Code...
$ch = curl_init();
$timeout = 10;
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
$data = curl_exec($ch);
curl_close($ch);
var_dump($data);
Hope it helps...
Turn your error reporting all the way to 11.
error_reporting(E_ALL);
This will tell you a little more about the issue you are facing. The most common culprit when working with curl on a dev environment and moving to a server is a missing curl package.
You can check to see if you have curl installed by doing the following:
if(!function_exists('curl_version')) {
throw new Exception('Curl package missing');
}
Thus "PHP Fatal error: Call to undefined function curl_init() in..." is a very common error that is thrown.
Some additional debugging tips are to..
print_r the response of curl_exec($handle);
print_r curl_error($handle) this will give you a curl error code.
setup a curl proxy using set_opt and CURLOPT_PROXY, set the value to your ip:port, open the port 8888 on your router and install charles proxy. This will show you an exact printout of the request and response.
I've read the previous answers and it seems that your code is fine but problem is connectivity to remote server, please try this:
<?
//debug
error_reporting(E_ALL);
ini_set('display_errors', 1);
//debug end
$ch = curl_init();
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($ch, CURLOPT_URL,"http://mysite/scripts/showsomething.php");
curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
$result=curl_exec ($ch);
curl_close ($ch);
echo result;
?>
what does it output ?
Notes:
Do you administer the other server?
if not, is there a possibility that your ip was blocked by the remote server?
Does script http://mysite/scripts/showsomething.php contains any errors ?
If you're able to edit it, please enable php errors on showsomething.php by adding the following code at the top of it :
error_reporting(E_ALL);
ini_set('display_errors', 1);
I've solved the problem by adding the following code.
The problem is SSL.
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
I hope it helps.
this is very weird, php curl download stops at 95% all the time. its driving me crazy.
here is the code that i'm using nothing fancy
$fp = fopen($file, 'w');
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "http://www.domain.com/");
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch,CURLOPT_HTTPHEADER,array("ETag: $rddash"));
curl_setopt($ch, CURLOPT_FILE, $fp);
curl_exec($ch);
curl_close($ch);
fclose($fp);
Something i noticed, the remote website is using Etag, so i used it but still not working.
what could be the reason the download stops before it completes??
Maybe a timeout issue in your php.ini settings. Use set_time_limit(0); in your code.
See the manual for more details.
Also check the PHP error log.
I have sites where stored some xml files, and I want to download to our server, we don't have ftp connection, so we can download through http. I always used file(url) is there any better way to download files through php
If you can access them via http, file() (which reads the file into an array) and file_get_contents() (which reads content into a string) are perfectly fine provided that the wrappers are enabled.
Using CURL could also be a nice option:
// create a new CURL resource
$ch = curl_init();
// set URL and other appropriate options
curl_setopt($ch, CURLOPT_URL, "http://www.server.com/file.zip");
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_BINARYTRANSFER, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
set_time_limit(300); # 5 minutes for PHP
curl_setopt($ch, CURLOPT_TIMEOUT, 300); # and also for CURL
$outfile = fopen('/mysite/file.zip', 'wb');
curl_setopt($ch, CURLOPT_FILE, $outfile);
// grab file from URL
curl_exec($ch);
fclose($outfile);
// close CURL resource, and free up system resources
curl_close($ch);