Downloading files via PHP - php

I have problem with downloading files via PHP.
The funny thing is that I can not trace the problem. The code works good for some websites and not good with other. It is loop in PHP that downloads the backup files from websites (there is delay with sleep before requests).
Why I can not trace the problem?
Because when I run manually the code, it works (downloads the file). And when it is run by CRON, sometimes it downloads the file, sometimes it does NOT download the file (only downloads 2 empty new lines).
The download is with curl (I have also tried with different code with fsockopen and fread).
Does anyone have an idea on how I can solve this?
Headers are removed with CURL by setting the correct option.
function fetch_url($url) {
$c = curl_init();
curl_setopt($c, CURLOPT_URL, $url);
curl_setopt($c, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($c, CURLOPT_TIMEOUT, 20);
if ($cookiejar != '') {
curl_setopt($c, CURLOPT_COOKIEJAR, $cookiejar);
curl_setopt($c, CURLOPT_COOKIEFILE, $cookiejar);
}
curl_setopt($c, CURLOPT_HEADER , false);
curl_setopt($c, CURLOPT_SSL_VERIFYHOST , false);
curl_setopt($c, CURLOPT_SSL_VERIFYPEER , false);
curl_setopt($c, CURLOPT_FOLLOWLOCATION , true);
curl_setopt($c, CURLOPT_AUTOREFERER , true);
curl_setopt($c, CURLOPT_USERAGENT, 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12');
$con = curl_exec($c);
curl_close($c);
return $con;
}
echo fetch_url('http://www.example.com/zip.zip');

Try using http://www.php.net/manual/en/function.curl-getinfo.php to display information about the curl request
echo curl_errno($c);
print_r(curl_getinfo($c));
Also, Maybe it's in your code elsewhere, but I'm not seeing any content type headers for your echoing of the file
$file = fetch_url('http://www.example.com/zip.zip');
header('Content-type: application/zip');
header('Content-Disposition: attachment; filename="zip.zip"');
header("Content-length: " . strlen($file));
echo $file;

Related

Transalte curl command to PHP

Hello I have this Linux command that downloads a compressed file
curl -L -O http://www.url.com
The problem is that when I do curl inside PHP I get the HTML code instead of the compressed file.
The PHP code is this:
$url = https://www.example.com
$filePath = '/app/storage/temp/' . $fileName;
$fp = fopen($filePath . 'me', "w");
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_FTPAPPEND, 0);
curl_setopt($ch, CURLOPT_FILE, $fp);
$data = curl_exec($ch);
fwrite($fp, $data);
curl_close($ch);
fclose($fp);
I can't share the real URL since it contains secret keys.
EDIT
The curl command downloaded the same HTML file as the curl command when I added the -L -O options to the curl command it started working, so the thing here is, how can I add those lines with PHP
Using CURL_FILE means that the output is written to the file handle. You don't need to also use fwrite (especially since without setting CURLOPT_RETURNTRANSFER, the return from curl_exec is just true or false).
If it is indeed possible to load from this URL, either remove the fwrite, or remove the CURLOPT_FILE and use:
curl_setopt($ch, CUROPT_RETURNTRANFER, TRUE)
That way, the return from curl_exec will be the loaded data.
Create an empty zip file where you want to download your file.
function downloadFile($url) {
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 2);
curl_setopt($ch, CURLOPT_TIMEOUT, 60);
set_time_limit(65);
$rawFileData = curl_exec($ch);
$info = curl_getinfo($ch);
if (curl_errno($ch)) {
return curl_error($ch);
}
curl_close($ch);
$filepath = dirname(__FILE__) . '/testfile.zip';
file_put_contents($filepath, $rawFileData); // Put the downloaded content in the file
return $info;
}
Hope this helps!
Guys sorry for making this hard for you since I couldn't give much information about my doubt.
The problem is solved, I realized that the URL did work with
curl -O -L htttp://www.example.com
and also by the web browser.
This last thing was actually the one that gave me the path:
Open the web browser
Click F12
Paste the URL and hit enter
I came to realize I needed to add some headers the headers were these:
accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
accept-encoding:gzip, deflate, br
accept-language:en-US,en;q=0.8,es-MX;q=0.6,es;q=0.4
user-agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36
After I added these headers to the curl inside PHP the result was the compressed zip file I was searching for.

How to use PHP CURL to bypass cross domain

I need PHP to submit paramaters from one domain to another. JavaScript is not an option for my situation. I'm now trying to use CURL with PHP, but have not been successful in bypassing the cross domain.
From domain_A, I have a page with the following PHP with CURL script:
if (_iscurl()){
echo "<p>CURL is enabled</p>";
$url = "http://domain_B/process.php?id=123&amt=100&jsonp=?";
$ch = curl_init();
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 0);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT,10);
curl_setopt($ch, CURLOPT_USERAGENT , "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1)");
curl_setopt($ch, CURLOPT_URL, $url );
$return = curl_exec($ch);
curl_close($ch);
echo "<p>Finished operations</p>";
}
else{
echo "CURL is disabled";
}
?>
I am not getting any results, so I am assuming that the PHP CURL script is not successful. Any ideas to fix this?
Thanks
Well, its bit late. But adding this answer for further readers who might face similar issue. This issue arises some times when we are sending php curl request from a domain hosted over http to a domain hosted over https (http over ssl).
Just add below code snippet before curl execution.
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$output = curl_exec($ch);
Using false in CURLOPT_RETURNTRANSFER doesn't return anything by curl. make it true(or 1)
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

PHP CURL - Problems storing and using cookies when scraping

I've been trying to write a script that retrieves Google trends results for a given keyword. Please note im not trying to do anything malicious I just want to be able to automate this process and run it a few times every day.
After investigating the Google trends page I discovered that the information is available using the following URL:
http://www.google.com/trends/trendsReport?hl=en-GB&q=keyword&cmpt=q&content=1
You can request that information mutliple times with no issues from a browser, but if you try with "privacy mode" after 4 or 5 requests the following is displayed:
An error has been detected You have reached your quota limit. Please
try again later.
This makes me think that cookies are required. So I have written my script as follows:
$cookiefile = $siteurl . '/wp-content/plugins/' . basename(dirname(__FILE__)) . '/cookies.txt';
$url = 'http://www.google.com/trends/trendsReport?hl=en-GB&q=keyword&cmpt=q&content=1';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookiefile);
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookiefile);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.8) Gecko/2009032609 Firefox/3.0.8');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
$x='error';
while (trim($x) != '' ){
$html=curl_exec($ch);
$x=curl_error($ch);
}
echo "test cookiefile contents = ".file_get_contents($cookiefile)."<br />";
echo $html;
However I just can't get anything written to my cookies file. So I keep on getting the error message. Can anyone see where I'm going wrong with this?
I'm pretty sure your cookie file should exist before you can use it with curl.
Try:
$h = fopen($cookiefile, "x+");

Error when copying img with a cURL script (php)

I made a nice simple userscript:
When I browse the web, I can "bookmark" any image in 1 click
My userscript
Grab the img src
Grab the url of the webpage
Copy the .jpg .png .gif to my server
Everything works perfectly, BUT in some cases, the script cannot copy the file...
Actually the file is created but do not contains the img data, it only contains the content of an error webpage:
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access /data/x/xxx_xxx_x.jpg on this server.</p>
<p>Additionally, a 403 Forbidden
error was encountered while trying to use an ErrorDocument to handle the request.</p>
<hr>
<address>Apache Server at xxxxxxxx.net Port 80</address>
</body></html>
The "copy" code (php):
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $urlimg);
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_BINARYTRANSFER, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
set_time_limit(300); # 5 minutes for PHP
curl_setopt($ch, CURLOPT_TIMEOUT, 300); # and also for CURL
$path = $dirpix.'/'.$aa.'/'.$mm;
if ( ! is_dir($path)) {
mkdir($path);
}
$outfile = fopen($path.'/'.$id.'.'.$ext, 'wb');
curl_setopt($ch, CURLOPT_FILE, $outfile);
curl_exec($ch);
fclose($outfile);
curl_close($ch);
Maybe the website blocks that kind of "copy" script?
Thanks!
2 things I can think of here are,
Set a user agent to your curl request. Because from what you say, you are able to view the image but curl is getting 403 error, it could very well be userAgent filtering on server side.
Add referer to your curl request. You can send the referer information from your userscript to the php script. You'd have to post or get window.location.href's value.
Try it below code it working fine in my server. it is tested code:-
<?php
$img[]='http://i.indiafm.com/stills/celebrities/sada/thumb1.jpg';
$img[]='http://i.indiafm.com/stills/celebrities/sada/thumb5.jpg';
$path="images/";
foreach($img as $i){
save_image($i, $path);
if(getimagesize($path.basename($i))){
echo '<h3 style="color: green;">Image ' . basename($i) . ' Downloaded OK</h3>';
}else{
echo '<h3 style="color: red;">Image ' . basename($i) . ' Download Failed</h3>';
}
}
//Alternative Image Saving Using cURL seeing as allow_url_fopen is disabled - bummer
function save_image($img,$fullpath='basename'){
if($fullpath!='basename'){
$fullpath = $fullpath.basename($img);
}
$ch = curl_init ($img);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_BINARYTRANSFER,1);
$rawdata=curl_exec($ch);
curl_close ($ch);
if(file_exists($fullpath)){
unlink($fullpath);
}
$fp = fopen($fullpath,'x');
fwrite($fp, $rawdata);
fclose($fp);
}
for correct work add
$agent= 'Accept:text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 User-Agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11';
curl_setopt($ch, CURLOPT_USERAGENT, $agent);
I had a hard time to access my DLink camera using this method.
But finally I found the issue: authentication.
Don't forget authentication.
This is the solution that worked for me, thanks to all contributors.
<?php
function download_image1($image_url, $image_file){
$fp = fopen ($image_file, 'w+'); // open file handle
$ch = curl_init($image_url);
$agent= 'Accept:image/jpeg,text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 User-Agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11';
curl_setopt($ch, CURLOPT_USERAGENT, $agent);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_BINARYTRANSFER,true);
curl_setopt($ch, CURLOPT_TIMEOUT, 400);
curl_setopt($ch, CURLOPT_AUTOREFERER, false);
curl_setopt($ch, CURLOPT_REFERER, "http://google.com");
curl_setopt($ch, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_1_1);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE); // Follows redirect responses.
curl_setopt($ch, CURLOPT_USERPWD, "user:password");
$raw=curl_exec($ch);
if ($raw === false) {
trigger_error(curl_error($ch));
}
curl_close ($ch);
$localName = $image_file; // The file name of the source can be used locally
if(file_exists($localName)){
unlink($localName);
}
$fp = fopen($localName,'x');
fwrite($fp, $raw);
fclose($fp);
}
download_image1("http://url_here/image.jpg","/path/filename.jpg"); // to access DLink cameras
// be sure you have rights to the path
?>
The code above probably has some redundance, since I am openning fopen twice. To be honest, I won't correct, sinve it is working!

Curl redirect,, not working?

I'm using the following code:
$agent= 'Mozilla/5.0 (Windows; U; Windows NT 5.1; pl; rv:1.9) Gecko/2008052906 Firefox/3.0';
$ch = curl_init();
curl_setopt($ch, CURLOPT_USERAGENT, $agent);
curl_setopt($ch, CURLOPT_URL, "www.example.com");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HEADER, 0);
$output = curl_exec($ch);
echo $output;
But it redirects to like this:
http://localhost/aide.do?sht=_aide_cookies_
Instead of to the URL page.
Can anyone help me solve my problem, please?
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
http://docs.php.net/function.curl-setopt says:
CURLOPT_FOLLOWLOCATION
TRUE to follow any "Location: " header that the server sends as part of the HTTP header (note this is recursive, PHP will follow as many "Location: " headers that it is sent, unless CURLOPT_MAXREDIRS is set).
If it's up to URL redirection only then see the following code, I've documented it for you so you can use it easily & directly, you've two main cURL options control URL redirection (CURLOPT_FOLLOWLOCATION/CURLOPT_MAXREDIRS):
// create a new cURL resource
$ch = curl_init();
// The URL to fetch. This can also be set when initializing a session with curl_init().
curl_setopt($ch, CURLOPT_URL, "http://www.example.com/");
// The contents of the "User-Agent: " header to be used in a HTTP request.
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; pl; rv:1.9) Gecko/2008052906 Firefox/3.0");
// TRUE to include the header in the output.
curl_setopt($ch, CURLOPT_HEADER, false);
// TRUE to return the transfer as a string of the return value of curl_exec() instead of outputting it out directly.
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
// TRUE to follow any "Location: " header that the server sends as part of the HTTP header (note this is recursive, PHP will follow as many "Location: " headers that it is sent, unless CURLOPT_MAXREDIRS is set).
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
// The maximum amount of HTTP redirections to follow. Use this option alongside CURLOPT_FOLLOWLOCATION.
curl_setopt($ch, CURLOPT_MAXREDIRS, 10);
// grab URL and pass it to the output variable
$output = curl_exec($ch);
// close cURL resource, and free up system resources
curl_close($ch);
// Print the output from our variable to the browser
print_r($output);
The above code handles the URL redirection issue, but it doesn't deal with cookies (your localhost URL seems to be dealing with cookies). If you wish to deal with cookies from the cURL resource, then you may have to give the following cURL options a look:
CURLOPT_COOKIE
CURLOPT_COOKIEFILE
CURLOPT_COOKIEJAR
For further details please follow the following link:
http://docs.php.net/function.curl-setopt

Categories