I want to know which one is better for trans-loading large files on my server. I have 2 options using curl or normal php fopen or fwrite. Below are both implementations. Could you please suggest which one is better and with reasons if possible.
Curl implementation
$fp = fopen (dirname(__FILE__) . '/localfile.tmp', 'w+');//This is the file where we save the information
$ch = curl_init(str_replace(" ","%20",$url));//Here is the file we are downloading, replace spaces with %20
curl_setopt($ch, CURLOPT_TIMEOUT, 50);
curl_setopt($ch, CURLOPT_FILE, $fp); // write curl response to file
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_exec($ch); // get curl response
curl_close($ch);
fclose($fp);
Normal php implementation
while(!feof($url)) {
fwrite($filename, fread($file, 1024 * 8 ), 1024 * 8 );
}
Related
I'm trying do retrieve and download a file (image) from a remote location.
Inside the php.ini the allow_url_fopen is enabled, but i can't download the image.
Code i'm using is described below
$local_file = "test.jpg";
$remote_file = "http://somehost:6346/cgi-bin/CGIProxy.fcgi?cmd=snapPicture2&usr=xxxx&pwd=xxxx";
$ch = curl_init();
$fp = fopen ($local_file, 'w+');
$ch = curl_init($remote_file);
curl_setopt($ch, CURLOPT_TIMEOUT, 50);
curl_setopt($ch, CURLOPT_FILE, $fp);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_ENCODING, "");
curl_exec($ch);
curl_close($ch);
fclose($fp);
with any other url that contains a real jpg file, it's working perfectly, i suppose that the issue is that the url use some special characters that doesn't like to curl.
If i try to execute the php snippet above,page load for almost 1 minute,and it seems that no error are displayed,the image test.jpg is created, but it's empty.
Do you have any suggestion?
Thanks!
Try this
$local_file = "test.jpg";
$remote_file = "http://somehost:6346/cgi-bin/CGIProxy.fcgi?cmd=snapPicture2&usr=xxxx&pwd=xxxx";
function getPage($url) {
$ch = curl_init();
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_REFERER, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
$result = curl_exec($ch);
curl_close($ch);
return $result;
}
function saveToFile($base, $decode=false, $output_file)
{
$ifp = fopen($output_file, "wb");
if ($decode){
fwrite($ifp, base64_decode($base));
}else{
fwrite($ifp, $base);
}
fclose($ifp);
return($output_file);
}
$remote_page = getPage($remote_file);
$saved_file = saveToFile($remote_page , false, $local_file);
when debugging issues like this, set CURLOPT_VERBOSE, it will probably reveal why the page loaded for almost 1 minute, with no apparent output.
i suppose that the issue is that the url use some special characters - this is fully possible, for example your username and password, they're supposed to be urlencoded. urlencoding is binary safe, meaning you can have any special characters you'd like, you just need to encode it properly. use urlencode() or http_build_query() for that, eg
$remote_file = "http://somehost:6346/cgi-bin/CGIProxy.fcgi?" . http_build_query ( array (
'cmd' => 'snapPicture2',
'usr' => 'username',
'pwd' => 'password'
) );
now http_build_query will properly urlencode any special characters in your username and password (for example, if your username is an email address, the # becomes %40).
if that doesn't fix it, what does CURLOPT_VERBOSE say?
also, final note, here you're sending the download request with credentials in a GET request. that's very unusual, the vast majority of websites want you to login with a POST request, and there are good security-related reasons for that, are you sure your website allows sending credentials in GET parameters? the vast majority of websites doesn't allow it... (and the best way to find out, is to record a browser logging in, does the browser use GET parameters, or POST parameters?)
I heard that the function readfile is vulnerable as a php function, so I disabled it, and I have a script to download files, which uses this function:
$url = strip_tags($_GET['path']);
$fileName = strip_tags($_GET['file']);
header ("Content-type: octet/stream");
header ("Content-disposition: attachment; filename=".$fileName.";");
header ("Content-Length: ".filesize($url.$fileName));
readfile($url.$fileName);
exit;
If it is vulnerable, is there a way to avoid it or replace it with another function, I tried cURL, but the download failed, I got from this link
Using cURL:
<?php
set_time_limit(0);
$fp = fopen (dirname(__FILE__) . '/localfile.tmp', 'w+');//This is the file where we save the information
$ch = curl_init(str_replace(" ","%20",$url));//Here is the file we are downloading, replace spaces with %20
curl_setopt($ch, CURLOPT_TIMEOUT, 50);
curl_setopt($ch, CURLOPT_FILE, $fp); // write curl response to file
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_exec($ch); // get curl response
curl_close($ch);
fclose($fp);
?>
Thanks in advance
readfile isn't vulnerable per se. A possible attack vector is only given, when you pass un-sanitized data as parameter to it.
I mean $url. Your example doesn't show where URL comes from.
Let's use the worst case and assume $url comes from $_POST, e.g.
$url = $_POST['url']; and combine it with a nice Path Traversal attack (https://www.owasp.org/index.php/Path_Traversal).
You could limit the file access to the current dir (__DIR__).
Or use realpath() and exclude/restrict what readfile can do.
Referencing: How to avoid path traversal attacks
Ok, i've tested you cURL example. It works!
<?php
$url = 'http://www.thetimes.co.uk/tto/news/rss';
set_time_limit(0);
$fp = fopen (dirname(__FILE__) . '/localfile.tmp', 'w+');//This is the file where we save the information
$ch = curl_init(str_replace(" ","%20",$url));//Here is the file we are downloading, replace spaces with %20
curl_setopt($ch, CURLOPT_TIMEOUT, 50);
curl_setopt($ch, CURLOPT_FILE, $fp); // write curl response to file
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_exec($ch); // get curl response
curl_close($ch);
fclose($fp);
This downloads the content of the news/rss feed URL and saves it to localfile.tmp.
trying to copy() .MP3 file from remote url but it always fails.
$link = str_replace(' ','%20','http://mp3hungama.com/music/download.php?song_id=80522');
if (!copy($link,'/home2/muser/tmp/newname.mp3')) {
echo 'copy failed !';
}
$link url redirects to http://mp3hungama.com/music/audio//Indian%20Movies/Indian%20Movies%20Hindi%20Mp3%20Songs/Singh%20Is%20Bling%20(2015)/songs/Cinema%20Dekhe%20Mamma%20#%20Mp3HunGama.Com.mp3
same code works for others random urls like www.example.com/download.php?id=2332. what's the specifically problem here or any other way to do this job ?
I've tested your code and I also couldn't download the file, then, I've used curl an it work as expected:
$local_file = "/home2/muser/tmp/newname.mp3";//This is the file where we save the information
$remote_file = "http://mp3hungama.com/music/download.php?song_id=80522"; //Here is the file we are downloading
$ch = curl_init();
$fp = fopen ($local_file, 'w+');
$ch = curl_init($remote_file);
curl_setopt($ch, CURLOPT_TIMEOUT, 50);
curl_setopt($ch, CURLOPT_FILE, $fp);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_ENCODING, "");
curl_exec($ch);
curl_close($ch);
fclose($fp);
NOTE:
Make sure /home2/muser/tmp/ has write permissions.
TIP:
In the future, if you need to encode/decode a url, use urlencode or urldecode instead of str_replace
This link
already redirects to second link. So it's working already.
I am doing a system where one of my sites goes to the other to get documents.
On the first site I am using Curl to make a request to get the file wanted:
I am using the solution from Download file from URL using CURL :
function collect_file($url){
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_VERBOSE, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_AUTOREFERER, false);
curl_setopt($ch, CURLOPT_REFERER, "http://example.com");
curl_setopt($ch, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_1_1);
curl_setopt($ch, CURLOPT_HEADER, 0);
$result = curl_exec($ch);
curl_close($ch);
echo $result;
return($result);
}
function write_to_file($text,$new_filename){
$fp = fopen($new_filename, 'w');
fwrite($fp, $text);
fclose($fp);
}
$curlUrl = 'http://site2.com/file-depository/14R4NP8JkoIHwIyjnexSUmyJibdpHs5ZpFs3NLFCxcs54kNhHj';
$new_file_name = "testfile-new.png";
$temp_file_contents = collect_file($curlUrl);
write_to_file($temp_file_contents,$new_file_name);
I am testing downloading an image. If i use a direct URL into $curlUrl , for instance http://site2.com/file-depository/image.png it works perfect.
What I am doing is that the URL http://site2.com/file-depository/14R4NP8JkoIHwIyjnexSUmyJibdpHs5ZpFs3NLFCxcs54kNhHj is then parsed and checked against a database to match the document requested, once there is a document matched I need to provide this document to the Curl response.
I have tried many ways to read the file but everytime i am getting a file on the other end but it is only 1kb in size (45 expected) and when trying to open it i get an error unkown file type etc.
On the second site, once the URL is validated here is what I have:
$file = readfile('some-image.png');
echo $file;
I am guessing there is part of the information which belongs to the file missing but can't figure it out, any pointers appreciated!
I have replaced
function write_to_file($text,$new_filename){
$fp = fopen($new_filename, 'w');
fwrite($fp, $text);
fclose($fp);
}
by file_put_contents($new_file_name,trim($temp_file_contents));
Please note the trim(), the issue was that I was apparently collecting some empty space in front of the file content.
I'm trying to download this image with PHP to edit it with GD. I found many solutions for image links, but this one is a download link.
Edit:
$curl = curl_init("http://minecraft.net/skin/Notch.png");
$bin = curl_exec($curl);
curl_close($curl);
$img = #imagecreatefromstring($bin);
This is my current code. It displays "301 Moved Permanently". Are there CURLOPTs I have to set?
$curl = curl_init("http://minecraft.net/skin/Notch.png");
// Moved? Fear not, we'll chase it!
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);
// Because you want the result as a string
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
$bin = curl_exec($curl);
curl_close($curl);
$img = #imagecreatefromstring($bin);
Here is an option to directly save the image to a file (instead of using imagecreatefromstring):
<?php
$fileName = '/some/local/path/image.jpg';
$fileUrl = 'http://remote.server/download/link';
$ch = curl_init($fileUrl); // set the url to open and download
$fp = fopen($fileName, 'wb'); // open the local file pointer to save downloaded image
curl_setopt($ch, CURLOPT_FILE, $fp); // tell curl to save to the file pointer
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true); // tell curl to follow 30x redirects
curl_exec($ch); // fetch the image and save it with curl
curl_close($ch); // close curl
fclose($fp); // close the local file pointer
fopen - depends on your php settings if url fopen is allowed.
or curl
see the fine php manual.