Curl chunked uploads with PHP - php

I have written a PHP script that is using CURL to transfer large files from one Windows server to another. Both are running IIS, which has a 2GB limit for file uploads. My process works great for files less than 2GB. However, it fails for files > 2GB.
I have heard that CURL may have the ability to send files in chunks, which may solve my problem. However, I can't find any good examples of how to do this. My current working code is posted below. Does anyone know how to modify it to send my files in chunks (i.e. 100MB chunks)?
$postVars = array("file1" => $fileContents1,
"file2" => $fileContents2,
"outputPath" => $outputPath);
// Initializing curl
$ch = curl_init();
// Configuring curl options
curl_setopt($ch, CURLOPT_URL,$url);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_USERPWD, $username . ":" . $password);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($ch, CURLOPT_VERBOSE, false);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_FORBID_REUSE, true);
curl_setopt($ch, CURLOPT_FRESH_CONNECT, true);
curl_setopt($ch, CURLOPT_BUFFERSIZE, '64000');
curl_setopt($ch, CURLOPT_BINARYTRANSFER, true);
curl_setopt($ch, CURLOPT_SSLVERSION, '3');
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-type: multipart/form-data'));
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)");
curl_setopt($ch, CURLOPT_POSTFIELDS,$postVars);
echo "Sending files...\n";
curl_setopt($ch, CURLOPT_TIMEOUT, 9999);
if(curl_exec($ch) === false)
{
echo 'Curl error: ' . curl_error($ch);
}
else
{
echo 'Operation completed without any errors';
}
$info = curl_getinfo($ch);
print_r($info);
curl_close($ch);

plupload is a javascript/php library, and it's quite easy to use and allows chunking of your desired size.
It uses HTML5 though. I found only the way to uploading a large file is PlUpload Library and make the FileManager Application. Hope it will help you also.

Related

Sending a file via PHP CURL to remote server

I'm trying to send a file using PHP's CURL functions.
$postdata = curl_file_create(realpath($filename), 'text/csv', $filename);
and then
$ch = curl_init();
curl_setopt($ch, CURLOPT_POST, TRUE);
curl_setopt($ch, CURLOPT_POSTFIELDS, $postdata);
curl_setopt($ch, CURLOPT_URL,$url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_TIMEOUT, 30);
curl_setopt($ch, CURLOPT_HTTPAUTH, CURLAUTH_BASIC);
curl_setopt($ch, CURLOPT_USERPWD, "$login:$password");
curl_setopt($ch, CURLOPT_VERBOSE, TRUE);
$result = curl_exec($ch);
$cinfo = curl_getinfo($ch);
$cerror = curl_error($ch);
var_dump($result);
var_dump($cinfo);
var_dump($cerror);
curl_close($ch);
This only dumps bool(false), so I'm not getting far with the debugging. Am I POSTing the filename as a string rather than the file contents? The php script and the file to be send are in the same directory. Do I need to specify the path in a certain format?
Update:
I added curl_error($ch) and figured out that the problem is a connection timeout. When running the script from my local machine it seems to be working, at least I get a 200 OK reply. Could my webhost somehow be blocking the connection?

Upload the file to the FTP server over HTTPS using curl in php

I am trying to upload the data [can be text or zip] to the ftp site.
As we have proxy in place in the environment so, I decided to upload the data using curl. Before I go head to set proxy server setting, I was testing the script on the environment without proxy.
I followed :-
FTP upload file to distant server with CURL and PHP uploads a blank file
However, I couldn't able to upload a file.
link where, I would like to upload is this format:-
https://fp.emc.com/.....
Do any one know, how to upload a file to ftp server over https using curl function of PHP?
<?php
$sendTo = 'https://ftp.emc.com/....?domain=XX&user=XXX&password=XXX';
$localfile ="23.txt";
$fp = fopen($localfile, 'r');
// Create CURL Connection
$ch = curl_init();
//curl_setopt($ch, CURLOPT_PROTOCOLS, CURLPROTO_HTTPS);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 2);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_USERPWD, "usr:passwd");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt ($ch, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)");
curl_setopt($ch, CURLOPT_FAILONERROR, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_URL, $sendTo);
//curl_setopt($ch, CURLOPT_HTTPHEADER, $header);
curl_setopt($ch, CURLOPT_UPLOAD, 1);
curl_setopt($ch, CURLOPT_INFILE, $fp);
curl_setopt($ch, CURLOPT_INFILESIZE, filesize($localfile));
echo $ch;
$m=curl_exec ($ch);
echo "m $m<br>";
$error_no = curl_errno($ch);
echo "error_no $error_no<br>";
curl_close ($ch);
if ($error_no == 0) {
$error = 'File uploaded succesfully.';
} else {
$error = 'File upload error.';
}
echo $error;
?>
Try this:
$fp = fopen($filepath, 'r');
$ftp_url = "ftp://user:password#ftpserver:21/" . $filename;
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $ftp_url);
curl_setopt($ch, CURLOPT_UPLOAD, 1);
curl_setopt($ch, CURLOPT_INFILE, $fp);
curl_setopt($ch, CURLOPT_INFILESIZE, filesize($filepath));
curl_setopt($ch, CURLOPT_PROXY, $proxy_server_ip);
curl_setopt($ch, CURLOPT_PROXYPORT, $proxy_server_port);
curl_setopt($ch, CURLOPT_PROXYTYPE, 'HTTP');
curl_setopt($ch, CURLOPT_PROXYUSERPWD, $proxy_login);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_exec($ch);
curl_close($ch);
It works for me.

Trello API PHP Curl

I am trying to integrate/replace my web edit queue into Trello.
I made an organization which is not public but have created a token for read/write access.
I haven't seen a good PHP wrapper for the Trello API (have looked at the two available and couldn't really get them up and running for my purposes.)
Anyway, what I'd like to do is provide rather rudimentary access to read and insert cards to a particular list.
I have gotten as far as using the API to return the results of a List using the following:
https://api.trello.com/1/lists/[mylistID]/cards?key=[myappkey]&token=[mytoken]
I get exactly what I want as a result, json of the cards in the list.
Now I;m trying to recreate that in PHP using CURL and I'm getting an error response of unauthorized or bad request from whatever I try in the following code:
$url = "https://api.trello.com/1/lists/[myboardID]/cards";
$trello_key = 'mykey';
$trello_list_id = 'mylistid';
$trello_member_token = 'mytoken';
$fields = "key=$trello_key&token=$trello_member_token&name=$name&idList=$trello_list_id";
e
# init curl
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_POSTFIELDS, $fields);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)");
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLINFO_HEADER_OUT, TRUE); // make sure we see the sended header afterwards
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_TIMEOUT, 0);
curl_setopt($ch, CURLOPT_POST, 1);
# dont care about ssl
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
# download and close
$output = curl_exec($ch);
$request = curl_getinfo($ch, CURLINFO_HEADER_OUT);
$error = curl_error($ch);
curl_close($ch);
So I'm just looking to see if anyone knows what I'm doing wrong. I feel like it should be simple but I've spent a couple hours on it and I think I need some help. Let me know if you have any ideas.
{i have left out obvious references to my API key, token, BoardID etc}
This actually seems to work for me. I was trying to use POST instead of the default GET with CURL. Still working on parsing the response but it seems like I'm on the right track. Got an "200 OK" in the response.
$url = 'https://api.trello.com/1/lists/[myListID]/cards?key=[MyApiKey]&token=[myToken]';
# init curl
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
//curl_setopt($ch, CURLOPT_POSTFIELDS, $encoded_fields);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)");
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLINFO_HEADER_OUT, TRUE); // make sure we see the sended header afterwards
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_TIMEOUT, 0);
//curl_setopt($ch, CURLOPT_POST, 1);
# dont care about ssl
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
# download and close
$output = curl_exec($ch);
$request = curl_getinfo($ch, CURLINFO_HEADER_OUT);
$error = curl_error($ch);
curl_close($ch);
echo 'This is output = '.$output .'<br />';
echo 'This is request = '.$request .'<br />';
echo 'This is error = '.$error .'<br />';
$url = 'https://api.trello.com/1/boards/[myListID]/cards?key=[MyApiKey]&token=[myToken]';
# init curl
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
//curl_setopt($ch, CURLOPT_POSTFIELDS, $encoded_fields);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)");
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLINFO_HEADER_OUT, TRUE); // make sure we see the sended header afterwards
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_TIMEOUT, 0);
//curl_setopt($ch, CURLOPT_POST, 1);
# dont care about ssl
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
# download and close
$output = curl_exec($ch);
$request = curl_getinfo($ch, CURLINFO_HEADER_OUT);
$error = curl_error($ch);
curl_close($ch);
echo 'This is output = '.$output .'<br />';
echo 'This is request = '.$request .'<br />';
echo 'This is error = '.$error .'<br />';

curl through proxy returns no content

I'm working on a PHP script right now which sends requests to our school's servers to get real-time information about class sizes for different courses. The script runs perfectly fine when I don't use a proxy, returning a string full of course numbers and available seats. However, I want to make this a service for the students, and I'm afraid that if I make too many requests my ip will get blocked. So I'm attempting to do this through a proxy, with no success. As soon as I add the CURLOPT_HTTPPROXYTUNNEL and CURLOPT_PROXY fields to my requests nothing gets returned. I'm not even sure how to troubleshoot it at this point since I'm not getting an error message of any kind. Does anyone know what's going on or how to at least troubleshoot it?
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$proxy = explode(':', $proxy);
curl_setopt($ch, CURLOPT_PROXY, $proxy[0]);
curl_setopt($ch, CURLOPT_PROXYPORT, $proxy[1]);
curl_setopt($ch, CURLOPT_HTTPPROXYTUNNEL, 1);
curl_setopt($ch, CURLOPT_COOKIEFILE, 'tempcookie.txt');
curl_setopt($ch, CURLOPT_COOKIEJAR, 'tempcookie.txt');
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_REFERER, $ref);
$exec = curl_exec($ch);
echo curl_error($ch);
print_r(curl_getinfo($ch));
echo $exec;
Proxy used for tests: 75.147.173.215:8080
I use the following code if I have to use a proxy with curl:
$proxy = "127.0.0.1:8080"; // or something like that
if($proxy !== null){
// no need to specify PROXYPORT again
curl_setopt($ch, CURLOPT_PROXY, $proxy);
// to make the request go through as though proxy didn't exist
curl_setopt($ch, CURLOPT_HTTPPROXYTUNNEL, 0);
}
You can set CURLOPT_STDERR and CURLOPT_VERBOSE curl options to save your errors in a file. Also, you may use curl_error() function. BTW, by default, curl should show all errors in STDERR.
Besides, for general check, you can simply specify selected proxy in you browser configuration properties, try to open specific service in browser and see, whether correct response is returned.
UPDATE:
CURLOPT_HTTPPROXYTUNNEL is used to make curl call CONNECT HTTP method when requesting proxy server (see here for details). I tested code without this option - it worked successfully.
Code I used:
$proxy = "75.147.173.215:8080";
$proxy = explode(':', $proxy);
$url = "http://google.com";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_PROXY, $proxy[0]);
curl_setopt($ch, CURLOPT_PROXYPORT, $proxy[1]);
curl_setopt($ch, CURLOPT_HEADER, 1);
$exec = curl_exec($ch);
echo curl_error($ch);
print_r(curl_getinfo($ch));
echo $exec;
Here is a well tested function which i used for my projects with detailed self explanatory comments
There are many times when the ports other than 80 are blocked by server firewall so the code appears to be working fine on localhost but not on the server , try using port 80 proxies
function get_page($url){
global $proxy;
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
//curl_setopt($ch, CURLOPT_PROXY, $proxy);
curl_setopt($ch, CURLOPT_HEADER, 0); // return headers 0 no 1 yes
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); // return page 1:yes
curl_setopt($ch, CURLOPT_TIMEOUT, 200); // http request timeout 20 seconds
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true); // Follow redirects, need this if the url changes
curl_setopt($ch, CURLOPT_MAXREDIRS, 2); //if http server gives redirection responce
curl_setopt($ch, CURLOPT_USERAGENT,
"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.7) Gecko/20070914 Firefox/2.0.0.7");
curl_setopt($ch, CURLOPT_COOKIEJAR, "cookies.txt"); // cookies storage / here the changes have been made
curl_setopt($ch, CURLOPT_COOKIEFILE, "cookies.txt");
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); // false for https
curl_setopt($ch, CURLOPT_ENCODING, "gzip"); // the page encoding
$data = curl_exec($ch); // execute the http request
curl_close($ch); // close the connection
return $data;
}

Unable to set CURLOPT_NOBODY without crashing curl_exec

I've been killing myself all day for this one bug. I can't tell you enough how much I'd appreciate any possible help on this.
Basically, I have a very simple script. It logs into a website, looks at a file's header to see if it is an image type and then it downloads it. It then repeats this three times.
The problem here is that I cannot set CURLOPT_NOBODY without curl_exec crashing the entire script with -no- errors. (I can't even call or get an curl_error!)It would seem that it is impossible for me to go from CURLOPT_NOBODY, true to CURLOPT_NOBODY, false. The loop below runs one time and then dies().
What could possibly be causing this bug?
Here is the script:
// Log into the Website
curl_setopt($ch, CURLOPT_URL, 'http://myexample.com/login');
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.0.12) Gecko/2009070611 Firefox/3.0.12");
curl_setopt($ch, CURLOPT_POSTFIELDS, $post_fields);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_COOKIEJAR, "cookie.txt");
curl_setopt($ch, CURLOPT_COOKIEFILE, "cookie.txt");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 5);
curl_exec($ch);
// Begin the Loop for Finding Images
for($i = 0; $i < 3; $i++) {
curl_setopt($ch, CURLOPT_URL, 'http://myexample.com/file.php?id=' . $i);
curl_setopt($ch, CURLOPT_NOBODY, true);
curl_setopt($ch, CURLOPT_HEADER, true);
$output = curl_exec($ch) or die('WHY DOES THIS DIE!!!');
$curl_info = curl_getinfo($ch);
echo '<br/>' . $output;
// (Normally checks for content type here) Download the File
curl_setopt($ch, CURLOPT_URL, 'http://myexample.com/file.php?id=' . $i);
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_NOBODY, false);
$filename = 'downloads/test-' . $i . '.jpg';
$fp = fopen($filename, 'w');
curl_setopt($ch, CURLOPT_FILE, $fp);
curl_exec($ch);
fclose($fp);
}
I'm running apache 2.2 and PHP version 5.2.13.
Thanks for any help- I can't tell you how much I'd appreciate it. I'm completely stuck here. :(
Looks to me like the curl library is getting confused, esp when you are reusing the resource.
you should do
$ch = curl_init();
// do stuff with curl
curl_close($ch);
$ch = curl_init();
// another curl call
curl_close($ch);
$ch = curl_init();
// yet another curl call
curl_close($ch);
I was given the same errors executing your script, but adding in the curl_close's and the curl_init to reinitialize, seem to fix the problem. I don't know if this is acceptable, if not. I'd use the fopen() to do your http downloading, its much more intuitive than using curl, unless you need something that isn't supported in fopen.

Categories