I use the following curl to multipart post contents of a csv file via curl to another server
I have 1500 records in the csv file and it takes around 5 minutes to post the data ..
Is there a way we can make the post of each record faster .
Will curl multi work here ? or will it just end up posting same content multiple times .
move_uploaded_file($_FILES['file']['tmp_name'], __DIR__.'/uploads/'. $_FILES["image"]['name']);
$file = "uploads/payables.csv";
$authorization = "Authorization: Bearer [some_token_key]";
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, "https://example.org/api/v1/imports.json");
curl_setopt($curl, CURLOPT_POST, 1);
curl_setopt($curl, CURLOPT_HTTPHEADER, [$authorization, 'Content-Type: text/csv']);
$cfile = new CurlFile($file, 'text/csv');
//curl file itself return the realpath with prefix of #
$data = array('data-binary' => $cfile);
curl_setopt($curl, CURLOPT_POSTFIELDS, $data);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
$curl_response = curl_exec($curl);
curl_close($curl);
Using curl_multi_*() is unlikely to help and is only going to be possible if you can shard the data at both ends of the exchange.
Since you've presented no analysis of where the bottleneck is, its impossible to say what will make this exchange go fast. Certainly its unlikely to be latency, but might be bandwidth, might be network configuration, might be capacity at either the client or the server, might be lots of other things or it might be a combination of factors.
You might consider trying to isolate the problem before you start trying to fix it.
If the issue is bandwidth (and some network configuration issues) then compressing the file should make a significant difference. Unfortunately, I don't think libcurl's option for enabling SSL compression is exposed in PHP - so you'd need to inject a transfer-encoding header and compress the file in your code. Compressing a file in PHP is easy, but only if you can load it all in memory. Compressing a stream is a bit more involved.
Related
I am working on a project where I use two separate servers, one for the development and one for the visible version. This is how the process works, and where I'm having troubles: Every morning, I run some VBA macros that collects data, compiles that data (mainly .xlsx files) and sends it to my development server via FTP. The visible server is supposed to use that data to display some informations etc, but FTP is blocked on that server.
Because of that, I have to copy everything from my development server to the visible server every morning, so that the data on the visible server is updated, and I'd like to automatize that.
I tried sending the data from the VBA macros directly to the visible server via HTTP Requests (WinHTTPRequest to be exact) but that didn't work.
I searched online and found that cURL can send HTTP requests through PHP, and i'd like to use that solution if possible, here is my current code:
send.php:
<?php
$request = curl_init('mysite/receive.php');
curl_setopt($request, CURLOPT_POST, true);
curl_setopt($request,
CURLOPT_POSTFIELDS,
array(
'file' => '#MyFileRealPath.xlsx;filename=file'
));
curl_setopt($request, CURLOPT_RETURNTRANSFER, true);
echo curl_exec($request);
curl_close($request);
?>
receive.php:
<?php
var_dump($_FILES);
?>
When I run send.php, I get:
array(0) { }
So the receive.php file does not get any file, does someone know how to fix that?
If what I'm trying to do is not possible, does someone know any other way that I could try to send the files from the development server to the visible one?
Thanks and sorry for my not perfect english, I'm not a native speaker.
Have a nice day!
I had something similar today when our server suddenly did not send a file over curl anymore.
I found that the PHP version was upgraded.
Apparently, in a previous version a new curl setting (CURLOPT_SAFE_UPLOAD) was introduced which disables using # symbol for file uploads, and "PHP 5.6.0 changes the default value to TRUE".
I think you need to add this setting using curl_setopt and disable it, like:
$request = curl_init('mysite/receive.php');
curl_setopt($request, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_SAFE_UPLOAD, false);
curl_setopt($request,
CURLOPT_POSTFIELDS,
array(
'file' => '#MyFileRealPath.xlsx;filename=file'
));
curl_setopt($request, CURLOPT_RETURNTRANSFER, true);
Alternatively, you can use CurlFile:
$request = curl_init('mysite/receive.php');
curl_setopt($request, CURLOPT_POST, true);
curl_setopt($request,
CURLOPT_POSTFIELDS,
array(
'file' => new CurlFile('MyFileRealPath.xlsx', 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet');
));
was searching stackoverflow for a solution, but couldn't find anything even close to what I am trying to achieve. Perhaps I am just blissfully unaware of some magic PHP sauce everyone is doing tackling this problem... ;)
Basically I have an array with give or take a few hundred urls, pointing to different XML files on a remote server. I'm doing some magic file-checking to see if the content of the XML files have changed and if it did, I'll download newer XMLs to my server.
PHP code:
$urls = array(
'http://stackoverflow.com/a-really-nice-file.xml',
'http://stackoverflow.com/another-cool-file2.xml'
);
foreach($urls as $url){
set_time_limit(0);
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_FAILONERROR, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_BINARYTRANSFER, false);
$contents = curl_exec($ch);
curl_close($ch);
file_put_contents($filename, $contents);
}
Now, $filename is set somewhere else and gives each xml it's own ID based on my logic.
So far this script is running OK and does what it should, but it does it terribly slow. I know my server can handle a lot more and I suspect my foreach is slowing down the process.
Is there any way I can speed up the foreach? Currently I am thinking to up the file_put_contents in each foreach loop to 10 or 20, basically cutting my execution time 10- or 20-fold, but can't think of how to approach this the best and most performance kind of way. Any help or pointers on how to proceed?
Your bottleneck (most likely) is your curl requests, you can only write to a file after each request is done, there is no way (in a single script) to speed up that process.
I don't know how it all works but you can execute curl requests in parallel: http://php.net/manual/en/function.curl-multi-exec.php.
Maybe you can fetch the data (if memory is available to store it) and then as they complete fill in the data.
Just run more script. Each script will download some urls.
You can get more information about this pattern here: http://en.wikipedia.org/wiki/Thread_pool_pattern
The more script your run the more parallelism you get
I use on paralel requests guzzle pool ;) ( you can send x paralel request)
http://docs.guzzlephp.org/en/stable/quickstart.html
I've written a script that imports data from an xml file into the mysql database by selecting it from the source disk and uploading it via a button submital. But what if a 3rd party application were to be used to automate this import. Would it be proper to check if a get parameter of a xml path exist and grab its content and import the same way i did before? or is there a better method?
by get parameter i mean like this:
http://domain.com/import.php?path=externaldomain.com/xml/page.xml
it depends on what kind of data you are importing. If you import data from an rss feed, this method is fine. But if you are going to import personal data this might not really be a good method.
I would suggest something more secure if you are working with critical data that others are not supposed to see. You can start thinking of importing the xml files through ftp, download them from behind a server secured folder. Ask the 3rd party application to upload the xml files to a secure location of your choosing. Anything that goes on behind some kind of security is better then the suggested method for personal data.
Firstly I'd advice you using cURL. Doesn't matter how huge is your XML will be, you'll have less problems with memory.
$fp = fopen('/var/www/vhosts/my.com/xml/feed.xml', 'w'); // opening file handler to write feed in
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'http://domain.com/xml/page.xml'); // setting URL to take XML from
curl_setopt($ch, CURLOPT_ENCODING, 'gzip'); // If result is gziped
curl_setopt($ch, CURLOPT_SSLVERSION, 3); // OpenSSL issue
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0); // Wildcard certificate
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 2);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 0); // disabling buffer output, bec. we want to write XML to the file first and don't need it to be returned into variable
curl_setopt($ch, CURLOPT_FILE, $fp); // here we should transfer opened file handler to the cURL and it should be writable!
$result = curl_exec($ch); // executing download
$reponse_code = (int)curl_getinfo($ch, CURLINFO_HTTP_CODE); // retrieving HTTP return code for our request. Was it successful or not.
Thus, you can download/save your XML feed even if it is behind SSL and GZIPed, directly to the file.
Using curl_getinfo() you can get diverse information about your request. If procedure supposed to be automated than it would be nice to decide what to do if request fails.
Than, if file is not large (I mean really large files above 200 - 300 Mb) you can just use SimpleXML (available only since PHP5) library and parse your data. If you are under PHP4 (it is still possible today) try to find libXML which is very useful too.
If file you retrieved is rather huge :) MySQL database with FILE permissions is your friend.
For some reason my curl call is very slow. Here is the code I used.
$postData = "test"
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $postData);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HEADER, false);
$result = curl_exec($ch);
Executing this code takes on average 250ms to finish.
However when I just open the url in a browser, firebug says it only takes about 80ms.
Is there something I am doing wrong? Or is this the overhead associated with PHP Curl.
It's the call to
curl_exec
That is taking up all the time.
UPDATE:
So I figured out right after I posted this that if I set the curl option
curl_setopt($ch, CURLOPT_POSTFIELDS, $postData);
It significantly slows down
curl_exec
The post data could be anything and it will slow it down.
Even if I set
curl_setopt($ch, CURLOPT_POST, false);
It's slow.
I'll try to work around it by just adding the parameters to the URI as a query string.
SECOND UPDATE:
Confirmed that if I just call the URI using GET and passing parameters
as a query string it is much faster than using POST and putting the parameters in the body.
CURL has some problems with DNS look-ups. Try using IP address instead of domain name.
Curl has the ability to tell exactly how long each piece took and where the slowness is (name lookup, connect, transfer time). Use curl_getinfo (http://www.php.net/manual/en/function.curl-getinfo.php) after you run curl_exec.
If curl is slow, it is generally not the PHP code, it's almost always network related.
try this
curl_setopt($ch, CURLOPT_IPRESOLVE, CURL_IPRESOLVE_V4 );
Adding "curl_setopt($ch, CURLOPT_POSTREDIR, CURL_REDIR_POST_ALL);" solved here. Any problem with this solution?
I just resolved this exact problem by removing the following two options:
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $postData);
Somehow on the site I was fetching, the POST request to over ten full seconds. If it's GET, it's less than a second.
So... in my wrapper function that does the Curl requests, it now only sets those two options when there is something in $postData
I just experienced a massive speed-up through compression. By adding the Accept-Encoding header to "gzip, deflate", or just to all formats which Curl supports, my ~200MB download took 6s instead of 20s:
curl_setopt($ch, CURLOPT_ENCODING, '');
Notes:
If an empty string, "", is set, a header containing all supported encoding types is sent.
you do not even have to care about decompression after the download, as this is done by Curl internally.
CURLOPT_ENCODING requires Curl 7.10+
The curl functions in php directly use the curl command line tool under *nix systems.
Therefore it really only depends on the network speed since in general curl itself is much faster than a webbrowser since it (by default) does not load any additional data like included pictures, stylesheets etc. of a website.
It might be possible that you are not aware, that the network performance of the server on which you were testing your php script is way worse than on your local computer where you were testing with the browser. Therefore both measurements are not really comparable.
generally thats acceptable when you are loading contents or posting to slower end of world. curl call are directly proportional to your network speed and throughput of your webserver
There is an epic lack of PHP cURL love on the Internet for beginners like me. I was wondering how to use cURL to download & display an ICS file (They're plain text to me...) in my PHP code. Unless fopen() is 1,000 times easier, I'd like to stick with cURL for this one.
If your webserver allows it, file_get_contents() is even easier.
echo file_get_contents('http://www.example.com/path/to/your/file.ics');
If you can not open URLs with file_get_contents() check out all the stuff on Stack Overflow, which I believe should be fine for a beginner.
If remote file_get_contents is not enabled, cURL can indeed do this.
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, 'http://example.com/file.ics');
// this is the key option - sets curl_exec to return the HTTP response
curl_setopt($curl, CURLOPT_RETURNTRANSFER, TRUE);
$file_contents = curl_exec($curl);