I am using flysystem with IRON IO queue and I am attempting to run a DB query that will be taking ~1.8 million records and while doing 5000 at at time. Here is the error message I am receiving with file sizes of 50+ MB:
PHP Fatal error: Allowed memory size of ########## bytes exhausted
Here are the steps I would like to take:
1) Get the data
2) Turn it into a CSV appropriate string (i.e. implode(',', $dataArray) . "\r\n")
3) Get the file from the server (in this case S3)
4) Read that files' contents and append this new string to it and re-write that content to the S3 file
Here is a brief run down of the code I have:
public function fire($job, $data)
{
// First set the headers and write the initial file to server
$this->filesystem->write($this->filename, implode(',', $this->setHeaders($parameters)) . "\r\n", [
'visibility' => 'public',
'mimetype' => 'text/csv',
]);
// Loop to get new sets of data
$offset = 0;
while ($this->exportResult) {
$this->exportResult = $this->getData($parameters, $offset);
if ($this->exportResult) {
$this->writeToFile($this->exportResult);
$offset += 5000;
}
}
}
private function writeToFile($contentToBeAdded = '')
{
$content = $this->filesystem->read($this->filename);
// Append new data
$content .= $contentToBeAdded;
$this->filesystem->update($this->filename, $content, [
'visibility' => 'public'
]);
}
I'm assuming this is NOT the most efficient? I am going off of these docs:
PHPLeague Flysystem
If anyone can point me in a more appropriate direction, that would be awesome!
Flysystem supports read/write/update stream
Please check latest API https://flysystem.thephpleague.com/api/
$stream = fopen('/path/to/database.backup', 'r+');
$filesystem->writeStream('backups/'.strftime('%G-%m-%d').'.backup', $stream);
// Using write you can also directly set the visibility
$filesystem->writeStream('backups/'.strftime('%G-%m-%d').'.backup', $stream, [
'visibility' => AdapterInterface::VISIBILITY_PRIVATE
]);
if (is_resource($stream)) {
fclose($stream);
}
// Or update a file with stream contents
$filesystem->updateStream('backups/'.strftime('%G-%m-%d').'.backup', $stream);
// Retrieve a read-stream
$stream = $filesystem->readStream('something/is/here.ext');
$contents = stream_get_contents($stream);
fclose($stream);
// Create or overwrite using a stream.
$putStream = tmpfile();
fwrite($putStream, $contents);
rewind($putStream);
$filesystem->putStream('somewhere/here.txt', $putStream);
if (is_resource($putStream)) {
fclose($putStream);
}
If you are working with S3, I would use the AWS SDK for PHP directly to solve this particular problem. Appending to a file is actually very easy using the SDK's S3 streamwrapper, and doesn't force you to read the entire file into memory.
$s3 = \Aws\S3\S3Client::factory($clientConfig);
$s3->registerStreamWrapper();
$appendHandle = fopen("s3://{$bucket}/{$key}", 'a');
fwrite($appendHandle, $data);
fclose($appendHandle);
Hi I am trying to export database table users to a csv file and I am using the following method in my controller:
public function getExport()
{
$table = User::all();
$output='';
foreach ($table as $row) {
$output.= implode(",",$row->toArray());
}
$headers = array(
'Content-Type' => 'text/csv',
'Content-Disposition' => 'attachment; filename="ExportFileName.csv"',
);
return Response::make(rtrim($output, "\n"), 200, $headers);
}
but when I open "ExportFileName.csv" I am getting not Columns Headers and also getting some other garbage in there that are not in the database ?
Whats the best and easy way to do this in Laravel 4
Thanks
You've not added the headers to the CSV. Before outputting the rows, you should do something like:
foreach ($table[0] as $column => $value) {
$output .= "$column,";
}
rtrim($output, ",");
$output .= "\n";
(There are better ways to do that).
You'll also need to account for special characters in the values:
Wrap every value in "" to deal with commas,
Escape and quotation marks, and
Delete or replace any \ns.
I work with yii-powered application. My goal is write controller action what exporting some data from mongodb to csv file using Yii 1.1: csvexport and CHttpRequest::sendFile
My code:
public function actionCatalogDataExport( $catalog_id )
{
// prepare all needed variables here
$data = ..., $headers = ..., $filename = ...
Yii::import('ext.csv.ECSVExport');
$csv = new ECSVExport($data);
$output = $csv->setHeaders($headers)->setDelimiter(',')->toCSV();
Yii::app()->getRequest()->sendFile($filename, $output, "text/csv", true);
}
This script works properly, but if I open resulting file via Excel I see something like that:
There are some problems with file encoding... I opened notepad++ and changed encoding to UTF-8 without BOM, now file looks good (language: Ru):
Tested this fixes but no success results:
header('Content-type: text/csv; charset=UTF-8'); // no effect
Yii::app()->getRequest()->sendFile(
$filename,
$output,
"text/csv; charset=UTF-8", // no effect
true
);
How can I achieve this immediately after yii send file action?
Try to add the encode to begin of the csv file like this:
$encode = "\xEF\xBB\xBF"; // UTF-8 BOM
$content = $encode . $csv->toCSV();
//var_dump($content);
Yii::app()->getRequest()->sendFile($filename, $content, "text/csv; charset=UTF-8", false);
By default setup params, Excel for Windows opens csv files using windows-1251 encoding. If I need to make correct data values using this encoding I must use iconv
foreach( $data as $key => &$value ) {
$value = iconv('UTF-8', 'windows-1251', $value);
}
// send file to user...
// ...and it works as I need.
I am trying to download a rapidshare file using its "download" subroutine as a free user. The following is the code that I use to get response from the subroutine.
function rs_download($params)
{
$url = "http://api.rapidshare.com/cgi-bin/rsapi.cgi?sub=download&fileid=".$params['fileid']."&filename=".$params['filename'];
$reply = #file_get_contents($url);
if(!$reply)
{
return false;
}
$result_arr = array();
$result_keys = array(0=> 'hostname', 1=>'dlauth', 2=>'countdown_time', 3=>'md5hex');
if( preg_match("/DL:(.*)/", $reply, $reply_matches) )
{
$reply_altered = $reply_matches[1];
}
else
{
return false;
}
foreach( explode(',', $reply_altered) as $index => $value )
{
$result_arr[ $result_keys[$index] ] = $value;
}
return $result_arr;
}
For instance; trying to download this...
http://rapidshare.com/files/440817141/AutoRun__live-down.com_Champ.rar
I pass the fileid(440817141) and filename(AutoRun__live-down.com_Champ.rar) to rs_download(...) and I get a response just as rapidshare's api doc says.
The rapidshare api doc (see "sub=download") says call the server hostname with the download authentication string but I couldn't figure out what form the url should take.
Any suggestions?, I tried
$download_url = "http://$the-hostname/$the-dlauth-string/files/$fileid/$filename"
and a couple other variations of the above, nothing worked.
I use curl to download the file, like the following;
$cr = curl_init();
$fp = fopen ("d:/downloaded_files/file1.rar", "w");
// set curl options
$curl_options = array(
CURLOPT_URL => $download_url
,CURLOPT_FILE => $fp
,CURLOPT_HEADER => false
,CURLOPT_CONNECTTIMEOUT => 0
,CURLOPT_FOLLOWLOCATION => true
);
curl_setopt_array($cr, $curl_options);
curl_exec($cr);
curl_close($cr);
fclose($fp);
The above curl code doesn't seem to work, nothing gets downloaded. Probably its the download url that is incorrect.
Also tried this format for the download url:
"http://rs$serverid$shorthost.rapidshare.com/files/$fileid/$filename"
With this curl writes a file entry but that is all it does(writes a 0/1 kb file).
Here is the code that I use to get the serverid, shorthost, among a few other values from rapidshare.
function rs_checkfile($params)
{
$url = "http://api.rapidshare.com/cgi-bin/rsapi.cgi?sub=checkfiles_v1&files=".$params['fileids']."&filenames=".$params['filenames'];
// the response from rapishare would a string something like:
// 440817141,AutoRun__live-down.com_Champ.rar,47768,20,1,l3,0
$reply = #file_get_contents($url);
if(!$reply)
{
return false;
}
$result_arr = array();
$result_keys = array(0=> 'file_id', 1=>'file_name', 2=>'file_size', 3=>'server_id', 4=>'file_status', 5=>'short_host'
, 6=>'md5');
foreach( explode(',', $reply) as $index => $value )
{
$result_arr[ $result_keys[$index] ] = $value;
}
return $result_arr;
}
rs_checkfile(...) takes comma seperated fileids and filenames(no commas if calling for a single file)
Thanks in advance for any suggestions.
You start by requesting ?sub=download&fileid=X&filename=Y, and it returns $hostname,$dlauth,$countdown,$md5hex.. since you're a free user you have to delay for $countdown seconds, and then call ?sub=download&fileid=X&filename=Y&dlauth=Z to perform the download.
There's a working implementation in python here that would probably answer any of your other questions.
I’ve run into a limitation in the cURL bindings for PHP. It appears there is no easy way to send the same multiple values for the same key for postfields. Most of the workarounds I have come across for this have involved creating the URL encoded post fields by hand tag=foo&tag=bar&tag=baz) instead of using the associative array version of CURLOPT_POSTFIELDS.
It seems like a pretty common thing to need to support so I feel like I must have missed something. Is this really the only way to handle multiple values for the same key?
While this workaround might be considered workable (if not really annoying), my main problem is that I need to be able to do multiple values for the same key and also support file upload. As far as I can tell, file upload more or less requires to use the associate arravy version of CURLOPT_POSTFIELDS. So I feel like I am stuck.
I have posted about this problem in more detail on the cURL PHP mailing list in the hopes that someone there has some ideas about this.
Suggestions or hints on where I can look for more information on this are greatly appreciated!
I ended up writing my own function to build a custom CURLOPT_POSTFIELDS string with multipart/form-data. What a pain.
function curl_setopt_custom_postfields($ch, $postfields, $headers = null) {
// $postfields is an assoc array.
// Creates a boundary.
// Reads each postfields, detects which are #files, and which values are arrays
// and dumps them into a new array (not an assoc array) so each key can exist
// multiple times.
// Sets content-length, content-type and sets CURLOPT_POSTFIELDS with the
// generated body.
}
I was able to use this method like this:
curl_setopt_custom_postfields($ch, array(
'file' => '#/path/to/file',
'tag' => array('a', 'b', 'c'),
));
I am not certain of CURLOPT_HTTPHEADER stacks, so since this method calls it, I made certain that the function would allow for the user to specify additonal headers if needed.
I have the full code available in this blog post.
If you use tag[] rather than tag for the name, PHP will generate an array for you, in other words, rather than
tag=foo&tag=bar&tag=baz
You need
tag[]=foo&tag[]=bar&tag[]=baz
Note that when urlencoded for transmission this should become
tag%5B%5D=foo&tag%5B%5D=bar&tag%5B%5D=baz
Vote for PHP Bug #51634.
Try #BeauSimensen's answer.
Guzzle can do this. See an example below.
$client = new \GuzzleHttp\Client();
$client->request('POST', $url, [
'multipart' => [
[ 'name' => 'foo', 'contents' => 'bar' ],
[ 'name' => 'foo', 'contents' => 'baz' ],
]
]);
I ran into the same issue. But I was able to solve it this way.
for($cnt = 0; $cnt < count($siteRows); $cnt++)
{
$curlParams['site_ids['.$cnt.']'] = $siteRows[$cnt]->site_id;
}
Works for files too:
for($cnt = 0; $cnt < count($imageRows); $cnt++)
{
$curlParams['product_images['.$cnt.']'] = '#'.$imageRows[$cnt]->full_path;
}
I got it working using:
curl_setopt($ch, CURLOPT_POSTFIELDS,array('tag[0]'=>'val0','tag[1]'=>'val1'));
then $_POST results in: $_POST['tag'][0] = 'val0' and $_POST['tag'][1] = 'val1'
I think the established standard for multiple values in one key (or the same key) is to have it concatenated with a delimiter, such as for multiple selections of option lists in form elements. I believe this delimiter is the tab character (\t) or the pipe symbol (|).
If the keyname is terminated with [] (like tag[]), PHP will automatically convert the values into an array for your convenience.
lImbus and paul, thank you for your input.
If I had control over the form I am posting to, I could probably find an alternate solution to this problem. However, I do not have any control over the form. And I am almost positive that the software reading the post is not PHP and does not obey the tag[] standards.
Even if it did, cURL does not seem to obey the tag[] syntax either. Basically, I tried the following and neither worked...
curl_setopt($ch, CURLOPT_POSTFIELDS, array('file' => '#/pathtofile', 'tag[]' => array('a', 'b', 'c'));
curl_setopt($ch, CURLOPT_POSTFIELDS, array('file' => '#/pathtofile', 'tag' => array('a', 'b', 'c'));
And again, I don't think that passing tag[] would work anyway as the form I am posting to is actually looking for 'tag' and not 'tag[]'.
I am really starting to get the feeling that the cURL PHP bindings really have no support for this. Which seems so surprising to me. It seems like it can do quite literally anything else, yet it is unable to do something simple like this?
DON'T USE GUZZLE:
# at your command line start php interactive
user#group:~:php -a
php > $arr=array('var' => array(1,2,3,4));
php > echo http_build_query($arr);
var%5B0%5D=1&var%5B1%5D=2&var%5B2%5D=3&var%5B3%5D=4
php > echo urldecode(http_build_query($arr));
var[0]=1&var[1]=2&var[2]=3&var[3]=4
So, you need http_build_query where you pass a hash array of key-values; your (array) variable is entered as a key with value a array instead a scalar value like 'var' => array(1,2,3,4). Now, http_build_query can format the post fields of curl command:
$fields = array('key1' => 'value1', 'var' => array(1,2,3,4));
$curlPost = \http_build_query($fields);
curl_setopt($ch, CURLOPT_POSTFIELDS, $curlPost);
that's 3 lines of code! how many 1000s of code lines are in Guzzle? (*)
So far, I used curl to:
manage Google OAuth protocol with success
connect with APIs like mailgun
handle paypal smart buttons
that's a replacement of million of lines with some 100s!
(*): the result of http_build_query can be formatted further according your needs.
I ran into the same problem in which I had to send a parameter which has to be an array from a PHP server to another server that does not use '[]' for mixing values with the same key along with a file.
In Laravel 8 I could achieve this goal with Http client (of course Http client uses guzzle).
Here is a sample of my code.
Illuminate\Support\Facades\Http::attach('file', $fileContents, 'file-name')
->post('https://destination' , [['name' => 'tag', 'content' => 'foo'], ['name' => 'tag', 'content' => 'bar']])
I found this answer online and want to post it here before it disappears:
http://yeehuichan.wordpress.com/2011/08/07/sending-multiple-values-with-the-same-namekey-in-curl-post/
function curl_setopt_custom_postfields($ch, $postfields, $headers = null) {
$algos = hash_algos();
$hashAlgo = null;
foreach ( array('sha1', 'md5') as $preferred ) {
if ( in_array($preferred, $algos) ) {
$hashAlgo = $preferred;
break;
}
}
if ( $hashAlgo === null ) { list($hashAlgo) = $algos; }
$boundary =
'----------------------------' .
substr(hash($hashAlgo, 'cURL-php-multiple-value-same-key-support' . microtime()), 0, 12);
$body = array();
$crlf = "\r\n";
$fields = array();
foreach ( $postfields as $key => $value ) {
if ( is_array($value) ) {
foreach ( $value as $v ) {
$fields[] = array($key, $v);
}
} else {
$fields[] = array($key, $value);
}
}
foreach ( $fields as $field ) {
list($key, $value) = $field;
if ( strpos($value, '#') === 0 ) {
preg_match('/^#(.*?)$/', $value, $matches);
list($dummy, $filename) = $matches;
$body[] = '--' . $boundary;
$body[] = 'Content-Disposition: form-data; name="' . $key . '"; filename="' . basename($filename) . '"';
$body[] = 'Content-Type: application/octet-stream';
$body[] = '';
$body[] = file_get_contents($filename);
} else {
$body[] = '--' . $boundary;
$body[] = 'Content-Disposition: form-data; name="' . $key . '"';
$body[] = '';
$body[] = $value;
}
}
$body[] = '--' . $boundary . '--';
$body[] = '';
$contentType = 'multipart/form-data; boundary=' . $boundary;
$content = join($crlf, $body);
$contentLength = strlen($content);
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
'Content-Length: ' . $contentLength,
'Expect: 100-continue',
'Content-Type: ' . $contentType,
));
curl_setopt($ch, CURLOPT_POSTFIELDS, $content);
}
And to use it:
curl_setopt_custom_postfields($ch, array(
'file' => '#a.csv',
'name' => array('James', 'Peter', 'Richard'),
));