Memory allocation problems while using a Stream through PHP - php

I'm using a stream in PHP by using the GuzzleHttp\Stream\Stream class. Whilst using it i'm getting PHP memory allocation problems. Is there a method i can use which doesn't use up much memory?
Problem
When i need to serve a Content-Range of 0-381855148 bytes (for example) this causes me a memory allocation issues. Is there a method how can i serve the content while not needing that much memory? Something that passes the data straight through, instead of "reserving" it in memory?
This is part of my code responsible for the error...
$stream = GuzzleHttp\Stream\Stream::factory(fopen($path, 'r'));
$stream->seek($offset);
while (!$stream->eof()) {
echo $stream->read($length);
}
$stream->close();
This is passed as a callback function for my stream.
Background
First i tried fixing the problem by providing a maximum chunk length for my stream. I did this by giving my stream a maximum offset. It's fixes the memory allocation problem, but new problems arise in Firefox when distributing my dynamic video content. Chrome doesn't have problems with it.
It's because Firefox asks for a "0-" Content-Range but i give a Content-Range "0-" back. Instead i need to give back the whole range (until maximum) but this causes the infamous "Allowed memory size of 262144 bytes exhausted (tried to allocate 576 bytes)" error.
Disclaimer: it's actually a little bit more technical. But i wanted to keep it simple.
Does someone knows a solution?
Thanks.

Found my answer on a different forum.
The reason for the memory exhausted problem was because of Guzzle and the way it is build (PSR-7).
A more in in depth article about the problem: https://evertpot.com/psr-7-issues/
I fixed it using this code:
if ($i = ob_get_level()) {
# Clear buffering:
while ($i-- && ob_end_clean());
if (!ob_get_level()) header('Content-Encoding: ');
}
ob_implicit_flush();
$fp = fopen($path, 'rb');
fseek($fp, $offset);
while ($length && !feof($fp)) {
$chunk = min(8192, $length);
echo fread($fp, $chunk);
$length -= $chunk;
}
fclose($fp);
Credits goes to djmaze

Related

Laravel file_get_contents allowed memory exhausted

I am using a console command to download some data locally and than dispatch an update job from that data. The issue I'm having is that the data downloaded is around 65MB for now. The line Storage::disk('local')->put($name, $content); specifically throws a php fatal error: allowed memory size of 134217728 bytes exhausted since I assume the put method creates a copy of $content going beyond 128MB.
Is there a way around this other than setting the memory limit to say 256MB?
Can I store this data in chunks maybe? I am not interested in working on the chunks themselfs. Is there some Laravel method that takes the reference &$contents to store the data?
I would prefer a "Laravel" solution if possible.
$name = basename(config('helper.db_url'));
$content = file_get_contents(config('helper.db_url'));
Storage::disk('local')->put($name, $content);
UpdatePostsTable::dispatch();
Log::info("Downloaded $name");

Memory problems with php://input

I have an API endpoint that can receive a POST JSON/XML body (or even raw binary data) inside the body content as payload that should be written immediately to a file on the filesystem.
For backwards compatibility reasons, it cannot be a multipart/form-data.
It works with no problems for body content up to a certain size (around 2.3GB with a 8GB script memory limit).
I've tried all of the followings:
both with and without setting the buffers' sizes
$filename = '/tmp/test_big_file.bin';
$input = fopen('php://input', 'rb');
$output = fopen($filename, 'wb');
stream_set_read_buffer($input, 4096);
stream_set_write_buffer($output, 4096);
stream_copy_to_stream($input, $output);
fclose($input);
fclose($output);
and
$filename = '/tmp/test_big_file.bin';
file_put_contents($filename, file_get_contents('php://input'));
and
$filename = '/tmp/test_big_file.bin';
$input = fopen('php://input', 'rb');
$output = fopen($filename, 'wb');
while (!feof($input)) {
fwrite($output, fread($input, 8192), 8192);
}
fclose($input);
fclose($output);
Unfortunately, none of them works. At one point, I get always the same error:
PHP Fatal error: Allowed memory size of 8589934592 bytes exhausted (tried to allocate 2475803056 bytes) in Unknown on line 0
Also unsetting enable_post_data_reading makes no difference and all the php.ini post/memory/whatever sizes are set to 8GB.
I'm using php-fpm.
Looking what's happening at the memory with free -mt, I can see that the memory used increases slowly at the beginning, going faster after a while, up to a point that no more free memory is left, so the error.
On the temp directory, the file is not directly stream-copied, but instead it is written on a temporary file named php7NARsX or other random strings which is not deleted after the script crashes, so that at the following free -mt check, the available memory is 2.3GB less.
Now my questions:
Why the stream is not copied directly from php://input to the output instead of loading it into memory? (also using php://temp as output stream leads to the same error)
Why is PHP using so much memory? I'm sending a 3GB payload, so why it needs more than 8GB?
Of course, any working solution will be much appreciated. Thank You!

Allowed memory size exhausted when parsing a PDF with TCPDF

I'm using TCPDF to generate PDFs, as well as parsing PDF in order to count their pages. For one specific file, the TCPDF parser throws an error:
FatalErrorException in tcpdf_filters.php line 357: Allowed memory size
of 536870912 bytes exhausted (tried to allocate 22416744 bytes)
The line mentioned in the error:
public static function decodeFilterFlateDecode($data) {
$decoded = #gzuncompress($data); // <== Breaks here
if ($decoded === false) {
self::Error('decodeFilterFlateDecode: invalid code');
}
return $decoded;
}
The file I'm trying to parse is a normal PDF, 4.55MB, non-password protected. What might be off with it, is that it has internal and external hyperlinks in it.
I've been looking for an answer for some time, and most of the suggestions focus on the memory_limit setting in php.ini. In my case it is set to 512M, so it definitely should be fine with the ~22MB shown in the error message.
Not sure how to debug this further. Any ideas?
UPDATE
After increasing the memory_limit to 1024M, the same memory error is thrown, but in a new place this time (so I guess we're moving forward):
if (preg_match('/^([\r]?[\n])/isU', substr($this->pdfdata, $offset), $matches) == 1) {
$offset += strlen($matches[0]);
if (preg_match('/(endstream)[\x09\x0a\x0c\x0d\x20]/isU', substr($this->pdfdata, $offset), $matches, PREG_OFFSET_CAPTURE) == 1) {
$objval = substr($this->pdfdata, $offset, $matches[0][1]);
$offset += $matches[1][1];
}
}
Breaks at the first if.
I think this is TCPDF being unable to parse this specific file correctly. I will try to locate what is exactly causing this.
The PDF: https://drive.google.com/file/d/0BzFpQ1iB9xTnbllGTE16azR0R0U/view?usp=sharing

NuSOAP varDump PHP Fatal error: Allowed memory size of 134217728 bytes exhausted

Sorry for my english :)
I have NuSOAP version 0.9.5. And I have had an php error when tried to get a big data:
PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 27255652 bytes)
Stack trace shows that problem was in varDump method.
My solution is:
I have changed varDump method (in nusoap.php) to:
function varDump($data) {
$ret_val = "";
if ($this->debugLevel > 0) {
ob_start();
var_dump($data);
$ret_val = ob_get_contents();
ob_end_clean();
}
return $ret_val;
}
and then reset
$GLOBALS['_transient']['static']['nusoap_base']['globalDebugLevel']
to 0 (from 9). In class.nusoap_base.php and nusoap.php.
This helped me.
Does anyone have any comments on this? Or maybe better solution?
Many thanks and respect to Aaron Mingle for the real solution found for NuSOAP out of memory problem. The solution can be found here:
https://sourceforge.net/p/nusoap/discussion/193578/thread/12965595/
I already implemented and immediately tested and I am happy now because it works perfect. In my case I had approx 45 MB SOAP message size (including ~30 pdf files in base64 encoded) and even 2 GB memory for PHP did not helped before. So I have tried Aaron Mingle's solution and it was the good solution with only 384 MB memory granted to PHP.
+1 to Alexey Choporov as well because his suggestion is also required. So both modification is a must have patch in NuSOAP working preperly with larger messages.

Allowed memory size exhausted error exporting from mongodb

I try to export some documents from mongodb to .csv. For some large lists, the files would be something like 40M, I get errors about memory limit:
Fatal error: Allowed memory size of 134217728 bytes exhausted
(tried to allocate 44992513 bytes) in
/usr/share/php/Zend/Controller/Response/Abstract.php on line 586
I wonder why this error happens. What consumes such an amount of memory? How do I avoid such error without changing memory_limit which is set 128M now.
I use something like this:
public static function exportList($listId, $state = self::SUBSCRIBED)
{
$list = new Model_List();
$fieldsInfo = $list->getDescriptionsOfFields($listId);
$headers = array();
$params['list_id'] = $listId;
$mongodbCursor = self::getCursor($params, $fieldsInfo, $headers);
$mongodbCursor->timeout(0);
$fp = fopen('php://output', 'w');
foreach ($mongodbCursor as $subscriber) {
foreach ($fieldsInfo as $fieldInfo) {
$field = ($fieldInfo['constant']) ? $fieldInfo['field_tag'] : $fieldInfo['field_id'];
if (!isset($subscriber->$field)) {
$row[$field] = '';
} elseif (Model_CustomField::isMultivaluedType($fieldInfo['type'])) {
$row[$field] = array();
foreach ($subscriber->$field as $value) {
$row[$field][] = $value;
}
$row[$field] = implode(self::MULTIVALUED_DELEMITOR, $row[$field]);
} else {
$row[$field] = $subscriber->$field;
}
}
fputcsv($fp, $row);
}
}
Then in my controller I try to call it something like this:
public function exportAction()
{
set_time_limit(300);
$this->_helper->layout->disableLayout();
$this->_helper->viewRenderer->setNoRender();
$fileName = $list->list_name . '.csv';
$this->getResponse()->setHeader('Content-Type', 'text/csv; charset=utf-8')
->setHeader('Content-Disposition', 'attachment; filename="'. $fileName . '"');
Model_Subscriber1::exportList($listId);
echo 'Peak memory usage: ', memory_get_peak_usage()/1024, ' Memory usage: ', memory_get_usage()/1024;
}
So I'm at the end of the file where I export data. It's rather strange that for the list I export with something like 1M documents, it exports successfully and displays:
> Peak memory usage: 50034.921875 Kb Memory usage: 45902.546875 Kb
But when I try to export 1.3M documents, then after several minutes I only get in export file:
Fatal error: Allowed memory size of 134217728 bytes exhausted
(tried to allocate 44992513 bytes) in
/usr/share/php/Zend/Controller/Response/Abstract.php on line 586.
The size of documents I export are approximately the same.
I increased memory_limit to 256M and tried to export 1.3M list, this is what it showed:
Peak memory usage: 60330.4609375Kb Memory usage: 56894.421875 Kb.
It seems very confusing to me. Isn't this data so inaccurate? Otherwise, why it causes memory exhausted error with memory_limit set to 128M?
While the size of the documents may be about the same, the size allocated by PHP to process them isn't directly proportional to the document size or number of documents. This is because different types require different memory allocation in PHP. You may be able to free some memory as you go, but I don't see any place where you can in your code.
The best answer is to probably just increase the memory limit.
One thing you could do is offload the processing to an external script and call that from PHP. Many languages do this sort of processing in a more memory efficient way than PHP.
I've also noticed that the memory_get_peak_usage() isn't always accurate. I would try an experiment to increase the mem_limit to say 256 and run it on the larger data set (the 1.3 million). You are likely to find that it reports below the 128 limit as well.
I could reproduce this issue in a similar case of exporting a CSV file, where my system should have had enough memory, as shown by memory_get_usage(), but ended up with the same fatal error:
Fatal error: Allowed memory size.
I circumvented this issue by outputting the CSV contents into a physical temporary file, that I eventually zipped, before reading it out.
I wrote the file in a loop, so that each iteration wrote only a limited chunk of data, so that I never exceded the memory limit.
After zipping, the compression ratio was such, that I could handle raw files of over 10 times the size I initially hit the wall at. All up, it was a success.
Hint: when creating your archive, don't unlink the archive component(s) before invoking $zip->close(), as this call seems to be the one doing the business. Otherwise you'll end up with an empty archive!
Code sample:
<?php
$zip = new ZipArchive;
if ($zip->open($full_zip_path, ZipArchive::CREATE) === TRUE) {
$zip->addFile($full_csv_path, $csv_name);
$zip->close();
$Response->setHeader("Content-type", "application/zip; charset=utf-8");
$Response->setHeader("Content-disposition", "attachment; filename=" . $zip_name);
$Response->setBody(file_get_contents($full_zip_path));
}
else {
var_dump(error_get_last());
echo utf8_decode("Couldn't create zip archive '$full_zip_path'."), "\r\n";
}
unset($zip);
?>
Attention: when adding items to the zip archive, don't prepend a leading slash to the item's name if using Windows based OS.
Discussion over the original issue:
The Zend file at the line quoted is the
public function outputBody()
{
$body = implode('', $this->_body);
echo $body;
}
from the outputBody() method of the Zend_Controller_Response_Abstract class.
It looks like, however you do it, through echo, or print, or readfile, the output is always captured, and stuck into the response body, even if your turn the response return feature off before the dispatch.
I even tried to use the clearBody() class method, within the echo loop, with in mind that each $response->sendResponse() followed by $response->clearBody() would release memory, but it failed.
The way Zend handles the sending of the response is such that I always got the memory allocation of the full size of the raw CSV file.
Yet to be determined how it would be possible to tell Zend not to "capture" the output buffer.

Categories