PHP Memory exhausted - php

As I know the solution for this is.
ini_set('memory_limit','-1');
What if even this is not enough.
Problem is I am using a loop and creating and destroying the variables used in loop. But still I have not found the exact reason behind this. that memory utilization after every loop execution increases. my loop is going to run almost 2000 to 10000 times. So even 4gb ram is not going to enough.
As I observed using top commond that memory is using 50mb at the begining of loops, once loop goes on it increases size 10 to 15 mb after every iteration. So my code is not getting executed completely.
ini_set('memory_limit', '-1');
ini_set('xdebug.max_nesting_level', 1000);
$ex_data = some data;
$config = some data;
$docConf = some data;
$codeNameIndex = some data;
$originalName = some data;
CONST LIMIT = 3000;
CONST START = 1000;
//till here it is using 55 to 6o mb memory
for ($i = self::START; $i < (self::START + self::LIMIT); $i++) {
$start_memory = memory_get_usage();
$object = new ImportProjectController();
$object->ex_data = $ex_data;
$object->config = $config;
$object->docConf = $docConf;
$StratProInsertDateTime = microtime(true);
try {
DB::connection()->getPdo()->beginTransaction();
$object->ex_data[$codeNameIndex[2]][$codeNameIndex[1]] = $originalName . '_' . $i;
$object->ex_data[$codeCodeIndex[2]][$codeCodeIndex[1]] = $originalCode . '_' . $i;
if (!$object->insert_project()) {
throw new Exception('error while inserting project');
}
if (!$object->insert_documents()) {
throw new Exception('error while inserting documents');
}
App::make('AccessController')->rebuildCache();
DB::connection()->getPdo()->commit();
} catch (Exception $ex) {
DB::connection()->getPdo()->rollBack();
echo $ex;
}
//it is increasing memory utilization every iteration.
echo "Memory used for inserting a ".$i."th project :- ";
echo memory_get_usage() - $start_memory.PHP_EOL;
unset($object->ex_data);
unset($object->config);
unset($object->docConf);
$object = null;
echo "Memory utilization before inserting project :- ";
echo memory_get_usage() - $start_memory.PHP_EOL;
}
$object->insert_project()
$object->insert_documents()
App::make('AccessController')->rebuildCache()
Methods do some database inserts.
As I am unsetting the $object variable at the end of loop. but still it is not releasing the memory. And I am sure there is nothing that occupying the memory in above method.
Swap: 0k total, 0k used, 0k free, 241560k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
27671 ec2-user 20 0 1489m 1.1g 9908 R 66.0 30.4 8:15.00 php
4307 mysql 20 0 852m 140m 5576 S 18.2 3.7 14:21.50 mysqld
Above is the top commond output, as you can clearly see the memory utilization goes to 1.1gb. and it is increasing..
Plz let me know if need more description.

I got answer from my colleague for this problem.
Laravel do a query logging, and all query keeps it in-memory, Thats why I was getting such issue. With the following code my script is running well with the use of only 250mb of memory. Hope this will help to others.
DB::disableQueryLog();

Related

phpspreadsheet write 10.000 record is too slow

i have a requirement to make report using XLSX file , this report may contains 10.000-1.000.000 rows of trx. i made decision using phpspreadsheet from https://phpspreadsheet.readthedocs.io/en/latest/
The problem is it takes too long to write 10.000 hours of which each record consist of 50 columns. its nearly 24 hours and the script still in running and the progress is 2300/10000, here my codes :
<?php
require 'vendor/autoload.php';
use PhpOffice\PhpSpreadsheet\Spreadsheet;
$client = new \Redis();
$client->connect('192.168.7.147', 6379);
$pool = new \Cache\Adapter\Redis\RedisCachePool($client);
$simpleCache = new \Cache\Bridge\SimpleCache\SimpleCacheBridge($pool);
\PhpOffice\PhpSpreadsheet\Settings::setCache($simpleCache);
$process_time = microtime(true);
if(!file_exists('test.xlsx')) {
$spreadsheet = new Spreadsheet();
$writer = new \PhpOffice\PhpSpreadsheet\Writer\Xlsx($spreadsheet);
$writer->save("test.xlsx");
unset($writer);
}
for($r=1;$r<=10000;$r++) {
$reader = new \PhpOffice\PhpSpreadsheet\Reader\Xlsx();
$spreadsheet = $reader->load("test.xlsx");
$rowArray=[];
for($c=1;$c<=50;$c++) {
$rowArray[]=$r.".Content ".$c;
}
$spreadsheet->getActiveSheet()->fromArray(
$rowArray,
NULL,
'A'.$r
);
$writer = new \PhpOffice\PhpSpreadsheet\Writer\Xlsx($spreadsheet);
$writer->save("test.xlsx");
unset($reader);
unset($writer);
$spreadsheet->disconnectWorksheets();
unset($spreadsheet);
}
$process_time = microtime(true) - $process_time;
echo $process_time."\n";
notes :
i propose CSV file but the clients only wants XLSX
without redis cache it gives memory error even only <400 record
im not intended to read .XLSX using php, only write action. looks like the library reads the entire spreadsheet
at example above it takes open-close file every 1 record file, when im doing open->writeall->close its shows memory error at the mid progress
at example above it takes open-close file every 1 record file, when im
doing open->writeall->close its shows memory error at the mid progress
I see you are opening (createReader) and saving (createWriter) each time when filling content in the loop. It may be the cause of slowing down the process. From your logic, eventually you are writing back the content to the same file, so it can just be open-one-time > write 50x10k records > close-and-save.
A quick test with re-arrange your coding as follows, which result in approximately 25 seconds using my local Xampp in Windows. I'm not sure if this meets your requirement or not, but I think it may consume more time if the content is some long string. My guess is that if you run on a powerful server, the performance might get significant improve in time wise.
$process_time = microtime(true);
$reader = new \PhpOffice\PhpSpreadsheet\Reader\Xlsx();
$spreadsheet = $reader->load($file_loc);
$row_count = 10000;
$col_count = 50;
for ($r = 1; $r <= $row_count; $r++) {
$rowArray = [];
for ($c = 1; $c <= $col_count; $c++) {
$rowArray[] = $r . ".Content " . $c;
}
$spreadsheet->getActiveSheet()->fromArray(
$rowArray,
NULL,
'A' . $r
);
}
$writer = new \PhpOffice\PhpSpreadsheet\Writer\Xlsx($spreadsheet);
$writer->save($target_dir . 'result_' . $file_name);
unset($reader);
unset($writer);
$spreadsheet->disconnectWorksheets();
unset($spreadsheet);
$process_time = microtime(true) - $process_time;
echo $process_time."\n";
Edited:
without redis cache it gives memory error even only <400 record
My quick test is without any cache settings. My guess for the memory issue is that you are opening the XLSX file every time you write the content for each row and then saving it back to the original file.
Every time you open the XLSX file, memory will be loaded and cached with all PhpSpreadsheet object info as well as the [previous content + (50 new columns added each time after saving)] and the memory grows in an exponential way; can you imagine that?
Finally, the memory clearing is way slower and results in memory errors.
1st time open and save
-> open: none
-> save: row A, 50 cols
2nd time open and save
-> open: row A, 50 cols
-> save: row A, 50 cols, row B, 50 cols
3nd time open and save
-> open: row A, 50 cols, row B, 50 cols
-> save: row A, 50 cols, row B, 50 cols, row C, 50 cols
so on and so forth...
memory might still keeping your previous loaded cache
and not releasing so fast (no idea how server is handling the memory, Orz)
and finally memory explode ~ oh no

Caching large Array causes memory exhaustion

So I'm trying to cache an array in a file and use it somewhere else.
import.php
// Above code is to get each line in CSV and put in it in an array
// (1 line is 1 multidimensional array) - $csv
$export = var_export($csv, true);
$content = "<?php \$data=" . $export . ";?>";
$target_path1 = "/var/www/html/Samples/test";
file_put_contents($target_path1 . "recordset.php", $content);
somewhere.php
ini_set('memory_limit','-1');
include_once("/var/www/html/Samples/test/recordset.php");
print_r($data);
Now, I've included recordset.php in somewhere.php to use the array stored in it. It works fine when the uploaded CSV file has 5000 lines, now if i try to upload csv with 50000 lines for example, i'm getting a fatal error:
Fatal error: Allowed memory size of 67108864 bytes exhausted (tried to allocate 79691776 bytes)
How can I fix it or is there a possible way to achieve what i want in a more convenient way? Speaking about the performance... Should i consider the CPU of the server? I've override the memory limit and set it to -1 in somewhere.php
There are 2 ways to fix this:
You need to increase memory(RAM) on the server as memory_limit can only use memory which is available on server. And it seems that you have very low RAM available for PHP.
To Check the total RAM on Linux server:
<?php
$fh = fopen('/proc/meminfo','r');
$mem = 0;
while ($line = fgets($fh)) {
$pieces = array();
if (preg_match('/^MemTotal:\s+(\d+)\skB$/', $line, $pieces)) {
$mem = $pieces[1];
break;
}
}
fclose($fh);
echo "$mem kB RAM found"; ?>
Source: get server ram with php
You should parse your CSV file in chunks & every time release occupied memory using unset function.

PHP MongoDB findOne Memory Leak

I have found some old posts regarding the problem of a memory leak with the php mongodb driver. But none gave a final solution or explenation for older versions of the php mongodb driver.
My Version of the Driver is 1.4.5 (stable)
with PHP 5.3.10
Code with Debug echos:
echo memory_get_usage()." in MB : ";
echo memory_get_usage()/1024/1024;
echo "<br>";
unset($cursor);
$dt = new DateTime($day." 00:00:00", new DateTimeZone($this->timezone));
$mongodate = new MongoDate($dt->getTimestamp());
// print_r($mongodate);
$cursor = $dc->findOne(array('keyword' => $keyword, 'date' => $mongodate));
echo "Cursor loaded Doc (".$cursor['_id'].") : ";
echo memory_get_usage()." in MB : ";
echo memory_get_usage()/1024/1024;
echo "<br>";
** Echos True Memory Usage**
3932160 in MB : 3.75
Cursor geladen Doc (534cdee3c30fd1b8ee0bb641) : 218305980 in MB : 208.1928062439
Code with Debug echos True Memory Usage:
echo memory_get_usage(true)." in MB : ";
echo memory_get_peak_usage(true)/1024/1024;
echo "<br>";
unset($cursor);
$dt = new DateTime($day." 00:00:00", new DateTimeZone($this->timezone));
$mongodate = new MongoDate($dt->getTimestamp());
// print_r($mongodate);
$cursor = $dc->findOne(array('keyword' => $keyword, 'date' => $mongodate));
/*
echo "<pre>";
print_r($cursor);
echo "</pre>";
*/
echo "Cursor loaded Doc (".$cursor['_id'].") : ";
echo memory_get_usage(true)." in MB : ";
echo memory_get_peak_usage(true)/1024/1024;
echo "<br>";
** Echos True Memory Usage**
3932160 in MB : 3.75
Cursor loaded Doc (534cdee3c30fd1b8ee0bb641) : 218628096 in MB : 224.5
So only one Documents causes an encrease by over 200 MB of memory.
bitrs3:PRIMARY> var doc = db.dailies.findOne({"_id" : ObjectId("534cdee3c30fd1b8ee0bb641")})
bitrs3:PRIMARY> Object.bsonsize(doc)
16754823
The document loaded is truely not small, it has 16754823 Bytes so reaches the maximum Bson Size of 16 MB
Still I am wondering if it is normal that the findOne + cursor operation which creates an array out of the results needs so much memory.
You can verify if this is "the cost of doing business with PHP" or if you have found a bug in the driver by serializing the array (using serialize() or even json_encode()) and save it to a file.
Then you unserialize() (or json_decode()) the content of the file and check the memory usage.
If the memory usage is similar you see the actual overhead of PHP typing.
<?php
$mc = new MongoClient;
$collection = $mc->selectCollection("myDB", "myCollection");
$d = $collection->findOne(array("my" => "criteria"));
var_dump(memory_get_usage() / 1024 / 1024);
file_put_contents("serialized.bin", serialize($d));
?>
Then loading it again:
<?php
$val = unserialize(file_get_contents("serialized.bin"));
var_dump(memory_get_usage() / 1024 / 1024);
?>
EDIT: To preemptively clarify any misunderstanding.
A 16Mb MongoDB document does generally not need hundreds of megabytes of memory.
If you however have hundreds of thousands of elements inside that object, then individual element overhead starts to count times every element you have.

PHP Download file, limit max speed and calculate downloading speed

I have written a script that gives you ability to download the file with my maximum file speed that I allow, however when I allow 'unlimited' speed like 10000kB/s then the ftell works strange, it behaves like it downloads with 10000kBps speed, which is not true and I can not make calculations in database like time remaining, current download speed and so on...
So browser downloads file after some time, but in database it is already like 'downloaded', how could I make some precision calculations even I set the unlimited speed so user can download a file at the speed of the network and the database values are also counted by his network speed not by the ftell(); which depends on $download_rate; ...?
Thanks in advance!
<?php
while(!feof($fopen)) {
//echo fread($fopen, 4096);
$this->get_allowed_speed_limit($download_rate);
//$download_rate = 350;
print fread($fopen, round($download_rate * 1024));
sleep(1); //needed for download speed limit
if(connection_status() != 0 || connection_aborted()) {
$bytes_transferred = ftell($fopen);
if($bytes_transferred < $bytes) {
//CANCELLED
$this->download_unsuccessfull($file_name);
} else {
//CANCELLED (but gets executed only on strange networks like eduroam in CZE)
$this->download_unsuccessfull($file_name);}
flush();
die;
} else {
$progress = ftell($fopen) / $bytes * 100;
if($progress >= 100) {
//DONE
$this->download_successfull($file_name);
flush();
} else {
//DOWNLOADING
if(ftell($fopen) != 0) {
$bytes_transferred = ftell($fopen);
$time_end = microtime(true);
$time = $time_end - $time_start;
$dl_speed = floor(($bytes_transferred / $time) / 1000);
///////HERE THE CALCULATIONS ARE TOTALLY WRONG, BECAUSE IT ALL DEPENDS ON THE INPUT OF $download_rate;
mysqli_query($con, "UPDATE `download_meter` SET `current_speed` = '".mysqli_real_escape_string($con, $bytes_transferred)."'");
$this->update_active_downloads($file_name, $bytes_transferred, $dl_speed);
}
flush();
}
}
//Activate this for delay download.
//flush();
//sleep(1);
}
?>
Limiting download speed is up to your webserver. PHP is too high level. It knows nothing of the outgoing data.
Apache: https://stackoverflow.com/a/13355834/247372
Nginx: http://www.nginxtips.com/how-to-limit-nginx-download-speed/
The same goes for measuring: the webserver will know and might tell you somehow. Logs, unix socket, after-the-fact, I don't know. Those links will know.
How about (re)adding that sleep(1); thing to the WHILE loop? From what I can see the script outputs the file almost all at once (as fast as it can) and there's nothing that pauses it so it can actually limit the download speed.
That way you will know that each second you send just 64kbytes (or whatever) and even though you can't be sure that the user can in fact recieve this much data/second (whoa, so fast!), it could be a bit more precise than what you have in there now.
Or am I getting this wrong?

PHP Memory Debugging

For one off my projects I need to import a very huge text file ( ~ 950MB ). I'm using Symfony2 & Doctrine 2 for my project.
My problem is that I get errors like:
Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 24 bytes)
The error even occurs if I increase the memory limit to 1GB.
I tried to analyze the problem by using XDebug and KCacheGrind ( as part of PHPEdit ), but I don't really understand the values :(
I'am looking for a tool or a method (Quick & Simple due to the fact that I don't have much time) to find out why memory is allocated and not freed again.
Edit
To clear some things up here is my code:
$handle = fopen($geonameBasePath . 'allCountries.txt','r');
$i = 0;
$batchSize = 100;
if($handle) {
while (($buffer = fgets($handle,16384)) !== false) {
if( $buffer[0] == '#') //skip comments
continue;
//split parts
$parts = explode("\t",$buffer);
if( $parts[6] != 'P')
continue;
if( $i%$batchSize == 0 ) {
echo 'Flush & Clear' . PHP_EOL;
$em->flush();
$em->clear();
}
$entity = $em->getRepository('MyApplicationBundle:City')->findOneByGeonameId( $parts[0] );
if( $entity !== null) {
$i++;
continue;
}
//create city object
$city = new City();
$city->setGeonameId( $parts[0] );
$city->setName( $parts[1] );
$city->setInternationalName( $parts[2] );
$city->setLatitude($parts[4] );
$city->setLongitude( $parts[5] );
$city->setCountry( $em->getRepository('MyApplicationBundle:Country')->findOneByIsoCode( $parts[8] ) );
$em->persist($city);
unset($city);
unset($entity);
unset($parts);
unset($buffer);
echo $i . PHP_EOL;
$i++;
}
}
fclose($handle);
Things I have tried, but nothing helped:
Adding second parameter to fgets
Increasing memory_limit
Unsetting vars
Increasing memory limit is not going to be enough. When importing files like that, you buffer the reading.
$f = fopen('yourfile');
while ($data = fread($f, '4096') != 0) {
// Do your stuff using the read $data
}
fclose($f);
Update :
When working with an ORM, you have to understand that nothing is actually inserted in the database until the flush call. Meaning all those objects are stored by the ORM tagged as "to be inserted". Only when the flush call is made, the ORM will check the collection and start inserting.
Solution 1 : Flush often. And clear.
Solution 2 : Don't use the ORM. Go for plain SQL command. They will take up far less memory than the object + ORM solution.
33554432 are 32MB
change memory limit in php.ini for example 75MB
memory_limit = 75M
and restart server
Instead of simply reading the file, you should read the file line by line. Every time you do read the one line you should process your data. Do NOT try to fit EVERYTHING in memory. You will fail. The reason for that is that while you can put the TEXT file in ram, you will not be able to also have the data as php objects/variables/whathaveyou at the same time, since php by itself needs much larger amounts of memory for each of them.
What I instead suggest is
a) read a new line,
b) parse the data in the line
c) create the new object to store in the database
d) goto step a, by unset(ting) the old object first or reusing it's memory

Categories