I have this PHP code below to generate a set of 12 digit unique random numbers(ranging from 100000 to a million) and save it in db. I am first fetching the existing codes from MySQL db(right now there are already a million of them), flipping the array, generating a new codes. Later I use array_diff_key and array_keys on $random and $existingRandom to get the new codes which are to be saved back to db.
// total labels is the number of new codes to generate
//$totalLabels is fetched as user input and could range from 100000 to a million
$codeObject = new Codes();
//fetch existing codes from db
$codesfromDB = $codeObject->getAllCodes();
$existingRandom = $random = array();
$existingRandom = $random = array_flip($codesfromDB);
$existingCount = count($random); //The codes you already have
do {
$random[mt_rand(100000000000,999999999999)] = 1;
} while ((count($random)-$existingCount) < $totalLabels);
$newCodes = array_diff_key($random,$existingRandom);
$newCodes = array_keys($newCodes);
The issue I am facing is that the array_flip function is running out of memory and causing my program to crash Error
"Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 72 bytes)"
My questions are below:
1) Can someone help me understand why the array_flip is running out of memory. Memory limit in php.ini file is 256M. Please show me calculation of the memory used by the function if possible. (Also if array_flip passes array_diff_key and array_keys run out of memory)
2) How do I optimize the code so that the memory used is under the limit. I even tried to break the array_flip operation in smaller chunks but even that is running out of memory.
$size = 5000;
$array_chunk = array_chunk($codesfromDB, $size);
foreach($array_chunk as $values){
$existingRandom[] = $random[] = array_flip($values);
}
3) Is what I am doing optimal would it be fair to further increase the memory limit in php.ini file. What are the things to keep in mind while doing that.
Here is my query as well to fetch the existing codes from db if needed:
$sql = "SELECT codes FROM code";
$stmt = $this->db->prepare($sql);
$stmt->execute();
$result = $stmt->fetchAll(PDO::FETCH_COLUMN, 0);
return $result;
Related
I wrote a one-off script that I use to parse PDFs saved on the database. So far it is working okay until I ran out of memory after parsing 2,700+ documents.
The basic flow of the script is as follows:
Get a list of all the document IDs to be parsed and save it as an array in the session (~155k documents).
Display a page that has a button to start parsing
Make an AJAX request when that button is clicked that would parse the first 50 documents in the session array
$files = $_SESSION['files'];
$ids = array();
$slice = array_slice($files, 0, 50);
$files = array_slice($files, 50, null); // remove the 50 we are parsing on this request
if(session_status() == PHP_SESSION_NONE) {
session_start();
}
$_SESSION['files'] = $files;
session_write_close();
for($i = 0; $i < count($slice); $i++) {
$ids[] = ":id_{$i}";
}
$ids = implode(", ", $ids);
$sql = "SELECT d.id, d.filename, d.doc_content
FROM proj_docs d
WHERE d.id IN ({$ids})";
$stmt = oci_parse($objConn, $sql);
for($i = 0; $i < count($slice); $i++) {
oci_bind_by_name($stmt, ":id_{$i}", $slice[$i]);
}
oci_execute($stmt, OCI_DEFAULT);
$cnt = oci_fetch_all($stmt, $data);
oci_free_statement($stmt);
# Do the parsing..
# Output a table row..
The response to the AJAX request typically includes a status whether the script has finished parsing the total ~155k documents - if it's not done, another AJAX request is made to parse the next 50. There's a 5 second delay between each request.
Questions
Why am I running out of memory when I was expecting that peak memory usage would be when I get a list of all the document IDs on #1 since it holds all of the possible documents not a few minutes later when the session array holds 2,700 elements less?
I saw a few questions similar to my problem and they suggested to either set the memory to unlimited which I don't want to do at all. The others suggested to set my variables to null when appropriate and I did that but I still ran out of memory after parsing ~2,700 documents. So what other approaches should I try?
# Freeing some memory space
$batch_size = null;
$with_xfa = null;
$non_xfa = null;
$total = null;
$files = null;
$ids = null;
$slice = null;
$sql = null;
$stmt = null;
$objConn = null;
$i = null;
$data = null;
$cnt = null;
$display_class = null;
$display = null;
$even = null;
$tr_class = null;
So I'm not really sure why but reducing the number of documents I'm parsing from 50 down to 10 for each batch seems to fix the issue. I've gone past 5,000 documents now and the script is still running. My only guess is that when I was parsing 50 documents I must have encountered a lot of large files which used up all of the memory allotted.
Update #1
I got another error about memory running out at 8,500+ documents. I've reduced the batches further down to 5 documents each and will see tomorrow if it goes all the way to parsing everything. If that fails, I'll just increase the memory allocated temporarily.
Update #2
So it turns out that the only reason why I'm running out of memory is that we apparently have multiple PDF files that are over 300MB uploaded on to the database. I increased the memory allotted to PHP to 512MB and this seems to have allowed me to finish parsing everything.
I am using Yii 1.1.14 with php 5.3 on centos 6 and I am using CDbCommand to fetch data from a very large table, the result set is ~90,000 records over 10 columns I am exporting it to a csv file and the file size is about 15MB,
the script always crashed without any error messages and only after some research I figured out that I need to raise the memory_limit in php.ini in order to be able to execute the script successfully.
The only problem is that for a successful execution I had to raise the memory limit to 512MB(!) which is a lot! and if 10 users will be executing the same script my server will not respond very well...
I was wondering if anyone might know of a way to reduce memory consumption on sql queries with Yii?
I know I can split the query to multiple queries using limits and offsets, but it just doesn't seem logical that a 15MB query will consume 512MB.
Here is the code:
set_time_limit(0);
$connection = new CDbConnection($dsn,$username,$password);
$command = $connection->createCommand('SELECT * FROM TEST_DATA');
$result = $command->queryAll(); //this is where the script crashes
print_r($result);
Any ideas would be greatly appreciated!
Thanks,
Instead of using readAll that will returns all the rows in a single array (the real memory problem is here), you should simply use a foreach loop (take a look at CDbDataReader), e.g. :
$command = $connection->createCommand('SELECT * FROM TEST_DATA');
$rows = $command->query();
foreach ($rows as $row)
{
}
EDIT : Using LIMIT
$count = Yii::app()->db->createCommand('SELECT COUNT(*) FROM TEST_DATA')->queryScalar();
$maxRows = 1000:
$maxPages = ceil($count / $maxRows);
for ($i=0;$i<$maxPages;$i++)
{
$offset = $i * $maxRows;
$rows = $connection->createCommand("SELECT * FROM TEST_DATA LIMIT $offset,$maxRows")->query();
foreach ($rows as $row)
{
// Here your code
}
}
I have a big table in my MySQL database. I want to go over one of it's column and pass it in a function to see if it exist in another table and if not create it there.
However, I always face either a memory exhausted or execution time error.
//Get my table
$records = DB::($table)->get();
//Check to see if it's fit my condition
foreach($records as $record){
Check_for_criteria($record['columnB']);
}
However, when I do that, I get a memory exhausted error.
So I tried with a for statement
//Get min and max id
$min = \DB::table($table)->min('id');
$max = \DB::table($table)->max('id');
//for loop to avoid memory problem
for($i = $min; $i<=$max; $i++){
$record = \DB::table($table)->where('id',$i)->first();
//To convert in array for the purpose of the check_for_criteria function
$record= get_object_vars($record);
Check_for_criteria($record['columnB']);
}
But going this way, I got a maximum execution time error.
FYI the check_for_criteria function is something like
check_for_criteria($record){
$user = User::where('record', $record)->first();
if(is_null($user)){
$nuser = new User;
$nuser->number = $record;
$nuser->save();
}
}
I know I could ini_set('memory_limit', -1); but I would rather find a way to limit my memory usage in some way or at least spreading it some way.
Should I run these operations in background when traffic is low? Any other suggestion?
I solved my problem by limiting my request to distinct values in ColumnB.
//Get my table
$records = DB::($table)->distinct()->select('ColumnB')->get();
//Check to see if it's fit my condition
foreach($records as $record){
Check_for_criteria($record['columnB']);
}
I made a method on my class to fetch and store in a array all the results the desired SQL statement has in it, and it works just fine. Now, after some months in production, I came across this error:
Fatal error: Allowed memory size of 67108864 bytes exhausted (tried to allocate 3503272 bytes) in C:\xxpathxx\class.DGC_Gerais.php on line 0
As I begin to inspect, I tracked the error to the mysql_free_result(), but upon commenting that line, it still doenst work.
Here's the _fetch method:
PUBLIC static function _fetch($sql){
$q = mysql_query($sql);
if(mysql_num_rows($q) > 0):
for($a=0 ; $linha = mysql_fetch_row($q) ; $a++){ // forEach row.
foreach($linha as $chave => $valor): // forEach field.
$result[$a][$chave] = $valor;
endforeach;
}
// mysql_free_result($q);
return $result;
else:
return false;
endif;
}
That code is extremely convoluted and can be simplified to:
public static function _fetch($sql) {
$q = mysql_query($sql);
if (mysql_num_rows($q) == 0) {
return false;
}
$result = array();
while ($linha = mysql_fetch_row($q)) {
$result[] = $linha;
}
return $result;
}
Does exactly the same without double loops.
The problem is that you're fetching all that data from the database and are storing it in $result, which means it needs to be stored in memory. And PHP limits the amount of memory available to scripts by default, so you're simply exceeding that limit. mysql_free_result has nothing as such to do with it. First try to fetch less data, or to process the data inside that while loop without needing to store everything in an array.
If that doesn't help, carefully turn up the memory limit with ini_set('memory_limit', $limit).
Ever since developing my first MySQL project about 7 years ago, I've been using the same set of simple functions for accessing the database (although, have recently put these into a Database class).
As the projects I develop have become more complex, there are many more records in the database and, as a result, greater likelihood of memory issues.
I'm getting the PHP error Allowed memory size of 67108864 bytes exhausted when looping through a MySQL result set and was wondering whether there was a better way to achieve the flexibility I have without the high memory usage.
My function looks like this:
function get_resultset($query) {
$resultset = array();
if (!($result = mysql_unbuffered_query($query))) {
$men = mysql_errno();
$mem = mysql_error();
echo ('<h4>' . $query . ' ' . $men . ' ' . $mem . '</h4>');
exit;
} else {
$xx = 0 ;
while ( $row = mysql_fetch_array ($result) ) {
$resultset[$xx] = $row;
$xx++ ;
}
mysql_free_result($result);
return $resultset;
}
}
I can then write a query and use the function to get all results, e.g.
$query = 'SELECT * FROM `members`';
$resultset = get_resultset($query);
I can then loop through the $resultset and display the results, e.g.
$total_results = count($resultset);
for($i=0;$i<$total_results;$i++) {
$record = $resultset[$i];
$firstname = $record['firstname'];
$lastname = $record['lastname'];
// etc, etc display in a table, or whatever
}
Is there a better way of looping through results while still having access to each record's properties for displaying the result list?
I've been searching around for people having similar issues and the answers given don't seem to suit my situation or are a little vague.
Your problem is that you're creating an array and filling it up with all the results in your result set, then returning this huge array from the function. I suppose that the reason for which this is not supported by any mysql_* function is that it's extremely inefficient.
You should not fill up the array with everything you get. You should step through the results, just like you do when filling up the array, but instead of filling anything, you should process the result and get to the next one, so that the memory for this one gets a chance to be freed.
If you use the mysql_* or mysqli_* functions, you should return the resource, then step through it right there where you're using it, the same way you're stepping through it to fill the array. If you use PDO, then you can return the PDOStatement and use PDOStatement::fetch() to step through it.