I am using an update function, where I insert some 40,000 rows to a mysql database. While making that array, I am getting out of memory error (tried to allocate 41 bytes).
The final function is like this:
function Tright($area) {
foreach ($area as $a1=>&$a2) {
mysql_query('INSERT INTO 0_right SET section=\''.$a2['sec_id'].'\', user_id=\''.$a2['user_id'].'\', rank_id=\''.$a2['rank_id'].'\', menu_id=\''.$a2['menu_id'].'\', droit = 1;');
}
}
I have two questions. Is it natural that this above work load becomes too much for php to handle?
If no, can anyone suggest where should I check? And if yes, is there a way to break that $area array to subarrays and execute the function, maybe that way I won't get the out of memory issue. Any other workaround?
Thanks guys.
Edit: #halfdan, #Patrick Fisher, both of you have spoken about making a single multi insert query. How do you do that, in this example please.
First, you should combine all the values into a single INSERT statement, instead of 40,000 different queries.
Second, yes, it is quite natural that you are running out of memory. You can increase this limit at runtime with ini_set(), e.g. ini_set('memory_limit', '16M');
To insert multiple values at once, your SQL should look something like this:
INSERT INTO 0_right (section, user_id, rank_id, menu_id, droit) VALUES
(1,1,1,1,1),
(1,2,1,1,1),
(1,3,1,1,1)
You can build the query like so:
$values = '';
foreach ($area as $a){
if ($values != ''){
$values .= ',';
}
$values .= "('{$a['sec_id']}', '{$a['user_id']}', '{$a['rank_id']}', '{$a['menu_id']}', 1)";
}
$sql = "INSERT INTO 0_right (section, user_id, rank_id, menu_id, droit) VALUES $values";
These are all nice answers to get around the problem. If this is a one time script, just bump up the RAM and make sure the script doesn't time out (max_execution_time in the php.ini file) and you should be fine.
It may run faster if it was one big insert statement, but then you'd pay the cost of constructing the huge query on the PHP side (so the out of memory issue will still be there and will be even worse with the string concatenation). But honestly, who cares if you're just running this once?
However, if you're to perform this operation all the time (e.g. on a webpage), I'd recommend other approaches... like restricting the size of the area, cutting the feature or storing the data differently.
PHP has a built-in (configurable) memory limit to prevent a single script that goes the wrong way from bogging down the whole machine. Depending on your version, that limit defaults to 8MB (pre-5.2.0), 16MB (5.2.0), or 128MB (5.3.0).
You can change the limit either via ini_set or in the php.ini.
Related
The company i work for uses Kayako to manage its support tickets. I was set to make an external webapp that takes all the tickets from one company and displays the history.
Im using mysqli_query to connect to the db.
$link = mysqli_connect($serverName, $userName, $password, $dbName);
$sql = "SELECT ticketid, fullname, creator, contents FROM swticketposts";
$result = mysqli_query($link, $sql) or die(mysqli::$error);
The problem is that the "contents" table in mySQL uses the datatype LONGTEXT.
Trying to read this data with php gives me either timeout or max memory usage errors.
Line 49 is the $result = mysqli_query etc line.
EDIT: Posted wrong error msg in original post
Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 8192 bytes)
Trying to work around the memory problem i added:
ini_set('memory_limit', '-1');
which gave me this timeout error instead:
Fatal error: Maximum execution time of 30 seconds exceeded in C:\xampp\htdocs\test\php\TicketCall.php on line 49
Thing is, when we view the tickets on the kayako platform, it reads the same contents table and does it instantly, so there must be a way to read these longtext's faster. How is beyond me.
Solutions i can't use:
Change the datatype to something smaller (would break our kayako system)
TLDR; Is there a way to read from longtext data types without killing memory and getting timout errors using php and mysql.
First of all, you might consider changing your query to something like this:
SELECT ticketid, fullname, creator, contents FROM swticketposts WHERE customer = 'something'
That will make your query return less data by filtering out the information from customers your report doesn't care about. It may save execution time on your query.
Second, you may wish to use
SUBSTRING(contents, 1, 1000) AS contents
in place of contents in your query. This will get you back part of the contents column (the first thousand characters); it may (or may not) be good enough for what you're trying to do.
Third, mysqli_ generally uses buffered querying. That means the entire result set gets slurped into your program's memory when you run the query. That's probably why your memory is blowing out. Read this. http://php.net/manual/en/mysqlinfo.concepts.buffering.php
You can arrange to handle your query row-by-row doing this:
$unbufResult = mysqli_query($link, $sql, MYSQLI_USE_RESULT) or trigger_error($link->error);
if ($unbufResult ) {
while ($row = $unbufResult ->fetch_assoc()) {
/* deal with the data from one row */
}
}
$unbufResult->close();
You need to be careful to use close() when you're done with these unbuffered result sets, because they use resources on your MySQL server until you do. This will probably prevent your memory blowout. (There's nothing special about using fetch_assoc() here; you can fetch each row with any method you wish.)
Fourth, php's memory and time limits are usually tuned for interactive web site operation. Sometimes report generation takes a long time. Put calls to set_time_limit() in and around your loop, something like this.
set_time_limit (300); /* five minutes */
$unbufResult = mysqli_query($link, $sql, MYSQLI_USE_RESULT) or trigger_error($link->error);
if ($unbufResult ) {
while ($row = $unbufResult ->fetch_assoc()) {
set_time_limit (30); /* reset time limit on each row */
/* deal with the data from one row */
}
}
$unbufResult->close();
set_time_limit (300); /* set time limit back to five minutes when done reading */
I'm inserting multiple rows in a table, and I get this message:
MySQL server has gone away
My Query:
INSERT INTO table
(a,b,c,d,e,f,g,h,i,j,k)
VALUES(1,2,3,4,5,6,7,8,9,10,11),(1,2,3,4,5,6,7,8,9,10,11), ...
ON DUPLICATE KEY UPDATE
c=VALUES(c),
d=VALUES(d),
e=VALUES(e),
f=VALUES(f),
g=VALUES(g),
h=VALUES(h),
i=VALUES(i),
j=VALUES(j)
Is it because I stuffed too many values inside a single query? (There are like 5000 pairs of values from a array which I implode with ,).
If this is the reason - then should I insert each row one by one? Is it slower than inserting them all at once?
The PHP code:
foreach($data as &$entry)
$entry = "('".implode("','", array(
$entry->ID,
addslashes($entry->field_1),
addslashes($entry->field_2),
...
))."')";
$data = implode(',', $data);
$query = "... VALUES{$data} ON ..."
$data is a array of STD type objects...
edit again :)
So I tried splitting my $data into smaller arrays of 100 elements each:
$data_chunks = array_chunk($data, 100);
foreach($data_chunks as $data_chunk)
insert_into_db($data_chunk);
and it works, I don't get that error anymore...
So that means the issue was the very long query string...
Now I'm even more confused:
Is there a length limit of the query, or maybe PHP arguments in general?
Is there any difference between inserting row by row than inserting multiple rows? Is it worth the array_chunk() ?
it could be that your query is taking too long to complete, mysql times out and closes the connection. You can alter the system variables to wait longer.
http://dev.mysql.com/doc/refman/5.0/en/gone-away.html
I think your problem is with *max_allowed_packet*, although the error seems to point in different direction. Try doing as suggested here: http://dev.mysql.com/doc/refman/5.5/en/packet-too-large.html
Or, before making any changes to mysql configuration, simply strlen() your query and find out how long(in bytes) it actually is.
I'm having a strange time dealing with selecting from a table with about 30,000 rows.
It seems my script is using an outrageous amount of memory for what is a simple, forward only walk over a query result.
Please note that this example is a somewhat contrived, absolute bare minimum example which bears very little resemblance to the real code and it cannot be replaced with a simple database aggregation. It is intended to illustrate the point that each row does not need to be retained on each iteration.
<?php
$pdo = new PDO('mysql:host=127.0.0.1', 'foo', 'bar', array(
PDO::ATTR_ERRMODE=>PDO::ERRMODE_EXCEPTION,
));
$stmt = $pdo->prepare('SELECT * FROM round');
$stmt->execute();
function do_stuff($row) {}
$c = 0;
while ($row = $stmt->fetch()) {
// do something with the object that doesn't involve keeping
// it around and can't be done in SQL
do_stuff($row);
$row = null;
++$c;
}
var_dump($c);
var_dump(memory_get_usage());
var_dump(memory_get_peak_usage());
This outputs:
int(39508)
int(43005064)
int(43018120)
I don't understand why 40 meg of memory is used when hardly any data needs to be held at any one time. I have already worked out I can reduce the memory by a factor of about 6 by replacing "SELECT *" with "SELECT home, away", however I consider even this usage to be insanely high and the table is only going to get bigger.
Is there a setting I'm missing, or is there some limitation in PDO that I should be aware of? I'm happy to get rid of PDO in favour of mysqli if it can not support this, so if that's my only option, how would I perform this using mysqli instead?
After creating the connection, you need to set PDO::MYSQL_ATTR_USE_BUFFERED_QUERY to false:
<?php
$pdo = new PDO('mysql:host=127.0.0.1', 'foo', 'bar', array(
PDO::ATTR_ERRMODE=>PDO::ERRMODE_EXCEPTION,
));
$pdo->setAttribute(PDO::MYSQL_ATTR_USE_BUFFERED_QUERY, false);
// snip
var_dump(memory_get_usage());
var_dump(memory_get_peak_usage());
This outputs:
int(39508)
int(653920)
int(668136)
Regardless of the result size, the memory usage remains pretty much static.
Another option would be to do something like:
$i = $c = 0;
$query = 'SELECT home, away FROM round LIMIT 2048 OFFSET %u;';
while ($c += count($rows = codeThatFetches(sprintf($query, $i++ * 2048))) > 0)
{
foreach ($rows as $row)
{
do_stuff($row);
}
}
The whole result set (all 30,000 rows) is buffered into memory before you can start looking at it.
You should be letting the database do the aggregation and only asking it for the two numbers you need.
SELECT SUM(home) AS home, SUM(away) AS away, COUNT(*) AS c FROM round
The reality of the situation is that if you fetch all rows and expect to be able to iterate over all of them in PHP, at once, they will exist in memory.
If you really don't think using SQL powered expressions and aggregation is the solution you could consider limiting/chunking your data processing. Instead of fetching all rows at once do something like:
1) Fetch 5,000 rows
2) Aggregate/Calculate intermediary results
3) unset variables to free memory
4) Back to step 1 (fetch next set of rows)
Just an idea...
I haven't done this before in PHP, but you may consider fetching the rows using a scrollable cursor - see the fetch documentation for an example.
Instead of returning all the results of your query at once back to your PHP script, it holds the results on the server side and you use a cursor to iterate through them getting one at a time.
Whilst I have not tested this, it is bound to have other drawbacks such as utilising more server resources and most likely reduced performance due to additional communication with the server.
Altering the fetch style may also have an impact as by default the documentation indicates it will store both an associative array and well as a numerical indexed array which is bound to increase memory usage.
As others have suggested, reducing the number of results in the first place is most likely a better option if possible.
I'm downloading large sets of data via an XML Query through PHP with the following scenario:
- Query for records 1-1000, download all parts (1000 parts has roughly 4.5 megs of text), then store those in memory while i query the next 1001 - 2000, store in mem (up to potentially 400k)
I'm wondering if it would be better to write these entries to a text field, rather than storing them in memory and once the complete download is done trying to insert them all up into the DB or to try and write them to the DB as they come in.
Any suggestions would be greatly appreciated.
Cheers
You can run a query like this:
INSERT INTO table (id, text)
VALUES (null, 'foo'), (null, 'bar'), ..., (null, 'value no 1000');
Doing this you'll do the thing in one shoot, and the parser will be called once. The best you can do, is running something like this with the MySQL's Benchmark function, running 1000 times a query that inserts 1000 records, or 1000000 of inserts of one record.
(Sorry about the prev. answer, I've misunderstood the question).
I think write them to database as soon as you receive them. This will save memory and u don't have to execute a 400 times slower query at the end. You will need mechanism to deal with any problems that may occur in this process like a disconnection after 399K results.
In my experience it would be better to download everything in a temporary area and then, when you are sure that everything went well, to move the data (or the files) in place.
As you are using a database you may want to dump everything into a table, something like this code:
$error=false;
while ( ($row = getNextRow($db)) && !error ) {
$sql = "insert into temptable(key, value) values ($row[0], $row[1])";
if (mysql_query ($sql) ) {
echo '#';
} else {
$error=true;
}
}
if (!error) {
$sql = "insert into myTable (select * from temptable)";
if (mysql_query($sql) {
echo 'Finished';
} else {
echo 'Error';
}
}
Alternatively, if you know the table well, you can add a "new" flag field for newly inserted lines and update everything when you are finished.
I got thousands of data inside the array that was parsed from xml.. My concern is the processing time of my script, Does it affect the processing time of my script since I have a hundred thousand records to be inserted in the database? I there a way that I process the insertion of the data to the database in batch?
Syntax is:
INSERT INTO tablename (fld1, fld2) VALUES (val1, val2), (val3, val4)... ;
So you can write smth. like this (dummy example):
foreach ($data AS $key=>$value)
{
$data[$key] = "($value[0], $value[1])";
}
$query = "INSERT INTO tablename (fld1, fld2) VALUES ".implode(',', $data);
This works quite fast event on huge datasets, and don't worry about performance if your dataset fits in memory.
This is for SQL files - but you can follow it's model ( if not just use it ) -
It splits the file up into parts that you can specify, say 3000 lines and then inserts them on a timed interval < 1 second to 1 minute or more.
This way a large file is broken into smaller inserts etc.
This will help bypass editing the php server configuration and worrying about memory limits etc. Such as script execution time and the like.
New Users can't insert links so Google Search "sql big dump" or if this works goto:
www [dot] ozerov [dot] de [ slash ] bigdump [ dot ] php
So you could even theoretically modify the above script to accept your array as the data source instead of the SQl file. It would take some modification obviously.
Hope it helps.
-R
Its unlikely to affect the processing time, but you'll need to ensure the DB's transaction logs are big enough to build a rollback segment for 100k rows.
Or with the ADOdb wrapper (http://adodb.sourceforge.net/):
// assuming you have your data in a form like this:
$params = array(
array("key1","val1"),
array("key2","val2"),
array("key3","val3"),
// etc...
);
// you can do this:
$sql = "INSERT INTO `tablename` (`key`,`val`) VALUES ( ?, ? )";
$db->Execute( $sql, $params );
Have you thought about array_chunk? It worked for me in another project
http://www.php.net/manual/en/function.array-chunk.php