I have large MySQL query (1.8M rows, 25 columns) and I need to make 2 dimensional array from it (memory table based on primary key).
Code works as expected, but $table creation takes a long time in PHP7.0.
What is the reason why PHP7.0 performs so much worse? My primary interest is in mysqli.
Thank you for any insights - PHP7 would save me much memory if I can fix performance.
mysqli code snippet
$start = microtime(true);
$vysledek = cluster::query("SELECT * FROM `table` WHERE 1");
$query_time = (microtime(true) - $start);
$start_fetch = microtime(true);
while($zaznam = mysqli_fetch_assoc ( $vysledek )){
$fetch_time+= (microtime(true) - $start_fetch);
$start_assign = microtime(true);
$table[$zaznam['prikey']] = $zaznam;
$assign_time+= (microtime(true) - $start_assign);
$start_fetch = microtime(true);
}
$total_time+= (microtime(true) - $start);
echo round($assign_time, 2).' seconds to set the array values\n';
echo round($query_time, 2).' seconds to execute the query\n';
echo round($fetch_time, 2).' seconds to fetch data\n';
echo round($total_time, 2).' seconds to execute whole script\n';
echo "Peak Memory Usage:".round(memory_get_peak_usage(true)/(1024 * 1024), 2)." MB\n";
mysqli results
Deb 7 PHP 5.4 mysqlnd 5.0.10
1.8 seconds to set the array values
8.37 seconds to execute the query
13.49 seconds to fetch data
24.42 seconds to execute whole script
Peak Memory Usage:8426.75 MB
Deb 8 PHP 5.6 mysqlnd 5.0.11-dev
1.7 seconds to set the array values
8.58 seconds to execute the query
12.55 seconds to fetch data
23.6 seconds to execute whole script
Peak Memory Usage: 8426.75 MB
Deb 8 PHP 7.0 mysqlnd 5.0.12-dev
0.73 seconds to set the array values
8.63 seconds to execute the query
126.71 seconds to fetch data
136.46 seconds to execute whole script
Peak Memory Usage:7394.27 MB
Deb 8 PHP 7.0 mysqlnd 5.0.12-dev extended benchmarking
I have extended benchmarking for section fetch to report every 100k lines with following results:
Lines fetched 100000 in 1.87s
Lines fetched 300000 in 5.24s
Lines fetched 500000 in 10.97s
Lines fetched 700000 in 19.17s
Lines fetched 900000 in 29.96s
Lines fetched 1100000 in 43.03s
Lines fetched 1300000 in 58.48s
Lines fetched 1500000 in 76.47s
Lines fetched 1700000 in 96.73s
Lines fetched 1800000 in 107.78s
DEB8 PHP7.1.0-dev libclient 5.5.50
1.56 seconds to set the array values
8.38 seconds to execute the query
456.52 seconds to fetch data
467.68 seconds to execute whole script
Peak Memory Usage:8916 MB
DEB8 PHP7.1.0-dev libclient 5.5.50 extended benchmarking
Lines fetched 100000 in 2.72s
Lines fetched 300000 in 15.7s
Lines fetched 500000 in 38.7s
Lines fetched 700000 in 71.69s
Lines fetched 900000 in 114.8s
Lines fetched 1100000 in 168.18s
Lines fetched 1300000 in 231.69s
Lines fetched 1500000 in 305.36s
Lines fetched 1700000 in 389.05s
Lines fetched 1800000 in 434.71s
DEB8 PHP7.1.0-dev mysqlnd 5.0.12-dev
1.51 seconds to set the array values
9.16 seconds to execute the query
261.72 seconds to fetch data
273.61 seconds to execute whole script
Peak Memory Usage:8984.27 MB
DEB8 PHP7.1.0-dev mysqlnd 5.0.12-dev extended benchmarking
Lines fetched 100000 in 3.3s
Lines fetched 300000 in 13.63s
Lines fetched 500000 in 29.02s
Lines fetched 700000 in 49.21s
Lines fetched 900000 in 74.56s
Lines fetched 1100000 in 104.97s
Lines fetched 1300000 in 140.03s
Lines fetched 1500000 in 180.42s
Lines fetched 1700000 in 225.72s
Lines fetched 1800000 in 250.01s
PDO code snippet
$start = microtime(true);
$sql = "SELECT * FROM `table` WHERE 1";
$vysledek = $dbh->query($sql, PDO::FETCH_ASSOC);
$query_time = (microtime(true) - $start);
$start_fetch = microtime(true);
foreach($vysledek as $zaznam){
$fetch_time+= (microtime(true) - $start_fetch);
$start_assign = microtime(true);
$table[$zaznam['prikey']] = $zaznam;
$assign_time+= (microtime(true) - $start_assign);
$start_fetch = microtime(true);
}
$total_time+= (microtime(true) - $start);
echo round($assign_time, 2).' seconds to set the array values\n';
echo round($query_time, 2).' seconds to execute the query\n';
echo round($fetch_time, 2).' seconds to fetch data\n';
echo round($total_time, 2).' seconds to execute whole script\n';
echo "Peak Memory Usage:".round(memory_get_peak_usage(true)/(1024 * 1024), 2)." MB\n";
PDO Results
Deb 7 PHP 5.4 mysqlnd 5.0.10
1.85 seconds to set the array values
12.51 seconds to execute the query
16.75 seconds to fetch data
31.82 seconds to execute whole script
Peak Memory Usage:11417.5 MB
Deb 8 PHP 5.6 mysqlnd 5.0.11-dev
1.75 seconds to set the array values
12.16 seconds to execute the query
15.72 seconds to fetch data
30.39 seconds to execute whole script
Peak Memory Usage:11417.75 MB
Deb 8 PHP 7.0 mysqlnd 5.0.12-dev
0.71 seconds to set the array values
35.93 seconds to execute the query
114.16 seconds to fetch data
151.19 seconds to execute whole script
Peak Memory Usage:6620.29 MB
Baseline comparison code
$start_query = microtime(true);
exec("mysql --user=foo --host=1.2.3.4 --password=bar -e'SELECT * FROM `profile`.`table`' > /tmp/out.csv");
$query_time = (microtime(true) - $start_query);
echo round($query_time, 2).' seconds to execute the query \n';
Execution time is similar for all systems at 19 seconds +-1 second variation.
Based on above observations I would say that PHP 5.X is reasonable as there is a bit more work executed than just dumping to the file.
all 3 servers are on same host (source and both test servers)
tests are consistent when repeated
there is already similar variable in memory ,I need to do it for comparison removed for testing, is not related to the problem
CPU is at 100% whole time
Both servers have 32G RAM and swappiness set to 1, goal is to perform it as memory operation
test server is dedicated, there is nothing else running
php.ini changed between major versions but all options relating to mysqli/PDO seems to be the same
Deb8 machine was downgraded to PHP5.6 and issue disappeared, after reinstalling PHP7 its back
Reported a bug at php.net - ID 72736 since I belive that it was proven that problem is in PHP and not in the system or any other configuration
Edit 1 : Added PDO Comparison
Edit 2 : Added benchmarking markers, edited PDO results as there was benchmarking error
Edit 3 : Major cleanup in original question, rebuild of code snipets for better indication of the error
Edit 4 : added point about Downgrade and upgrade of PHP
Edit 5 : added extended benchmarking for DEB8 PHP7.0
Edit 6 : included php7 config
Edit 7 : performance measurement for PHP 7.1 dev with both libraries- compiled with configs from bishop, removed my php-config
Edit 8 : added comparison against CLI command, minor clean-ups
For cross-reference: With the release of PHP 7.1 on 1st Dec 2016 this issue should be resolved (in PHP 7.1).
PHP 7.0: Even in the ticket it's written that PHP-7.0 has been patched, I've not yet seen in the recent change-log (7.0.13 on 10 Nov 2016, since patch incorporation date) that this is part of the current PHP 7.0.x release. Perhaps with the next release.
The bug is tracked upstream (thanks to OP's report): Bug #72736 - Slow performance when fetching large dataset with mysqli / PDO (bugs.php.net; Aug 2016).
As the problem appears to be in the fetch (not the array creation), and we know the driver is running mysqlnd (which is a driver library independently written by the PHP team, not provided by MySQL AB aka Oracle), then recompiling PHP using libmysqlclient (which is the MySQL AB aka Oracle provided interface) may improve the situation (or at least narrow the problem space).
First thing I'd suggest is writing a small script that can be run from the CLI that demonstrates the problem. This will help to eliminate any other variables (web server modules, opcache, etc).
Then, I'd suggest rebuilding PHP with libmysqlclient to see if performance improves. Quick guide to rebuilding PHP (for the technically competent):
Download the source for the PHP version you want
Decompress and go into the PHP code directory
Run ./buildconf
Run ./configure --prefix=/usr --with-config-file-path=/etc/php5/apache2 --with-config-file-scan-dir=/etc/php5/apache2/conf.d --build=x86_64-linux-gnu --host=x86_64-linux-gnu --sysconfdir=/etc --localstatedir=/var --mandir=/usr/share/man --enable-debug --disable-static --with-pic --with-layout=GNU --with-pear=/usr/share/php --with-libxml-dir=/usr --with-mysql-sock=/var/run/mysqld/mysqld.sock --enable-dtrace --without-mm --with-mysql=shared,/usr --with-mysqli=shared,/usr/bin/mysql_config --enable-pdo=shared --without-pdo-dblib --with-pdo-mysql=shared,/usr CFLAGS="-g -O2 -fstack-protector-strong -Wformat -Werror=format-security -O2 -Wall -fsigned-char -fno-strict-aliasing -g" LDFLAGS="-Wl,-z,relro" CPPFLAGS="-D_FORTIFY_SOURCE=2" CXXFLAGS="-g -O2 -fstack-protector-strong -Wformat -Werror=format-security"
Run make && make test
Walk away
Run sapi/cli/php -i and confirm the version and presence of libmysqlclient
Run your test again. Any better?
Related
I've had a server reboot and something changed.
Suddenly tiny SQL queries using PDO to MYSQL require roughly 90MB of memory.
It uses 3 times the highest input buffer.
MEM: 3586264 / 4718592
MEM: 96740584 / 98304000
MEM: 96740584 / 98304000
The code is as simple as possible:
$a=memory_get_peak_usage(false);
$b=memory_get_peak_usage(true);
echo "MEM: $a / $b\n";
$pdo->query("SELECT * FROM `results` WHERE `customer_id` = '456' AND `jobname` = 'job1' LIMIT 1")
$a=memory_get_peak_usage(false);
$b=memory_get_peak_usage(true);
echo "MEM: $a / $b\n";
$pdo->query("SELECT * FROM `results` WHERE `customer_id` = '456' AND `jobname` = 'job1' LIMIT 1")
$a=memory_get_peak_usage(false);
$b=memory_get_peak_usage(true);
echo "MEM: $a / $b\n";
Of course I checked the query, the content is indeed tiny, it's just a single row of a few varchar fields.
PDO is initialized like this: $opt = [PDO::ATTR_EMULATE_PREPARES => false, PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION, PDO::MYSQL_ATTR_MAX_BUFFER_SIZE => 1024 * 1024 * 30];
I noticed the error when I increased the MAX_BUFFER_SIZE to 60MB.
Suddenly my application was way beyond 140MB memory.
It seems to use max_buffer_size as MIN_buffer_size*3
It seems like PDO is using 3 times the max_buffer_size variable as storage buffer for tiny queries.
Which makes no sense at all ..
It's PHP 5.6.30
I just don't know where to look anymore, I debugged it down to a single query and now I am stuck, that's a PHP internal allocation.
Update
The issue was likely present before reboot, just unnoticed.
I don't know what triggered it but changing the library to php-mysqlnd solved it.
No memory usage anymore.
It seems that's a driver bug in the php-mysql driver, it allocates massive amounts of memory based on the MAX_BUFFER attribute and does not release it anymore when using PDO.
I switched to the native driver mysqlnd and the issue vanished.
In Short: How to break the code after the code takes more than 53 seconds to execute, something like this:
while (5 < 200)
{
// do some work here, which will take around 1-6 seconds
if (code this loop is running for more than 53 seconds)
break;
}
If you want to know, why i want to do this:
Ok, this is What I am doing: I am copying data from 7 JSON pages on the web and inserting them into my MySQL database.
The code is something like this:
$start_time = microtime(TRUE); // Helps to count PHP script execution time
connect to database
$number = get inserted number into database
while ($number > ($number+20) )
{
7 Open JSON File link like this - "example.com/$number?api=xxxyyzz"
Use Prepared PDO statements to insert data into MySQL
}
// Code to count PHP script execution time
$end_time = microtime(TRUE);
$time_taken = $end_time - $start_time;
$time_taken = round($time_taken,5);
echo '<p>Page generated in '.$time_taken.' seconds.</p>';
So in my case, It takes around 5.2 seconds to complete one whole loop of adding all data. But some JSON files are empty, so it takes only 1.4 second to complete 1 loop, if they are empty.
So like that, I want to complete millions of loops (add Data from millions of JSON files). So if my code runs for 24/7, it will take me 1 month to complete my task.
But after the code runs for 90 seconds, i get this error:
I am using a CRON job to do the task. And looks like server gives the same error to CRON job.
So, I want to do the CRON job to run every 1 minute, so I do not get timed out error.
But I am afraid of this: What If the script added data in half rows and 1 mintue gets over, and it do not add data into other half rows. Then after the starting on the new minute, the code start from the next $number.
So, If i can break; out of the loop after 53 seconds (If the code starts another loop at 52 seconds, then break at the end of it, that will be around 58-59 seconds).
I mean, i will put the break; code just before the loop end (before }). So i do not exit the loop, while the data got inserted into half of the rows.
I guess that your PHP's max_execution_time is equal to 90 seconds, you can specify max_execution_time by set_time_limit, but I don't think it is a good approach for this.
Have a try pcntl or pthreads, it would save you a lot of time.
Please anyone help me to solved my case :
What should I add to my code, if the output from the code below is insert on the table in POSTGRES, and row that will be inserted are more than 1000 rows. When I running using this code below it appears the error :
Fatal error: Maximum execution time of 60 seconds exceeded in D: \ xampp \ htdocs \ BillingDeliveryInfo \ page2.php on line 45
Because it took so long query maybe
while($msisdn = pg_fetch_row($qaccount)){
if ($msisdn[4]==2){
$insert1="insert into delin_bdi_rf
values ('$msisdn[1]',
'$m1[1] $msisdn[1] $m1[2] $msisdn[2] $m1[3] $msisdn[3] $m1[4]',
CURRENT_DATE);";
}else{
$insert2="insert into delin_bdi_rf
values ('$msisdn[1]',
'$m2[1] $msisdn[1] $m2[2] $msisdn[3] $m2[3]',
CURRENT_DATE);";
}
$qwer=$insert1.$insert2;
$n=pg_query($qwer);
}
Try adding this in the beginning:
ini_set('max_execution_time', 300); //300 seconds = 5 minutes
It increases maximum execution time of script.
I was doing bulk inserts in the RealTime Index using PHP and by Disabling AUTOCOMIT ,
e.g.
// sphinx connection
$sphinxql = mysqli_connect($sphinxql_host.':'.$sphinxql_port,'','');
//do some other time consuming work
//sphinx start transaction
mysqli_begin_transaction($sphinxql);
//do 50k updates or inserts
// Commit transaction
mysqli_commit($sphinxql);
and kept the script running overnight, in the morning i saw
PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate
212334 bytes) in
so when i checked the nohup.out file closely , i noticed , these lines ,
PHP Warning: mysqli_query(): MySQL server has gone away in /home/script.php on line 502
Warning: mysqli_query(): MySQL server has gone away in /home/script.php on line 502
memory usage before these lines was normal , but memory usage after these lines started to increase, and it hit the php mem_limit and gave PHP Fatal error and died.
in script.php , line 502 is
mysqli_query($sphinxql,$update_query_sphinx);
so my guess is, sphinx server closed/died after few hours/ minutes of inactivity.
i have tried setting in sphinx.conf
client_timeout = 3600
Restarted the searchd by
systemctl restart searchd
and still i am facing same issue.
So how can i not make sphinx server die on me ,when no activity is present for longer time ?
more info added -
i am getting data from mysql in 50k chunks at a time and doing while loop to fetch each row and update it in sphinx RT index. like this
//6mil rows update in mysql, so it takes around 18-20 minutes to complete this then comes this following part.
$subset_count = 50000 ;
$total_count_query = "SELECT COUNT(*) as total_count FROM content WHERE enabled = '1'" ;
$total_count = mysqli_query ($conn,$total_count_query);
$total_count = mysqli_fetch_assoc($total_count);
$total_count = $total_count['total_count'];
$current_count = 0;
while ($current_count <= $total_count){
$get_mysql_data_query = "SELECT record_num, views , comments, votes FROM content WHERE enabled = 1 ORDER BY record_num ASC LIMIT $current_count , $subset_count ";
//sphinx start transaction
mysqli_begin_transaction($sphinxql);
if ($result = mysqli_query($conn, $get_mysql_data_query)) {
/* fetch associative array */
while ($row = mysqli_fetch_assoc($result)) {
//sphinx escape whole array
$escaped_sphinx = mysqli_real_escape_array($sphinxql,$row);
//update data in sphinx index
$update_query_sphinx = "UPDATE $sphinx_index
SET
views = ".$escaped_sphinx['views']." ,
comments = ".$escaped_sphinx['comments']." ,
votes = ".$escaped_sphinx['votes']."
WHERE
id = ".$escaped_sphinx['record_num']." ";
mysqli_query ($sphinxql,$update_query_sphinx);
}
/* free result set */
mysqli_free_result($result);
}
// Commit transaction
mysqli_commit($sphinxql);
$current_count = $current_count + $subset_count ;
}
So there are a couple of issues here, both related to running big processes.
MySQL server has gone away - This usually means that MySQL has timed out, but it could also mean that the MySQL process crashed due to running out of memory. In short, it means that MySQL has stopped responding, and didn't tell the client why (i.e. no direct query error). Seeing as you said that you're running 50k updates in a single transaction, it's likely that MySQL just ran out of memory.
Allowed memory size of 134217728 bytes exhausted - means that PHP ran out of memory. This also leads credence to the idea that MySQL ran out of memory.
So what to do about this?
The initial stop-gap solution is to increase memory limits for PHP and MySQL. That's not really solving the root cause, and depending on t he amount of control you have (and knowledge you have) of your deployment stack, it may not be possible.
As a few people mentioned, batching the process may help. It's hard to say the best way to do this without knowing the actual problem that you're working on solving. If you can calculate, say, 10000 or 20000 records instad of 50000 in a batch that may solve your problems. If that's going to take too long in a single process, you could also look into using a message queue (RabbitMQ is a good one that I've used on a number of projects), so that you can run multiple processes at the same time processing smaller batches.
If you're doing something that requires knowledge of all 6 million+ records to perform the calculation, you could potentially split the process up into a number of smaller steps, cache the work done "to date" (as such), and then pick up the next step in the next process. How to do this cleanly is difficult (again, something like RabbitMQ could simplify that by firing an event when each process is finished, so that the next one can start up).
So, in short, there are your best two options:
Throw more resources/memory at the problem everywhere that you can
Break the problem down into smaller, self contained chunks.
You need to reconnect or restart the DB session just before mysqli_begin_transaction($sphinxql)
something like this.
<?php
//reconnect to spinx if it is disconnected due to timeout or whatever , or force reconnect
function sphinxReconnect($force = false) {
global $sphinxql_host;
global $sphinxql_port;
global $sphinxql;
if($force){
mysqli_close($sphinxql);
$sphinxql = #mysqli_connect($sphinxql_host.':'.$sphinxql_port,'','') or die('ERROR');
}else{
if(!mysqli_ping($sphinxql)){
mysqli_close($sphinxql);
$sphinxql = #mysqli_connect($sphinxql_host.':'.$sphinxql_port,'','') or die('ERROR');
}
}
}
//10mil+ rows update in mysql, so it takes around 18-20 minutes to complete this then comes this following part.
//reconnect to sphinx
sphinxReconnect(true);
//sphinx start transaction
mysqli_begin_transaction($sphinxql);
//do your otherstuff
// Commit transaction
mysqli_commit($sphinxql);
A few days ago wrote a php script that goes through all my music and reads id3 tags for each song and inserts those in a mysql database. Here's a snippet for context:
exec ( "find /songs -type f -iname '*mp3'", $song_path );
$number_of_songs = count($song_path);
for($i=0; $i<$number_of_songs; $i++){
//read id3 tags
//store id3 tags into database
}
I changed the php directive max_execution_time in apache2/php.ini to gain a better understanding of what this directive does.
When I set max_execution_time = 10, my php script runs for about 45 seconds and successfully reads the id3 tags for about 150 songs (Out of thousands of songs) and inserts those tags into a mysql database before terminating the script and outputting the following to the screen:
Fatal error: Maximum execution time of 10 seconds exceeded in /websites/.../public_html/GetID3()/getid3/module.audio.mp3.php on line 1894
From the documentation, 'The maximum execution time is not affected by system calls, stream operations etc' http://www.php.net/manual/en/info.configuration.php#ini.max-execution-time
What can I infer from the difference between the
maximum_execution_time being set at 10 seconds and the script
running for a total of 45 seconds before terminating? Does this
mean out of the 45 seconds, 35 were spent doing non-php related
activities like reading the id3 tags, inserting data into mysql etc..., while 10 were spent
doing php related activities like iterating the loop?
Is there a way I can calculate the execution time and print it to
the screen?
EDIT
Using the timer Dagon suggested, I called the getTime() function at the end of the loop, there were about 100+ iterations of the loop. Here is the output to my browser:
0.1163 seconds
0.8142 seconds
1.1379 seconds
1.5555 seconds
...
76.7847 seconds
77.2008 seconds
77.6071 seconds
Fatal error: Maximum execution time of 10 seconds exceeded in /websites/.../public_html/GetID3()/getid3/module.audio.mp3.php on line 505
<!-- Paste this at the top of the page -->
<?php
$mic_time = microtime();
$mic_time = explode(" ",$mic_time);
$mic_time = $mic_time[1] + $mic_time[0];
$start_time = $mic_time;
?>
<!-- Write Your script(executable code) here -->
enter code here
<!-- Paste this code at the bottom of the page -->
<?php
$mic_time = microtime();
$mic_time = explode(" ",$mic_time);
$mic_time = $mic_time[1] + $mic_time[0];
$endtime = $mic_time;
$total_execution_time = ($endtime - $start_time);
echo "Total Executaion Time ".$total_execution_time." seconds";
?>
i don't believe the script is actully running for more than 10 seconds, you need to put a proper timer in it
<!-- put this at the top of the page -->
<?php
function getTime() {
$timer = explode( ' ', microtime() );
$timer = $timer[1] + $timer[0];
return $timer;
}
$start = getTime();
?>
<!-- put other code and html in here -->
<!-- put this code at the bottom of the page -->
<?php
$end = getTime();
$time_took= round($end - $start,4).' seconds';
echo $time_took;
?>
This type of script should really be executed in the CLI environment, not in a php process executed by your web server. As per the manual docs on how the PHP command line environment differs from other PHP SAPIs:
PHP in a shell environment tends to be used for a much more diverse
range of purposes than typical Web-based scripts, and as these can be
very long-running, the maximum execution time is set to unlimited.
While it doesn't answer your question, it does solve your problem :)
Seems like you not only try to measure the script duration, but also try to limit it. And in your case max_execution_time is not what you want.
Basically, "The maximum execution time is not affected by system calls, stream operations etc" is correct. If you need to limit real-time script length, you need to implement your own time measurement. People usually write some benchmark class for it, which after all will be helpful in optimizing the script, but simple
$timer['start'] = time();
$timer['max_exec_time'] = 10; // seconds
at start and
if (time() > $timer['start'] + $timer['max_exec_time'])
break; // or exit; etc
at the end of the loop or anywhere else you want should be enough.