Starting multiple PHP processes - php

Question

So I have a script which does the following:
for ($i = $start;$i<=$end;$i++){
shell_exec("php /home/mysql_box/script.php phrases $i {$argv[3]} > /dev/null 2>&1 &");
}
On one of my servers I would like this script to run 400 times. As part of the call it connects to a Mysql server on a different box but for some wierd reason hits about 130 connections and then errors saying "Mysql server has gone away".
I appreciate this is quite common and the MySQL manual states:
You can also encounter this error with applications that fork child processes, all of which try to use the same connection to the MySQL server. This can be avoided by using a separate connection for each child process.
However I wouldn't be using the same connection surely? My wait_timeout is the default - but huge so why would some connections simply not be able to connect?
A couple of other points from the manual:
The connection uses the IP not host name
I didn't start mysql with skip-networking
The query is a single one liner - its not big
I am using Ubuntu
I am connecting with PDO
I hope I've given enough info to help.

Answer 1

So I believe I have discovered the issue and I hope this helps anyone else. The reason being was that the initial query that was run on each loop took aprox 15 seconds. For some reason this reduced the amount of concurrent new connections and as such crashed out. The solution was to therefore reduce the query time required and so if you experience this problem as well - play around with the queries themselves.

Answer 2

You are spawning 400 php processes and letting them to run in background. Having more than 130 simultaneous connections might be hard for your MySQL server handling.
You might want to wait a little to launch all this process:
for ($i = $start;$i<=$end;$i++){
shell_exec("php /home/mysql_box/script.php phrases $i {$argv[3]} > /dev/null 2>&1 &");
if ($i%50 === 0) //every 50 process wait 1 second, so you give a chance that they're finished, and hopefully your db server will be able to handle more connections
sleep(1);
}

Answer 3

I think that firing 400 queries almost simultaneusly by the scripts invoked in the for loop is a bad idea per se.
Even if each query takes little time to execute the loop will act faster.
You may hit the maximun numer of allowed connection set on the server running mysql: max_connections. I'm not 100% sure this is the issue you have because normally the error you would get is Too many connections: see Too many connections
You may also hit the maximum number of network connections allowed by the destination server (this is more likely to lead to a Mysql server has gone away)
Consider also you're spawning 400 processes at once each one running the PHP intepreter. Quite heavy for the local server.
The quickest dirty fix is to set some delay between scripts invocation.
for ($i = $start;$i<=$end;$i++){
shell_exec("php /home/mysql_box/script.php phrases $i {$argv[3]} > /dev/null 2>&1 &");
usleep( 1000 * 1000 * 100 ); // 0.1 SECS
}
However this is leading you versus a "syncronous" approach which may not be what you want.
The proper solution would be to implement a system to know when a spawned script has ended. Then start running the scripts in parallel but avoid more than ex. 50 running simultaneously.
You may consider implementing multithreading directly in your PHP scripts, see:
PThreads / PHP PThreads
PHPThread
GPhpThread
If "hitting the server more gently" (you may even try as a problem-isolation-strategy to run the scripts with high delay or syncronously) doesn't solve the problem (but you should issue less queries at once anyway in my opinion) then the problem falls into the MySQL server has gone away category.
I quote from the linked docs what may be the causes in my opinion (for the full list go to ht elink above)
You tried to run a query after closing the connection to the server. This indicates a logic error in the application that should be corrected.
A client application running on a different host does not have the necessary privileges to connect to the MySQL server from that host.
You have encountered a timeout on the server side and the automatic reconnection in the client is disabled (the reconnect flag in the MYSQL structure is equal to 0).
You may also see the MySQL server has gone away error if MySQL is started with the --skip-networking option.