I'm using PHP with MYSQL database as both are open source and easy to use.
I'm getting problem when I execute insert and/or update of millions of row one after another
while this operation perform I got the MYSQL error that:
'max_user_connections' active connections
which is the best way to solve this problem.
I don't want to use another database or language other then PHP.
connect_db();
$query = "insert into table(mobno,status,description,date,send_deltime,sms_id,msg,send_type) values('".$to."','".$status."','".$report."','','".$timewsha1."','".$smsID."','','".$type."')";
$result = mysql_query($query) or ("Query failed : " . mysql_error());
this query will execute thousand of times.
and then server give connection error.
First of all, try to know from your hosting server administrator about the max consecutive active connections available to the MySQL database. This is the most basic & primary information to have knowledge about.
If your page(s) load in a decent amount of time and release the connection once the page is loaded, it should be fine. The problem occurs when your script takes some long time to retrieve information from the database or maintains the connections.
Since you are executing INSERT and / or UPDATE operations of millions of rows, so you may have some problem.
Additionally, if you fail to close connections in your script(s), it is possible that someone will load a page and instead of closing the connection when the page is loaded, it is left open. No one else can then use that connection. So please make sure that at the end of execution of all the MySQL / SQL queries, the database connection is closed. Also please make sure that your server provides more than 250 connections, since 100 connections is available in almost all the servers generally.
Also make sure that you are not using the persistent connections (which is available when using the built-in function "mysql_pconnect()"), since this will lock up the user until the connection is manually closed.
Hope it helps.
//this loop is for preparing the subquery for mutiple records
for(// this loop for getting data for mutiple records){
$sub_query[] = "('".$to."','".$status."','".$report."','','".$timewsha1."','".$smsID."','','".$type."')";
}
$query = "insert into table(mobno,status,description,date,send_deltime,sms_id,msg,send_type) values ";
$query .= implode(',',$sub_query);
mysql_query($query );
So, a remote app calls into this script, sends it a list of values, and then this query is executed once, right? It's not in a foreach or for or while loop? When you say it will be executed millions of times, you don't mean in one sitting I mean. If it is in a loop, then move the db connect outside of the loop, otherwise it will attempt to connect again each time the loop iterates, and also, remember to call mysql_close at the end of the script, just in case.
mysql_pconnect() would create a persistent connection and that is the way to go if you don't want to exhaust your server's connection pool.
Related
I have two files:
config.php:
$con = mysqli_connect("$dbhost","$dbuser","$dbpass","$dbname");
ads.php (config.php require_once):
$query1 = mysqli_query($con,"SELECT * From XXXX where id = 'XXXX'");
$query2 = mysqli_query($con,"SELECT * FROM XXXX2 WHERE id = 'XXXX2'");
I have more than 40 different queries mysqli_query() using $con. Please keep in mind that all my logic is procedural style, not object oriented.
My two questions are:
am I connecting to the DB using a different connection on every query? Or just once when the first one is executed? Is there a better way to do this?
I understand that it is not mandatory to close a mysql connection since it closes itself, however since we have millions of consults sometimes a few can hang and stay for more than 10 seconds after the user left the PHP file. If I wanted to close a connection, should I put mysqli_close($con) at the end of the file (after the 40 consults) or at the end of each consult (40 times)?
Any help would be appreciated. Thanks
You are reusing the connection initiated in mysqli_connect.
A better way could be to use or create a DataHandler and implement PDOs. It could allow you to perform timing analytics on queries, automate binding parameters, destroy the connection once its finished with etc.
Call mysqli_close($con) at the end of the file. Also null $con to mark for garbage collection.
Once finished with the information from each step you could also call;
mysqli_free_result($query1) to free memory associated to the statement handle
I'm updating an old script using php mysql functions to use mysqli and I've already found an interesting question (mysqli_use_result() and concurrency) but it doesn't clarify one thing:
Lets say five users connects to a webpage, they connected at same time and user1 was the first to select the data from a huge MyIsam database (15000 records of forum posts with left join to 'users' and 'attachments' table).
while the php scripts retrieves the result, the other users will not be able to get results, is that right?
Also using the same situation above, when user1 fully received it's result, an 'UPDATE set view_count = INC(1)' query is sent and the table is locked I suppose, and this same query will fail for the other users?
About the article you've quoted. It just means, that you should not do this:
mysqli_query($link, $query);
mysqli_use_result($link);
// lots of 'client processing'
// table is blocked for updates during this
sleep(10)
mysqli_fetch_* .....
In such situtations you are adviced to do so:
mysqli_query($link, $query);
mysqli_store_result($link);
// lots of 'client processing'
// table is NOT blocked for updates during this
sleep(10)
mysqli_fetch_* .....
The article further says, that if a second query will be issued - after calling mysql_use_result() and before fetching the results from the query it will fail. This is meant per connection - per script. So other user's queries won't fail during this.
while the php scripts retrieves the result, the other users will not be able to get results, is that right?
No this is not right. MySQL supports as many parallel connections as you have configured in my.ini max_connections. Concurrent reads are handled by the mysql server. Client code has not to worry about that unless the max connection limit is reached and mysqli_connect() would fail. If your application reaches a point where this happens frequently you'll in most cases first try to tweak your mysql config so that mysql allows more parrallel connections. If a threshold is reached you'll use an attempt like replication or mysql cluster.
Also using the same situation above, when user1 fully received it's result, an 'UPDATE set view_count = INC(1)' query is sent and the table is locked I suppose, and this same query will fail for the other users?
When there are concurrent reads and writes this is of course a performance issue. But the MySQL server handle this for you, meaning the client code has not worry about it as long as connecting to mysql works. If you have really high load you'll mostly use master slave Replication or MySQL cluster.
and this same query will fail for the other users?
A database server usually a bit more intelligent than a plain text file.
So, your queries won't fail. They'd wait instead.
Though I wouldn't use mysqli_use_result() at all.
Why not just fetch your results already?
This question already has answers here:
"Commands out of sync; you can't run this command now" - Caused by mysqli::multi_query
(3 answers)
Closed 7 months ago.
I am currently doing the following:
$mysqli = new mysqli($server, $username, $password, $database);
$mysqli->multi_query($multiUpdates);
while ($mysqli->next_result()) {;} // Flushing results of multi_queries
$mysqli->query($sqlInserts);
Is there a faster way to dump the results?
I do not need them and just want to run the next query however I get the error:
Commands out of sync; you can't run this command now
Problem is the while ($mysqli->next_result()) {;} takes about 2 seconds which is a waste for something I don't want.
Any better solutions out there?
Found a faster solution which saves about 2-3 seconds when updating 500 records and inserting 500 records.
function newSQL() {
global $server, $username, $password, $database;
$con = new mysqli($server, $username, $password, $database);
return $con;
}
$mysqli = newSQL();
$mysqli->multi_query($multiUpdates);
$mysqli->close();
$mysqli = newSQL();
$mysqli->query($sqlInserts);
$mysqli->close();
Not sure how practical it is but works well for speed.
If closing and reopening the connection works for you, then you might be better off changing the order:
$mysqli = newSQL();
$mysqli->query($sqlInserts);
$mysqli->multi_query($multiUpdates);
$mysqli->close();
If you don't care which runs first, the query runs more predictably. As soon as it finishes, MySQL will return the results of the insert statement to the PHP client (probably mysqlnd). The connection will then be clear and can accept the next request. No need to close and reopen after a query. So you save the time it takes to close and reopen the connection with this order.
The multi_query is more complicated. It returns the results of the first update before the PHP code continues. At this point, we don't know if the later updates have run or not. The database won't accept any more queries on this connection until it has finished with the multi_query, including passing the results back with next_result. So what happens when you close the query?
One possibility is that it blocks until the multi_query is finished but does not require the results. So closing the connection essentially skips the part where the database server returns the results but still has to wait for them to be generated. Another possibility is that the connection closes immediately and the database continues with the query (this is definitely what happens if the connection is simply dropped without formally closing it, as the database server won't know that the connection is broken until it finishes or times out, see here or here).
You'll sometimes see the claim that query and multi_query take the same time. This is not true. Under some circumstances, multi_query can be slower. Note that with a normal query (using the default MYSQLI_STORE_RESULT), the database can simply return the result as soon as it finishes. But with multi_query (or with MYSQLI_USE_RESULT), it has to retain the result on the database server. If the database server stores the result, it may have to page it out of memory or it may deliberately store the result on disk. Either way, it frees up the memory but puts the result in a state where it takes more time to access (because disk is slower than memory).
NOTE for other readers: multi_query is harder to use safely than query. If you don't really know what you are doing, you are probably better off using PDO than mysqli (because PDO does more of the work for you) and you are almost certainly better off doing your queries one at a time. You should only use multi_query if you understand why it increases the risk of SQL injections and are avoiding it. Further, one usually doesn't need it.
The only real advantage to multi_query is it allows you to do your queries in one block. If you already have queries in a block (e.g. from a database backup), this might make sense. But it generally doesn't make sense to aggregate separate queries into a block so as to use multi_query. It might make more sense to use INSERT ON DUPLICATE KEY UPDATE to update multiple rows in one statement. Of course, that trick won't work unless your updates have a unique key. But if you do, you might be able to combine both the inserts and the updates into a single statement that you can run via query.
If you really need more speed, consider using something other than PHP. PHP produces HTML in response to web requests. But if you don't need HTML/web requests and just want to manipulate a database, any shell language will likely be more performant. And certainly multithreaded languages with connection pools will give you more options.
Is there a Mysql statement which provides full details of any other open connection or user? Or, an equally detailed status report on myisam tables specifically. Looking at Mysql's SHOW TABLE STATUS documentation, it's missing some very important information for my purpose.
What I'm trying to do: remote odbc connection one is inserting several thousand records, which due to a slow connection speed can take up to an hour. Tcp connection two, using PHP on the server's localhost, is running select queries with aggregate functions on that data. Before allowing connection two to run those queries, I'd like connection two to first check to make sure there's no pending inserts on any other connection on those specific tables so it can instead wait until all data is available. If the table is currently being written to, I'd like to spit back to the user of connection two an approximation of how much longer to wait based on the number of pending inserts. Ideally by table, I'd like to get back using a query the timestamp when connection one began the write, total inserts left to be done, and total inserts already completed. Instead of insert counts, even knowing number of bytes written and left to write would work just fine here.
Obviously since connection two is a tcp connection via a PHP script, all I can really use in that script is some sort of query. I suppose if I have to, since it is on localhost, I can exec() it if the only way is by a mysql command line option that outputs this info, but I'd rather not. I suppose I could simply update a custom-made transaction log before and after this massive insert task which the PHP script can check, but hopefully there's already a built-in Mysql feature I can take advantage of.
Edit: "Transaction" was the wrong word - a language collision. I'm not actually using Mysql transactions. What I meant was currently pending tasks, queries, and requests.
You can issue SHOW FULL PROCESSLIST; to show the active connections.
As for the rest, mysql doesn't know how many inserts are left, and how long they'll take. (And if you're using MyISAM tables, they dont support transactions). The server have no way of knowing whether your PHP scripts intend to send 10 more inserts, or 10000 - and if you're doing something like insert into xxx select ... from ... - mysql doesn't track/expose info on how much/many is done/is left .
You're better off handling this yourself via other tables where you update/insert data about when you started aggregating data, track the state,when it finished etc.
If the transactions are being performed on InnoDB tables, you can get full transaction details with SHOW INNODB STATUS. It's a huge blob of output, but part of it is transactions/lock status for each process/connection.
When you run a query like so:
$query = "SELECT * FROM table";
$result = odbc_exec($dbh, $query);
while ($row = odbc_fetch_array($result)) {
print_r($row);
}
Does the resource stored in $result point to data that exists on the server running php? Or is pointing to data in the database? Put another way, as the while loop does it's thing ,is PHP talking to the DB every iteration or is it pulling that $row from some source on the application side?
Where this is mattering to me is I have a database I'm talking to over VPN using ODBC with PHP. This last weekend something strange has happened where huge pauses are happening during the while loop. So between iterations, the script will stop execution for seconds and up to minutes. It seems to be completely random where this happens. I'm wondering if I need to talk to the server over VPN each iteration and maybe the connection is flaky or if something has gone wrong with my ODBC driver (FreeTDS).
mysql_query and odbc_exec both return a resource which (quote from php.net) "is a special variable, holding a reference to an external resource." This suggests the server is talking with the database server every iteration, I am not sure though.
However, there are 2 connections we are talking about here. The first being your connection with the PHP server, and the second one being the connection between the PHP server and the database server. If both servers have a fast connection, the strange behaviour you are experiencing might not have anything to do with your VPN.
The resource identifies the internal data structure used by PHP for interacting with the external resource.
In the case of the resource returned by mysql_query(), this data structure will include the rows returned by the query (and won't return until all the data has been returned or the conenction fails). However this behaviour is specific to MySQL - there is no requirement that the DBMS return the data before it is explicitly requested by the client.
If there is some strange problem causing lots of latency in your setup, then the only obvious solution would be to compile the results of the query at the database side then deliver them to your PHP code, aither batched or as a whole (think webservice).
C.