I'm updating an old script using php mysql functions to use mysqli and I've already found an interesting question (mysqli_use_result() and concurrency) but it doesn't clarify one thing:
Lets say five users connects to a webpage, they connected at same time and user1 was the first to select the data from a huge MyIsam database (15000 records of forum posts with left join to 'users' and 'attachments' table).
while the php scripts retrieves the result, the other users will not be able to get results, is that right?
Also using the same situation above, when user1 fully received it's result, an 'UPDATE set view_count = INC(1)' query is sent and the table is locked I suppose, and this same query will fail for the other users?
About the article you've quoted. It just means, that you should not do this:
mysqli_query($link, $query);
mysqli_use_result($link);
// lots of 'client processing'
// table is blocked for updates during this
sleep(10)
mysqli_fetch_* .....
In such situtations you are adviced to do so:
mysqli_query($link, $query);
mysqli_store_result($link);
// lots of 'client processing'
// table is NOT blocked for updates during this
sleep(10)
mysqli_fetch_* .....
The article further says, that if a second query will be issued - after calling mysql_use_result() and before fetching the results from the query it will fail. This is meant per connection - per script. So other user's queries won't fail during this.
while the php scripts retrieves the result, the other users will not be able to get results, is that right?
No this is not right. MySQL supports as many parallel connections as you have configured in my.ini max_connections. Concurrent reads are handled by the mysql server. Client code has not to worry about that unless the max connection limit is reached and mysqli_connect() would fail. If your application reaches a point where this happens frequently you'll in most cases first try to tweak your mysql config so that mysql allows more parrallel connections. If a threshold is reached you'll use an attempt like replication or mysql cluster.
Also using the same situation above, when user1 fully received it's result, an 'UPDATE set view_count = INC(1)' query is sent and the table is locked I suppose, and this same query will fail for the other users?
When there are concurrent reads and writes this is of course a performance issue. But the MySQL server handle this for you, meaning the client code has not worry about it as long as connecting to mysql works. If you have really high load you'll mostly use master slave Replication or MySQL cluster.
and this same query will fail for the other users?
A database server usually a bit more intelligent than a plain text file.
So, your queries won't fail. They'd wait instead.
Though I wouldn't use mysqli_use_result() at all.
Why not just fetch your results already?
Related
When the web server receives a request for my PHP script, I presume the server creates a dedicated process to run the script. If, before the script exits, another request to the same script comes, another process gets started -- am I correct, or the second request will be queued in the server, waiting for the first request to exit? (Question 1)
If the former is correct, i.e. the same script can run simultaneously in a different process, then they will try to access my database.
When I connect to the database in the script:
$DB = mysqli_connect("localhost", ...);
query it, conduct more or less lengthy calculations and update it, I don't want the contents of the database to be modified by another instance of a running script.
Question 2: Does it mean that since connecting to the database until closing it:
mysqli_close($DB);
the database is blocked for any access from other software components? If so, it effectively prevents the script instances from running concurrently.
UPDATE: #OllieJones kindly explained that the database was not blocked.
Let's consider the following scenario. The script in the first process discovers an eligible user in the Users table and starts preparing data to append for that user in the Counter table. At this moment the script in the other process preempts and deletes the user from the Users table and the associate data in the Counter table; it then gets preempted by the first script which writes the data for the user no more existing. These data become in the head-detached state, i.e. unaccessible.
How to prevent such a contention?
In modern web servers, there's a pool of processes (or possibly threads) handling requests from users. Concurrent requests to the same script can run concurrently. Each request-handler has its own connection to the DBMS (they're actually maintained in a pool, but that's a story for another day).
The database is not blocked while individual request-handlers are using it, unless you block it explicitly by locking a table or doing a request like SELECT ... FOR UPDATE. For more information on this deep topic, read about transactions.
Therefore, it's important to write your database queries in such a way that they won't interfere with each other. For example, if you need to learn the value of an auto-incremented column right after you insert a row, you should use LAST_INSERT_ID() or mysqli_insert_id() instead of trying to query the data base: another user may have inserted another row in the meantime.
The system test discipline for scaled-up web sites usually involves a rigorous load test in order to shake out all this concurrency.
If you're doing a bunch of work on a particular entity, in your case a User, you use a transaction.
First you do
BEGIN
to start the transaction. Then you do
SELECT whatever FROM User WHERE user_id = <<whatever>> FOR UPDATE
to choose the user and mark that user's row as busy-being-updated. Then you do all the work you need to do to fill out various rows in various tables relating to that user.
Finally you do
COMMIT
If you messed things up, or don't want to go through with the change, you do
ROLLBACK
and all your changes will be restored to their state right before the SELECT ... FOR UPDATE.
Why does this work? Because if another client does the same SELECT .... FOR UPDATE, MySQL will delay that request until the first one either gets COMMIT or ROLLBACK.
If another client works with a different userid, the operations may proceed concurrently.
You need the InnoDB access method to use transactions: MyISAM doesn't support them.
Multiple reads can be done concurrently, if there is a write operation then it will block all other operations. A read will block all writes.
I'm using the mysql_insert_id within my code to get an auto increment.
I have read around and it looks like there is no race condition regarding this for different user connections, but what about the same user? Will I be likely to run into race condition problems when connecting to the database using the same username/user but still from different connection sessions?
My application is PHP. When a user submits a web request my PHP executes code and for that particular request/connection session I keep a persistent SQL connection open in to MySQL for the length of that request. Will this cause me any race condition problems?
None for any practical purpose, If you execute the last_id request right after executing your insert then there is practically not enough time for another insert to spoil that. Theoretically might be
possible
According to PHP Manual
Note:
Because mysql_insert_id() acts on the last performed query, be sure to
call mysql_insert_id() immediately after the query that generates the
value.
Just in case you want to double check you can use this function to confirm your previous query
mysql_info
The use of persistent connections doesn't mean that every request will use the same connection. It means that each apache thread will have its own connection that is shared between all requests executing on that thread.
The requests will run serially (one after another) which means that the same persistent connection will not be used by two threads running at the same time.
Because of this, your last_insert_id value will be safe, but be sure that you check the result of your inserts before using it, because it will return the last_insert_id of the last successful INSERT, even if it wasn't the last executed INSERT.
Now I must start by saying, I can't copy the string. This is a general question.
I've got a query with several joins in that takes 0.9 seconds when run using the mysql CLI. I'm now trying to run the same query on a PHP site and it's taking 8 seconds. There are some other big joins on the site that are obviously slower, but this string is taking much too long. Is there a PHP cache for database connections that I need to increase? Or is this just to be expected.
PHP doesn't really do much with MySQL; it sends a query string to the server, and processes the results. The only bottleneck here is if it's an absolutely vast query string, or if you're getting a lot of results back - PHP has to cache and parse them into an object (or array, depending on which mysql_fetch_* you use). Without knowing what your query or results are like, I can only guess.
(From comments): If we have 30 columns and around, say, a million rows, this will take an age to parse (we later find that it's only 10k rows, so that's ok). You need to rethink how you do things:-
See if you can reduce the result set. If you're paginating things, you can use LIMIT clauses.
Do more in the MySQL query; instead of getting PHP to sort the results, let MySQL do it by using ORDER BY clauses.
Reduce the number of columns you fetch by explicitly specifying each column name instead of SELECT * FROM ....
Some wild guesses:
The PHP-version uses different parameters and variables each query: MySQL cannot cache it. While the version you type on the MySQL-CLI uses the same parameter: MySQL can fetch it from its cache. Try adding the SQL_NO_CACHE to your query on CLI to see if the result requires more time.
You are not testing on the same machine? Is the MySQL database you test the PHP-MySQL query with and the CLI the same machine? I mean: you are not testing one on your laptop and the other one on some production server, are you?
You are testing over a network: When the MySQL server is not installed on the same host as your PHP app, you will see some MySQL connection that uses "someserver.tld" instead of "localhost" as database host. In that case PHP will need to connect over a network, while your CLI already has that connection, or connects only local.
The actual connection is taking a long time. Try to run and time the query from your PHP-system a thousand times after each other. Instead of "connect to server, query database, disconnect", you should see the query timing when it is "connect to server, query database thousand times, disconnect". Some PHP-applications do that: they connect and disconnect for each and every query. And if your MySQL server is not configured correctly, connecting can take a gigantic amount of the total time.
How are you timing it?
If the 'long' version is accessed through a php page on a website, could the additional 7.1 seconds not just be the time it takes to send the request and then process and render the results?
How are you connecting? Does the account you're using use a hostname in the grant tables? If you're connectinv via TCP, MySQL will have to do a reverse DNS lookup on your IP to figure out if you're allowed in.
If it's the connection causing this, then do a simple test:
select version();
if that takes 8seconds, then it's connection overhead. If it return instantly, then it's PHP overhead in processing the data you've fetched.
The function mysql_query should should take the same time as mysql client. But any extra mysql_fetch_* will add up.
I'm using PHP with MYSQL database as both are open source and easy to use.
I'm getting problem when I execute insert and/or update of millions of row one after another
while this operation perform I got the MYSQL error that:
'max_user_connections' active connections
which is the best way to solve this problem.
I don't want to use another database or language other then PHP.
connect_db();
$query = "insert into table(mobno,status,description,date,send_deltime,sms_id,msg,send_type) values('".$to."','".$status."','".$report."','','".$timewsha1."','".$smsID."','','".$type."')";
$result = mysql_query($query) or ("Query failed : " . mysql_error());
this query will execute thousand of times.
and then server give connection error.
First of all, try to know from your hosting server administrator about the max consecutive active connections available to the MySQL database. This is the most basic & primary information to have knowledge about.
If your page(s) load in a decent amount of time and release the connection once the page is loaded, it should be fine. The problem occurs when your script takes some long time to retrieve information from the database or maintains the connections.
Since you are executing INSERT and / or UPDATE operations of millions of rows, so you may have some problem.
Additionally, if you fail to close connections in your script(s), it is possible that someone will load a page and instead of closing the connection when the page is loaded, it is left open. No one else can then use that connection. So please make sure that at the end of execution of all the MySQL / SQL queries, the database connection is closed. Also please make sure that your server provides more than 250 connections, since 100 connections is available in almost all the servers generally.
Also make sure that you are not using the persistent connections (which is available when using the built-in function "mysql_pconnect()"), since this will lock up the user until the connection is manually closed.
Hope it helps.
//this loop is for preparing the subquery for mutiple records
for(// this loop for getting data for mutiple records){
$sub_query[] = "('".$to."','".$status."','".$report."','','".$timewsha1."','".$smsID."','','".$type."')";
}
$query = "insert into table(mobno,status,description,date,send_deltime,sms_id,msg,send_type) values ";
$query .= implode(',',$sub_query);
mysql_query($query );
So, a remote app calls into this script, sends it a list of values, and then this query is executed once, right? It's not in a foreach or for or while loop? When you say it will be executed millions of times, you don't mean in one sitting I mean. If it is in a loop, then move the db connect outside of the loop, otherwise it will attempt to connect again each time the loop iterates, and also, remember to call mysql_close at the end of the script, just in case.
mysql_pconnect() would create a persistent connection and that is the way to go if you don't want to exhaust your server's connection pool.
Is there a Mysql statement which provides full details of any other open connection or user? Or, an equally detailed status report on myisam tables specifically. Looking at Mysql's SHOW TABLE STATUS documentation, it's missing some very important information for my purpose.
What I'm trying to do: remote odbc connection one is inserting several thousand records, which due to a slow connection speed can take up to an hour. Tcp connection two, using PHP on the server's localhost, is running select queries with aggregate functions on that data. Before allowing connection two to run those queries, I'd like connection two to first check to make sure there's no pending inserts on any other connection on those specific tables so it can instead wait until all data is available. If the table is currently being written to, I'd like to spit back to the user of connection two an approximation of how much longer to wait based on the number of pending inserts. Ideally by table, I'd like to get back using a query the timestamp when connection one began the write, total inserts left to be done, and total inserts already completed. Instead of insert counts, even knowing number of bytes written and left to write would work just fine here.
Obviously since connection two is a tcp connection via a PHP script, all I can really use in that script is some sort of query. I suppose if I have to, since it is on localhost, I can exec() it if the only way is by a mysql command line option that outputs this info, but I'd rather not. I suppose I could simply update a custom-made transaction log before and after this massive insert task which the PHP script can check, but hopefully there's already a built-in Mysql feature I can take advantage of.
Edit: "Transaction" was the wrong word - a language collision. I'm not actually using Mysql transactions. What I meant was currently pending tasks, queries, and requests.
You can issue SHOW FULL PROCESSLIST; to show the active connections.
As for the rest, mysql doesn't know how many inserts are left, and how long they'll take. (And if you're using MyISAM tables, they dont support transactions). The server have no way of knowing whether your PHP scripts intend to send 10 more inserts, or 10000 - and if you're doing something like insert into xxx select ... from ... - mysql doesn't track/expose info on how much/many is done/is left .
You're better off handling this yourself via other tables where you update/insert data about when you started aggregating data, track the state,when it finished etc.
If the transactions are being performed on InnoDB tables, you can get full transaction details with SHOW INNODB STATUS. It's a huge blob of output, but part of it is transactions/lock status for each process/connection.