I need to do two updates to rows but I need to make sure they are done together and that no other query from another user could interfere with them. I know about SELECT...FOR UPDATE but I imagine after the first update it will of course be unlocked which means someone could interfere with the second update. If someone else updates that row first, the update will work but will mess up the data. Is there anyway to ensure that the two updates happen how they are supposed to? I have been told about transactions but as far as I know they are only good for making sure the two updates actually happen and not whether they happen "together," unless I am mistaken and the rows will be locked until the transaction is committed?
Here are the queries:
SELECT z FROM table WHERE id='$id'
UPDATE table SET x=x+2 WHERE x>z
UPDATE table SET y=y+2 WHERE y>z
I made a mistake and didn't give full information. That was my fault. I have updated the queries. The issue I have is that z can be updated as well. If z is updated after the SELECT but before the other two updates, the data can get messed up. Does doing the transaction BEGIN/COMMIT work for that?
Learn about TRANSACTION
http://dev.mysql.com/doc/refman/5.0/en/commit.html
[... connect ...]
mysql_query("BEGIN");
$query1 = mysql_query('UPDATE table SET x=x+2 WHERE x>y');
$query2 = mysql_query('UPDATE table SET y=y+2 WHERE y>y');
if($query1 && $query2) {
mysql_query("COMMIT");
echo 'Save Done. All UPDATES done.';
} else {
mysql_query("ROLLBACK");
echo 'Error Save. All UPDATES reverted, and not done.';
}
There are various levels of transaction, but basically as per ACID properties you should expect that within a given transaction, all reads and updates are performed consistently meaning it will be kept in a valid state, but more importantly a transaction is isolated in that work being done in another transaction (thread) will not interfere with your transaction (your grouping of select & update SQL statements): this allows you to take a broad assumption that you are the only thread of execution within the system allowing you to commit that group of work (atomically) or roll it all back.
Each database may handle the semantics differently (some may lock rows or columns, some may re-order, some may serialize) but that's the beauty of a declarative database interface: you worry about the work you want to get done.
As stated, on MySQL InnoDB is transactional and will support what is mentioned above so ensure your tables are organized with InnoDB, other non-transactional engines (e.g. MyISAM) are not transactional and thus will force you to manage those transactional semantics (locking) manually.
One approach would be to lock the entire table:
LOCK TABLE `table` WRITE;
SELECT z FROM `table` WHERE id='$id';
UPDATE `table` SET x=x+2 WHERE x>z;
UPDATE `table` SET y=y+2 WHERE y>z;
UNLOCK TABLES;
This will prevent other sessions from writing, and reading, from the table table during the SELECTs and UPDATEs.
Whether this is an appropriate solution does depend on how appropriate it is for sessions to wait to read or write from the table.
Related
I did a lot of research and I found a lot of information about all the relevant topics. However, I am not confident that I do now understand, how to put all this information together properly.
This application is written in PHP.
For queries I use PDO.
The MySQL database is configured as InnoDB.
What I need
SELECT ... FROM tableA;
// PHP looks at what comes back and does some logic.
INSERT INTO tableA ...;
INSERT INTO tableB ...;
Conditions:
The INSERTs need to be atomic. If one of them fails I want to roll back.
No reads and writes from/to tableA are allowed to happen between the SELECT and the INSERT from/into tableA.
This to me looks like a very simple problem. Yet I am not able to figure out, how to do this properly. So my question is:
What is the best approach?
This is an outline for my current plan, heavily simplified:
try {
SET autocommit = 0;
BLOCK TABLES tableA WRITE, tableB WRITE;
SELECT ... FROM tableA;
INSERT INTO tableA ...;
INSERT INTO tableB ...;
COMMIT;
UNLOCK TABLES;
SET autocommit = 1;
}
catch {
ROLLBACK;
UNLOCK TABLES;
SET autocommit = 1;
}
I feel like there is a lot that could be done better, but I don't know how :/
Why do it like this?
I need some kind of transaction to be able to do a rollback if INSERTs fail.
I need to lock tableA to make sure that no other INSERTs or UPDATEs take place.
Transactions and table locks don't work well together
(https://dev.mysql.com/doc/refman/8.0/en/lock-tables-and-transactions.html)
I want to use autocommit as a standard throughout the rest of my application, which is why I set it back to "1" at the end.
I am really not sure about this: But I somewhere picked up, that after locking a table, I can (from within the current connection) only query to this table until I unlock it (this does not make sense to me). This is why I locked tableB too, altough otherwise I wouldn't need to.
I am open for completely different approaches
I am open for any suggestion within the framework conditions PHP, MySQL, PDO and InnoDB.
Thank You!
Edit 1 (2018-06-01)
I feel like my problem/question needs some more clarification.
Starting point:
If have two tables, t1 and t2.
t1 has multiple columns of non-unique values.
The specifics of t2 are irrelevant for this problem.
What I want to do:
Step by step:
Select multiple columns and rows from t1.
In PHP analyse the retrieved data. Based on the results of this analysis put together a dataset.
INSERT parts of the dataset into t1 and parts of it into t2.
Additional information:
The INSERTs into the 2 tables must be atomic. This can be achieved using transactions.
No INSERTs from a different connection are allowed to occur between steps 1 and 3. This is very important, because every single INSERT into t1 has to occur with full awareness of the current state of the table. I'll best describe this in more detail. I will leave t2 out of this for now, to make things easier to understand.
Imagine this sequence of events (connections con1 and con2):
con1: SELECT ... FROM t1 WHERE xyz;
con1: PHP processes the information.
con2: SELECT ... FROM t1 WHERE uvw;
con2: PHP processes the information.
con1: INSERT INTO t1 ...;
con2: INSERT INTO t1 ...;
So both connections see t1 in the same state. However, they select different information. Con1 takes the information gathered, does some logic with it and then INSERTs data into a new row in t1. Con2 does the same, but using different information.
The problem is this: Both connections INSERTed data based on calculations that did not take into account whatever the other connection inserted into t1, because this information wasn't there when they read from t1.
Con2 might have inserted a row into t1 that would have met the WHERE-conditions of con1's SELECT-statement. In other words: Had con2 inserted its row earlier, con1 might have created completely different data to insert into t1. This is to say: The two INSERTs might have completely invalidated each others inserts.
This is why I want to make sure, that only one connection can work with the data in t1 at a time. No other connection is allowed to write, but also no other connection is allowed to read until the current connection is done.
I hope this clarifies things a bit... :/
Thoughts:
My thoughts were:
I need to make the INSERTs into the 2 tables atomic. --> I will use a transaction for this. Something like this:
try {
$pdo->beginTransaction();
// INSERT INTO t1 ...
// INSERT INTO t2 ...
$pdo->commit();
}
catch (Exception $e) {
$pdo->rollBack();
throw $e;
}
I need to make sure, no other connection writes to or reads from t1. This is where I decided that I need LOCK TABLES.
Assuming I had to use LOCK TABLES, I was confronted with the problem that LOCK TABLES is not transaction aware. Which is why I decided to go with the solution proposed here (https://dev.mysql.com/doc/refman/8.0/en/lock-tables-and-transactions.html) and also in multiple answers on stackoverflow.
But I wasn't happy with how the code looked like, which is why I came here to ask this (meanwhile rather lengthy) question.
Edit 2 (2018-06-01)
This process will not run often. So there is no significant need for high performance and effiency. This, of course, also means that the chances of two of those processes infering with eachother are rather minute. Stil, I'd like to make sure nothing can happen.
Case 1:
BEGIN;
INSERT ..
INSERT ..
COMMIT;
Other connections will not see the inserted rows until after the commit. That is, BEGIN...COMMIT made the two inserts "atomic".
If anything fails, you still need the try/catch to deal with it.
Do not use LOCK TABLES on InnoDB tables.
Don't bother with autocommit; BEGIN..COMMIT overrides it.
My statements apply to (probably) all frameworks. (Except that some do not have "try" and "catch".)
Case 2: Lock a row in anticipation of possibly modifying it:
BEGIN;
SELECT ... FROM t1 FOR UPDATE;
... work with the values SELECTed
UPDATE t1 ...;
COMMIT;
This keeps others away from the rows SELECTed until after the COMMIT.
Case 3: Sometimes IODKU is useful to do two things in a single atomic statement:
INSERT ...
ON DUPLICATE KEY UPDATE ...
instead of
BEGIN;
SELECT ... FOR UPDATE;
if no row found
INSERT ...;
else
UPDATE ...;
COMMIT;
Class 4: Classic banking example:
BEGIN;
UPDATE accounts SET balance = balance - 1000.00 WHERE id='me';
... What if crash occurs here? ...
UPDATE accounts SET balance = balance + 1000.00 WHERE id='you';
COMMIT;
If the system crashes between the two UPDATEs, the first update will be undone. This keeps the system from losing track of the funds transfer.
Case 5: Perhaps close to what the OP wants. It is mostly a combination of Cases 2 and 1.
BEGIN;
SELECT ... FROM t1 FOR UPDATE; -- see note below
... work with the values SELECTed
INSERT INTO t1 ...;
COMMIT;
Notes on Case 5: The SELECT..FOR UPDATE must include any rows that you don't want the other connection to see. This has the effect of delaying the other connection until this connection COMMITs. (Yes, this feels a lot like LOCK TABLES t1 WRITE.)
Case 6: The "processing" that needs to be inside the BEGIN..COMMIT will take too long. (Example: the typical online shopping cart.)
This needs a locking mechanism outside of InnoDB's transactions. One way (useful for shopping cart) is to use a row in some extra table, and have everyone check it. Another way (more practical within a single connection) is to use GET_LOCK('foo') and it's friends.
General Discussion
All of the above examples lock only the row(s) involved, not the entire table(s). This makes the action much less invasive, and allows for the system to handle much more activity.
Also, read about MVCC. This is a general technique used under the cover to let one connection see the values of the table(s) at some instant in time, even while other connections are modifying the table(s).
"Prevent inserts" -- With MVCC, if you start a SELECT, it is like getting a snapshot in time of everything you are looking at. You won't see the INSERTs until after you complete the transaction that the SELECT is in. You can have your cake and eat it, too. That is, it appears as if the inserts were blocked, but you get the performance benefit of them happening in parallel. Magic.
I am building a PHP RESTful-API for remote "worker" machines to self-assign tasks. The MySQL InnoDB table on the API host holds pending records that the workers can pick up from the API whenever they are ready to work on a record. How do I prevent concurrently requesting worker system from ever getting the same record?
My initial plan to prevent this is to UPDATE a single record with a uniquely generated ID in a default NULL field, and then poll for the details of the record where the unique ID field matches.
For example:
UPDATE mytable SET status = 'Assigned', uniqueidfield = '3kj29slsad'
WHERE uniqueidfield IS NULL LIMIT 1
And in the same PHP instance, the next query:
SELECT id, status, etc FROM mytable WHERE uniqueidfield = '3kj29slsad'
The resulting record from the SELECT statement above is then given to the worker. Would this prevent simultaneously requesting workers from getting the same records shown to them? I am not exactly sure on how MySQL handles the lookups within an UPDATE query, and if two UPDATES could "find" the same record, and then update it sequentially. If this works, is there a more elegant or standardized way of doing this (not sure if FOR UPDATE would need to be applied to this)? Thanks!
Nevermind my previous answer. I believe I understand what you are asking. I'll reword it so maybe it is clearer to others.
"If I issue two of the above update statements at the same time, what would happen?"
According to http://dev.mysql.com/doc/refman/5.0/en/lock-tables-restrictions.html, the second statement would not interfere with the first one.
Normally, you do not need to lock tables, because all single UPDATE
statements are atomic; no other session can interfere with any other
currently executing SQL statement.
A more elegant way is probably opinion based, but I don't see anything wrong with what you're doing.
This table has two fields, Price and amount.
I am performing a query on this innodb table which Selects for update all records whose price is less than 100 (query 1).
Whilst this query is going on, I want to be able to insert or update a row in the table which has a price greather 100. Is this certain to work?
At the same time, I want to guarantee that any attempts to update or insert rows into the table, whose price is less than 100 will not happen until (query 1) is complete. Is this also certain?
If I want to perform ordinary select * from queries on this table whilst queries of type 1 are happening, and queries in question 2, will I be able to do these reads or will these reads have to wait until the locks are off? This has to be certain. edit: I would like these selects to be able to access any record in the table
Finally, are indexes crucial to get this working? Or will it work as is?
Many thanks.
If your table only has two fields (price and amount), this seems… bad. Surely there should be some sort of other identifier in there?
Yes. SELECT FOR UPDATE only locks the records it selected.
Maybe. Depending on your transaction isolation mode, MySQL may lock an index range to block inserts into the range that was scanned as part of your query.
This depends on the transaction isolation mode you use. If you are using a higher isolation mode, locking the rows for update will prevent them from being read, as your transaction may or may not end up making changes to them, depending on whether it is committed. See the MySQL manual's section on "SELECT ... FOR UPDATE and SELECT ... LOCK IN SHARE MODE Locking Reads" for details.
Yes. The index range locking mentioned in part 2 depends on there being an index. Running unindexed queries is a bad idea, anyway — avoid it.
I have mysql table fg_stock. Most of the time concurrent access is happening in this table. I used this code but it doesn't work:
<?php
mysql_query("LOCK TABLES fg_stock READ");
$select=mysql_query("SELECT stock FROM fg_stock WHERE Item='$item'");
while($res=mysql_fetch_array($select))
{
$stock=$res['stock'];
$close_stock=$stock+$qty_in;
$update=mysql_query("UPDATE fg_stock SET stock='$close_stock' WHERE Item='$item' LIMIT 1");
}
mysql_query("UNLOCK TABLES");
?>
Is this okay?
"Most of the time concurrent access is happening in this table"
So why would you want to lock the ENTIRE table when it's clear you are attempting to access a specific row from the table (WHERE Item='$item')? Chances are you are running a MyISAM storage engine for the table in question, you should look into using the InnoDB engine instead, as one of it's strong points is that it supports row level locking so you don't need to lock the entire table.
Why do you need to lock your table anyway?????
mysql_query("UPDATE fg_stock SET stock=stock+$qty_in WHERE Item='$item'");
That's it! No need in locking the table and no need in unnecessary loop with set of queries. Just try to avoid SQL Injection by using intval php function on $qty_in (if it is an integer, of course), for example.
And, probably, time concurrent access is only happens due to non-optimized work with database, with the excessive number of queries.
ps: moreover, your example does not make any sense as mysql could update the same record all the time in the loop. You did not tell MySQL which record exactly do you want to update. Only told to update one record with Item='$item'. At the next iteration the SAME record could be updated again as MySQL does not know about the difference between already updated records and those that it did not touched yet.
http://dev.mysql.com/doc/refman/5.0/en/internal-locking.html
mysql> LOCK TABLES real_table WRITE, temp_table WRITE;
mysql> INSERT INTO real_table SELECT * FROM temp_table;
mysql> DELETE FROM temp_table;
mysql> UNLOCK TABLES;
So your syntax is correct.
Also from another question:
Troubleshooting: You can test for table lock success by trying to work
with another table that is not locked. If you obtained the lock,
trying to write to a table that was not included in the lock statement
should generate an error.
You may want to consider an alternative solution. Instead of locking,
perform an update that includes the changed elements as part of the
where clause. If the data that you are changing has changed since you
read it, the update will "fail" and return zero rows modified. This
eliminates the table lock, and all the messy horrors that may come
with it, including deadlocks.
PHP, mysqli, and table locks?
Just to give you an example:
I have a PHP script that manages users votes.
When a user votes, the script makes a query to check if someone has already voted for the same ID/product. If nobody has voted, then it makes another query and insert the ID into a general ID votes table and another one to insert the data into a per user ID votes table. And this kind of behavior is repeated in other kind of scripts.
The question is, if two different users votes simultaneously its possible that the two instances of the code try to insert a new ID (or some similar type of query) that will give an error??
If yes, how I prevent this from happening?
Thanks?
Important note: I'm using MyISAM! My web hosting don't allow InnoDB.
The question is, if two different users votes simultaneously its possible that the two instances of the
code try to insert a new ID (or some similar type of query) that will give an erro
Yes, you might end up with two queries doing the insert. Depending on the constraints on the table, one of them will either generate an error, or you'll end up with two rows in your database.
You could solve this, I believe, with applying some locking;
e.g. if you need to add a vote to the product with id theProductId:(pseudo code)
START TRANSACTION;
//lock on the row for our product id (assumes the product really exists)
select 1 from products where id=theProductId for update;
//assume the vote exist, and increment the no.of votes
update votes set numberOfVotes = numberOfVotes + 1 where productId=theProductId ;
//if the last update didn't affect any rows, the row didn't exist
if(rowsAffected == 0)
insert into votes(numberOfVotes,productId) values(1,theProductId )
//insert the new vote in the per user votes
insert into user_votes(productId,userId) values(theProductId,theUserId);
COMMIT;
Some more info here
MySQL offers another solution as well, that might be applicable here, insert on duplicate
e.g. you might be able to just do:
insert into votes(numberOfVotes,productId) values(1,theProductId ) on duplicate key
update numberOfVotes = numberOfVotes + 1;
If your votes table have a unique key on the product id column, the above will
do an insert if the particular theProductId doesn't exist, otherwise it will do an update, where it increments the numberOfVotes column by 1
You could probably avoid a lot of this if you created a row in the votes table at the same time you added the product to the database. That way you could be sure there's always a row for your product, and just issue an UPDATE on that row.
The question is, if two different
users votes simultaneously its
possible that the two instances of the
code try to insert a new ID (or some
similar type of query) that will give
an error??
Yes, in general this is possible. This is an example of a very common problem in concurrent systems, called a race condition.
Avoiding it can be rather tricky, but in general you need to make sure that the operations cannot interleave in the way you describe, e.g. by locking the database for a while.
There are several practical solutions to this, all with their own advantages and risks (e.g. dead locks). See the Wikipedia article for a discussion and further pointers to information.
The easiest way:
LOCK TABLES table1 WRITE, table2 WRITE, table3 WRITE
-- check for record, insert if not exists, etc...
UNLOCK TABLES
If voting does not occur many times per second, then the above should be sufficient.
InnoDB tables offer transactions, which might be useful here as well. Others have already commented on it, so I won't go into any detail.
Alternatively, you could solve it at the code level via using some sort of shared memory mutex that disables concurrent execution of that section of PHP code.
This when the singleton pattern come in handy. It ensure that a code is executed only by one process at an instant.
http://en.wikipedia.org/wiki/Singleton_pattern
You have to make a singleton class for the database access this will prevent you from the type of error you describing.
Cheers.