We want to prevent some concurrency issues in a database (we use event sourcing and want to insert an event to the event log in case the event is a valid one).
The problem is that we need to check if a certain operation is allowed (requiring a SELECT query and some checks in php) and then run a INSERT query to actually perform the operation.
Now, we can simply LOCK the entire table, do our checks and if they succeed, then insert (and remove the lock).
The problem with this is that it locks the entire table, which is overkill (there will be lots of queries on this table). What I would like to do is to lock all queries that want to do this select-insert operation for a specific object_id, but allow queries for all other object_id's to continue as if there is no lock.
I searched a bit but couldn't find a lock attribute command. There seems to be a lock row command in innoDB, but it's not really what we want (I think I'm not 100% sure what it does).
We can of course try to manually handle the locks (check if there exists some column with object_id in some seperate lock table and wait untill there is none), but that feels a bit fishy and error prone.
So, here's the actual question: is it possible to lock a table for a specific value of a column (object_id)?
It would be awesome if the lock only held for the specific SELECT-INSERT queries, and not for standalone SELECT's, but that doesn't matter that much for now.
Consider manual arbitrary locks with GET_LOCK();
Choose a name specific to the rows you want locking. e.g. 'xxx_event_id_y'. Where 'xxx' is a string specific to the procedure and table and 'y' is the event id.
Call SELECT GET_LOCK('xxx_event_id_y',30) to lock the name 'xxx_event_id_y'.. it will return 1 and set the lock if the name becomes available, or return 0 if the lock is not available after 30 seconds (the second parameter is the timeout).
Use DO RELEASE_LOCK('xxx_event_id_y') when you are finished.
Be aware; You will have to use the same names in each transaction that you want to wait and calling GET_LOCK() again in a transaction will release the previously set lock.
GET_LOCK() docs
I actually use this method to lock our application cache too (even when it doesn't use the DB), so it has scope outside the database as well.
Migrate tables to innodb if not already done, and use transactions.
Related
Im trying to find a solution to a MariaDB race condition.
There are some cases where multiple process get executed almost at the same time (with the same timestamp that comes from some units in the field) and they need to read the same row each one (but only the first one needs to perform the action).
I was thinking to lock the row (only the row, locking the entire table is not an option), so the first process will read it and check the timestamp of the latest update, perform the task needed and once is unlocked the other processes will be able to read it, and if it has the same timestamp just ignore it.
It will affect performance, but only for those few cases where this happens.
I've been trying to do this in MariaDB, but I can't find the how... starting a transaction and executing FOR UPDATE seems to lock the entire table, because I'm not able to fetch another rows meanwhile.
And reading MariaDB documentation I see this
When LOCK IN SHARE MODE is specified in a SELECT statement, MariaDB will wait until all transactions that have modified the rows are committed. Then, a write lock is acquired. All transactions can read the rows, but if they want to modify them, they have to wait until your transaction is committed.
So basically, what I want to do can't be done in MariaDB?
Any ideas?
Thank you,
I've seen many posts explaining the usage of Select FOR UPDATE and how to lock a row, however I haven't been able to find any that explain what occurs when the code tries to read a row that's locked.
For instance. Say I use the following:
$con->autocommit(FALSE);
$ps = $con->prepare( "SELECT 1 FROM event WHERE row_id = 100 FOR UPDATE");
$ps->execute();
...
//do something if lock successful
...
$mysqli->commit();
In this case, how do I determine if my lock was successful? What is the best way to handle a scenario when the row is locked already?
Sorry if this is described somewhere, but all I seem to find are the 'happy path' explanations out there.
In this case, how do I determine if my lock was successful? What is the best way to handle a scenario when the row is locked already?
If the row you are trying to lock is already locked - the mysql server will not return any response for this row. It will wait², until the locking transaction is either commited or rolled back.
(Obviously: if the row has been deleted already, your SELECT will return an empty result set and not lock anything)
After that, it will return the latest value, commited by the transaction that was holding the lock.
Regular Select Statements will not care about the lock and return the current value, ignoring that there is a uncommited change.
So, in other words: your code will only be executed WHEN the lock is successfull. (Otherwhise waiting² until the prior lock is released)
Note, that using FOR UPDATE will also block any transactional SELECTS for the time beeing locked - If you do not want this, you should use LOCK IN SHARE MODE instead. This would allow transactional selects to proceed with the current value, while just blocking any update or delete statement.
² the query will return an error, after the time defined with innodb_lock_wait_timeout http://dev.mysql.com/doc/refman/5.5/en/innodb-parameters.html#sysvar_innodb_lock_wait_timeout
It then will return ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction
In other words: That's the point where your attempt to acquire a lock fails.
Sidenode: This kind of locking is just suitable to ensure data-integrity. (I.e. that no referenced row is deleted while you are inserting something that references this row).
Once the lock is released any blocked (or better call it delayed) delete statement will be executed, maybe deleting the row you just inserted due to Cascading on the row on which you just held the lock to ensure integrity.
If you want to create a system to avoid 2 users modifying the same data at the same time, you should do this at an application level and look at pessimistic vs optimistic locking approches, because it is no good idea to keep transactions running for a long period of time. (I think in PHP your database connections are automatically closed after each request anyway, causing an implicit commit on any running transaction)
I have an application, running on php + mysql plattform, using Doctrine2 framework. I need to execute 3 db queries during one http request: first INSERT, second SELECT, third UPDATE. UPDATE is dependent on result of SELECT query. There is a high probability of concurrent http requests. If such situation occurs, and DB queries get mixed up (eg. INS1, INS2, SEL1, SEL2, UPD1, UPD2), it will result in data inconsistency. How do I assure atomicity of INS-SEL-UPD operation? Do I need to use some kind of locks, or transactions are sufficient?
The answer from #YaK is actually a good answer. You should know how to deal with locks in general.
Addressing Doctrine2 specifically, your code should look like:
$em->getConnection()->beginTransaction();
try {
$toUpdate = $em->find('Entity\WhichWillBeUpdated', $id, \Doctrine\DBAL\LockMode::PESSIMISTIC_WRITE);
// this will append FOR UPDATE http://docs.doctrine-project.org/en/2.0.x/reference/transactions-and-concurrency.html
$em->persist($anInsertedOne);
// you can flush here as well, to obtain the ID after insert if needed
$toUpdate->changeValue('new value');
$em->persist($toUpdate);
$em->flush();
$em->getConnection()->commit();
} catch (\Exception $e) {
$em->getConnection()->rollback();
throw $e;
}
The every subsequent request to fetch for update, will wait until this transaction finishes for one process which has acquired the lock. Mysql will release the lock automatically after transaction is finished successfully or failed. By default, innodb lock timeout is 50 seconds. So if your process does not finish transaction in 50 seconds it will rollback and release the lock automatically. You do not need any additional fields on your entity.
A table-wide LOCK is guaranteed to work in all situations. But they are quite bad because they kind of prevent concurrency, rather than deal with it.
However, if your script holds the locks for a very short time frame, it might be an acceptable solution.
If your table uses InnoDB engine (no support for transactions with MyISAM), transaction is the most efficient solution, but also the most complex.
For your very specific need (in the same table, first INSERT, second SELECT, third UPDATE dependending on result of SELECT query):
Start a transaction
INSERT your records. Other transactions will not see these new rows until your own transaction is committed (unless you use a non-standard isolation level)
SELECT your record(s) with SELECT...LOCK IN SHARE MODE. You now have a READ lock on these rows, no one else may change these rows. (*)
Compute whatever you need to compute to determine whether or not you need to UPDATE something.
UPDATE the rows if required.
Commit
Expect errors at any time. If a dead-lock is detected, MySQL may decide to ROLLBACK you transaction to escape the dead-lock. If another transaction is updating the rows you are trying to read from, your transaction may be locked for some time, or even time-out.
The atomicity of your transaction is guaranteed if you proceed this way.
(*) in general, rows not returned by this SELECT may still be inserted in a concurrent transaction, that is, the non-existence is not guaranteed throughout the course of the transaction unless proper precautions are taken
Transactions won't prevent thread B to read the values thread A has not locked
So you must use locks to prevent concurrency access.
#Gediminas explained how you can use locks with Doctrine.
But using locks can result in dead locks or lock timeouts.
Doctrine renders these SQL errors as RetryableExceptions.
These exceptions are often normal if you are in a high concurrency environment.
They can happen very often and your application should handle them properly.
Each time a RetryableException is thrown by Doctrine, the proper way to handle this is to retry the whole transaction.
As easy as it seems, there is a trap. The Doctrine 2 EntityManager becomes unusable after a RetryableException and you must recreate a new one to replay your whole transaction.
I wrote this article illustrated with a full example.
How to implement pessimistic locking in a php/mysql web application?
web-user opens a page to edit one dataset (row)
web-user clicks on the button "lock", so other users are able to read but not to write this dataset
web-user makes some modifications (takes maybe 1 to 30 minutes)
web-user clicks "save" or "cancel" and the "lock" is removed
Are there standard methods in php/mysql for this scenario? What happens if the web-user never clicks on "save"/"cancel" but closes the internet-exploror?
You need to implement a LOCKDATE and LOCKWHO field in your table. Ive done that in many applications outside of PHP/Mysql and it's always the same way.
The lock is terminated when the TTL has passed, so you could do a substraction of dates using NOW and LOCKDATE to see if the object has been locked for more than 30 minutes or 1h as you wish.
Another factor is to consider if the current user is the one locking the object. So thats why you also need a LOCKWHO. This can be a user_id from your database, a session_id from PHP. But keep it to something that identifies a user, an ipaddress is not a good way to do it.
Finaly, always think of a mass-unlock feature that simply resets all LOCKDATEs and LOCKWHOs...
Cheers
I would write the locks in one centralized table instead of adding fields to all tables.
Example table structure :
tblLocks
TableName (The name of tha locked table)
RowID (Primary key of locked table row)
LockDateTime (When the row was locked)
LockUser (Who locked the row)
With this approach you can find all locks that are made by a user without having to scan all tables. You could kill all locks when user logs out for example.
Traditionally this is done with a boolean locked column on the record in the database that is flagged appropriately.
It is a function of this sort of locking that the lock has to be released, and circumstances may prevent this happening naturally (system crashes, user stupidity, dropped network packets, etc etc etc). This is why you would need to provide some manual unlock method and/or impose a time limit (maybe with a cron job?) on how long a record can be locked for. You could implement some kind of AJAX poll to keep the record locked if the browser is still open? At any rate, you would probably be best to verify the data in the record is the same as it was when the lock was aquired before you modify it.
This limitation of this type of behaviour is particularly prevalent in web applications, but is true of anything that uses this approach - Sage Line 50, for one, is a bugger for it, I regularly have to delete lock files after machine/application crashes.
I have something similar to an online chat client. When one user sends a message it will be stored in a MySQL database table with id=currentID+1. I have another client long polling and waiting for message 'id=currentID+1'.
After this exchange, that row is never used again.
Do I need a lock in this case? The thing I'm worried about is that the reading PHP side will be able to see the row and read its values before the writing PHP side finishes creating the row.
MySQL won't make the row available until it's done reading (it automatically acquires a table lock in the case of MyISAM, or a row lock in the case of INNODB. So no, you should be ok so long as you're only inserting the row (and not later calling updates on it, etc)...
Writes in MySQL are atomic. Other queries cannot "see" the row until it is completely written, assuming you're using a single INSERT statement to do this.
the new row will only be seen by the select query after its inserted.
inserts are atomic
and yes myisam imploys table level locking, while innodb imploys row level locking.
and set of statements in a transaction are considered atmoic, there effect is not visible to any read until the transaction is committed, u can do a select shared to see uncommitted data.