I have one table that is read at the same time by different threads.
Each thread must select 100 rows, execute some tasks on each row (unrelated to the database) then they must delete the selected row from the table.
rows are selected using this query:
SELECT id FROM table_name FOR UPDATE;
My question is: How can I ignore (or skip) rows that were previously locked using a select statement in MySQL ?
I typically create a process_id column that is default NULL and then have each thread use a unique identifier to do the following:
UPDATE table_name SET process_id = #{process.id} WHERE process_id IS NULL LIMIT 100;
SELECT id FROM table_name WHERE process_id = #{process.id} FOR UPDATE;
That ensures that each thread selects a unique set of rows from the table.
Hope this helps.
Even though it is not the best solution, as there is no way that I know to ignore locked rows, I select a random one and try to obtain a lock.
START TRANSACTION;
SET #v1 =(SELECT myId FROM tests.table WHERE status is NULL LIMIT 1);
SELECT * FROM tests.table WHERE myId=#v1 FOR UPDATE; #<- lock
Setting a small timeout for the transaction, if that row is locked the transaction is aborted and I try another one. If I obtain the lock, I process it. If (bad luck) that row was locked, it is processed and the lock is released before my timeout, I then select a row that has already been 'processed'! However, I check a field that my processes set (e.g. status): if the other process transaction ended OK, that field tells me that work has already been done and I do not process that row again.
Every other possible solution without transactions (e.g. setting another field if the row has no status and ... etc.) can easily provide race conditions and missed processes (e.g. one thread abruptly dies, the allocated data is still tagged, while a transaction expires; ref. comment here
Hope it helps
Related
I have MySQL (InnoDB) table with the column is_locked which shows current state of the record (is it being handled by system now, or not).
On the other hand, I have many nodes that perform SELECT * FROM table_name WHERE is_locked = 0 and then handles got rows from this table.
In my code I do this:
System takes the row from DB (SELECT * FROM table_name WHERE is_locked = 0)
System lockes the row by command UPDATE table_name SET is_locked = 1 WHERE id = <id>
Problem:
Nodes are working very fast, all of them may get the same row, before first of them will update the row and set is_locked to 1
I found out LOCKING of the tables, but I don't think it is the right way.
Can anybody tell me, how to handle such cases?
I recommend two things:
Limit your select to one, as you're dealing with concurrency issues, it is better to take smaller "bites" with each iteration
Use transactions, this allows you to start the transaction, get the record, lock it and then commit the transaction. This will force mysql to enforce your concurrency locks.
I'm working on a project using a MySQL database as the back-end (accessed from PHP). Sometimes, I select a row, do some operations on it, and then update the record in the database.
I am worried that another user could have initiated a similar process on the same row right after the first select, and his changes could overwrite some of the changes the first user did (because the second user's select did not yet include those changes).
Is this an actual problem? Should I lock the table, and won't this severely impact my application's performance? Any other solutions?
Just to be thorough with my information, I also have some CRON jobs running that could also be modifying the same data.
Thanks!
I can think of two solutions, other than explicitly using transactions:
Use SELECT .. FOR UPDATE : http://dev.mysql.com/doc/refman/5.0/en/innodb-locking-reads.html
Manually change a value so the row is not select by other queries:
SET #update_id := 0;
UPDATE table_name SET status = 'IN_PROCESS', id = (SELECT #update_id := id) WHERE status = 'WAITING' AND [your condition] LIMIT 1;
SELECT #update_id;
Here, the rows to be selected must have the value of status="WAITING". And when this query runs, it selects the ID, and changes the value of 'status', so the row can't be selected by other queries.
Does the following code really disable concurrent execution?
LOCK TABLES codes WRITE;
INSERT INTO codes VALUES ();
SELECT mid FROM codes ORDER BY expired DESC LIMIT 0,1;
UNLOCK TABLES;
The PHP will execute SQL. The PHP file will be requested by many users via HTTP. Would it really execute in isolation for every users?
mid is something which has to be unique for every user so I think I should use MySQL locks to achieve that.
If you have a table with an auto incremented key and you use mysql_insert_id just after insertion, it is guaranteed to be unique and it won't mix user threads (it fetches the ID on the connection you give). No need to lock the table.
http://php.net/manual/en/function.mysql-insert-id.php
i have a code like this
reserve.php
$r=mysql_query("select count(*) from ticket");
$rec=mysql_fetch_array($r);
if ($rec[0]==0)
{
insert into ticket values .....
}
i have 1 ticket only.
two users request reserve.php.
"a" user request reserve.php and available ticket is 0 . but before insert, for "b" user available ticket is 0 yet. so two users reserve ticket.
table is Innodb.
how to prevent this? transaction , lock table or what?
In these situations I usually just use an UPDATE statement and check how many records were affected (mysql_affected_rows) by the update.
UPDATE tickets SET ticket_count=ticket_count-1 WHERE ticket_id=123 AND ticket_count>0
If someone else decremented the counter first, then no update occurs and you don't give the user a ticket. Autocommit should be enabled so the update is a self-contained "transaction".
Alternatively, you can change your SELECT to be SELECT ... FOR UPDATE
http://dev.mysql.com/doc/refman/5.0/en/innodb-locking-reads.html
I would do it in a single transaction, without a roundtrip back to the application level - both SELECT count() .... and INSERT INTO being sent in a single command from PHP and embedded into a TRANSACTION with EXCLUSIVE LOCK
I have to do some network IO based on every row in a table with more than 70 million rows. Since high TPS is needed i have created a php script that does this task for a single row in table. I plan to call this php script using a cron job about 40 times every second. How do I do this so that no two script access the same row.
To do it purely based on the table, you will need to set something In the table - a boolean, timestamp, deleting the row, etc - that indicates that you've processed the row. After that, a transaction is all you need.
START TRANSACTION;
SELECT * FROM table WHERE processing = 0 ORDER BY id ASC LIMIT 1 FOR UPDATE;
UPDATE table SET processing = 1 WHERE id = $id_of_what_we_got;
COMMIT;
-- process row here
-- optionally, tell the db we're done
UPDATE table SET processing = 2 WHERE id = $id_of_what_we_got;
Just make sure to use the same MySQL connection (PHP resource) for the entire transaction.
Further reading:
http://dev.mysql.com/doc/refman/5.0/en/set-transaction.html
http://dev.mysql.com/doc/refman/5.0/en/innodb-locking-reads.html
https://github.com/ryandotsmith/Queue-Classic/blob/master/lib/queue_classic/durable_array.rb