I sometimes gets mysql deadlock errors saying:
'Deadlock found when trying to get lock; try restarting transaction'
I have a queues table where multiple php processes are running simultaneously selecting rows from the table. However, for each process i want it grab a unique batch of rows each fetch so i don't have any overlapping rows being selected.
so i run this query: (which is the query i get the deadlock error on)
$this->db->query("START TRANSACTION;");
$sql = " SELECT mailer_queue_id
FROM mailer_queues
WHERE process_id IS NULL
LIMIT 250
FOR UPDATE;";
...
$sql = "UPDATE mailer_queues
SET process_id = 33044,
status = 'COMPLETED'
WHERE mailer_queue_id
IN (1,2,3...);";
...
if($this->db->affected_rows() > 0) {
$this->db->query("COMMIT;");
} else{
$this->db->query("ROLLBACK;");
}
I'm also:
inserting rows to the table (with no transactions/locks) at the same time
updating rows in the table (with no transactions/locks) at the same time
deleting the rows from the table (with no transactions/locks) at the same time
As well, my updates and deletes only update and delete rows where they have a process_id assigned to them ...and where i perform my transactions that "SELECT rows ... FOR UPDATE" are where the process_id = null. In theory they should never be overlapping.
I'm wondering if there is a proper way to avoid these deadlocks?
Can a deadlock occur because one transaction is locking the table for too long while its selecting/update and the another process is trying to perform the same transaction and just timesout?
any help is much appreciated
Deadlocks occur when two or more processes requests locks in such a way that the resources being locked overlap, but occur in different orders, so that each process is waiting for a resource that's locked by another process, and that other process is waiting for a lock that the original process has open.
In real world terms, consider a construction site: You've got one screwdriver, and one screw. Two workers need to drive in a screw. Worker #1 grabs the screwdriver, and worker #2 grabs the screw. Worker #1 goes to grab the screw as well, but can't, because it's being held by worker #2. Worker #2 needs the screwdriver, but can't get it because worker #1 is holding it. So now they're deadlocked, unable to proceed, because they've got 1 of the 2 resources they need, and neither of them will be polite and "step back".
Given that you've got out-of-transaction changes occurring, it's possible that one (or more) of your updates/deletes are overlapping the locked areas you're reserving inside the transactions.
You might want to try LOCK TABLES before starting the transaction, thereby assuring you have explicit control over the tables. The lock will wait until all activity on the particular tables has completed.
I think everyone on net has explained very well about the deadlock.
Mysql provide very good log to check all the last dead lock happened and which
queries were stuck at that time.
Check this mysql documentation page and search for LATEST DETECTED DEADLOCK
its a great logs, helped finding many subtle deadlocks.
Related
I am pretty new to database design and PostgreSQL in general, but I am aware of the general concept behind row versioning, transactions and exclusive locks in Postgres (e.g. this article gives a pretty good overview).
My current problem is that a) I am not sure why I get so many exclusive locks show up in my PG database log files and b) why these locks are happening at all.
I run PostgreSQL 10 (+ PostGIS extension) with about 300 Million rows over 5 tables (200GB). I have about 5 scripts (4x PHP and 1x Python Psycopg2) running 24/7 that make a lot of inserts (and DO UPDATE with COALESCE, in case the entry already exist). However, as far as I understand, PHP Postgres extension commits automatically after each SQL Query and in my Python Script, increasing commits does not significantly reduce locks. I have a couple of triggers that dynamically update rows, but as far as I can tell from the log files they are not the cause of locks. It should generally be very rare that two or more of my scripts insert/update the same row at the same time.
This is an example log entry:
2018-01-31 01:04:02 CET [808]: [258-1] user=user1,db=maindb,app=[unknown],client=::1 LOG: process 808 still waiting for ExclusiveLock on page 0 of relation 26889 of database 16387 after 1015.576 ms
2018-01-31 01:04:02 CET [808]: [259-1] user=user1,db=maindb,app=[unknown],client=::1 DETAIL: Process holding the lock: 680. Wait queue: 1728, 152, 808.
2018-01-31 01:04:02 CET [808]: [260-1] user=user1,db=maindb,app=[unknown],client=::1 STATEMENT:
INSERT INTO "table1" (...)
VALUES (...)
ON CONFLICT (...)
DO UPDATE SET
...;
I have similar log entries about every 2-3 Minutes. Are they problematic? What do they mean exactly, are the locks finally resolved or is the data of the transaction lost? There is no log entry that states that locks are resolved or updates are finally committed to the database.
The second type of frequent log entry is similar to this:
2018-01-31 07:22:16 CET [2504]: [16384-1] user=,db=,app=,client= LOG: checkpoint complete: wrote 9999 buffers (3.8%); 0 WAL file(s) added, 0 removed, 7 recycled; write=269.842 s, sync=0.218 s, total=270.123 s; sync files=85, longest=0.054 s, average=0.002 s; distance=66521 kB, estimate=203482 kB
2018-01-31 07:22:46 CET [2504]: [16385-1] user=,db=,app=,client= LOG: checkpoint starting: time
Does this mean Auto-Vaccum or Auto-Commit that resolves all locks?
My general question: should I be concerned and do something or simply leave things as they are?
After a while, I found out what is causing these locks and also how to solve them. All Exclusive locks happened on one relation in the database:
...ExclusiveLock on page 0 of relation 26889 of database...
What is 26889?
SELECT relname FROM pg_class WHERE OID=26889
Result: idx_post_hashtags
It was all caused by a GIN index over a particular column with Array (Text). Furthermore, this GIN index was useless, as the array had a variable length and looking up any specific array value did not benefit from the index. I dropped it: all Exclusive Locks gone!
Reading the logs carefully really helps.
I have a daemon that runs background jobs requested by our webservice. We have 4 workers running simultaneously.
Sometimes a job is executed twice at the same time, because two workers decided to run that job. To avoid this situation we tried several things:
Since our jobs comes from our databases, we added a flag called executed, that prevents other works to get a job that has already been started to execute; This does not solve the problem, sometimes the delay with our database is enough to have simultaneous executions;
Added memcached in the system (all workers run in the same system), but somehow we had simultaneous jobs running today -- memcached does not solve for multiple servers as well.
Here is the following logic we are currently using:
// We create our memcached server
$memcached = new Memcached();
$memcached->addServer("127.0.0.1", 11211);
// Checkup every 5 seconds for operations
while (true) {
// Gather all operations TODO
// In this query, we do not accept operations that are set
// as executed already.
$result = findDaemonOperationsPendingQuery();
// We have some results!
if (mysqli_num_rows($result) > 0) {
$op = mysqli_fetch_assoc($result);
echo "Found an operation todo #" . $op['id'] . "\n";
// Set operation as executed
setDaemonOperationAsDone($op['id'], 'executed');
// Verifies if operation is happening on memcached
if (get_memcached_operation($memcached, $op['id'])) {
echo "\tOperation id already executing...\n";
continue;
} else {
// Set operation on memcached
set_memcached_operation($memcached, $op['id']);
}
... do our stuff
}
}
How this kind of problem is usually solved?
I looked up on the internet and found out a library called Gearman, but I'm not convinced that it will solve my problems when we have multiple servers.
Another thing I thought was to predefine a daemon to run the operation at insertion, and create a failsafe exclusive daemon that runs operations set by daemons that are out of service.
Any ideas?
Thanks.
An alternative solution to using locks and transactions, assuming each worker has an id.
In your loop run:
UPDATE operations SET worker_id = :wid WHERE worker_id IS NULL LIMIT 1;
SELECT * FROM operations where executed = 0 and worker_id = :wid;
The update is a single operation which is atomic and you are only setting worker_id if it is not yet set so no worries about race conditions. Setting the worker_id makes it clear who owns the operation. The update will only assign one operation because of the LIMIT 1.
You have a typical concurrency problem.
Worker 1 reads the table, select a job
Worker 1 update the table to mark the job as 'assigned' or whatever
Oh but wait, between 1 and 2, worker 2 read the table as well, and since the job wasn't yet marked a 'assigned', worker 2 selected the same job
The way to solve this is to use transactions and locks, in particular SELECT.. FOR UPDATE. It'll go like this:
Worker 1 starts a transaction (START TRANSACTION) and tries to acquire an exclusive lock SELECT * FROM jobs [...] FOR UPDATE
Worker 2 does the same. Except he has to wait because Worker 1 already has the lock.
Worker 1 updates the table to say he's now working on the job and commit the transaction immediately. This releases the lock for other workers to select jobs. Worker 1 can now safely start working on this job.
Worker 2 can now read the table and acquire a lock. Since the table has been updated, worker 2 will select a different job.
EDIT: Specific comment about your PHP code:
Your comment says you are fetching all the jobs that needs to be done at once in each worker. You should only select one, do it, select one, do it, etc.
You are setting the flag 'executed' when in fact it's not (yet) executed. You need a 'assigned' flag, and a different 'executed' flag.
I am writing an application which shall track the financial transactions (as in a bank), to maintain the balance amount. I am using Denormalizing technique to keep the performance in check(and not have to calculate the balance at runtime) as discussed Here and Here.
Now, I am facing a Race Condition if two people simultaneously did a transaction related to same entity, the balance calculation as discussed above, shall return/set inconsistent data, as discussed Here and Here, And also as suggested in the answers..
I am going for Mysql Transactions.
Now My question is,
What Happens to the other similar queries when a mysql Transaction is underway?
I wish to know if other transactions fail as in Error 500 or are they queued and executed, once the first transaction finishes.
I also need to know how to deal with the either result from the php point of view.
And since these transactions are going to be an element of a larger set of operation in php with many prior insert queries, should I also device a mechanism to roll-back those successfully executed queries too, since I want Atomicity not only as in individual queries but also as in whole operation logic(php).
Edit 1 :-
Also, if former is the case, should I check for the error, and wait a few second and try that specific transaction query again after some time?
Edit 2 :-
Also Mysql Triggers is not an option for me.
With code like this, there is no race condition. Instead, one transaction could be aborted (ROLLBACK'd).
BEGIN;
SELECT balance FROM Accounts WHERE acct_id = 123 FOR UPDATE;
if balance < 100, then ROLLBACK and exit with "insufficient funds"
UPDATE Accounts SET balance = balance - 100 WHERE acct_id = 123;
UPDATE Accounts SET balance = balance + 100 WHERE acct_id = 456;
COMMIT;
And check for errors at each step. If error, ROLLBACK and rerun the transaction. On the second time, it will probably succeed. If not, then abort -- it is probably a logic bug. Only then should you give http error 500.
When two users "simultaneously" try to do similar transactions, then one of these things will happen:
The 'second' user will be stalled until the first finishes.
If that stall is more than innodb_lock_wait_timeout, your queries are too slow or something else. You need to fix the system.
If you get a "Deadlock", there may be ways to repair the code. Meanwhile, simply restarting the transaction is likely to succeed.
But it will not mess up the data (assuming the logic is correct).
There is no need to "wait a second" -- unless you have transactions that take "a second". Such would be terribly slow for this type of code.
What I am saying works for "real money", non-real money, non-money, etc.; whatever you need to tally carefully.
I have a process that selects the next item to process from a MySQL InnoDB Table based on some criteria. When a row has been selected as the next to process, it's processing field is set to 1 while processing is happening outside the database. I do this so that many processors can be run at once, and they won't process the same row.
If I use transactions to execute the following queries, are they guaranteed to be executed together ( eg. Without any other MySQL connections executing queries. )? If they are not, then multiple processors could get the same id from the SELECT query and then processing will be redundant.
Pseudo Code Example
Prepare Transaction...
$id = SELECT id
FROM companies
WHERE processing = 0
ORDER BY last_crawled ASC
LIMIT 1;
UPDATE companies
SET processing = 1
WHERE id = $id;
Execute Transaction
I've been struggling to accomplish this fast enough using a single UPDATE query ( see this question ). Assume that is not an option for the purposes of this question.
You still have a possibility of a race condition, even though you execute the SELECT followed by the UPDATE in a single transaction. SELECT by itself does not lock anything, so you could have two concurrent sessions both SELECT and get the same id. Then both would attempt to UPDATE, but only one would "win" - the other would have to wait.
To get around this, use the SELECT...FOR UPDATE clause, which creates a lock on the rows it returns.
Prepare Transaction...
$id = SELECT id
FROM companies
WHERE processing = 0
ORDER BY last_crawled ASC
LIMIT 1
FOR UPDATE;
This means that the lock is created as the row is selected. This is atomic, which means no other session can sneak in and get a lock on the same row. If they try, their transaction will block on the SELECT.
UPDATE companies
SET processing = 1
WHERE id = $id;
Commit Transaction
I changed your "execute transaction" pseudocode to "commit transaction." Statements within a transaction execute immediately, which means they create locks and so on. Then when you COMMIT, the locks are released and any changes are committed. Committed means they can't be rolled back, and they are visible to other transactions.
Here's a quick example of using mysqli to accomplish this:
$mysqli = new mysqli(...);
$mysqli->report_mode = MYSQLI_REPORT_STRICT; /* throw exception on error */
$mysqli->begin_transaction();
$sql = "SELECT id
FROM companies
WHERE processing = 0
ORDER BY last_crawled ASC
LIMIT 1
FOR UPDATE";
$result = $mysqli->query($sql);
while ($row = $result->fetch_array(MYSQLI_ASSOC)) {
$id = $row["id"];
}
$sql = "UPDATE companies
SET processing = 1
WHERE id = ?";
$stmt = $mysqli->prepare($sql);
$stmt->bind_param("i", $id);
$stmt->execute();
$mysqli->commit();
Re your comment:
I tried an experiment and created a table companies, filled it with 512 rows, then started a transaction and issues the SELECT...FOR UPDATE statement above. I did this in the mysql client, no need to write PHP code.
Then, before committing my transaction, I examined the locks reported:
mysql> show engine innodb status\G
=====================================
2013-12-04 16:01:28 7f6a00117700 INNODB MONITOR OUTPUT
=====================================
...
---TRANSACTION 30012, ACTIVE 2 sec
2 lock struct(s), heap size 376, 513 row lock(s)
...
Despite using LIMIT 1, this report shows transaction appears to lock every row in the table (plus 1, for some reason).
So you're right, if you have hundreds of requests per second, it's likely that the transactions are queuing up. You should be able to verify this by watching SHOW PROCESSLIST and seeing many processes stuck in a state of Locked (i.e. waiting for access to rows that another thread has locked).
If you have hundreds of requests per second, you may have outgrown the ability for an RDBMS to function as a fake message queue. This isn't what an RDBMS is good at.
There are a variety of scalable message queue frameworks with good integration with PHP, like RabbitMQ, STOMP, AMQP, Gearman, Beanstalk.
Check out http://www.slideshare.net/mwillbanks/message-queues-a-primer-international-php-conference-fall-2012
That depends. There are (in general) differet isolation levels in SQL. In MySQL you can change which one to use using SET TRANSACTION ISOLATION LEVEL.
While "SERIALIZABLE" (which is the strictest one) still doesn't imply that no other actions are executed in between the ones from your transaction, it DOES make sure that there is no difference if simultanious transactions are executed one after another or not - if it would make a difference, on transaction is rolled back and executed later.
Note however that the stricter the isolation is, the more locking and rollbacks has to be done. So makre sure you really need that before using it.
What I want to do is to execute the same script every few minutes with cron.
The script needs to process some data read from the database, so obviously I need it work on diffrent row each time.
My concept was to use row locking to make sure each instance work on different row, but it doesn't seem to work that way.
Is it even possible to use row locks this way? Any other solutins?
Example:
while($c < $limit) {
$sql=mysql_query("SELECT * FROM table WHERE ... LIMIT 1 FOR UPDATE");
$data=mysql_fetch_assoc($sql);
(process data)
mysql_query("update table set value=spmething, timestamp=NOW()");
$c++;
}
Basically what i need is SCRIPT1 reads R1 from the table; SCRIPT2 reads R2 (next non-locked row matching criteria)
EDIT:
Let's say for example that:
1) the table stores a list of URL
2) the script checks if URL responses, and updates it's status (and timestamp) in database
This should essentially be treated as two separate problems:
Finding a job for each worker to process. Ideally this should be very efficient and pre-emptively avoid failures in step 2, which comes next.
Ensuring that each job gets processed at most once or exactly once. No matter what happens the same job should not be concurrently processed by multiple workers. You may want to ensure that no jobs are lost due to buggy/crashing workers.
Both problems have multiple workable solutions. I'll give some suggestions about my preference:
Finding a job to process
For low-velocity systems it should be sufficient just to look for the most recent un-processed job. You do not want to take the job yet, just identify it as a candidate. This could be:
SELECT id FROM jobs ORDER BY created_at ASC LIMIT 1
(Note that this will process the oldest job first—FIFO order—and we assume that rows are deleted after processing.)
Claiming a job
In this simple example, this would be as simple as (note I am avoiding some potential optimizations that will make things less clear):
BEGIN;
SELECT * FROM jobs WHERE id = <id> FOR UPDATE;
DELETE FROM jobs WHERE id = <id>;
COMMIT;
If the SELECT returns our job when queried by id, we've now locked it. If another worker has already taken this job, an empty set will be returned, and we should look for a different job. If two workers are competing for the same job, they will block each other from the SELECT ... FOR UPDATE onwards, such that the previous statements are universally true. This will allow you to ensure that each job is processed at most once. However...
Processing a job exactly once
A risk in the previous design is that a worker takes a job, fails to process it, and crashes. The job is now lost. Most job processing systems therefor do not delete the job when they claim it, instead marking it as claimed by some worker and implement a job-reclaim system.
This can be achieved by keeping track of the claim itself using either additional columns in the job table, or a separate claim table. Normally some information is written about the worker, e.g. hostname, PID, etc., (claim_description) and some expiration date (claim_expires_at) is provided for the claim e.g. 1 hour in the future. An additional process then goes through those claims and transactionally releases claims which are past their expiration (claim_expires_at < NOW()). Claiming a job then also requires that the job row is checked for claims (claim_expires_at IS NULL) both at selection time and when claiming with SELECT ... FOR UPDATE.
Note that this solution still has problems: If a job is processed successfully, but the worker crashes before successfully marking the job as completed, we may eventually release the claim and re-process the job. Fixing that requires a more advanced system which is left as an exercise for the reader. ;)
If you are going to read the row once, and only once, then I would create an is_processed column and simply update that column on the rows that you've processed. Then you can simply query for the first row that has is_processed = 0