I have a daemon that runs background jobs requested by our webservice. We have 4 workers running simultaneously.
Sometimes a job is executed twice at the same time, because two workers decided to run that job. To avoid this situation we tried several things:
Since our jobs comes from our databases, we added a flag called executed, that prevents other works to get a job that has already been started to execute; This does not solve the problem, sometimes the delay with our database is enough to have simultaneous executions;
Added memcached in the system (all workers run in the same system), but somehow we had simultaneous jobs running today -- memcached does not solve for multiple servers as well.
Here is the following logic we are currently using:
// We create our memcached server
$memcached = new Memcached();
$memcached->addServer("127.0.0.1", 11211);
// Checkup every 5 seconds for operations
while (true) {
// Gather all operations TODO
// In this query, we do not accept operations that are set
// as executed already.
$result = findDaemonOperationsPendingQuery();
// We have some results!
if (mysqli_num_rows($result) > 0) {
$op = mysqli_fetch_assoc($result);
echo "Found an operation todo #" . $op['id'] . "\n";
// Set operation as executed
setDaemonOperationAsDone($op['id'], 'executed');
// Verifies if operation is happening on memcached
if (get_memcached_operation($memcached, $op['id'])) {
echo "\tOperation id already executing...\n";
continue;
} else {
// Set operation on memcached
set_memcached_operation($memcached, $op['id']);
}
... do our stuff
}
}
How this kind of problem is usually solved?
I looked up on the internet and found out a library called Gearman, but I'm not convinced that it will solve my problems when we have multiple servers.
Another thing I thought was to predefine a daemon to run the operation at insertion, and create a failsafe exclusive daemon that runs operations set by daemons that are out of service.
Any ideas?
Thanks.
An alternative solution to using locks and transactions, assuming each worker has an id.
In your loop run:
UPDATE operations SET worker_id = :wid WHERE worker_id IS NULL LIMIT 1;
SELECT * FROM operations where executed = 0 and worker_id = :wid;
The update is a single operation which is atomic and you are only setting worker_id if it is not yet set so no worries about race conditions. Setting the worker_id makes it clear who owns the operation. The update will only assign one operation because of the LIMIT 1.
You have a typical concurrency problem.
Worker 1 reads the table, select a job
Worker 1 update the table to mark the job as 'assigned' or whatever
Oh but wait, between 1 and 2, worker 2 read the table as well, and since the job wasn't yet marked a 'assigned', worker 2 selected the same job
The way to solve this is to use transactions and locks, in particular SELECT.. FOR UPDATE. It'll go like this:
Worker 1 starts a transaction (START TRANSACTION) and tries to acquire an exclusive lock SELECT * FROM jobs [...] FOR UPDATE
Worker 2 does the same. Except he has to wait because Worker 1 already has the lock.
Worker 1 updates the table to say he's now working on the job and commit the transaction immediately. This releases the lock for other workers to select jobs. Worker 1 can now safely start working on this job.
Worker 2 can now read the table and acquire a lock. Since the table has been updated, worker 2 will select a different job.
EDIT: Specific comment about your PHP code:
Your comment says you are fetching all the jobs that needs to be done at once in each worker. You should only select one, do it, select one, do it, etc.
You are setting the flag 'executed' when in fact it's not (yet) executed. You need a 'assigned' flag, and a different 'executed' flag.
Related
I have a process that selects the next item to process from a MySQL InnoDB Table based on some criteria. When a row has been selected as the next to process, it's processing field is set to 1 while processing is happening outside the database. I do this so that many processors can be run at once, and they won't process the same row.
If I use transactions to execute the following queries, are they guaranteed to be executed together ( eg. Without any other MySQL connections executing queries. )? If they are not, then multiple processors could get the same id from the SELECT query and then processing will be redundant.
Pseudo Code Example
Prepare Transaction...
$id = SELECT id
FROM companies
WHERE processing = 0
ORDER BY last_crawled ASC
LIMIT 1;
UPDATE companies
SET processing = 1
WHERE id = $id;
Execute Transaction
I've been struggling to accomplish this fast enough using a single UPDATE query ( see this question ). Assume that is not an option for the purposes of this question.
You still have a possibility of a race condition, even though you execute the SELECT followed by the UPDATE in a single transaction. SELECT by itself does not lock anything, so you could have two concurrent sessions both SELECT and get the same id. Then both would attempt to UPDATE, but only one would "win" - the other would have to wait.
To get around this, use the SELECT...FOR UPDATE clause, which creates a lock on the rows it returns.
Prepare Transaction...
$id = SELECT id
FROM companies
WHERE processing = 0
ORDER BY last_crawled ASC
LIMIT 1
FOR UPDATE;
This means that the lock is created as the row is selected. This is atomic, which means no other session can sneak in and get a lock on the same row. If they try, their transaction will block on the SELECT.
UPDATE companies
SET processing = 1
WHERE id = $id;
Commit Transaction
I changed your "execute transaction" pseudocode to "commit transaction." Statements within a transaction execute immediately, which means they create locks and so on. Then when you COMMIT, the locks are released and any changes are committed. Committed means they can't be rolled back, and they are visible to other transactions.
Here's a quick example of using mysqli to accomplish this:
$mysqli = new mysqli(...);
$mysqli->report_mode = MYSQLI_REPORT_STRICT; /* throw exception on error */
$mysqli->begin_transaction();
$sql = "SELECT id
FROM companies
WHERE processing = 0
ORDER BY last_crawled ASC
LIMIT 1
FOR UPDATE";
$result = $mysqli->query($sql);
while ($row = $result->fetch_array(MYSQLI_ASSOC)) {
$id = $row["id"];
}
$sql = "UPDATE companies
SET processing = 1
WHERE id = ?";
$stmt = $mysqli->prepare($sql);
$stmt->bind_param("i", $id);
$stmt->execute();
$mysqli->commit();
Re your comment:
I tried an experiment and created a table companies, filled it with 512 rows, then started a transaction and issues the SELECT...FOR UPDATE statement above. I did this in the mysql client, no need to write PHP code.
Then, before committing my transaction, I examined the locks reported:
mysql> show engine innodb status\G
=====================================
2013-12-04 16:01:28 7f6a00117700 INNODB MONITOR OUTPUT
=====================================
...
---TRANSACTION 30012, ACTIVE 2 sec
2 lock struct(s), heap size 376, 513 row lock(s)
...
Despite using LIMIT 1, this report shows transaction appears to lock every row in the table (plus 1, for some reason).
So you're right, if you have hundreds of requests per second, it's likely that the transactions are queuing up. You should be able to verify this by watching SHOW PROCESSLIST and seeing many processes stuck in a state of Locked (i.e. waiting for access to rows that another thread has locked).
If you have hundreds of requests per second, you may have outgrown the ability for an RDBMS to function as a fake message queue. This isn't what an RDBMS is good at.
There are a variety of scalable message queue frameworks with good integration with PHP, like RabbitMQ, STOMP, AMQP, Gearman, Beanstalk.
Check out http://www.slideshare.net/mwillbanks/message-queues-a-primer-international-php-conference-fall-2012
That depends. There are (in general) differet isolation levels in SQL. In MySQL you can change which one to use using SET TRANSACTION ISOLATION LEVEL.
While "SERIALIZABLE" (which is the strictest one) still doesn't imply that no other actions are executed in between the ones from your transaction, it DOES make sure that there is no difference if simultanious transactions are executed one after another or not - if it would make a difference, on transaction is rolled back and executed later.
Note however that the stricter the isolation is, the more locking and rollbacks has to be done. So makre sure you really need that before using it.
What I want to do is to execute the same script every few minutes with cron.
The script needs to process some data read from the database, so obviously I need it work on diffrent row each time.
My concept was to use row locking to make sure each instance work on different row, but it doesn't seem to work that way.
Is it even possible to use row locks this way? Any other solutins?
Example:
while($c < $limit) {
$sql=mysql_query("SELECT * FROM table WHERE ... LIMIT 1 FOR UPDATE");
$data=mysql_fetch_assoc($sql);
(process data)
mysql_query("update table set value=spmething, timestamp=NOW()");
$c++;
}
Basically what i need is SCRIPT1 reads R1 from the table; SCRIPT2 reads R2 (next non-locked row matching criteria)
EDIT:
Let's say for example that:
1) the table stores a list of URL
2) the script checks if URL responses, and updates it's status (and timestamp) in database
This should essentially be treated as two separate problems:
Finding a job for each worker to process. Ideally this should be very efficient and pre-emptively avoid failures in step 2, which comes next.
Ensuring that each job gets processed at most once or exactly once. No matter what happens the same job should not be concurrently processed by multiple workers. You may want to ensure that no jobs are lost due to buggy/crashing workers.
Both problems have multiple workable solutions. I'll give some suggestions about my preference:
Finding a job to process
For low-velocity systems it should be sufficient just to look for the most recent un-processed job. You do not want to take the job yet, just identify it as a candidate. This could be:
SELECT id FROM jobs ORDER BY created_at ASC LIMIT 1
(Note that this will process the oldest job first—FIFO order—and we assume that rows are deleted after processing.)
Claiming a job
In this simple example, this would be as simple as (note I am avoiding some potential optimizations that will make things less clear):
BEGIN;
SELECT * FROM jobs WHERE id = <id> FOR UPDATE;
DELETE FROM jobs WHERE id = <id>;
COMMIT;
If the SELECT returns our job when queried by id, we've now locked it. If another worker has already taken this job, an empty set will be returned, and we should look for a different job. If two workers are competing for the same job, they will block each other from the SELECT ... FOR UPDATE onwards, such that the previous statements are universally true. This will allow you to ensure that each job is processed at most once. However...
Processing a job exactly once
A risk in the previous design is that a worker takes a job, fails to process it, and crashes. The job is now lost. Most job processing systems therefor do not delete the job when they claim it, instead marking it as claimed by some worker and implement a job-reclaim system.
This can be achieved by keeping track of the claim itself using either additional columns in the job table, or a separate claim table. Normally some information is written about the worker, e.g. hostname, PID, etc., (claim_description) and some expiration date (claim_expires_at) is provided for the claim e.g. 1 hour in the future. An additional process then goes through those claims and transactionally releases claims which are past their expiration (claim_expires_at < NOW()). Claiming a job then also requires that the job row is checked for claims (claim_expires_at IS NULL) both at selection time and when claiming with SELECT ... FOR UPDATE.
Note that this solution still has problems: If a job is processed successfully, but the worker crashes before successfully marking the job as completed, we may eventually release the claim and re-process the job. Fixing that requires a more advanced system which is left as an exercise for the reader. ;)
If you are going to read the row once, and only once, then I would create an is_processed column and simply update that column on the rows that you've processed. Then you can simply query for the first row that has is_processed = 0
As I have installed cron run the same script every minute. At the same time a script is executed several times.
There is such a part:
$query = mysql_query("select distinct `task_id` from tasks_pending where `checked`='0' and `taken`='0' limit 50");
Then, the obtained values set "taken = 1"
Since several processes are executed at the same time the request Return the same data for different processes. Is it possible to somehow disable this part of her time to can perform only one process?
Sorry for bad English.
Use SELECT FOR UPDATE and it will block another process to select the same rows, before they are updated
You might want to lock the table before doing anything with it, and unlock it afterwards using the [UN]LOCK TABLES statements.
Otherwise, you could use a SELECT FOR UPDATE query within a transaction scope.
I have a php script that executes mysql pdo queries. There are a few reads and writes to the same table in this script.
For sake of example let's say that there are 4 queries, a read, write, another read, another write, each read takes 10 second to execute, and each write takes .1 seconds to execute.
If I execute this script from the cli nohup php execute_queries.php & twice in 1/100th of a second, what would the execution order of the queries be?
Would all the queries from the first instance of the script need to finish before the queries from the 2nd instance begin to run, or would the first read from both instances start and finish before the table is locked by the write?
NOTE: assume that I'm using myisam and that the write is an update to a record (IE, entire table gets locked during the write.)
Since you are not using transactions, then no, the won't wait for all the queries in one script to finish an so the queries may get overlaped.
There is an entire field of study called concurrent programming that teaches this.
In databases it's about transactions, isolation levels and data locks.
Typical (simple) race condition:
$visits = $pdo->query('SELECT visits FROM articles WHERE id = 44')->fetch()[0]['visits'];
/*
* do some time-consuming thing here
*
*/
$visits++;
$pdo->exec('UPDATE articles SET visits = '.$visits.' WHERE id = 44');
The above race condition can easily turn sour if 2 PHP processes read the visits from the database one millisecond after the other, and assuming the initial value of visits was 6, both would increment it to 7 and both would write 7 back into the database even though the desired effect was that 2 visits increment the value by 2 (final value of visits should've been 8).
The solution to this is using atomic operations (because the operation is simple and can be reduced to one single atomic operation).
UPDATE articles SET visits = visits+1 WHERE id = 44;
Atomic operations are guaranteed by the database engines to take place uninterrupted by other processes/threads. Usually the database has to queue incoming updates so that they don't affect each other. Queuing obviously slows things down because each process has to wait for all processes before it until it gets the chance to be executed.
In a less simple operation we need more than one statement:
SELECT #visits := visits FROM articles WHERE ID = 44;
SET #visits = #visits+1;
UPDATE articles SET visits = #visits WHERE ID = 44;
But again even at the database level 3 separate atomic statements are not guaranteed to yield an atomic result. They can be overlap with other operations. Just like the PHP example.
To solve this you have to do the following:
START TRANSACTION
SELECT #visits := visits FROM articles WHERE ID = 44 FOR UPDATE;
SET #visits = #visits+1;
UPDATE articles SET visits = #visits WHERE ID = 44;
COMMIT;
I sometimes gets mysql deadlock errors saying:
'Deadlock found when trying to get lock; try restarting transaction'
I have a queues table where multiple php processes are running simultaneously selecting rows from the table. However, for each process i want it grab a unique batch of rows each fetch so i don't have any overlapping rows being selected.
so i run this query: (which is the query i get the deadlock error on)
$this->db->query("START TRANSACTION;");
$sql = " SELECT mailer_queue_id
FROM mailer_queues
WHERE process_id IS NULL
LIMIT 250
FOR UPDATE;";
...
$sql = "UPDATE mailer_queues
SET process_id = 33044,
status = 'COMPLETED'
WHERE mailer_queue_id
IN (1,2,3...);";
...
if($this->db->affected_rows() > 0) {
$this->db->query("COMMIT;");
} else{
$this->db->query("ROLLBACK;");
}
I'm also:
inserting rows to the table (with no transactions/locks) at the same time
updating rows in the table (with no transactions/locks) at the same time
deleting the rows from the table (with no transactions/locks) at the same time
As well, my updates and deletes only update and delete rows where they have a process_id assigned to them ...and where i perform my transactions that "SELECT rows ... FOR UPDATE" are where the process_id = null. In theory they should never be overlapping.
I'm wondering if there is a proper way to avoid these deadlocks?
Can a deadlock occur because one transaction is locking the table for too long while its selecting/update and the another process is trying to perform the same transaction and just timesout?
any help is much appreciated
Deadlocks occur when two or more processes requests locks in such a way that the resources being locked overlap, but occur in different orders, so that each process is waiting for a resource that's locked by another process, and that other process is waiting for a lock that the original process has open.
In real world terms, consider a construction site: You've got one screwdriver, and one screw. Two workers need to drive in a screw. Worker #1 grabs the screwdriver, and worker #2 grabs the screw. Worker #1 goes to grab the screw as well, but can't, because it's being held by worker #2. Worker #2 needs the screwdriver, but can't get it because worker #1 is holding it. So now they're deadlocked, unable to proceed, because they've got 1 of the 2 resources they need, and neither of them will be polite and "step back".
Given that you've got out-of-transaction changes occurring, it's possible that one (or more) of your updates/deletes are overlapping the locked areas you're reserving inside the transactions.
You might want to try LOCK TABLES before starting the transaction, thereby assuring you have explicit control over the tables. The lock will wait until all activity on the particular tables has completed.
I think everyone on net has explained very well about the deadlock.
Mysql provide very good log to check all the last dead lock happened and which
queries were stuck at that time.
Check this mysql documentation page and search for LATEST DETECTED DEADLOCK
its a great logs, helped finding many subtle deadlocks.