I'm creating a web application where every row of a table needs to be processed. I'm spawning one child PHP process per table row. I'm implementing a safety mechanism, so if a PHP process is interrupted processing a row, a new PHP process will spawned to process said row. To do this I'm going to create a new table where all PHP processes check in every 10 seconds or so. I need MySQL to delete all rows that haven't been checked into for 5 minutes or more, so my application will know to create a new PHP child to process that row.
I know it's possible to get MySQL to run queries on an interval, but I don't know how.
~Enter stackoverflow~
Edit: I was hoping to learn how to do this 100% MySQL. Is there no way to set MySQL to run a query every hour, or at a specific time each day or such?
Crontab. You can run the query directly using the mysql client (mysql -uusername -ppassword dbname -e 'query here') or schedule a PHP script which runs the query.
DELETE FROM table WHERE checked_into < CURRENT_TIMESTAMP - INTERVAL 5 MINUTE
MySQL Events are tasks that run according to a schedule. Therefore, we sometimes refer to them as scheduled events. ... Conceptually, this is similar to the idea of the Unix crontab (also known as a “cron job”) or the Windows Task Scheduler.
http://dev.mysql.com/doc/refman/5.1/en/events-overview.html
And here is the lovely syntax: http://dev.mysql.com/doc/refman/5.1/en/create-event.html
One way to run MySQL queries on a certain interval would be to set up a cron job. Assuming you've got full access to your webserver, this should be doable. You'd just make a PHP page that does the SQL operations you want to occur every X time interval, and then set the script to run on that interval via cron jobs. More specifics: http://en.wikipedia.org/wiki/Cron
I think what you are looking for is an event scheduler, first introduced in MySQL 5.1.
On a side note, maybe you should redesign your program a little to avoid the extra layer of event scheduler:
Instead of deleting a row, where a process has not checked in for a while, just have a column with a check in timestamp. Then if some row has a very old check in timestamp, you can spawn a new PHP process for it.
Related
I have a database that has 20 rows each row I had set a Boolean value to it, so it is by default zero and when a row gets viewed its value changes to 1
I want the database to send any kind of signal that when 10 rows their value change from zero to 1, a certain PHP file fires up and starts a process that will affect only these 10 rows
How can I do that?
Thanks in advance
I would say, query from the php file every set amount of time to your database
The other way, database to execute a php file is almost impossible.
If you are using mySQL as database, a trigger could invoke the sys_exec() UDF available here: https://github.com/mysqludf/lib_mysqludf_sys#readme
So, there might be a possibility, actually, via an UDF function that would launch the php executable/script; not that easy, but seems possible ;-)
Invoking php from mysql is impossible, all you can do is set cron jobs for it. Cron job check mysql after certain interval of time and run the respected code
Every database is only a storage and it is its purpose in the system. Don't try to trigger any external process by the storage. The communication with the storage should be only a one way.
Rather think how to trigger your process from outside. Generally, there are two approaches:
a script that will check your database data in some interval like 1s, 10s, 1min or whatever would fit for a particular process
the current process that is updating your data can check your data and trigger another process if needed.
You can not trigger external file/script from mysql.
What you can do is create a cron job which run after certain interval of time which check database and perform certain operations.
I have to populate and update one of my MySql database table using a complex and expensive query, based on selection from other table's data. This table doesn't need to be always fully updated when i make a query on it, but i'd like to have a cyclic update every 5 minutes.
This automatic update should be infinite and i need to be sure that it never stops.
After some research, i've found some solution, but i don't know which is better for security and performance.
One of these could be my goal:
Don't create table and make complex query from php every time to get the desired result
Create a php script that repeats cyclically and update table db, maybe using Cron Job.
Update table using a sql event
I think that first solution could be to expensive since query is complex and there could be many request every second, but the result is always updated. I don't have experience about Cron Job, so i can't know if it could be a good idea or not. For the third solution, i still don't have database privileges to run events, but i'd like to know if it could be a valid solution.
All other solutions are welcome, thanks.
Do not use cron. Think about what will happen if one instance goes beyond 5 minutes and the next starts up. Eventually you will have hundreds of copies bogged down stumbling over each other.
Instead have a single job in a loop doing the update. (OK, you could have a cron job to perform a "keep-alive" task of restarting the query if it dies.)
The job would
CREATE TABLE new ...
INSERT INTO new SELECT complex-stuff...
RENAME TABLE real TO old, new TO real;
DROP TABLE old;
loop.
I would opt for Cron Job.
It doesn't clog any request, since it's executed from the operating system.
You can define which user executes the script (cron -u apache -e).
Easy to define interval. (i.e. every 5 minutes */5 * * * * php /path/to/script.php).
It's loggable.
Additional Notes
I had a cron job running under root and it worked just fine. My problem was that the project had a private logging mechanism that each log file would be created by apache user. By running it from root, sometimes the file would be created by root and after that, the scripts being executed by apache would not be able to APPEND to the log.
I also had an emailing script that would run once every 2 minutes that got stuck for 1h. Turns out, because of a bug in the application, an invalid email address (somethingwithoutatsign.com) was inserted into the database, which made the PHPMailer library throws errors. After that, I added a catch block that would send an email to me whenever an exception was thrown. Now, if the script stops running because of bad execution, I get to know right away.
I have a MySQL table called cronjobs which holds entires for every cronjob needed (e.g. delete old emails, update profile age, and so on). For every cronjob there is a defined code block which gets executed if the cronjob is due (I got different intervals for different cronjobs).
For the execution of the due cronjobs, I got a PHP script which is executed by UNIX crontab every minute (calls execute_cronjobs_due.sh which calls "php -f /path/to/file/execute_cronjobs_due.php").
When executing execute_cronjobs_due.php all cronjobs get marked that they are going to be executed, so that another call of execute_cronjobs_due.php wouldn't cause a parallel execution of the same cronjob getting already executed.
Now the problem: Sometimes the execution takes more than 60 seconds but the crontab program does not call execute_cronjobs_due.sh after these 60 seconds. What actually happens is that execute_cronjobs_due.sh is called right after the execution of the execution of the previous crontab. And if an execution takes more than 120 seconds, the next two executions are initialize simultaneously.
Timeline:
2015-06-15 10:00:00: execution execute_cronjobs_due.sh (takes 140 seconds)
2015-06-15 10:02:20: two simultaneous executions of execute_cronjobs_due.sh
Since it is executed exactly simultaneous, there is no use of marking the cronjob that they are being executed since the selects (which should actually exclude the marked once) are executed at the exact same time. So the update occurs right after both already selected the due cronjobs.
How can I solve this problem, so that there are no simultaneous executions of cronjobs? Can I use MySQL table locks?
Thank you very much for your help in advance,
Frederic
Yes you could use mysql table locks, but this may be overkill for your situation. Anyway to do that in most generic way
Make sure that you have autocommit off
LOCK TABLES cronjobs;
do your stuff
UNLOCK TABLES
for exact syntax and details read the docs obviusly https://dev.mysql.com/doc/refman/5.0/en/lock-tables.html , I personally never used table level locking so maybe there are some catches involved I am not aware of.
What I would do, if you use InnoDB table engine is to go with optimistic locking:
start transaction as a first thing in your script
get some id of script or whatever, might be process pid (getmypid()) or combination of host+pid. Or just generate guid if you don't know which will be perfect
do something like UPDATE cronjobs SET executed_by = my_id WHERE executed_by is null and /* whatever condition to get jobs to run */
then SELECT * FROM cronjobs where executed_by = my_pid
do your stuff on whatever above select returned
UPDATE cronjobs set executed_by = null where executed_by = my_pid
This should be as easy to do, easier to track what happens and scale in the future (i.e. you can have few instances running running in parallel as long as they execute different scripts)
With this solution second script will not fail (technically), it will just run 0 jobs.
Minus is that you will have to clean jobs that were claimed but script failed to mark them as finished, but you probably have to do it anyway with current solution. The easiest way would be to add a timestamp column that would track when was the job claimed last time and expire it after i.e. 15 minutes or an hour depending on business requirements (short pseudocode: first update will do SET executed_by = my_id, started_at = NOW() where executed_by is null or (executed_by is not null and started_at < NOW() - 1 hour))
How can I solve this problem, so that there are no simultaneous executions of cronjobs?
There are multiple ways to solve this. They might be helpful as-well:
My suggestion is to keep it simple and use either a file-locking or file-exist checking approach.
file_exist() + PID based CronHelper Class
http://abhinavsingh.com/how-to-use-locks-in-php-cron-jobs-to-avoid-cron-overlaps/
flock() based: https://stackoverflow.com/a/5428665/1163786
when you want to avoid IO, store the locking-state into memcache
database transactions: see below and #sakfa's answer
lock cronjobs across a distributed system using Redis as central: https://github.com/kvz/cronlock & http://kvz.io/blog/2012/12/31/lock-your-cronjobs/
Can I use MySQL table locks?
Yes, but it's a bit overkill.
You would use a "cronjob processing table" with a cronjob status column ("ToDo, Started, Complete" or "Todo, Running, Done") and a PID column.
Then you select jobs and mark their state by using transactions.
That makes sure that "Selecting a job from Todo" and "marking it as running/started" is done in one step. In the end, you might still have multiple exec's of your "central cronjob processing script", but jobs are NOT selected multiple times for processing.
I have a requirement where I need to Update some fields in database on monthly basis, I know I can use Cron Jobs, But I don't have access to use Cron, So is there any other way doing this in PHP, MySQL, JQuery or any other ??
I just need to update & Insert some rows in Database Tables on the scheduled time Automatically.
If you have site or something else with regular visits you can check with each visit is it time to run you job or not. If I not mistake wordpress's jobs are working in this way.
Have you ever heard of MySQL Event Scheduler?
Info
Creating event
Do you have access to it? If you want to check whether event_scheduler is running you can type show processlist - there should be a process run by User 'event_scheduler'. If not you can always run it.
Of course the best way is to add event_scheduler = ON in mysql config file (in mysqld section).
I have no idea where to start with this one:
I have a database that stores postID and Date.
What I want to do is have my website auto delete all rows where Date is less than today. This script can't have any user input at all. No button clicks, nothing. The script must run every day at midnight.
I've been looking all over the place for something that does this and I've found absolutely nothing.
You can use PHP script and use cron job on your cpanel.
Example:
cronjobcommand.php
<?php
include 'your_db_connection';
mysql_query("DELETE FROM your_table_name WHERE Date < NOW()");
?>
I have attached a screenshot below for your more reference.
For those out there who are on a shared hosting, like 1and1's, and can't use cron, here are 2 alternatives :
mysql events enable you to place a time trigger on mysql, which will execute when you'll want, without having to be fired by any kind of user input
if you cannot create mysql events because you're on 1and1 :(, an alternative is to use webcron
You just need to tell webcron the url of the php script you'd like to be run, and they'll trigger it for you at the intervals you want
Why using cronjobs everyday?? Why not filter data on output. For example in your select check if post date equals today with adding a simple where:
SELECT * FROM `posts`
WHERE (DATE(`post_date`) = DATE(NOW()));
This way you're not required to do your database managements/cronjobs on any special time and it will be used just for database managements. Afterwards you can delete unnecessary data at any time using by mysql command like:
DELETE FROM `posts` WHERE (
DATE(`post_date`) < DATE(NOW())
)
Most hosts provide a cron(8) service that can execute commands at specific times. You use the crontab(1) program to manage the crontab(5) file the describes when to run which commands.
There's a lot of functionality available to you, but if you write a program (shell script, php script, C program, whatever) that runs the appropriate MySQL commands, you can call the program via cron(8) in an entirely hands-off fashion.
Run crontab -e to edit your current crontab(5) file. If none exists, hopefully you'll get one with a helpful header. If not, copy this:
# m h dom mon dow command
The columns indicate the minute, hour, day of month, month, and day of week to execute commands. All the numbers in the columns are essentially ANDed together to decide when to run commands.
Thus, midnight every night would look like this:
0 0 * * * /path/to/executable
It's remarkably flexible, so put some time into the documentation, and you'll find many uses for it.
You should set cron job (scheduled tack.) for it.
A cron job is an automated program setup for Linux and Unix operating systems. It allows the user to execute several commands or functions at a specific time and date.
you have cron Job in your cpanel setup. first you need to make a php script with your logic for delete record after each date. take date from server and write script for delete.
then go to cron tab in your cpanel and do settings for time interval to run cron and give path of your php script file.
MySQL doesn't have a task scheduler. So you have to use the task scheduler of your Operating System (CRON under Linux), or to lunch a basic task checker sub-script during the script of the main page (on another page that is supposed to display the changing data).