Hello fellow programmers! :)
I want to be able to set up some php script to be run after some events, triggered by user. Let's say, user creates a forum thread, that should be closed after 48 hours automatically. It is equivalent to an update to MySQL row:
UPDATE threads SET closed = '1' WHERE threads.id = 'x'.
Hence, this problem should not necessarily be solved exclusively with php.
This kind of questions pop up from time to time, but everything I found was to set up a cron job to run every 'x' amount of time, that checks if the time has come to close the thread. The problem is, that running this checks often cause higher system load than if you schedule a script to be run once at a given time. Not to forget, that there could be hundreds or even thousands of threads, each with it's own time to be closed. We can avoid checking every single thread by creating some sort of queue, for instance in MySQL, so the script selects from the DB entries with "time_to_close < NOW()" and closes these. Another drawback is, that I would like the thread to be closed exactly after 48 hours. In that case the script should be run every second and should take very little time to be executed completely.
Alternatively to cron job I think following method can also be useful:
check at every access to the Thread if it should be closed. This also causes higher load, especially if the thread is accessed very often.
So is there any efficient way to schedule a (php) script run depending on the time of a specific event? While writing this question I stumbled upon MySQL event scheduler. Together with procedures, that can provide additional flow control (close thread only if there was no activity since 48 hours) I think my idea can be implemented. I am not familiar with these functions of MySQL, so I would appreciate any help on this topic.
With best regards,
system__failure.
I know a lot of websites compensate for this kind of behavior by doing this on a per user request basis. The overhead is not that bad and your records are always displaying correctly (unless you have a design problem.) This also works because most hosts don't give you cron access. It is very rare you will need to schedule a job in php. There are a few exceptions like report generations every hour. But trying to catch user actions with cron is not a good idea.
Related
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 4 years ago.
Improve this question
I'm developing a tool for my company, that in very broad strokes is intended to message users some information (a link to a picture) if they decide they want to be notified when it comes online.
If it were just alerting them when it's online is easy, because you don't have to schedule it, just check the list to see if anyone wants to be messaged about the picture when it comes online, and do it.
But we also have "We've got your petition, the picture is not here yet but we'll message you when it is" kind of message, and a few "not here yet" that I have to launch days later if the picture isn't online yet. And all these scheduled jobs need to be canceled if in any moment the picture comes online, and we send the message with it's link.
I'll try to explain it in as much detail as I can:
The user asks to be notified
Our web takes note of the petition. (adds it to the DB)
Check if the file is already online. At this point in time the file might not have been uploaded yet, but may be mere seconds away from it. So if the picture is not online yet when the petition is made, we want to wait 1-2 minutes to send the "Picture not yet here" message. Just so we don't send a "not yet here" message and 5 seconds later a "here's your picture" one.
We want to wait a few hours (1-3), to send a new message asking them to be patient.
After another set amount of time (7 days, approx) we want to send a last message, letting them know that the picture might never reach them because it's not being uploaded.
In any given time, even after point 5, if the picture comes online we want to cancel all these schedules and send the message with the picture.
I've been trying to learn how to do this, and through my search I've learned of three possible ways to achieve this functionality:
Option A: A single Cronjob executing every minute, that sweeps the database table searching if it's time to send one of those messages.
This option is easy to understand, although I'm afraid it might tax the database too much. I can use the shifty control panel that 1and1 has to set up that single Cronjob and call it a day.
Option B: Programatically write Cronjobs for every message that gets scheduled.
This sounds like it would be more "efficient", or at least less taxing on the DB, but I'm not sure Cronjob is supposed to work like that. It's normally used to schedule tasks that repeat themselves, isn't it? This would need a whole lot of functions to work (read the Crontab, add a line, search a line, edit a line, delete a line). Problem being here that I don't know how to edit the crontab that's on 1and1 servers via php. I have tried to contact them but their support has not been helpful at all.
Option C: The "at" function in linux.
This I just learned about. It looks like it would do what I want: schedule a task that happens only once, and it's structure seems pretty easy to handle. The problem here is threefold: 1- I don't know if PHP can execute Command Lines, 2- I don't know if the server at 1and1 has the "at" program installed, 3- I don't know if I can get a Command Line to execute a PHP file with the arguments to make it work.
And if any of these can be done, I don't know how.
As you see there are plenty of things I don't know about, but I've been trying to inform myself and learn. I just ask here because I'm at the end of the rope.
These options I listed are not an exhaustive list, they are just the methods I've found.
Which method would serve my purpose better? And how to do it?
Relevant facts:
Our host and database are located within 1and1, in a virtual server (meaning, we don't have a complete server for us, but share one with other clients)
Although we have "Unlimited" database space and queries, there is still a hard limit of how many queries you can do in a certain limit.
I'm new-ish to using linux, and I have not worked with PHP for years (until I've got this job!), so it would be better if your explanation doesn't assume deep knowledge on my part.
Programatically write Cronjobs for every message that gets scheduled.
God no, the biggest issue you'll have with that is that you will have to anticipate in advance what kinds of messages you'll have to send when, and that clashes with your ability to easily cancel messages. You will generally have to worry about a lot of state to manage (more on that later), which is a lot of hassle. You also need to ensure your scheduled jobs are cleaned up again afterwards, since cron can only set repeating tasks.
The "at" function in linux.
This is basically cron but for non-repeating tasks. That's better, but is still stateful. Especially with shared hosts it's also somewhat unpredictable whether your code will always execute on the same machine or when a machine might reboot. In those circumstances you may lose your scheduled jobs, so this is a no-go.
A single Cronjob executing every minute, that sweeps the database table searching if it's time to send one of those messages.
Yes, this is the way to do it. The biggest advantage here is that it's stateless, in the sense that it will always pick up on the exact current contents of your database, so it allows you to easily manage what your job should be doing on its next run and not having to anticipate that at the time you schedule an event.
I'm afraid it might tax the database too much.
It's one query per minute (if you write it well). Presumably every single page load of your website will incur one or multiple queries, and a properly built site should be able to handle hundreds to thousands of loads per second. One more query per minute isn't going to tank it. If it does, you have bigger issues.
I would personally choose the option A, I'm using it already on a project I worked on.
In your case, having the data on a shared hosting, I would create a cronjob that runs every minute (using an online service) and hits a php file somewhere in your folders, checking in a database table if anything must be done.
You should write some code that handles all the notifications you want to send and when, creating, for each of them, a row in the db table with the time of execution and all the details ready to be used to create the notification and to send it out.
The entire thing would work more or less as follow:
- Something happens that requires the creation of a notification to be sent out in 5 minutes: the row is created in the db table with the unix time or date of 5 minutes from now.
- A notification needs to be sent out 3 days from now, you use the same procedure as above.
The cronjob runs every minute and checks for expired orders (anything with date <= now), if any, a script takes care of these rows and execute the orders (sending out only the notifications required).
The database wouldn't be bothered too much, having to perform only 1 query per minutes (only checking for expired orders).
I have an application where I intend users to be able to add events at any time, that is, chunks of code that should only run at a specific time in the future determined by user input. Similar to cronjobs, except at any point there may be thousands of these events that need to be processed, each at its own specific due time. As far as I understand, crontab would not be able to handle them since it is not meant to have massive number of cronjobs, and additionally, I need precision to the second, and not the minute. I am aware it is possible to programmatically add cronjobs to crontab, but again, it would not be enough for what I'm trying to accomplish.
Also, I need these to be real time, faking them by simply checking if there are due items whenever the pages are visited is not a solution; they should also fire even if no pages are visited by their due time. I've been doing some research looking for a sane solution, I read a bit about queue systems such as gearman and rabbitmq but a FIFO system would not work for me either (the order in which the events are added is irrelevant, since it's perfectly possible one adds an event to fire in 1 hour, and right after another that is supposed to trigger in 10 seconds)
So far the best solution that I found is to build a daemon, that is, a script that will run continuously checking for new events to fire. I'm aware PHP is the devil, leaks memory and whatnot, but I'm still hoping nonetheless that it is possible to have a php daemon running stably for weeks with occasional restarts, so as long as I spawn new independent processes to do the "heavy lifting", the actual processing of the events when they fire.
So anyway, the obvious questions:
1) Does this sound sane? Is there a better way that I may be missing?
2) Assuming I do implement the daemon idea, the code naturally needs to retrieve which events are due, here's the pseudocode of how it could look like:
while 1 {
read event list and get only events that are due
if there are due events
for each event that is due
spawn a new php process and run it
delete the event entry so that it is not run twice
sleep(50ms)
}
If I were to store this list on a MySQL DB, and it certainly seems the best way, since I need to be able to query the list using something on the lines of "SELECT * FROM eventlist where duetime >= time();", is it crazy to have the daemon doing a SELECT every 50 or 100 milliseconds? Or I'm just being over paranoid, and the server should be able to handle it just fine? The amount of data retrieved in each iteration should be relatively small, perhaps a few hundred rows, I don't think it will amount for more than a few KBs of memory. Also the daemon and the MySQL server would run on the same machine.
3) If I do use everything described above, including the table on a MySQL DB, what are some things I could do to optimize it? I thought about storing the table in memory, but I don't like the idea of losing its contents whenever the server crashes or is restarted. The closest thing I can think of would be to have a standard InnoDB table where writes and updates are done, and another, 1:1 mirror memory table where reads are performed. Using triggers it should be doable to have the memory table mirror everything, but on the other hand it does sound like a pain in the ass to maintain (fubar situations can easily happen if some reason the tables get desynchronized).
I've search on the web and apparently there is no way to launch a php script without user interaction.
Few advisors recommend me Cron but I am not sure this is the right way to go.
I am building a website where auctions are possible just like ebay. And after an amount of time the objects are not available anymore and the auction is considered as finished.
I would like to know a way to interact with the database automatically.
When do you need to know if an object is available? -> Only if someone asks.
And then you have the user interaction you are searching for.
It's something different if you want to, let's say, send an email to the winner of an auction. In this case you'd need some timer set to the ending time of the auction. The easiest way to do this would be a cron job...
There are several ways to do this. Cron is a valid one of them and the one I would recommend if its available.
Another is to check before handling each request related to an object whether it is still valid. If it is not, you can delete it from the database on-the-fly (or do whatever you need to) and display a different page.
Also you could store the time at which your time-based script was run last in the database and compare that time with the current time. If the delay is large enough, you can run your time based code. However, this is prone to race conditions if multiple users hit the page at the same time, so the script may run multiple times (maybe this can be avoided using locks or anything though).
To edit cronjobs from the shell: crontab -e
A job to run every 10 minutes: */10 * * * * curl "http://example.com/finished.php"
TheGeekStuff.com cron Examples
Use heartbeat/bot implement
ation
Cron job that runs pretty frequently or a program that starts on boot and runs continuously (maybe sleeping periodically) is the way to go. With a cron job you'll need to make sure that you don't have two running at any given time or write it such that it doesn't matter if you have more than one working at any given time. With "resident" program you'll need to figure out how to handle the case when it crashes unexpectedly.
I wouldn't rely on this mechanism to actually close the auction, though. That should be handled in your database/web site. That is, the auction has a close time and either the database constraints or your code makes it impossible to bid on a closed auction. Notifying the winner and seller, setting up the payment process, etc. are things your service/scheduled task could do.
I've heard about cron job and don't think the actual creation of it will be that hard to make but I've some concerns about how this will work with a large script.
Without going too much off-topic on my project i will stick with the basics about my situation. I need to make a script that every day performs a CURL fetch for data on a remote website and updates an database for each featured member on my website with it. In short, it's approximatively at this time 1000 times the script need to be executed but it will be a larger number as times goes by.
As you can guess, this will take a long time to preform so i'm worried about how the execution will work in a manner of not crashing in the middle of it.
My first thought was to perhaps split the users into groups and make the executions on a small amount of users each time but don't know how this is manageable ( will read on further about the topic when i got some form of confirmation on this).
So, to my question. Do you think there is any way for me to make this happen and do you perhaps have any suggestions on how to make this to work efficiently? All help i can get is appreciated. Thank you for your time.
bigger cron-jobs with php and mysql needs to be fragmented, since there is no way for you to 'nice' them, (reduce their os priority). Even if you nice the script, the mysql-requests will be executed without this concern.
From what you're describing there's two aspects to consider:
Congestion of network bandwith
Congestion of database throughput
I'd recommend a fragmented solution where you call your script from cron more often, and let the script execute only a small amount of the total job. The job should further be canceled (postponed to next run) if i/o-bandwith or cpu-usage is above any limit that may affect response-time to visitors.
regards,
/t
One Way:
I'm usually against putting logic in the database, but in this case a stored procedure might help. It will run your job faster (since it's a large one) and also you want to lock the tables as you do it. That way, if the script that calls the stored procedure gets hit by cron before the original job was over with it wont edit your database while the first one is running.
The actual time can i not give an
straight answer on but based on
previously experiences this will take
longer then the max execution time.
So solve that problem. There's a reason you can have a different php.ini for the command line interface. Then you can simply focus on processing all users in one script.
I solved this program using the files of cron job as differents cron jobs with small pieces. If you are using PHP you can set a cron job to domain/cronjob1.php, domain/cronjob2.php limiting the database lets say 10 with
$sql="SELECT * FROM table LIMIT 10";
to cronjob 1 and the rest in cronjo2
How can I set up a program in which a certain piece of data for a user is updated every hour. One example I can give is Mafia Wars. When you obtain property, your money is incremented every set amount of time based on which property it is. I'm not asking to spit out code for me, but rather to guide me in the right direction for a solution. I tried looking into cron jobs, but that only runs a script on a set time. Different users are going to be using this, and they may have different times to update their information. So thus, cron jobs are not applicable here.
You could still have cron jobs, just lots of them (not one per user, but maybe one per minute).
Also, Mafia Wars strikes me as not very interactive, so it may be enough to just update the data (after the fact) when the user (or some other part of the system) next looks at it. So when you log in after 37 hours, you get all the updates for the last 37 hours retroactively applied. Cheap trick, but if there is no need for a consistent global view, that might work, too.
A solution that I came up with when wondering how to implement such a thing is that whenever the player saves the game, the game saves the current time. Then, when the player loads the game back up, it calculates how many minutes have passed and figures out how much money the game should give the player. Then, you could update the SQL database to reflect the changes.
Why do you dismiss cron jobs? Have a cron job that runs a script in short intervals. Within this script, include logic to check which specific updates on the database have to be done.
A cron job that runs something is your friend.
What that something is, is up to you. It could be a PHP script that runs some mysql queries or procedures, or it could straight mysql command from the command line.
Either way, Cron (and other similiar tools) are exactly the bill for these tasks. It's lightweight, on nearly every server in the land, lots of help avaliable for it, and it 99.9999% of the time, it just works!