I have a php script run as a cron job that executes a set of simple tasks that loops for each user in the database and takes about 30 mins to complete. This process starts over every hour and needs to be as fast and efficient as possible. The problem Im having, is like with any server script, execution time varies and I need to figure out the best cron time settings.
If I run cron every minute, I need to stop the last loop of the script 20 seconds before the end of the minute to make sure that the current loop finishes in time. Over the course of the hour this adds up to a lot of wasted time.
Im wondering if its a bad idea to simple remove the php execution time limit and run the script once an hour and let it run to completion.... is this a bad idea?
Instead of setting the max_execution_time you could also use set_time_limit() to reset the counter on every loop. This will ensure your script is never running out of time unless there is something serious hanging within the current loop (and taking longer than the max_execution_time).
Basically this should make your script run as long as it needs while giving it a 30 seconds timeout between two set_time_limit() calls.
Assuming you'd like the work done ASAP, don't use cron. Cron is good for things that need to happen at specific times. It's often abused to simulate a background process that would ideally process work as soon as work appears. You should probably write a daemon that runs continuously. (Note: you could also look at a message/work-queue type system, there are nice libraries out there to do this too)
You can write a daemon from scratch using the pcntl functions (since you don't care about multiple worker processes, it's super-easy to get a process running in the background.), or cheat and just make a script that runs forever and run it via screen, or leverage some solid library code like PEAR's System:Daemon or nanoserv
Once the daemonization stuff is taken care of, all you really care about is having a loop that runs forever. You'll want to take care that your script doesn't leak memory, or consume too many resources.
Generally, you can do something like:
<?PHP
// some setup code
while(true){
$todo = figureOutIfIHaveWorkToDo();
foreach($todo as $something){
//do stuff with $something
//remember to clean up resources so you don't leak memory!
usleep(/*some integer*/);
}
usleep(/* some other integer */);
}
And it'll work pretty well.
Setting the time limit to 0 and letting it do its thing is fairly typical of PHP based cronjobs (in my experience), but this is also the point when you should ask yourself a few important questions, such as "Should I rewrite this job in a compiled language?" and "Am I using all of my tools (database, etc) to their maximum efficiency?"
That said, maybe better than completely removing the time limit would be to set it to the upper limit you actually want. If that means 48 minutes, then set_time_limit(48 * 60);
I really think you shouldn't set the time out to 0, that is just looking for trouble. At most, set it to 59*60 seconds, but setting it to 0 might cause security problems, if a script hangs, it will hang almost forever until the server host stops the execution. It is considered bad practice to do so.
I have used the php command-line interface for similar long running tasks in the past. You probably do not want to remove the execution time limit for any request.
Sounds like a great idea if there's little chance that it will take more than an hour. Note, however, that the wrong bug can be a really good way of making it take longer than expected..
To avoid all sorts of nasty problems, you should have a guard file with the process ID of the script. On startup, you should check to make sure the file doesn't exist, or if it does that the process ID in the file doesn't exist (through a kill( pid, 0 ) call). If these conditions are met, create a new file with the script's PID and delete the file when you're done.
This is the same trick that many daemons use to ensure it isn't already running. If the daemon was killed suddenly, the file will still exist but the PID of the process therein is unlikely to be running.
Depending on what your script does, it can lead to problems if you remove the time limit. If per example, you are polling an external server that is unresponsive while the job is running, and that your cron takes 2 hours instead of 30 minutes to complete, you may get a stack of PHP processes being fired up even if the previous ones haven't completed yet. This can cause system instability and crashes.
You probably have two options:
Make sure that no other instance of your script is running beforehand, otherwise exit() on start.
Consider changing your cronjob into a daemon.
Does it have to run hourly like clockwork?
If not split the job (you mentioned it was more than one simple task) do each task every hour?
Or split it per user, do A-M on hour, then N-Z the next?
Related
I have some limitations with my host and my scripts can't run longer than 2 or 3 seconds. But the time it will take to finish will certainly increase as the database gets larger.
So I thought about making the script stop what it is doing and call itself after 2 seconds, for example.
Firstly I tried using cURL and then I made some attempts with wget. But there is always a problem with waiting for the response and timeouts (with cURL, for example, I just need to ping the script, not wait for a response) or permissions with the server (functions that we use to run wget such as exec seems to be disabled on my server, or something like that).
What do you think is the best idea to make a PHP script ping/call itself?
On Unix/LInux systems I would personally recommend to schedule CRON JOBS to keep running the scripts at certain intervals
May be this SO Link will help you
Php scripts generally don't call other php scripts. It is possible to spawn a background process as illustrated here, but I don't think that's what you're after. If, so you'd be better off using cron as was discussed above.
Calling a function every X amount of seconds with the same script is certainly possible, but this does the opposite of what you want since it would only extend the run time of the script in question.
What you seem to be asking is, contrary to your comment, somewhat paradoxical. A process that calls method() every so often is still a long running process and is subject to the same restrictions as any other process on the server, regardless of the fact that it may be sitting idle for short intervals.
As far as I can see your options are:
Extend the php max_execution_time directive, or have your sysadmin do so if they are willing
Revise your script so that it completes within the time limit
Move to a new server
I have a site where auctions end a varying times. I need to send an automated email to the seller and the buyer after the auction is finished to notify them of the auction ending and the results. Obviously I can't really wait for someone to load the page to run the script so is there a good way to automate this by checking the current time and comparing that to the time of the auction end and running that script?
The site is on a UNIX server so a cron job is an option, but I'm concerned that running a cron job like that will put quite a load on the server.
A cron job runs at most once per minute.
Whatever load it generates on the server really depends on the kind of script you're going to run. Btw, I'm assuming that you're using cli to run the script (rather than just doing a curl http://mysite.com.
If your script takes longer than one minute (you should monitor this), simply either:
Increase the interval time between runs or,
Use a lock file to make sure no two instances of your script can run at the same time.
if (($fp = fopen('/tmp/mylockfile', "r+")) === false) {
die("Could not open lock file");
}
if (!flock($fp, LOCK_EX | LOCK_NB)) {
die("Could not obtain lock");
}
// run your code here
// release the lock and close file
fclose($fp);
OTOH If the script needs to run more than once per minute, you would need a different mechanism entirely.
Q: What is the best way to run a PHP script at a particular time, or interval?
A: Use cron
Q: Does a cronjob create a big load on the server?
A: Depends off course off your script. But checking if an auction should be closed, close it and send two emails shouldn't be to difficult. Be sure to create some kind of lockfile to make sure that if your script runs longer than the interval set, it isn't run twice.
Q: running a script with shorter intervals than 1 minute
A: Can't answer this one for you. Sorry :)
Use Cron. It allows you to run any command at most once per minute: http://clickmojo.com/code/cron-tutorial.html
As far as server load goes, it generally won't be a concern unless you are running a massive number of database calls very often on a very low-end server. I speak in generalities, but the idea is sound.
If you are using something else (besides PHP) to run your auction timer mechanism, I recommend you attach some code to that timer mechanism that also executes a mail-sending script when the timer runs down to zero and determines a winner.
Run the PHP script as a command line script. This will not put a load on the webserver - just a load on the server and you can easily run it via CRON.
If you add #!/usr/bin/php to the top of the script and change the execute bit on the file with chmod +x scriptname.php you can directly execute the script without passing it through php
http://php.net/manual/en/features.commandline.php
A couple of things you need to do this:
Store something in your auction information indicating whether you've sent this e-mail yet or not (could be a boolean or a date for when it was sent which might be null). Although I have to assume you need to do something besides send this e-mail? Like mark the auction as closed so no more bidding can take place?
A bit of code that finds auctions which need this e-mail sent: e.g. they've ended and have not yet been reminded.
Something to repeatedly execute the bit of code in 2. You could use cron. Alternatively you can write a pretty simple daemon for unix that runs constantly in a loop of (wait at least a few ms or more; do some stuff). The latter is a lot more work but in my opinion scales much better. See http://pear.php.net/package/System_Daemon for some useful tools if you're interested in this approach.
One thing to consider is how much you want to be careful about accidentally double-sending this e-mail. If you're only running this code in a single thread it's pretty easy but if you ever want to build out to the point where you have several different distributed machines that create and send these e-mails you have to be a bit more careful. If you're running it out of cron can you guarantee one run of it will always be finished before another one starts?
I have a PHP-script running on my server via a cronjob. The job runs every minute. In the php script i have a loop that executes, then waits one sevond and loops again. Essentially creating a script to run once every second.
Now I'm wondering, if i make the cronjob run only once per hour and have the script still loop for an entire hour or possible an entire day.. Would this have any impact on the servers cpu and or memory and if so, will it be positive or negative?
I spot a design flaw.
You can always have a PHP script permanently running in a loop performing whatever functionality you require, without dependency upon a webserver or clients.
You are obviously checking something with this script, any incites into what? There may be better solutions for you. For example if it is a database consider SQL triggers.
In my opinion it would have a negative impact. since the scripts keeps using resources.
cron is called on a time based scale that is already running on the server.
But cronjob can only run once a minute at most.
Another thing is if the script times out, fails, crashes for whatever reason you end up with not running the script for at max one hour. Would have a positive impact on server load but not what you're looking for i guess? :)
maybe run it every 2 or even 5 minutes to spare server load?
OR maybe change the script so it does not wait but just executes once and calling it from cron job. should have a positive impact on server load.
I think you should change script logic if it is possible.
If tasks your script executes are not periodic but are triggered by some events, the you can use some Message Queue (like Gearman).
Otherwise your solution is OK. Memory leaks can occurs, but in new PHP versions (5.3.x) Garbage Collector is pretty good. Some extensions can lead to memory leaks. Or your application design can lead to hungry memory usage (like Doctrine ORM loaded objects cache).
But you can control script memory usage by tools like monit and restart your script when mempry limit reaches some point or start script again when your script unexpectedly shuts down.
As a possible alternative to using cron jobs, I found the sleep function. I have never used this before.
If I tell my script to run inside a kind of loop, and inside that loop I have an instruction like this
# sleeps for 86400 seconds or one day
sleep(86400);
will my script be launched again after 1 day? even if don't access it on my web browser again within that period?
I think is not possible, but I'm here to ask an expert about it.
The main problem with using PHP this way is, in my experience, not web server timeouts (there are ways to handle that with difficulty varying on the server and the platform) but memory leaks.
Straightforward PHP code tends to leak a lot of memory; most of the scripts I wrote were able to do hundreds of times as many work after I did some analysis and placed some unsets. And I was never able to prevent all the leaks this way. I'm also told there are memory leaks in the standard library, which, if true, makes it impossible to write daemons that would run for a long time in loops.
The script will timeout. You need to set it so that it won't timeout using set_time_limit.
I wouldn't do this I would either use a cron (that is a link) job if it is a regular task or an at (that is a link) job if the job is added at the run time of your script.
cron allows you to run a recurring job every day at 1pm for example whereas at allows you to schedule a job to run once for now +1day for example.
I have written a PHP 5.3 wrapper for the at queue if you choose to go down that route. It is available on GitHub https://github.com/treffynnon/PHP-at-Job-Queue-Wrapper
There is also time_sleep_until(). Maybe more useful to wake up on a specific time...
If you access the script through a web browser, it will be terminated after 30 seconds.
If you start the PHP script on the command line, this could work.
It would work, but your "startup time" will be subject to drift. Let's say your job takes 10 seconds to run, then sleeps 86400, runs another 10, sleeps 86400, etc.. You start it exactly at midnight on day 1. On Day 2 it'll run at 12:00:10am, on day 3 it's 12:00:20am, etc...
You can do some fancy math internally to figure out how long the run took, and subtract that from the next sleep call, but at the point, why not use cron? With cron the script will exit after each run, cleaning up memory and resources used. With your sleep method, you'll have to be VERY careful that you're not leaking resources somewhere, or things will eventually grind to a halt.
I had a similar problem before and found a php cron parsing class that will allow you to execute php similar to running crons. You can tie it to a commonly accessed script on your site if you don't have access to run crons directly.
I actually use this script as part of a larger cron job script:
a cron job runs every hour
an xml file for each sub-cron with a cron-like time component(i.e.- * */2 * * * php /home..)
the sub-cron script that will run if the current time meets the criteria of the sub-cron time component
a user interface is setup so that I don't have to manually add/remove sub-crons from the main cron
The cronParser class is here.
Many correct answers, but: Using sleep() means your script keeps running, and keeps using memory. Raising the default timeout of 30s will work, but again, this is bad idea. I suggest you use crontasks.
This is why legitimate cron jobs were invented. Just use crontab. Using a PHP script to do it will be EXTRAORDINARILY unreliable, buggy, and poorly timed.
Hope this is insightful.
I have a php script that can take anything from 2 to 10 minutes to execute. It fetches info from around the web so its time depends on how fast lots of other things are talking.
I used to run the script on a cron every 15 minutes, but sometimes it only takes 2 minutes to run.
So I wondered if I can somehow make it run perpetually - setting itself going again as soon as it finishes its task? That way, however long it takes, it will always start agaiun straight away.
Seems like you're running into cron job issues. I would instead turn your script into a daemon, that way it can run perpetually without fear of overlaps or finishing too fast.
Here's a tutorial how to do it.
http://kevin.vanzonneveld.net/techblog/article/create_daemons_in_php/
Put the entire thing in an infinite loop? Either in the PHP code itself, or make a launch script that does that.
Writing a daemon is probably the best solution.
If you're lazy, and on linux/unix, you can just script an infinite loop, and set it running inside a screen session.