I'm trying to figure out the most efficient way to running a pretty hefty PHP task thousands of times a day. It needs to make an IMAP connection to Gmail, loop over the emails, save this info to the database and save images locally.
Running this task every so often using a cron isn't that big of a deal, but I need to run it every minute and I know eventually the crons will start running on top of each other and cause memory issues.
What is the next step up when you need to efficiently run a task multiple times a minute? I've been reading about beanstalk & pheanstalk and I'm not entirely sure if that will do what I need. Thoughts???
I'm not a PHP guy but ... what prevents you from running your script as a daemon? I've written many a perl script that does just that.
Either create a locking mechanism so the scripts won't overlap. This is quite simple as scripts only run every minute, a simple .lock file would suffice:
<?php
if (file_exists("foo.lock")) exit(0);
file_put_contents("foo.lock", getmypid());
do_stuff_here();
unlink("foo.lock");
?>
This will make sure scripts don't run in parallel, you just have to make sure the .lock file is deleted when the program exits, so you should have a single point of exit (except for the exit at the beginning).
A good alternative - as Brian Roach suggested - is a dedicated server process that runs all the time and keeps the connection to the IMAP server up. This reduces overhead a lot and is not much harder than writing a normal php script:
<?php
connect();
while (is_world_not_invaded_by_aliens())
{
get_mails();
get_images();
sleep(time_to_next_check());
}
disconnect();
?>
I've got a number of scripts like these, where I don't want to run them from cron in case they stack-up.
#!/bin/sh
php -f fetchFromImap.php
sleep 60
exec $0
The exec $0 part starts the script running again, replacing itself in memory, so it will run forever without issues. Any memory the PHP script uses is cleaned up whenever it exits, so that's not a problem either.
A simple line will start it, and put it into the background:
cd /x/y/z ; nohup ./loopToFetchMail.sh &
or it can be similarly started when the machine starts with various means (such as Cron's '#reboot ....')
fcron http://fcron.free.fr/ will not start new job if old one is still running, Your could use # 1 command and not worry about race conditions.
Related
I need to write a server-side program that lives on the server, and is checking a database consistently for new entries.
When a new entry shows up in the database, the program should process the data and put the results somewhere else.
It is important to hi-light that the process isn't instigated by new entries showing up, but by the program checking for new entries on its own.
Some people I've spoken to brought up cron jobs, I was curious what if this is the solution for me? I see that it has limitations, it won't run less than every minute. I was hoping for the program to run every 5 seconds, would I be better off writing a shell script or is that a bootleg fix?
I'm not sure if this is conventional (?) but...
Use a database trigger on INSERT that runs an external program (PHP, Python, .. whatever). Which database are you using? I think this post is old but might be of help: http://crazytechthoughts.blogspot.co.uk/2011/12/call-external-program-from-mysql.html
There is a technique I've frequently used when dealing with queues that I've been processing.
#!/bin/sh
php -f checkDBAndAct.php
sleep 5
exec $0
The exec $0 part starts the script running again, replacing itself in memory, so it will run forever without issues. Any memory the PHP script uses is cleaned up whenever it exits, so that's not a problem either.
A simple line will start it, and put it into the background:
cd /x/y/z ; nohup ./loopToProcessDB.sh &
or it can be similarly started when the machine starts with various means (such as Cron's '#reboot ....')
-- from https://stackoverflow.com/a/2686100/6216
An extended version is on http://PHPscaling.com and https://gist.github.com/alister/1386212
Though I'd use an actual queue system, rather than a DB, as there are a number of downsides to bending a database to this task.
Is there a way to make php work forever without cron.
What I want it for is to unban users after a few hours by running a mysql query, thanks
If you don't have access to cron jobs on your server (I guess you are running on a shared hosting?), the best alternative is to run an "external cron". Have a look at www.setcronjob.com. I have been using this for a couple of months now and it is pretty stable.
You can set it up such that it calls a script on your website every whenever you want. (Example: http://www.yoursite.com/script.xxx)
In the script, you can run a MySQL query to check which users have been banned for a couple of hours and then unban them.
You can start your script from the command line and let it run in the background. You will have to design this script in such a way that it never exits and just loops forever using the sleep() function to avoid unnecessary processor load. Since php scripts invoked from the command line have no max execution time the script will run until you manually kill it off with the kill command.
Once you've written the script you can start it with:
nohup php myscript.php &
nohup makes the script still run once you log out of the console session that you started it from, otherwise it would kill off then. The & symbol at the end starts the script as a new process in the background so that you can continue using the console.
I have a site where auctions end a varying times. I need to send an automated email to the seller and the buyer after the auction is finished to notify them of the auction ending and the results. Obviously I can't really wait for someone to load the page to run the script so is there a good way to automate this by checking the current time and comparing that to the time of the auction end and running that script?
The site is on a UNIX server so a cron job is an option, but I'm concerned that running a cron job like that will put quite a load on the server.
A cron job runs at most once per minute.
Whatever load it generates on the server really depends on the kind of script you're going to run. Btw, I'm assuming that you're using cli to run the script (rather than just doing a curl http://mysite.com.
If your script takes longer than one minute (you should monitor this), simply either:
Increase the interval time between runs or,
Use a lock file to make sure no two instances of your script can run at the same time.
if (($fp = fopen('/tmp/mylockfile', "r+")) === false) {
die("Could not open lock file");
}
if (!flock($fp, LOCK_EX | LOCK_NB)) {
die("Could not obtain lock");
}
// run your code here
// release the lock and close file
fclose($fp);
OTOH If the script needs to run more than once per minute, you would need a different mechanism entirely.
Q: What is the best way to run a PHP script at a particular time, or interval?
A: Use cron
Q: Does a cronjob create a big load on the server?
A: Depends off course off your script. But checking if an auction should be closed, close it and send two emails shouldn't be to difficult. Be sure to create some kind of lockfile to make sure that if your script runs longer than the interval set, it isn't run twice.
Q: running a script with shorter intervals than 1 minute
A: Can't answer this one for you. Sorry :)
Use Cron. It allows you to run any command at most once per minute: http://clickmojo.com/code/cron-tutorial.html
As far as server load goes, it generally won't be a concern unless you are running a massive number of database calls very often on a very low-end server. I speak in generalities, but the idea is sound.
If you are using something else (besides PHP) to run your auction timer mechanism, I recommend you attach some code to that timer mechanism that also executes a mail-sending script when the timer runs down to zero and determines a winner.
Run the PHP script as a command line script. This will not put a load on the webserver - just a load on the server and you can easily run it via CRON.
If you add #!/usr/bin/php to the top of the script and change the execute bit on the file with chmod +x scriptname.php you can directly execute the script without passing it through php
http://php.net/manual/en/features.commandline.php
A couple of things you need to do this:
Store something in your auction information indicating whether you've sent this e-mail yet or not (could be a boolean or a date for when it was sent which might be null). Although I have to assume you need to do something besides send this e-mail? Like mark the auction as closed so no more bidding can take place?
A bit of code that finds auctions which need this e-mail sent: e.g. they've ended and have not yet been reminded.
Something to repeatedly execute the bit of code in 2. You could use cron. Alternatively you can write a pretty simple daemon for unix that runs constantly in a loop of (wait at least a few ms or more; do some stuff). The latter is a lot more work but in my opinion scales much better. See http://pear.php.net/package/System_Daemon for some useful tools if you're interested in this approach.
One thing to consider is how much you want to be careful about accidentally double-sending this e-mail. If you're only running this code in a single thread it's pretty easy but if you ever want to build out to the point where you have several different distributed machines that create and send these e-mails you have to be a bit more careful. If you're running it out of cron can you guarantee one run of it will always be finished before another one starts?
In the past, I ran a bunch of scripts each as a separate cron job. Now I'd like to run a controller script with one cron job, then have that call the scripts separately (and in parallel, all at the same time), so I don't have to create a new cron job every time I add another script.
I looked up pcntl_fork() but we don't have that installed. Can fsockopen() do this as well?
A few questions:
I saw this example, http://phplens.com/phpeverywhere/?q=node/view/254, that uses fsockopen(). Will this allow me to run PHP scripts in parallel? Note, the scripts don't interact, but I would still like to know if any of them exited prematurely with an error.
Secondly the scripts I'm running aren't externally accessible, they are internal only. The script was previously run like so: php -f /path/to/my/script1.php. It's not a web-accessible path. Would the example in #1 work with this, or only web-accessible paths?.
Thanks for any advice you can offer.
You can use proc_open to run multiple processes without waiting for each process to finish.
You will have a process handle, you can terminate each process at any time and you can read the standard output of each process.
You can also communicate via pipes, which is optional.
Passing 1st param php /your/path/to/script.php param1 "param2 x" means starting a separate PHP process.
proc_open (see Example #1)
Ultimately you will want to use an infinite while loop + usleep (or sleep) to avoid maxing out on the CPU. Break when all processes finish, or after you killed them.
Edit: you can know if a process has exited prematurely.
Edit2: a simpler way of doing the above is popen
Please correct me if I'm wrong, but if I understand things correctly, the solution Tiberiu-Ionut Stan proposed implies that starting the processes with proc_open and waiting for them to finish will not be run as a cron script, but is part of a running program/service, right?
As far as I understand the cron jobs, the controller script user920050 was thinking of using would be started by cron on a schedule and each new instance would launch the processes all over again, do the waiting for them to finish and probably run in parallel with other cron-launched instances of the controller script.
I am developing a website that requires a lot background processes for the site to run. For example, a queue, a video encoder and a few other types of background processes. Currently I have these running as a PHP cli script that contains:
while (true) {
// some code
sleep($someAmountOfSeconds);
}
Ok these work fine and everything but I was thinking of setting these up as a deamon which will give them an actual process id that I can monitor, also I can run them int he background and not have a terminal open all the time.
I would like to know if there is a better way of handling these? I was also thinking about cron jobs but some of these processes need to loop every few seconds.
Any suggestions?
Creating a daemon which you can make calls to and ask questions would seem the sensible option. Depends on wether your hoster permits such things, especially if you're requiring it to do work every few seconds, then definately an OS based service/daemon would seem far more sensible than anything else.
You could create a daemon in PHP, but in my experience this is a lot of hard work and the result is unreliable due to PHP's memory management and error handling.
I had the same problem, I wanted to write my logic in PHP but have it daemonised by a stable program that could restart the PHP script if it failed and so I wrote The Fat Controller.
It's written in C, runs as a daemon and can run PHP scripts, or indeed anything. If the PHP script ends for whatever reason, The Fat Controller will restart it. This means you don't have to take care of daemonising or error recovery - it's all handled for you.
The Fat Controller can also do lots of other things such as parallel processing which is ideal for queue processing, you can read about some potential use cases here:
http://fat-controller.sourceforge.net/use-cases.html
I've done this for 5 years using PHP to run background tasks and its no different to doing in any other language. Just use CRON and lock files. The lock file will prevent multiple instances of your script running.
Also its important to monitor your code and one check I always do to prevent stale lock files from preventing scripts to run is to have second CRON job to check if if the lock file is older than a few minutes and if an instance of the PHP script is running, if not it then removes the lock file.
Using this technique allows you to set your CRON to run the script every minute without issues.
Use the System::Daemon module from PEAR.
One solution (that I really need to try myself, as I may need it) is to use cron, but get the process to loop for five mins or so. Then, get cron to kick it off every five minutes. As one dies, the next one should be finishing (or close to finishing).
Bear in mind that the two may overlap a bit, and so you need to ensure that this doesn't cause a clash (e.g. writing to the same video file). Some simple inter-process communication may be useful, even if it is just writing to a PID file in the temp directory.
This approach is a bit low-tech but helps avoid PHP hanging onto memory over the longer term - sort of in-built task restarts!