I have a MVC framework with a controller in it. A controller downloads images from server. I need to refresh my database with those images every 5 minutes. So, I planned to create php script which downloads the file and persists it to my database. In order to do this every 5 minutes. I will be setting up Cron job.
Now the question is,
What is the best practise to handle errors inside php script?
Because Cron will keep executing at every 5 minutes without knowing that the last queried image is already lost and not being saved.
How do I notify myself that something unusual happend and I need to maintain the DB consistency by my self (Which I don't mind for few instances).
What is the best practise to handle errors inside php script? Because
Cron will keep executing at every 5 minutes without knowing that the
last queried image is already lost and not being saved.
use asserts as described here: http://php.net/manual/en/function.assert.php
How do I notify myself that something unusual happend and I need to
maintain the DB consistency by my self (Which I don't mind for few
instances).
use mail() in asserts
Use try-catch along with database transactions (if possible). You can dump errors to error_log() and either set that up to generate email or add email to your error handler.
In addition to the other comments. I have often found it useful in cron scripts that could run into problems or take longer than the desired execution interval to where multiple execution instances could be running, to provide some text file that indicates last execution time, execution success, etc. that you can inspect to determine if the script should run as scheduled. It could be something as simple as writing a file at script start and deleting it on successful execution, and then checking for this file on next execution to decide whether to run or not.
Related
I'm pretty new to PHP. This is my first time actually, so I apologize in advance if the question sounds dumb.
I have a php script which fetches data from an external API and updates my database regularly. I know that the best way to do this is to use a cron job. However, I am using an infinite loop which sleeps for a particular time between each update.
Now, I need to allow the user (admin) to start / stop the script and also allow them to change the time interval between each update. The admin does this through a UI. What is the best way to do this? How do I execute the script in the background when the user clicks start and how do I stop the script?
Thanks.
I think the ideal solution would be the following:
Have the background job run as a cronjob every minute (instead of a loop which can cause memory leaks). PHP was not designed to be a daemon.
Set a DB flag in the cronjob for on/off, everytime it runs it checks if its on or off, if off it exists if on it continue.
In the UI you turn that flag on or off depending on what the admin needs.
I think that is the best way to go (and easiest).
You might want to take a look at Threading in PHP -> http://www.php.net/manual/en/class.thread.php
An other option is to either set a SESSION (which is a little tricky, since you won't catch it in the same call) or you generate a file wich tells the script to stop/exit/fail/what ever if that file exists.
So here's the lowdown:
The client i'm developing for is on HostGator, which has limited their max_execution_time to 30 seconds and it cannot be overridden (I've tried and confirmed it cannot be via their support and wiki)
What I'm have the code doing is take an uploaded file and...
loop though the xml
get all feed download links within the file
download each xml file
individually loop though each xml array of each file and insert the information of each item into the database based on where they are from (i.e. the filename)
Now is there any way I can queue this somehow or split the workload into multiple files possibly? I know the code works flawlessly and checks to see if each item exists before inserting it but I'm stuck getting around the execution_limit.
Any suggestions are appreciated, let me know if you have any questions!
The timelimit is in effect only when executing PHP scripts through a webserver, if you execute the script from CLI or as a background process, it should work fine.
Note that executing an external script is somewhat dangerous if you are not careful enough, but it's a valid option.
Check the following resources:
Process Control Extensions
And specifically:
pcntl-exec
pcntl-fork
Did you know you can trick the max_execution_time by registering a shutdown handler? Within that code you can run for another 30 seconds ;-)
Okay, now for something more useful.
You can add a small queue table in your database to keep track of where you are in case the script dies mid-way.
After getting all the download links, you add those to the table
Then you download one file and process it; when you're done, you check them off (delete from) from the queue
Upon each run you check if there's still work left in the queue
For this to work you need to request that URL a few times; perhaps use JavaScript to keep reloading until the work is done?
I am in such a situation. My approach is similar to Jack's
accept that execution time limit will simply be there
design the application to cope with sudden exit (look into register_shutdown_function)
identify all time-demanding parts of the process
continuously save progress of the process
modify your components so that they are able to start from arbitrary point, e.g. a position in a XML file or continue downloading your to-be-fetched list of XML links
For the task I made two modules, Import for the actual processing; TaskManagement for dealing with these tasks.
For invoking TaskManager I use CRON, now this depends on what webhosting offers you, if it's enough. There's also a WebCron.
Jack's JavaScript method's advantage is that it only adds requests if needed. If there are no tasks to be executed, the script runtime will be very short and perhaps overstated*, but still. The downsides are it requires user to wait the whole time, not to close the tab/browser, JS support etc.
*) Likely much less demanding than 1 click of 1 user in such moment
Then of course look into performance improvements, caching, skipping what's not needed/hasn't changed etc.
I'm using PHP/MySQL, although I think this question is essentially language/db ambivalent. I have a PHP script that connects to one API, gets the response data, parses it, and then sends it to a different API for storage in its database. Sometimes this process fails because of an error with one of the APIs. I would therefore like to easily track its success/failure.
I should clarify that "success" in this case is defined as the script getting the data it needs from the first API and successfully having it processed by the second API. Therefore, "failure" could result from 3 possible things:
First API throws an error
Second API throws an error
My script times out.
This script will run once a day. I'd like to store the success or failure result in a database so that I can easily visit a webpage and see the result. I'm currently thinking of doing the following:
Store the current time in a variable at the start of the script.
Insert that timestamp into the database right away.
Once the script has finished, insert that same timestamp into the database
again.
If the script fails, log the reason for failure in the DB.
I'd then gauge success or failure based on whether a single timestamp has two entries in the database, as opposed to just one.
Is this the best way to do it, or would something else work better? I don't see any reason why this wouldn't work, but I feel like some recognized standard way of accomplishing this must exist.
A user declared shutdown function might be an alternative: using register_shutdown_function() you can decalre a callback to be executed when the script terminates, whethe rsuccessfully, user-aborted, or timed-out
You could use a lock file :
at the very beginning of your script, you create a lock file somewhere on the filesystem
at the very end of your script, if everything worked good, you delete it from filesystem
Then you've just to monitor the directory where you've placed these files. With the lock file's creation date you can find which day didn't work.
You can combine this system by a monitoring script that sends alerts if lock files are present and have a creation date older than a given interval (let's say 1 hour for example if your script usually runs in a few minutes).
How can I make a scheduler in PHP without writing a cron script? Is there any standard solution?
Feature [For example]: sent remainder to all subscriber 24hrs b4 the subscription expires.
The standard solution is to use cron on Unix-like operating systems and Scheduled Tasks on Windows.
If you don't want to use cron, I suppose you could try to rig something up using at. But it is difficult to imagine a situation where cron is a problem but at is A-OK.
The solution I see is a loop (for or while) and a sleep(3600*24);
Execute it through a sending ajax call every set interval of yours through javascript
Please read my final opinion at the bottom before rushing to implement.
Cron really is the best way to schedule things. It's simple, effective and widely available.
Having said that, if cron is not available or you absolutely don't want to use it, two general approaches for a non-cron, Apache/PHP pseudo cron running on a traditional web server, is as follows.
Check using a loadable resource
Embed an image/script/stylesheet/other somewhere on each web page. Images are probably the best supported by browsers (if javascript is turned off there's no guarantee that the browser will even load .js source files). This page will send headers and empty data back to the browser (a 1x1 clear .gif is fine - look at fpassthru)
from the php manual notes
<?php
header("Content-Length: 0");
header("Connection: close");
flush();
// browser should be disconnected at this point
// and you can do your "cron" work here
?>
Check on each page load
For each task you want to automate, you would create some sort of callable API - static OOP, function calls - whatever. On each request you check to see if there is any work to do for a given task. This is similar to the above except you don't use a separate URL for the script. This could mean that the page takes a long time to load while the work is being performed.
This would involve a select query to your database on either a task table that records the last time a task has run, or simply directly on the data in question, in your example, perhaps on a subscription table.
Final opinion
You really shouldn't reinvent the wheel on this if possible. Cron is very easy to set up.
However, even if you decide that, in your opinion, cron is not easy to set up, consider this: for each and every page load on your site, you will be incurring the overhead of checking to see what needs to be done. True cron, on the other hand, will execute command line PHP on the schedule you set up (hourly, etc) which means your server is running the task checking code much less frequently.
Biggest potential problem without true cron
You run the risk of not having enough traffic to your site to actually get updates happening frequently enough.
Create a table of cronjob. In which keep the dates of cron job. Keep a condition, if today date is equal to the date in the creonjob table. then call for a method to execute. This works fine like CRON job.
Basically I'm designing an alert system in PHP with CodeIgniter. I want the alerts to be "scheduled" by inserting a database row and then sent out in batches periodically. The alerts are sent out via email.
The only way I can think to achieve this is to require a cron script to call a CLI-only controller every minute or so that checks for scheduled alerts and sends them if it finds anything.
Something is telling me that having a cron script run so often is poorly designed but I can't think of any other way to do it.
Advice?
Making the alerts asynchronous By separating the alert task from the rest of the code is desirable.
I don't see a problem with a cron script , I would ask why do you need to do it so often?
Setting a threshold is a good idea , but you would still need to check at the end of day or whenever to make sure it has fired anyway.
You could rather then use the whole overhead of controller (and rest of framework to get there) - maybe just make a lightweight CLI php script
Ok, currently this is what you're doing
Timed Cron Job
|
|-check for any new updates (in database)
|
|-Send them
Some Event happens to trigger alert
|
|-Store alert to send later in database
Instead just do this
Some Event happens to trigger alert
|
|-Send it
Something is triggering your db insert
So instead of inserting, just call your email routine.
Am I missing something?
One simple workaround is to have your insertion script check if a threshold number of alerts has been queued since the last batch send. It wouldn't be a timed job, but it'd mean no more than the threshold # of messages is waiting at any time.
But if you do need it to be done regularly, and your site isn't busy enough to generate sufficient alerts to usually fall in under the time limit, then a cron job is your only practical alternative.