I have a PHP web page which serves the RSS feed, but it takes about 15-20 seconds to generate a response (which then will be cached for 10 minutes on the server for faster responses).
How could I set a cron job timing for this operation? I am having problem with this. I think if I call the page before 10 minutes it will run cached page so I won't get latest updated page, is this true? And if I call that page after 10 mins then will I have to wait for 15-20 seconds to get a response?
How do I manage to make this process where I will get updated feed with swift response? I haven't tried cron job before, this is my first time, so i find this confusing.
My cron command is : */10 * * * * wget http//www.example.com/multifeed.php
Is it right?
You won't have a perfect cron to bring the fresh data as soon as it's available, That's a limitation you'll have to live with I think.
What I would do is run this cron every 2 minutes and try to get new data, I would check to see if the last update is different than what I already have, and if it is, update the file, if it's not, do nothing and wait for the next cron.
This method will provide at most two minutes of stale data.
Another option is to check the mtime of hte file : http://php.net/manual/en/function.filemtime.php
Basically, I visit your page, We check the mtime of the file, if it's greater than 10 minutes, we fetch fresh data, this, combined with the cron, can provide a way for users to always see fresh data. If the freshness of the information isn't that important(can you live with two minutes of stale data?), if it's not that important, simply run the two minute cron.
Hope it helps.
Related
I have a script that parses a large amount of historical data through MySQL queries and other PHP manipulation.
Essentially it is a "Order Report" generator that outputs a lot of data and forms it into an array, and then finally encoding it into json and saving it on my server.
I would like to optimize these more, but to generate a 30 day report takes about 10-15 seconds, 90 days takes 45-seconds to a minute to complete, 180 days anywhere from 2-3 minutes, and for an entire year it takes typically over 5 minutes for the script to finish.
Like eBay and Amazon, I am thinking, due to loading times, a 'File Queue' system is needed, as the user isn't going to want to wait 5 minutes in a loading scenario waiting for a yearly report.
I understand about ajax requests, and how this can be done, or have even read about hidden iframes that perform the request. My concern is, what if the user changes the page (if normal ajax request), or exits the browser, the script execution will not finish.
My only other thought would be to possibly have a cron job along with a MySQL table that inserts the users request, but thinking forward the CRON would have to run about once every minute, all day, every single day, non-stop to check if a report request was issued.
This seems plausible, but perhaps not ideal as having a cron run every minute 24/7/365 seems it'd produce a bit of baggage in its continuity (especially since reports being generated would not be conducted very often).
Can anyone recommend/suggest a route of action to take that won't be a burden to my server, and will still run and complete regardless of user action?
After some research, I haven't found anwers for what I'm trying to achieve.
I found some information about the php sleep() function, and some inforamtions about CRON jobs, but none of these options seem to solve the problem.
Here is what I try to do : I have a php file which may receive webhooks. I need to wait some time (15 minutes for example) before "reacting" to this webhook. So, basically, my script should :
1 - receive the webhook (Already done with current code)
2 - wait some time
3 - do some actions (Already done)
I've already done what I needed without the wait part, and it works very well, bu now I don't know how to do with it...
If I understood it well, Cron Jobs are executed periodically, while I want to wait some time only when the webhook is received.
I thought about the sleep() function, but I'm afraid it may use to much ressources...
How can I do this ?
Update : It seems I can't use the sleep function more than 30 seconds (max execution time I guess ?)
If I was you, I'd build a queueing system so that when a webhook comes in, a row in a database is inserted. I'd add a column for 'execution_time' and set that to 15 mins from when the webhook came in. You can then setup a cron job that runs every minute but only fires where the execution_time is 'now'.
use cron
“At every 15th minute past every hour.”
Minute Hour Day Month Weekday Command
15 * * * * php /homelink/path/filename.php
Note:
You can set an email address for the cron. Every trigger it will send an email notification. If receive email means cron is working, If not then need to check your code. Also you can debug using the appropriate echo's.
I have a cron job that calls a script that iterates through some items and submits them as posts to Facebook Graph API every minute. The issue is, each call takes a few seconds. If there is more than 10 posts to be sent to the API in a given minute, the script runs longer than a minute and then starts causing issues when the script starts running again at the next minute.
The general process is like this:
1. Each facebook profile posts every hour
2. Each of these profiles have a 'posting minute', which is the minute of the hour that they are posted at
3. A cron job runs every minute to see which profiles should be posted do, any given minute, and then posts to them
My question: Is it possible to continue the script immediately after calling the $facebook->api(...) method below, rather than waiting for it to complete before continuing? So that it can ensure to post to all profiles within the given minute, rather than potentially risk having too many profiles to post to and the script exceeding 60 seconds.
$profilesThatNeedToBePostedTo = getProfilesToPostTo(date(i));
foreach($profilesThatNeedToBePostedTo as $profile)
{
$facebook->api($endPoint, 'POST', $postData); // $postData and $endPoint omitted for brevity
}
what you are asking if in PHP you can do Asynchronous calls. There are some solutions here: Asynchronous PHP calls
Just to point there are not the most reliable options.
Is not limiting number of post to 9 each time reduce the times you have exceeded the 60 seconds. But what happens if the Facebook Graph is slow? what do you do?
Or run you code for 60 seconds and once the 60 seconds time is reached close the function. And let it run again the next time.
I am not sure if I understand your question hundred percent clearly. You meant that you want to run the cronjob every minute. However, it takes longer than 1 minute to call facebook API. Do you mean that foreach loop may delay your cronjob?
If so, I wonder if you did a right cronjob script, better to show your cronjob script. Because I think php script should not affect your sh script(cronjob)
i have script which must execute after every n minutes. n minutes is dynamic so that i could not set a cron job to call the script (at a specific time).
so what i did was i stored the time after every n minutes in an array so that when the script is executed, it will first check whether the current time is in the array. if it is found in the array, it continues to executes otherwise it exits.
to execute the script, i must use a cron job to run every minute to check the time in the array. unfortunately, my web host only allows 5 minutes as the least interval. so every time the script is called, i check whether the values between $current_time and $current_time + (4*60) // 4 minutes is found in the array. if it is, and if needed, i use time_sleep_until to delay the script until the time reaches the value found in the array.
so if my script executes at 10:05 and the value found in the array is 10:06, i let the script sleep until 10:06 before it continues to execute. however, if the sleep time is more than a minute or so, i get a Mysql server gone away.
how can i prevent this? or is there a better way to do this?
thanks!
A couple choices, which is better I do not know.
One, is make sure your script works with CLI and after that minute is up call it with the http://www.php.net/exec function (if your host allows it).
Two, is setup a script, with a possible hash as a key and use a header redirect after the minute is up, this would call the script brand new so a new MySQL connection is made.
A third option is to set the script up like in two, except setup a schedule task / cron job on your computer that opens that page (it would have to be in the webroot) and calls it every minute or however you want. This is not a set method, but depends on how much your computer is on.
Fourth, similar to the third but use a free cron job hosting service like: http://www.onlinecronjobs.com/en
Hope that helps. If I think of other options I will update.
I've got a PHP script on a shared webhost that selects from ~300 'feeds' the 40 that haven't been updated in the last half hour, makes a cURL request and then delivers it to the user.
SELECT * FROM table WHERE latest_scan < NOW() - INTERVAL 30 MINUTE ORDER BY latest_scan ASC LIMIT 0, 40;
// Make cURL request and process it
I want to be able to deliver updates as fast as possible, but don't want to bog down my server or the servers I'm fetching from (it's only a handful).
How often should I run the cron job, and should I limit the number of fetches per run? To how many?
It would be a good thing to "rate" how often each feed actually changes so if something has an average time of 24 hours per change, then you just fetch is every 12 hours.
Just store #changes and #try's and pick the ones you need to check... you can run the script every minute and let some statistics do the rest!
On a shared host you might also run into script run time issues. For instance, if your script runs longer than 30 seconds the server may terminate. If this is the case for your host, you might want to do some tests/logging of how long it takes to process each feed and take that into consideration when you figure out how many feeds you should process at the same time.
Another thing I had to do to help fix this was mark the "last scan" as updated before I processed each individual request so that a problem feed would not continue to fail and be picked up for each cron run. If desired, you can update the entry again on failure and specify a reason (if known) why the failure occurred.