I'm using laravel 5 and i have about 2000 instagram account's. And i need to check their current followers/following status every day and save them in my database.
The cron job was a perfect choice because i can say at what time of the day i want to run it, and it worked fine when i had less number of accounts.
But now my cron job often closes after i have download data for around 600 usernames. I read somewhere on internet that cron jobs are good for short running tasks that needs to be scheduled at specific times.
My tasks will grow with number of account's and currently it needs around 2 or 3 hours to finish. What is the best way to do this ?
Put some column such as 'in_process' check into your DB regarding Instagram account for which you are processing followers/status.
Run another job if one is killed. Now your job should process accounts which haven't been processed yet.
Related
Currently, I have done the following:
I created one scheduled task which runs daily to get the Scheduled time from Mysql DB for the currentdate and store it into the .txt file
SELECT workflow_id, DATE_FORMAT(schedule_datetime,'%H:%i')TIMEONLY FROM scheduling_event
where DATE(schedule_datetime) = CURDATE()
Created one more scheduled task that runs each 5mins to check if the scheduled time present in the .txt file matches the CURRENT TIME if yes then it calls the scheduled_program.php file.
The issue here is - this is not an efficient way if nothing is scheduled on the current date. So Is there any way to create/update a dynamic scheduled task instead of running each 5mins? ie: the first scheduled task will run and take the scheduled time on the current date then it will create a task based on the scheduled time. if the day ends delete all the scheduled tasks for the day.
Note: Number of the scheduled task is not fixed. imusing Windows 10, php7.
I am trying to achieve, run a scheduled_program php file on schedule Date and TIME
It looks like you're trying to solve a problem that's not serious. I guess you don't want to waste your computer's time running a cron job every five minutes if it has nothing to do.
But here's the thing:
a cron job
that runs a php program
that does a single query
to retrieve a list of workflows
and run them
has negligible cost if you run the cron job every five minutes, or even every single minute, and there are no workflows to run.
On the other hand, debugging and troubleshooting cronjobs is hard, especially in production.
So, I respectfully suggest you keep this system as simple as you possibly can. You will have to explain it over the telephone to somebody in the middle of the night at least once. That's the unfortunate truth of scheduled tasks. The dynamic scheduled task system you propose is not simple.
I am working on a web application that requires php code to run at a specific date/time.
Some hypothetical examples would be sending a user an email at 09:00 on their birthday or modifying a database entry (mySQL) at a predetermined date and time.
What would be the conventional way to implement this kind of scheduling feature?
I've seen cron-jobs been used for similar requirements but would this be feasible for a large amount of scheduled tasks?
Depends on the size of your project, but usually we don't have one script do all the cron jobs, because things need to happen at different times. Some emails might send at 9pm, some database cleanup might happen at midnight, and sessions might expire every few minutes. The best thing to do is set up separate cron jobs for each thing. There are a couple exceptions for this:
Some tasks need to run very frequently, like every 30 seconds. For that we set up a cron task which checks "ps" to see whether its own process is already running, and have it just wait 30 seconds and loop.
Some tasks make more sense to run when a queue is full. For that we trigger the task when a certain user-based script executes the 100th or 200th or 300th time. For example when a user pings 100 times without doing anything else, we log them out without using a cron task, but there is a separate cron task which checks if they've been inactive for 10 minutes.
I have a database where I have to save a lot of data.
All this data recovered from an Unofficial API that retrieve data from official site.
Unofficial API is the only one I can use, because official API is really short developed, but Unofficial API is really really slow.
I couldn't do anything to speed up this process.
To automate data update, I've created php pages that recovers data from Unofficial API, combine it and store it into the database.
These pages refresh every 30 seconds continuing process in order to avoid server time limit, set to 30 secs and not modifiable.
Than I've created multiple cron jobs.
But there's a problem.
Cron jobs work for at least 30/60/90/120 seconds, but the whole data update process can goes on for 20/30 mins.
Do you have any good idea on how to solve this problem?
I'll be more clear, sorry.
The problem is this:
- script is long about 20/30 mins, divided in multiple auto-loads, this recover and load all the data into the database;
- a cron job works for max 120 seconds than it returns a time-out error.
I need, if possibile, to find a way to have all data loaded into database, four time a day.
Do you have any idea?
Second update.
This is not a question related to TIME LIMIT of SERVER, I've overcome this problem cutting my script in pieces.
This is a metter of CRON JOB TIME LIMIT.
I cannot do anything (that I know) to accelerate script execution, so my data recover is really long and I need time execute it all, but cron jobs are too fast.
I am developing software that is similar to fantasy football applications.
I have user accounts with information pertaining to each of them stored in a relational database in mysql. I have a php script that I want to run after all of the games are finished to update every member's stats at once (there could be thousands of users).
After researching it seems like the best solution would be to set up a cron tab using Unix.
I am wondering if this would be the correct solution?
If I set up this crontab to run every Tuesday would it run the PHP script which would update the user information?
Is a cron tab easy to set up with Unix for a simple website?
Setting up crons is really simple. If you have shell access, simply type #crontab -e and it will bring you to the cron settings. Check out this link: Crontab - Quick Reference.
What you simply do is create a PHP script and link it through the cron tab. If you are running a server with cPanel, there is a built-in cron manager too, if you prefer to do it in a GUI.
You did not give enough information for me to say anything conclusive about this, but you want to balance the script out so that it either 1) runs frequently enough so that it has to update small enough changes--for example, if 100 records are changed every hour, running the script every day will mean it has to update 2400 records, whereas if you run it once a week, it has to update 16800 recods, which probably is so severe that it will raise your server load. And/ or 2) break up your updates in chunks. For example if records only change once a week, but at that time 16800 records change, then you might want to either have your scripts run for 1000 updates and put it to sleep for a while, run some more, sleep, etc (this is a bad habit, but it is the simplest I suppose), or you run your cron script frequently, every time it does 1000 updates, and at the end of it, it will leave a trace (for example, the last ID number) of the last change it has updated into the database so that the script can pick up on that the next time it runs.
If you want me to draft up some more elaborate examples the illustrate the two different approaches, please let me know.
I have a Cron Job with PHP which I want to set up on my webhost, but at the moment the script takes about 20 seconds to run with only 3 users data being refreshed. If I get a 1000 users - gonna take ages. Is there an alternative to Cron Job? Will my web host let me run a cron job which takes, for example, 10 minutes to run?
Your cron job can be as long as you want.
The main problem for you is that you must ensure the next cron job execution is not occuring while the first one is still running. You have a lot of solutions to avoid it, basically use a semaphore.
It can be a lock file, a record in database. Your cron job should check if the previous one is finished or not. A good thing is maybe sending you an email if he cannot run because of a long previous job (this way you'll have some notice alerting you that something is maybe getting wrong) By default cron jobs with bad error dstatus on exit are outputing all the standard output to the email of the account running the job, depending on how is configured the platform you could use this behavior or build an smtp connexion on the job (or store the alert in a database table).
If you want some alternatives to cron jobs you should have a look at work queues. You can mix work queues with a cron job, or use work queue in apache-php envirronment, lot of solutions, but the main idea is to make on single queue of things that should be done, and execute them one after the other (but be careful, if you handle theses tasks very slowly you'll get a big fat waiting queue).
A cron job shouldn't have any bearing on how long it's 'job' takes to complete. If you're jobs are taking 20 seconds to complete, it's PHP's fault, not cronjob.
Will my web host let me run a cron job which takes, for example, 10 minutes to run?
Ask your webhost.
If you want to learn about optimizing php scripts, take a look at Profiling PHP Code.