Let's say a website who needs to display the updated content every 5 minutes. For sure we can use a cron job to schedule a PHP script like
$weather = file_get_contents("https://weather.com/country/state/city/day/hour/minute.json");
$upload_to_database = send_data_to_db();
Let's say this simple script takes the data and send it to the MYSQL and then the data is being displayed on frontend by fetching data using AJAX every few minutes or maybe sending the notifications after analyzing the data and so on...
Now, what if we have thousands of cities. How can I create cron jobs for those cities automatically?
I hope you did understand my question.
Another example can be SERP Rank tracker for thousands of keywords and so on...
Instead of creating many separate cron jobs which each do one retrieve/update job, just create one generic cron job which does all the retrieval/updating work.
If you want them to run independently and simultaneously, you could spawn separate processes from that one cron job. You could do that dynamically from PHP, so you can use a current list of cities and spawn a separate process for each.
For example, instead of running "php updatecity.php Washington" (as an example of how you run your php script that updates a particular city), run this:
nohup php updatecity.php "$city" > /dev/null 2>&1 &
This will launch a separate php process silently in the background, running your updatecity.php script with a $city parameter as argument.
Make sure your processes cannot stall or keep running, otherwise you may end up flooding your server with tons of unterminated processes.
Related
Trying to make a dice-roll function in my telegram bot.
How it works right now:
When a user sends "roll" bot replies with sendDice method and sends another message with result like "you rolled 5, you won and blah-blah .."
> how it looks <
The problem is — the second message should not appear instantly, ideally after dice-roll animation is finished.
My first and obvious try on that was to add "sleep(3)" before sending the second message, and it worked fine, until I realized it completely delays the execution of my script for those 3 seconds. (if two users rolled at the same time, one of the users has to wait until another guy's roll will be finished). So it's not cool
What can I use? :c
The easiest option is to add the "task" to the "queue". The queue can be a table in the database with timestamps and chat id, when and to whom to send a message. Start another process, for example, which is started by cron, and it works for one minute. During that minute, he goes to the database and checks to see if there is something that needs to be sent now.
Crontab config
Open crontab
sudo crontab -e
Add next string
* * * * * php /path/to/cron.php >> /path/to/log/file/for/debug.log 2>&1
Cron run your script every 1 minute.
Cron.php "live" 60 second
cron.php:
$now = time();
$expectedTime = $now + 60;
while (true) {
Worker::run();
if ($expectedTime < time()) {
die(0);
}
}
Where Worker::run() your method, which get records from db, check timestamp and send message
From Wikipedia:
In computing, [..] fork is an operation whereby a process creates a
copy of itself.
When your PHP script runs, you can create multiple processes that interact with each other. Those processes run concurrently and asynchronous. This way you can have one process waiting to send the message, while the rest of the script continues to run.
Instead of starting another process, you could also start another thread. The technical difference between the two is explained here:
Forking vs Threading
PHP offers Process Control Extensions for both forking and threading. You might want to check out the example in the PHP documentation for pcntl_fork().
Depending on your needs, you might want to use a framework designed to handle concurrency throughout your application. If that is the case, I would recommend amphp.
What is the best way to perform cron-job automation for multiple users?
Example:
A cron-job needs to run every 10 minutes an call a PHP script that connects to an external API (via curl) and collects data (site visitors and other data) a user has received on an external web property. We need to check periodically every 10 minutes via the API if there is any new data available for that user and fetch it -- and that for EACH user in the web-app.
Such cron-job PHP script call to the API usually takes 1-3 seconds per user, but could occasionally take up to 30 seconds or more to complete for one user (in exceptional situations).
Question...
What is the best way to perform this procedure and collect external data like that for MULTIPLE users? Hundreds, even thousands of users?
For each user, we need to check for data every 10 minutes.
Originally I was thinking of calling 10 users in a row in a loop with one cron-job call, but since each user collection can take 30 seconds...for 10 users a loop could take several minutes and...the script could timeout? Correct?
Do you have some tips and suggestions on how to perform this procedure for many users most efficiently? Should separate cron jobs be used for each user? Instead of a loop?
Thank you!
=== EDIT ===
Let's say one PHP script can call the API for 10 users within 1 minute... Could I create 10 cron-jobs that essentially call the same PHP script simultaneously, but each one collecting a different batch of 10 users? This way I could potentially get data for 100 users within one minute? No?
It could look like this:
/usr/local/bin/php -q get_data.php?users_group=1
/usr/local/bin/php -q get_data.php?users_group=2
/usr/local/bin/php -q get_data.php?users_group=3
and so on...
Is this going to work?
=== NOTE ===
Each user has a unique Access Key with the external API service, so one API call can only be for one user at a time. But the API could receive multiple simultaneous calls for different users at once.
If it takes 30 seconds a user and you have more than 20 users, you won't finish before you need to start again. I would consider using GearMan or other job server to handle each of these requests in an async way. GearMan can also wait for jobs to complete, so you should be able to loop over all the requests you need to make and then wait for them to finish. You can probably accomplish the same thing with PHP's pthread implementation, however, that's going to be significantly more difficult.
I'm creating a website and in my website I have a MySQL database. In the database there is a table that has two property "date" and "time". I want to run a php code on a date and time according to records of table, automatically. How can I do it?
I see three options, listed here in order of difficulty to implement, easiest to hardest.
Create a script that is scheduled to execute every now and then (exact timing would depend on your needs for accuracy), which checks the database to see if there are any items in there that should have been executed in the time span since it's last execution. If there are, execute them.
Write a daemon application/script to monitor your database, and execute the tasks when needed. This one is not all that different from #1 except that it wouldn't require scheduling or manual triggering. That could be handled within the application/script itself.
Create a script that checks the database for changes, and schedules/reschedules/removes any tasks accordingly. This script would then be scheduled to execute every now and then and/or be executed manually by the application that manages the database.
Write a php script (that queries the database and runs the script) powered by crontab (for periodically running your 'query'-script) in a shell (on linux)
I want to create a feature on my website that allows users to schedule a post at a specific time. The posts will be placed in a queue and then posted at the specified time. What is the proper method for handling this? Do I need to create a separate cron job for each individual scheduled post?
Typically this is done with a single cron job acting as "trigger". That trigger (a php script) checks the list of pending jobs in the database and executes one or more.
It might spawn sub requests for each job to improve robustness, but this adds load to the system.
It is important to mark started jobs inside the database so that in case of concurrent trigger processes (whyever, for example system load), a single job is not started twice. This is especially true for jobs that crash...
You might also want to implementing a locking strategy, so that only one single concurrent trigger request is possible.
Here's what I'm trying to accomplish in high-level pseudocode:
query db for a list of names (~100)
for each name (using php) {
query a 3rd party site for xml based on the name
parse/trim the data received
update my db with this data
Wait 15 seconds (the 3rd party site has restrictions and I can only make 4 queries / minute)
}
So this was running fine. The whole script took ~25 minutes (99% of the time was spent waiting 15 seconds after every iteration). My web host then made a change so that scripts will timeout after 70 seconds (understandable). This completely breaks my script.
I assume I need to use cronjobs or command line to accomplish this. I only understand the basic us of cronjobs. Any high level advice on how to split up this work in a cronjob? I am not sure how a cronjob could parse through a dynamic list.
cron itself has no idea of your list and what is done already, but you can use two kinds of cron-jobs.
The first cron-job - that runs for example once a day - could add your 100 items to a job queue.
The second cron-job - that runs for example once every minute in a certain period - can check if there are items in the queue, execute one (or a few) and remove it from the queue.
Note that both cron-jobs are just triggers to start a php script in this case and you have two different scripts, one to set the queue and one to process part of a queue so almost everything is still done in php.
In short, there is not much that is different. Instead of executing the script via modphp or fcgi, you are going to execute it via command line php /path/to/script.php.
Because this is a different environment than http, some things obviously don't work. Sessions, cookies, get and post variables. Output gets send to stdout instead of the browser.
You can pass arguments to your script by using $argv.