I have to send bulk emails to the users. I think of having an endless loop in a cron job, where I want to fetch a few dozens or hundreds users and send emails one by one - updating the the table, that the email was sent. And also I should put some sleep interval, as soon as each packet of dozen(or hundred) users received the email. Basically it looks like
while(1 != 0) {
$notifications = // fetch notifications, where email is not sent
foreach($notifications as $notification) {
// 1) send email
// 2) update table - email was sent
}
sleep(5);
}
Now, is this all right to use, or it is considered a bad practice ?
I know, I can also use multiple crons, lets say every one minute, but to prevent overlapping when using lock file, as soon as the cron starts and the lock file exists(so another cron is still running) it should either
a) wait for some time to the first cron to finish, to start,
or
b) just return empty, allowing the next cron to do the job ASA the ongoing one is done.
The problem with a) is that, what if the crons take lot more time than expected, then after some time I will have bunch of crons in a "waiting" state. About the b) case, what if immediately after the second cron is done(returning empty), the first cron ends, so I will have a gap of ~ one minute, and I need to send emails to users as soon as possible.
also, qsn 2, what is better in performance wise, one cron in loop vs multiple crons?
Thanks
What you are describing a daemon, not a cron task.
There are lots of daemons that run continuously, so no, it's not a bad practice to do that.
If you want the daemon automatically restarted if it crashes, you could have a watchdog task, which continuously checks that the daemon is running, and starts a daemon process if one isn't running.
Another alternative (as you describe) is to have crontask that occasionally attempts to start the daemon; the startup should detect whether the daemon process is already running. If it's already running, leave it be, and just exit. If it's not running, then start another one (in the background, as a detached process. Either way, the crontask completes quickly.
(And it doesn't matter one whit whether the daemon connects to MySQL.)
Personally, I dislike endless loops. I prefer a cron job running every 5 minutes for example.
And you can optimize your script for send max emails quantity in cron job time.
You need to estimate how many emails you will send per minute. I'll assume 1 email per second.
So my idea is:
Query for 290 notifications [10 seconds delay to get and update notifications] and mark them as "sending" status (to prevent next cron dont pick them).
Send emails and save result in array (for later update).
When finished, update notifications status (sent or error).
Just my 2 cents.
Related
Here is the scenario, I'm using Beanstalkd queue to send an email to a huge list of emails (50000+), each email has to have some unique content so the fired job loops over all the addresses, generates the content and sends the mail.
Sometimes the user may want to cancel the operation in the middle of sending, so for example while the job is running and after the mail was sent to say 20000 addresses, the user clicks on "Stop" which should "delete" the job.
what I have done so far is that I managed to get the running job instance, Queue::Push returns the job ID, so I save this ID saved in DB, and when I want to stop the job this is what I tried to do
$phean= Queue::getPheanstalk();
$res = $phean->peek($Job_ID); // returns a Pheanstalk_Job
$job = new \Illuminate\Queue\Jobs\BeanstalkdJob(app() , $phean, $res , 'default') ;
$res = $job->delete() // returns NOT_FOUND ??
$data = $job->getRawBody() // returns correct data, so I'm sure this is the right job instance
so why do I get NOT_FOUND, although when I use
supervisorctl tail -f queuename
I can see that the job is still running and outputting content
Any help ? If there is a better approach than trying to get the job and delete it this way I'm open to suggestions, I thought about saving the job ID in database (ID , status), and when I want to delete it I alter the ID status, and In the loop that is running within the job it checks every time, or maybe every 10 times, and if the status is equal to 1 for example then $job->delete(), but this will be so slow as it will hit the db in every loop.
So you have a a main job, which you reserve() and hold open, and in that job you create many emails directly.
Since the job you are trying to delete is currently reserved, you can't delete it. Even if you could, how would the currently running job be informed by Beanstalkd?
Instead, I would have the main loop check for any jobs on some separate control tube (you could do a quick check every say 10 or 100 emails sent) - just to request a new job, but not waiting if there isn't anything there. If there is a job there, then the main process cleans up and exits.
Another idea is not to actually send email in the main loop, but instead put the details of what emails to send, one per address, into the queue. Other processes read that mass-queue and start sending emails, but again, also read a control tube (with a higher priority message that would be returned ahead of the lower-priority email/details message). If there's anything in the control tube, stop sending emails. You would need at least as many STOP messages in the control tube as you have workers.
I have 5 cron jobs running a PHP file. The PHP file checks the MySQL database for items that require processing. Since cron launches the scripts all at the same time, it seems that some of the items are processed twice, or even sometimes up to five times.
Upon SELECting the file in one of the scripts, it immediately sends an UPDATE query so that other jobs shouldn't run it again. But looks like it's still double processing.
What can I do to prevent the other scripts from processing an item that was previously selected by the other cron jobs?
This issue is called "race condition". In this case it happens due to SELECT and UPDATE, though called one after another, are not a single operation. Therefore, there is a chance that two jobs do SELECT the same job, then first does UPDATE, and then second does UPDATE. And so they proceed to run this job simultaneously.
There is a workaround, however.
You could add a field to your table containing ID of current cron job worker (if you run it all on one machine, it may be PID). In worker you do UPDATE first, trying to reserve a job for it:
UPDATE jobs
SET worker = $PID, status = 'processing'
WHERE worker IS NULL AND status = 'awaiting' LIMIT 1
Then you verify you successfully reserved a job for this worker:
SELECT * FROM jobs WHERE worker = $PID
If it did not return you a row, it means other worker was first to reserve it. You can try again from step 1 to aquire another job. If it did return a row, you do all your processing, and then final UPDATE in the end:
UPDATE jobs
SET status = 'done', worker = NULL
WHERE id = $JOB_ID
I think you have a typical problem to use semaphores. Take a look at this article:
http://www.re-cycledair.com/php-dark-arts-semaphores
The idea would be at first of each script, ask for the same semaphore and wait until it be free. Then SELECT and UPDATE the DB as you do it, free the semaphore and start the process. This is the only way you can be sure that no more than one script is reading the DB while another one is about to write on it.
I would start again. This train of thought:
it takes time to process one item. about 30 seconds. if i have five cron jobs, five items are processed in 30 seconds
This is just plain wrong and you should not write your code with this in mind.
By that logic why not make 100 cron jobs and do 100 per 30 seconds? Answer, because your server is not RoadRunner and it will fall over and fail.
You should
Rethink your problem, this is the most important as it will help with 1 and 2.
Optimise your code so that it does not take 30 seconds.
Segment your code so that each job is only doing one task at a time which will make it quicker and also ensure that you do not get this 'double processing' effect.
EDIT
Even with the new knowledge of this being on a third party server my logic still stands, do not start multiple calls that you are not in control of, in fact this is now even more important.
If you do not know what they are doing with the calls then you cannot be sure they are in the right order, when or if they are processed. So just make one call to ensure you do not get double processing.
A technical solution would be for them to improve the processing time or for you to cache the responses - but that may not be relevant to your situation.
I'm working on a service where the user schedules his tweets so for example i want to post a tweet tomorrow at 12:30 PM. How can that be done ? is cron jobs the best way to do so ? like running a cronjob every 5 minutes and see if there are tweets to post in this interval ? Are there any alternatives ?
Running a cron job is definitely the easiest solution, however there are other approaches available, one such approach would be to use a queue like Amazons SQS
This lets you simply throw things onto a queue to be processed later, by default they are immediately added to the queue in a state ready to be processed immediately, however you can add items to the queue with a timestamp they should remain dormant until. This would be your users predefined tweet time.
You could then have a script running that is constantly listening to the queue for any new tweets that need to be sent, as soon as a queue item becomes available, it will be processed.
The downside to this is of course that it's more effort, the upsides though are that you can scale easier since you can have multiple machines processing tweets and they wont ever send out the same tweet twice (whereas if two machines are running the same cron, there's the chance they'll both send out the tweet)
I have a website in PHP and users can schedule message to be sent. I can sent message with command similar to this one:
php sendMsg.php 249
where number is ID of the message
Many people suggested to use cron jobs, but since I don't want to run this in interval cron is no option(only once - for example after 3 hours).
My idea was as follows:
$seconds = $hours*60*60;
exec('sleep '.$seconds.'; php sendMsg.php 249');
But this wont work because it will block php for further executing. What is the simplest way to achieve this?
You said you don't want to use a cron job because you only want the message sent once, but this is mis-understanding the way that a cron job would be written for this kind of task
Consider a situation where you have many users creating many messages to be sent at various given points in time.
You don't want to have a PHP program sitting running on your server all that time for each of those messages; it would be wasteful of server resources, even if they were all just sleep()ing for the duration.
Instead, one would use a cron job to run a short-lived PHP program once every minute (or whatever interval suits you).
Your message creation program would not be written to acually send the message; instead it would insert it into a database, along with the time it needs to be sent.
Meanwhile, the cronjob PHP program would scan this database every minute to see if there are any messages that are due to send but have not yet been sent. It would then send those messages and mark them as 'sent' on the DB.
This is the standard way to write this kind of thing, so it's not surprising that people are recommending it to you.
Doing it this way means that you never have a program running on your system for longer than necessary. Both PHP programs do their job quickly and exit, meaning that no-one is kept waiting for them.
It also makes it much more robust. Imagine if your server had to be rebooted. If you had a bunch of PHP programs running for hours waiting for their moment to send their message, they'd all be lost. On the other hand, if they had saved their message to a DB, the cron job would find them and send them correctly once the server was restarted.
Put the schedule in a database. Run a cronjob every minute or so, check the database if a message should be sent within this minute, and send it.
Is there a reason you don't want to use a cron job? That would be the simplest and most efficient way of sending the messages.
I would think that a cronjob ist still the right way
Create a table where the to be send messages are stored, with a timestamp when to be send and a flag for isSend
Create a cronjob - start php skript every 1 minute , which sends the messages with timestamp < current time and isSend = false
Ignore suggestions of cron, if you want to simply wait a period of time then use the at scheduler:
$hours = 2;
$command = sprintf('echo "php sendMsg.php 249" | at now + %d hours', $hours);
exec($command);
I have a few ideas about this but here is what I need to do and just wanted some second opinions really.
I am writing a small auction site in PHP/SQL, but I have come up against a hurdle.
When an item finishes, much like eBay, I need to be able to tell that it's finished and send out the emails to who has won it and who has sold it.
The only way I can think of is to schedule a piece of code to keep checking what auctions have ended but surely there is a better way?
The solution can be in multiple parts :
A script that is launched via Cron (every 5 minutes could be good, even less...). It detects the finished auction and put them in a queue.
A script, that pretty much runs continuously, and that processes items in the queue.
Note that :
You have to ensure that an auction is still open before displaying the page ! (a simple test) That way people can't join in after it closes.
For each script, you can use PHP, or any other language
Advantages :
The cron job is very fast, low on resources, and if there are a lot of auction to process, there is no risk it will be run in parallel (and then conflicts)
The queue system ensure that your system won't crash because there is too much going on... It will process the queue as fast as possible, but if it is not fast enough, the website will continue to run. You can however end up with emails being sent hours or days after the auction is closed. But the "limit" is way more predictible, and won't crash your system.
You can extend it in the future with multithreading processing of the queue, distributed processing... This is a scalable architecture.
This architecture is fun.
Additionnal informations :
Regarding the daemon script, I doesn't have to run continuously. What you can do is : at the end of the cron job, if there are items in the queue, then it checks if the other script (processing) is running. If yes then exit. If the other script is not running, it launches it...
The daemon script gets an item out of the queue and process it. At the end, if there are still items in the queue, it processes it, else, it exits.
With this system, everything is optimal and everyone loves each other !
To check if the other script is running, you can use a file and write in it "1" or "0" (= running / not running). The first script reads it, the second writes it... You can also use the database to do it. Or you can maybe use system tools or shell command...
Please be kind to share the SQL script that query the highest bidder based on the bidding end date (how to know the bidding is over) and award the product to the highest bidder
I would setup a cron job to run every 10-20-30-60 minutes etc to send out emails and update the auction details.
If you're script is fast, running it every minute or so may be alright.
Be aware that many shared hosting will only allow you to send out a certain number of emails per hour.
Do these emails need to be sent out instantly?,
I can see 2 possible problems and goals you are trying to achive:
Visual: You want that when a user browse your website, without updating or refreshing the page, it keeps updating the page so that if an audition ends, it appears something like "Audition ended, the item goes to...".
Solution: You should use Javascript and AJAX. (I assume you are already using it for countdowns or something). Make an AJAX call every 5 seconds (could be enough) and update the content.
Pratical: You want that if an audition is ended an user cannot join it. Solution: You can do it just with PHP and mysql. You could create a fields where you store the audition start timestamp and then make a simple if (time() >= ($timestamp + $duration)) {} (where $timestamp is the start of the audition and $duration is the duration of the audition) to block possible bad users trying to do it.