My script sends notification emails of a new comment, this could be to 50 members and 50 emails need to be sent, which could take 20 seconds which is way to long for the user to wait! What's the best way in PHP to do this, is there a way to do asynchronously?
A simple way might be to store the necessary information (email addresses, content) in a database and have a batch process run every minute or so with a cron job. The batch process can query the database for pending emails and, if any are to be sent, go through them and then delete the database entries.
is there a way to do asynchronously?
Yes, there is!
exec('wget PATH_TO_YOUR_SCRIPT_THAT_SENDS_THE_NEWSLETTER > /dev/null &');
Note that the database alternative is a pretty good one too. But this should work too if you're on Linux (and doesn't require a database).
I'd use something like RabbitMQ. Your website acts like a producer sending the email requests to Rabbit; then have a consumer running that processes the requests from Rabbit.
Advantages - if your consumer falls over then when you restart it will pickup from where it left off (last acknowledged request).
Indeed it can be done asynchronously.
The simplest way is to insert the email data into a database rather than actually running the emails, and then have a cron job that periodically actually sends the emails.
There are of course other ways too, but that will probably be the most straight forward.
You can use cURL POST to start an asynchronous script. Set the timeout to a short period so your script can resume after the POST request has been made. You can set the email information in the POST request or store it in a data base table.
Related
I am using the PHPMailer library to handle the sending of emails from within my application.
The problem is, when some emails are triggered to be sent (such as when a contact form has been submitted, a new user registers, etc), it could take 1-3 seconds for the page to load while the email is sending. If there is ever a problem sending the mail, the delay can be more.
I was thinking of saving any emails that need to be sent into a pending_emails table in my database, then just have a cron job ran every minute which would send out all those emails, then remove them from the table.
My question is, does this seem like a logical thing to do? Are there any potential resource concerns I should have with a cron job running every minute vs sending the email in runtime? (I need to run the cron job often, as someone may be waiting on an urgent message, for example "reset password" email)
You got everything right already.
Sending at runtime, just when you respond to the user's HTTP request, is the easiest thing to do. But the response is slowed down a bit by this, of course. That's not too bad in a small application, because sending email is faster than one might think. It definitely works.
Implementing a message queue is the more elegant and scalable approach, of course. But it takes a little more work. Your idea of using a pending_emails database table is totally valid. There are libraries and components for such queues, but you don't have to use them.
This is a very opinion based question so you're going to get a lot of different, conflicting answers because there are some who might tell you its ok to make a user wait 1-3 seconds since its not that long but I tend to disagree with that. What I typically do instead, however, is use a Queue.
There are ways to create a queue WITHOUT using 3rd party software, but there are some excellent tools out there such as RabbitMQ, Iron.io or Beanstalkd which can be extremely helpful to performing tasks in the background. These services push your task into a queue and these items in the queue are processed in a timely manner in the background, but the user gets an almost immediate response (depending on what you're doing). This is how I usually handle more resource intensive tasks, like sending an email, in the background to avoid holding up the response to a user.
Best of luck.
Look into threading (PHP Threading). I would suggest you create a new thread which invokes the sending of the email. This way, you can return a response to the user without waiting for the email to be sent, and the email sending process would run in parallel in another thread.
In my application, the user needs to register through a form, where I have to send three mails and do some other (huge) database checks. It takes a lot of time, is it possible to make the whole task as background process or some other alternates is there?
If your database activities take too long then you need to rethink your design. However if the delay is due to emails, then just store the emails in DB or in files. Create a cron job that sends out these queued emails every 5/10/15 minutes(and then delete them).
maybe you can once a user is registered flag him as pending in your database.
Then you could defer the work in a python or php routine running in the background continuously who would look for any pending request, do the check, send the emails and finally update the database accordingly.
the user during this time would be in a registered but pending status, but at least from a visitor point of view, he is not stuck waiting for everything to be processesed.
You cannot make a PHP script that has been started over a webserver process a background process.
I would check if I can optimize the database (probably, you have insufficient indizes), and if that doesn't fly, build a second process that gets started regularily (maybe once every five minutes or so) on the CLI side with a cronjob, showing the user a "Thank you for your registration" page...
As per my comment elsewhere, spawning a long running process from PHP is a practical solution bearing in mind a few caveats if the performance problems are unavoidable.
However "send 3 mails" should not take an appreciable amount of time (I don't know what the database checks are). You need to spend some time looking at optimizing the existing process.
Other ways to solve the problem would be conventional batch processing, offloading the heavy lifting to a multi-process/multi-threaded daemon via a network call or asynchronous messaging system, or even a single threaded job processor using a message queue.
I have a web application where users can create topics and also comment on other topics (similar to what we have here on stackoverflow). I want to be able to send notifications to participating users of a discussion.
I know the easiest way to go about it is to hook the notification to the script executed when a user interacts with a discussion. In as much as that seems very easy, I believe its not the most appropriate way as the user will need to wait till all the emails notifications (notification script finishes execution) are sent till he gets the status of his action.
Another alternative I know of is to schedule the execution of the notification script using cronjob. In order for the notification to be relevant, the script will be scheduled to execute every 3 to 7 minutes so as to make sure the users get notification in a reasonable time.
Now my concern is, will setting cronjob to run a script every 3 minutes consume reasonable system resource putting into consideration my application is still running on a shared hosting platform?
Also, am thinking is it possible to have a scenario where by the comment script will trigger or notify a notification script to send notifications to specified email addresses while the comment script continues it's execution without having to wait for the completion of the notification script. If this can be achievable, then I think it will be the best choice for me.
Thank you very much for your time.
Unless your notification script is enormously resource-intensive and sends dozens or hundreds of messages out on each run, I would not worry about scheduling it every 3-7min on a shared host. Indeed, if you scheduled it for 3 minutes and found performance sagging on your site, then increase it to 4min for a 25% reduction in resources. It's pretty unlikely to be a problem though.
As far as starting a background process, you can achieve that with a system call to exec(). I would direct you to this question for an excellent answer.
IMO adding a "hook" to each "discussion interaction" is by far the cleanest approach, and one trick to avoid making users wait is to send back a Content-Length header in the HTTP response. Well-behaved HTTP clients are supposed to read the specified number of octets and then close the connection, so if you send back your "status" response with the proper Content-Length HTTP header (and set ignore_user_abort) then the end user won't notice that your server-side script actually continues on its merry way, generating email notifcations (perhaps even for several minutes) before exiting.
I have a php script which queries a list of clients from a mysql database, and goes to each client's IP address and picks up some information which is then displayed on the webpage.
But, it takes a long time, if the number of clients is too high. Is there anyway, I can send those url requests (file_get_contents) in parallel?
Lineke Kerckhoffs-Willems wrote a good article about Multithreading in PHP with CURL. You can use that instead of file_get_contents() to get needed information.
I would use something like Gearman to assign them as jobs in a queue for workers to come along and complete if this needs to scale.
As another option I have also written a PHP wrapper for the Unix at queue, which might be a fit for this problem. It would allow you to schedule the requests so that they can run in parallel. I have used this method successfully in the past to handle the sending of bulk email, which has similar blocking problems to your script.
I have a list of around 5,000 to 10,000 (individual user supplied) email addresses from people all over the world, each associated with their username and language codes. I also have a single message translated into the different languages of the users that I want to email. Now, I would like to send a single plain text email to each of the address, with the actual text of the email varying based on the user language, and personalised with the username of the person I'm emailing.
Because of the personalised requirement, and the fact that they will only be emailed once (per year or two with a overlapped but different user list), formal mailing list is probably (& preferably) out. 3rd party bulk email service is also out.
Ignoring programming time, what is the least manually time consuming way to do this in (preferably) PHP? (I am writing the script(s), but not necessarily the person that end up "pressing the button" to send it.) The ideal result is the person sending only having to type a single command to run the script (supplying the email list) and all the email will be sent with no more user intervention. This mean I am looking to avoid things like setting up cron jobs to run the script repeatedly until the email list is exhausted.
When this was done before a year ago, I wrote a PHP script that simply read in the email list line by line processing the username, email address, and language code and build the desired email text out of that before supplying it to PHPMailer to send individually. The problem I had was the script timing out and me not knowing where it got up to so that I can trim the email list at the right place to start again. I ended up manually splitting up the 1 email list into several sub-list that was short enough so that the script doesn't time out. How do I either avoid timing out, or keep track of where the script is up to email address wise so that it can be restarted manually and no person is sent emails more than once?
What other issues are there to take into account, such as avoiding blacklisting etc.?
You need to read about the function set_time_limit and maybe ini_set('memory_limit', xxMB') for memory;
You can run a php cli script from your web process, which forks (pcntl_fork) then kills the parent (the parent cli process exits). The webserver thread running the script can then continue with other code or exit.
Now the cli child process (A) can fork (and become the parent) and monitor a child (B) which sends out the emails. If the child B dies, the now parent A process can fork again and the new child continues where the previous left off.
You have to keep track of who you sent email to in a file/db/shared memory or through paired sockets (socket_create_pair) with the parent process.
I hope you get the idea.
PHP's not the right tool for the job here - you'll want something that runs independent of a web server (although potentially triggered by it) or you'll certainly run into timeouts.
For example, you could have PHP system() out to another script (Perl? Python?), which reads job information from a database then forks into the background to do its work.
You said no Cron scripts - do you not have access to Cron on your server? Or is your concern not wanting to manually set up the Cron job every time a mail job has to be sent out?
You might want to look at http://pear.php.net/package/Mail_Queue - it's something I've been using recently. You can have your user-called script dump all the emails at once into the mail queue. This is much faster than actually sending the emails on the fly since you won't have to deal with communication between servers, but rather just your database.
Then you have a Cron script that, every few minutes, calls the Mail_Queue::sendMailsInQueue command with a limit on the number of emails it sends per Cron call. You won't have to mess with the Cron script if that's your concern - if the queue is empty, it'll just exit. Your email will then trickle out of the server at a reasonable pace per call. This will prevent timeouts due to the limit on emails per cron call. It will also help avoid trouble with other mail servers, who might not be happy to suddenly be hammered by a lot of emails from you at once.