I'm working on an application that will handle a lot of email sending, and I'm looking for a minimal email queue solution.
What the sending code will do, is get the "To", "From", "Subject", "Text", "Format" fields from the queue, generate the headers, and send the email. If the sent is not successful, it can be retried. I would like a priority system too, with, at least, two levels of priority.
I've been thinking and the ideas I got are these:
MySQL: as everything else in the system goes through MySQL, I thought in using a MySQL table as a queue. The problem is that the sender must be always looking on the table, which causes a processor high load.
Files: a queue can be done through XML files in a directory. This is bad for everything (performance, server life...)
FIFOs: I've used FIFOs in C applications, but probably this is too low level for a high level application, and raw data is a bit harder to process (sizes, order of parameters...).
So I'm looking for ideas on how to do this email queue in an easy way. The system is done in PHP, and I'd like it to be in PHP, if there's solution.
Thanks in advance.
I've developed an email queue system for PHP that does exactly what you're asking for, check it our here. https://github.com/tin-cat/emailqueue
I've done something similar to this before to send about 200,000 emails in a day. Since they weren't time critical, I generated them (with Mail_Mime), and stored them all into a database, with Mail_Queue, sending them out with a shell script that kept re-running itself if the load average of the machine was OK.
Today, I'd do it with a Symfony-based system around Swiftmailer and White October SwiftMailer DB Bundle.
To have it avoid the database (which isn't optimum, but it does Just Work) I would use the DBBundle as a base and instead have it go through a queue system, such as Beanstalkd (it wouldn't be a big job to send it to a queue instead of a database table). The sending system can just delete the job if it decides it's 'too old'. Adding priorities to the queue job is also very easy - it's built right into Beanstalkd.
You could also elect to simply have the message in the queue be, "send user X an update email" - and the queue-runner goes to the original DB to assemble the email just before it is sent.
Related
I'm creating a site that sends out a daily news email to about 800 users, at a time they can specify. My problem is that my script takes a long time to run and times out, so I'm looking for some advice on how I could be approaching this better.
Current approach:
Users are placed in a 'mailing queue' database table with their ID, receive time, and a sent flag.
I'm then running a CRON script every minute which does the following:
Grab all from mailing queue with a 'receive time' less than or equal to now, that haven't sent
Loop through the users, joining a preferences table to get their chosen categories (up to about 30 per user).
For each category, find the latest 3 articles
Prepare an HTML email with this content using PHPMailer
PHPMailer is using Mailgun SMTP to avoid overloading my SMTP server
Send mail to user, mark as sent in database
My observations so far are:
When testing the script by running in-browser, it runs incredibly slowly for a few minutes then times out (without sending any emails).
When running every minute via CRON, it sends way over the number of emails (about 1400) over the course of 40 minutes, I guess because the script is overlapping itself and the sent flag is not reliably updated.
The majority of users are set to receive their email at the same time, so I'm doing 'worst case scenario' testing on this basis
Questions
Is my script far too heavy, by querying the database and generating the HTML email content for each user on the fly? I'm wondering if it would be better to generate the content ahead of time and store against the user in the mailing queue.
Would a queue manager like Beanstalkd help? I've had a look into it, but am struggling to see how to implement into my routine.
Ultimately I need the emails to be sent reliably to each user at the time they expect.
Any advice much appreciated!
You can do this in PHP, but you probably shouldn't. You're trying to build a mail server when there are much better ways of doing this, which mainly involve using a mail server.
Sending high volumes of email during page loads is not workable – it can be troublesome even for single messages, yet many still try. Approach it like this:
Store your list in a database.
When you want to send, generate a record representing each message to send (essentially a copy of the list).
Have a daemon (a long-running task) or cron job that sends the messages in chunks.
Create messages one at a time and submit them to a local mail server.
Use DKIM signatures.
As each message is sent, mark them as sent in the database, but you need to be very aware of how database transactions and locks work for this to work safely and avoid duplicates – do this right and overlapping processes work just fine.
You can generate messages as fast as you like, and your mail server will deal with queuing, onward delivery, retries, bounces.
Use VERP addressing, and feed bounces into a bounce handler (be warned, writing these is not fun!), and have that prevent sending to bouncing addresses in future.
This approach works well – it is exactly how my own system ([Smartmessages.net](https://info.smartmessages.net/ , which is built in PHP) works, and I can sustain over 200 messages per second using multiple message generators running in parallel (database transactions FTW!).
If you find all this a bit too much (it is very difficult), you're probably better off using a commercial sending service (like my own) or hosting a DIY solution, such as Mailcoach by my good friends at Spatie. Either of these would work well, and your list is pretty small – I'm often handling lists of over 100,000.
I'm just curious how would you handle a scenario where a lot of PDF's has to be generated on the server and be send to the user by email. You're not able to temper with the PDF because it needs to be 100% secure or close to that number.
For example the PDF contains the order you just made in a webshop, proof of purchase or something like that.
The application will have a lot of concurrent users. For this question I will use Laravel as a base platform for the web application.
I had the idea of running a cron job at night that will generate all this PDF's at once and send per e-mail.
What is considered best practise in this scenario?
For example the PDF contains the order you just made in a webshop, proof of purchase or something like that.
Given that these will presumably occur throughout the day, a queue may be a better solution than a cron. Every time someone does an action that'd require a PDF, fire off a queue job. A background process will check for queued jobs and process them.
This avoids having a giant backlog, protects you in the case a cron fails, and gets PDFs out to the clients in a more timely fashion.
I am using the PHPMailer library to handle the sending of emails from within my application.
The problem is, when some emails are triggered to be sent (such as when a contact form has been submitted, a new user registers, etc), it could take 1-3 seconds for the page to load while the email is sending. If there is ever a problem sending the mail, the delay can be more.
I was thinking of saving any emails that need to be sent into a pending_emails table in my database, then just have a cron job ran every minute which would send out all those emails, then remove them from the table.
My question is, does this seem like a logical thing to do? Are there any potential resource concerns I should have with a cron job running every minute vs sending the email in runtime? (I need to run the cron job often, as someone may be waiting on an urgent message, for example "reset password" email)
You got everything right already.
Sending at runtime, just when you respond to the user's HTTP request, is the easiest thing to do. But the response is slowed down a bit by this, of course. That's not too bad in a small application, because sending email is faster than one might think. It definitely works.
Implementing a message queue is the more elegant and scalable approach, of course. But it takes a little more work. Your idea of using a pending_emails database table is totally valid. There are libraries and components for such queues, but you don't have to use them.
This is a very opinion based question so you're going to get a lot of different, conflicting answers because there are some who might tell you its ok to make a user wait 1-3 seconds since its not that long but I tend to disagree with that. What I typically do instead, however, is use a Queue.
There are ways to create a queue WITHOUT using 3rd party software, but there are some excellent tools out there such as RabbitMQ, Iron.io or Beanstalkd which can be extremely helpful to performing tasks in the background. These services push your task into a queue and these items in the queue are processed in a timely manner in the background, but the user gets an almost immediate response (depending on what you're doing). This is how I usually handle more resource intensive tasks, like sending an email, in the background to avoid holding up the response to a user.
Best of luck.
Look into threading (PHP Threading). I would suggest you create a new thread which invokes the sending of the email. This way, you can return a response to the user without waiting for the email to be sent, and the email sending process would run in parallel in another thread.
I've got a web application with approx. 30k users and growing fast. We want to enable some functionality over email, which means assigning each user a unique email address (for security reasons). Emails sent to these unique addresses should be forwarded to a PHP script. The PHP script needs access to all parts of the email - the headers, body and any attachments. I'm not sure what the right way is to set this up. We're currently using sendmail and would prefer to stick with that (this is linux, if it matters). I've been able to get an individual email forwarding to a script, but it is not scalable (in my mind) to create 30k aliases and then a new one each time a new user signs up. I'd be much happier with some type of catch-all / RegEx-based solution that just tells the PHP script the unique email address that was the recipient, which would let us look-up the user.
30k accounts does not give us any information about the traffic you expect to handle. But let's assume it is another question.
You could create catch all type of account, which would simply accept any email in domain you create your users' addresses in (basically *#domain). Then you could launch your processing script on each incoming mail (you usually would be able to fetch whole mail body from stdin stream (in PHP accessible via php://stdin), and process it as you like. But this is bad if you want to use PHP (or almost any scripting language), as each mail would need to spawn new PHP process, init all the extensions and so on. It all this takes time and resources, and one bigger spam flood and your box is almost toasted.
Alternatively, you could collect your mail in mailbox as any other mail and then have your script getting it from there (either directly, digging into maildir or via mail protocols like IMAP or POP). This looks better from administration standpoint and as long as your mail server can accept incoming mails, your whole system still works. You can even try to "partition" your system by setting some filter rules (on MTA level) and deliver incoming mails not to single mailbox but more. Then you could process these mails in parallel with less hassle (no need to bother about processing same mail with more than one instance etc). "Partition" schema could be quite anything, i.e. depending on your full address naming scheme, if letter based it'd be separate mailbox for a*#, b*# or [a-d]*#] (depends on traffic), does not really matter.
Alternatively, you could try to put bodies of incoming mails into database and then process as described above, without any need of POP/IMAP handling.
Whatever you choose I recommend NOT putting your processing script of any form directly in mail receiving process chain. It usually is not a problem to have 1 minute delay (so you can have any sort of daemon running and fetching mails for processing, cronjob etc), users can wait that much w/o complaining - it looks like instant processing is most likely not mandatory in your case as well. But if it is, then PHP might not be the best tool choice I am afraid.
EDIT: (to answer comment question and not hit comment size limit)
I am not familiar with Sendmail enough to give you precise configuration directives here (I am on customized qmail), but some general hints would apply I hope. First, I believe Sendmail supports virtual users (which means mail boxes for users not present in /etc/passwd). If so, then MTA usually should let you either query DB (to find out if the target email address is valid and where is that user's home dir to put mail into) with some built-in providers. Or (which is even better) invoke external program to do that. The latter is a winner here because there are no users really, so you could write small script or (again, better for performance) C app that would do the trick. Since all you need is to "separate" incoming mails into several physical mailboxes based on some simple predefined rules, couple of if()s + substr()s and you are done. Couple of lines, no DB needed, pretty quick. Or, even better, you could even try the dumbest roundrobin approach here as well - it is in fact irrelevant if mail hits accountA or accountB as long as 50% of mails ends in A and other 50% in B (or 25% if you go for 4 target accounts etc) and that's the only condition you should care I think. You can try to use procmail for that, but I think it is pointless and is not best for performance.
As for putting mails into database. It depends on MTA - by default nobody does so as it slower than putting mails into maildirs (as DB is usually other host), but there still are couple of solutions. Either look for 3rd party patches for Sendmail that do that or add another script/app that would "copy" mails from physical mailboxes into DB. You may say - but it would be slower. Sure, but again - I do not know if non-realtime processing really matters in your case (my blind guess - it does not), so why really bother. If you save couple of milliseconds here but loose minutes on further maintenance of the whole solution, then choice is quite obvious.
BTW: just in case - whenever I wrote mailbox I definitely meant container for user's mails, NOT mailbox file format. As for file format maildir (or something derrivative) is much, much better choice.
Do you really want to fork off a PHP for every incoming email? That could get expensive if you're dealing with decent mail volume. Why not have a custom callback in your mail server that processes the incoming mail and inserts into a database, and then have a worker script that continually polls for new messages to process?
The mail-to-db part should be much lighter weight than doing all the processing, and this gives you a free queue so that if you get a bunch of requests at nearly the same time, your system doesn't fall over trying to process them all at once.
did a similar implementation of this, though aon a smaller scale... should scale to meet your purposes though.
two seperate problems
(1) mail - catch-all as mentioned is a good solution, though you will have to deal with spam and malformed email addresses (which may be a good thing or a bad thing). If you own the server, simply aliasing the "ficticious" accounts and directing the mail server to drop them all in the same central mailbox will accomplish the task as you don't need seperate actual accounts, and the header information will remain intact. the specifics depend on your mail solution, but any mail server should be more than equipped to handle this task and the volume, and you should be able to script the alias creation... otherwise back to catch-all
(2) PHP parsing and processing - PHP's builtin pop3 functions can easily handle checking the (now) single email account and processing the messages; Some examples of functions I used to go through this are;
function pop3_login($host,$port,$user,$pass,$folder="INBOX",$ssl=false){
$ssl=($ssl==false)?"/novalidate-cert":"";
return (imap_open("{"."$host:$port/pop3$ssl"."}$folder",$user,$pass,OP_SILENT));
}
function pop3_stat($connection){
$check = imap_mailboxmsginfo($connection); # changed from $imap
return ((array)$check);
}
function pop3_retr($connection,$message,$section='1'){
return(quoted_printable_decode(imap_fetchbody($connection,$message,$section)));
}
function pop3_dele($connection,$message){
return(imap_delete($connection,$message));
}
function pop3_close($connection){
return(imap_close($connection,CL_EXPUNGE));
}
all very generic, and there are probably classes out there if you don't need the control. Bottom line, PHP has all you need to quickly parse and process mail messages.
I have a list of around 5,000 to 10,000 (individual user supplied) email addresses from people all over the world, each associated with their username and language codes. I also have a single message translated into the different languages of the users that I want to email. Now, I would like to send a single plain text email to each of the address, with the actual text of the email varying based on the user language, and personalised with the username of the person I'm emailing.
Because of the personalised requirement, and the fact that they will only be emailed once (per year or two with a overlapped but different user list), formal mailing list is probably (& preferably) out. 3rd party bulk email service is also out.
Ignoring programming time, what is the least manually time consuming way to do this in (preferably) PHP? (I am writing the script(s), but not necessarily the person that end up "pressing the button" to send it.) The ideal result is the person sending only having to type a single command to run the script (supplying the email list) and all the email will be sent with no more user intervention. This mean I am looking to avoid things like setting up cron jobs to run the script repeatedly until the email list is exhausted.
When this was done before a year ago, I wrote a PHP script that simply read in the email list line by line processing the username, email address, and language code and build the desired email text out of that before supplying it to PHPMailer to send individually. The problem I had was the script timing out and me not knowing where it got up to so that I can trim the email list at the right place to start again. I ended up manually splitting up the 1 email list into several sub-list that was short enough so that the script doesn't time out. How do I either avoid timing out, or keep track of where the script is up to email address wise so that it can be restarted manually and no person is sent emails more than once?
What other issues are there to take into account, such as avoiding blacklisting etc.?
You need to read about the function set_time_limit and maybe ini_set('memory_limit', xxMB') for memory;
You can run a php cli script from your web process, which forks (pcntl_fork) then kills the parent (the parent cli process exits). The webserver thread running the script can then continue with other code or exit.
Now the cli child process (A) can fork (and become the parent) and monitor a child (B) which sends out the emails. If the child B dies, the now parent A process can fork again and the new child continues where the previous left off.
You have to keep track of who you sent email to in a file/db/shared memory or through paired sockets (socket_create_pair) with the parent process.
I hope you get the idea.
PHP's not the right tool for the job here - you'll want something that runs independent of a web server (although potentially triggered by it) or you'll certainly run into timeouts.
For example, you could have PHP system() out to another script (Perl? Python?), which reads job information from a database then forks into the background to do its work.
You said no Cron scripts - do you not have access to Cron on your server? Or is your concern not wanting to manually set up the Cron job every time a mail job has to be sent out?
You might want to look at http://pear.php.net/package/Mail_Queue - it's something I've been using recently. You can have your user-called script dump all the emails at once into the mail queue. This is much faster than actually sending the emails on the fly since you won't have to deal with communication between servers, but rather just your database.
Then you have a Cron script that, every few minutes, calls the Mail_Queue::sendMailsInQueue command with a limit on the number of emails it sends per Cron call. You won't have to mess with the Cron script if that's your concern - if the queue is empty, it'll just exit. Your email will then trickle out of the server at a reasonable pace per call. This will prevent timeouts due to the limit on emails per cron call. It will also help avoid trouble with other mail servers, who might not be happy to suddenly be hammered by a lot of emails from you at once.