I have about 50 000 application names and e mail addresses stored in a CSV. I want to send mails to all 50 000 e mail addresses and the name of the application will also be included in the mail. This requires the CSV to be uploaded first. I searched for a similar kind of a scenario but was unable to find an answer.
SLOW DOWN!
50,000 is a large distribution list. While you can do this yourself i would strongly suggest using some existing sofware to do it yourself, better still, a hosted solution.
If you do do this yourself, you MUST throttle the sending rate. Flat out, you'll send about 11,000-20,000 in an hour (estimate) but your target should be 2,000 per hour max. Otherwise you'll overload the mail queue, not be able to handle the bounces and quickly get yourself on a blacklist that takes you days to get off it. If you're on a shared hosting account - throttle back to 1,000 per hour (or clear it with your hosts first).
A common "free" solution is http://www.phplist.com (and they even offer a hosted version if this is a one-off). Hosted versions also offer more surely that the mail server won't get blocked by spam - there are several around, just google and check on prices.
If you are rolling your own, to enable throttling, upload to DB (see other answers) and also add a "sent" column, every minutee grab 10/20 that have not been sent, mark as sent and send them. Then wait another minute and repeat. You can do this via a cron or via a page refresh. (PHP List will handle this for you!)
But I'd strongly suggest a hosted solution or strucutred software than rolling your own for something this large.
Have a look at fgetcsv. After you figured out how to read the data, you'll have to send it. I'd not recommend using PHPs send function. Instead use a mail library.
There are functions available to parse the CSV file in PHP and after parsing you will get the data in an array of PHP variable. Take the array index in a loop and initiate the mail function with the data.
Related
I'm creating a site that sends out a daily news email to about 800 users, at a time they can specify. My problem is that my script takes a long time to run and times out, so I'm looking for some advice on how I could be approaching this better.
Current approach:
Users are placed in a 'mailing queue' database table with their ID, receive time, and a sent flag.
I'm then running a CRON script every minute which does the following:
Grab all from mailing queue with a 'receive time' less than or equal to now, that haven't sent
Loop through the users, joining a preferences table to get their chosen categories (up to about 30 per user).
For each category, find the latest 3 articles
Prepare an HTML email with this content using PHPMailer
PHPMailer is using Mailgun SMTP to avoid overloading my SMTP server
Send mail to user, mark as sent in database
My observations so far are:
When testing the script by running in-browser, it runs incredibly slowly for a few minutes then times out (without sending any emails).
When running every minute via CRON, it sends way over the number of emails (about 1400) over the course of 40 minutes, I guess because the script is overlapping itself and the sent flag is not reliably updated.
The majority of users are set to receive their email at the same time, so I'm doing 'worst case scenario' testing on this basis
Questions
Is my script far too heavy, by querying the database and generating the HTML email content for each user on the fly? I'm wondering if it would be better to generate the content ahead of time and store against the user in the mailing queue.
Would a queue manager like Beanstalkd help? I've had a look into it, but am struggling to see how to implement into my routine.
Ultimately I need the emails to be sent reliably to each user at the time they expect.
Any advice much appreciated!
You can do this in PHP, but you probably shouldn't. You're trying to build a mail server when there are much better ways of doing this, which mainly involve using a mail server.
Sending high volumes of email during page loads is not workable – it can be troublesome even for single messages, yet many still try. Approach it like this:
Store your list in a database.
When you want to send, generate a record representing each message to send (essentially a copy of the list).
Have a daemon (a long-running task) or cron job that sends the messages in chunks.
Create messages one at a time and submit them to a local mail server.
Use DKIM signatures.
As each message is sent, mark them as sent in the database, but you need to be very aware of how database transactions and locks work for this to work safely and avoid duplicates – do this right and overlapping processes work just fine.
You can generate messages as fast as you like, and your mail server will deal with queuing, onward delivery, retries, bounces.
Use VERP addressing, and feed bounces into a bounce handler (be warned, writing these is not fun!), and have that prevent sending to bouncing addresses in future.
This approach works well – it is exactly how my own system ([Smartmessages.net](https://info.smartmessages.net/ , which is built in PHP) works, and I can sustain over 200 messages per second using multiple message generators running in parallel (database transactions FTW!).
If you find all this a bit too much (it is very difficult), you're probably better off using a commercial sending service (like my own) or hosting a DIY solution, such as Mailcoach by my good friends at Spatie. Either of these would work well, and your list is pretty small – I'm often handling lists of over 100,000.
I've got a web application with approx. 30k users and growing fast. We want to enable some functionality over email, which means assigning each user a unique email address (for security reasons). Emails sent to these unique addresses should be forwarded to a PHP script. The PHP script needs access to all parts of the email - the headers, body and any attachments. I'm not sure what the right way is to set this up. We're currently using sendmail and would prefer to stick with that (this is linux, if it matters). I've been able to get an individual email forwarding to a script, but it is not scalable (in my mind) to create 30k aliases and then a new one each time a new user signs up. I'd be much happier with some type of catch-all / RegEx-based solution that just tells the PHP script the unique email address that was the recipient, which would let us look-up the user.
30k accounts does not give us any information about the traffic you expect to handle. But let's assume it is another question.
You could create catch all type of account, which would simply accept any email in domain you create your users' addresses in (basically *#domain). Then you could launch your processing script on each incoming mail (you usually would be able to fetch whole mail body from stdin stream (in PHP accessible via php://stdin), and process it as you like. But this is bad if you want to use PHP (or almost any scripting language), as each mail would need to spawn new PHP process, init all the extensions and so on. It all this takes time and resources, and one bigger spam flood and your box is almost toasted.
Alternatively, you could collect your mail in mailbox as any other mail and then have your script getting it from there (either directly, digging into maildir or via mail protocols like IMAP or POP). This looks better from administration standpoint and as long as your mail server can accept incoming mails, your whole system still works. You can even try to "partition" your system by setting some filter rules (on MTA level) and deliver incoming mails not to single mailbox but more. Then you could process these mails in parallel with less hassle (no need to bother about processing same mail with more than one instance etc). "Partition" schema could be quite anything, i.e. depending on your full address naming scheme, if letter based it'd be separate mailbox for a*#, b*# or [a-d]*#] (depends on traffic), does not really matter.
Alternatively, you could try to put bodies of incoming mails into database and then process as described above, without any need of POP/IMAP handling.
Whatever you choose I recommend NOT putting your processing script of any form directly in mail receiving process chain. It usually is not a problem to have 1 minute delay (so you can have any sort of daemon running and fetching mails for processing, cronjob etc), users can wait that much w/o complaining - it looks like instant processing is most likely not mandatory in your case as well. But if it is, then PHP might not be the best tool choice I am afraid.
EDIT: (to answer comment question and not hit comment size limit)
I am not familiar with Sendmail enough to give you precise configuration directives here (I am on customized qmail), but some general hints would apply I hope. First, I believe Sendmail supports virtual users (which means mail boxes for users not present in /etc/passwd). If so, then MTA usually should let you either query DB (to find out if the target email address is valid and where is that user's home dir to put mail into) with some built-in providers. Or (which is even better) invoke external program to do that. The latter is a winner here because there are no users really, so you could write small script or (again, better for performance) C app that would do the trick. Since all you need is to "separate" incoming mails into several physical mailboxes based on some simple predefined rules, couple of if()s + substr()s and you are done. Couple of lines, no DB needed, pretty quick. Or, even better, you could even try the dumbest roundrobin approach here as well - it is in fact irrelevant if mail hits accountA or accountB as long as 50% of mails ends in A and other 50% in B (or 25% if you go for 4 target accounts etc) and that's the only condition you should care I think. You can try to use procmail for that, but I think it is pointless and is not best for performance.
As for putting mails into database. It depends on MTA - by default nobody does so as it slower than putting mails into maildirs (as DB is usually other host), but there still are couple of solutions. Either look for 3rd party patches for Sendmail that do that or add another script/app that would "copy" mails from physical mailboxes into DB. You may say - but it would be slower. Sure, but again - I do not know if non-realtime processing really matters in your case (my blind guess - it does not), so why really bother. If you save couple of milliseconds here but loose minutes on further maintenance of the whole solution, then choice is quite obvious.
BTW: just in case - whenever I wrote mailbox I definitely meant container for user's mails, NOT mailbox file format. As for file format maildir (or something derrivative) is much, much better choice.
Do you really want to fork off a PHP for every incoming email? That could get expensive if you're dealing with decent mail volume. Why not have a custom callback in your mail server that processes the incoming mail and inserts into a database, and then have a worker script that continually polls for new messages to process?
The mail-to-db part should be much lighter weight than doing all the processing, and this gives you a free queue so that if you get a bunch of requests at nearly the same time, your system doesn't fall over trying to process them all at once.
did a similar implementation of this, though aon a smaller scale... should scale to meet your purposes though.
two seperate problems
(1) mail - catch-all as mentioned is a good solution, though you will have to deal with spam and malformed email addresses (which may be a good thing or a bad thing). If you own the server, simply aliasing the "ficticious" accounts and directing the mail server to drop them all in the same central mailbox will accomplish the task as you don't need seperate actual accounts, and the header information will remain intact. the specifics depend on your mail solution, but any mail server should be more than equipped to handle this task and the volume, and you should be able to script the alias creation... otherwise back to catch-all
(2) PHP parsing and processing - PHP's builtin pop3 functions can easily handle checking the (now) single email account and processing the messages; Some examples of functions I used to go through this are;
function pop3_login($host,$port,$user,$pass,$folder="INBOX",$ssl=false){
$ssl=($ssl==false)?"/novalidate-cert":"";
return (imap_open("{"."$host:$port/pop3$ssl"."}$folder",$user,$pass,OP_SILENT));
}
function pop3_stat($connection){
$check = imap_mailboxmsginfo($connection); # changed from $imap
return ((array)$check);
}
function pop3_retr($connection,$message,$section='1'){
return(quoted_printable_decode(imap_fetchbody($connection,$message,$section)));
}
function pop3_dele($connection,$message){
return(imap_delete($connection,$message));
}
function pop3_close($connection){
return(imap_close($connection,CL_EXPUNGE));
}
all very generic, and there are probably classes out there if you don't need the control. Bottom line, PHP has all you need to quickly parse and process mail messages.
I'm using a homemade PHP script with PHPmailer for sending email, currently I just do one select statement in MYSQL and then run a 100-150 email send request from PHPmailer via a for loop.
my problem is that I have to port this script to a 7000 user mailing list, and I know it's not going to hold up since sending to 150 takes about 8 secs of execution time - also sending 7000 emails might mark me as a spammer or I won't be able to handle all responces.
I thought about splitting the 7000 into groups of like 200-500 and sending an email to each group every day until sending everyone - to do that i'll have to write a PHP script that handles the limits and remembers when & who on the list hasn't received that email yet and is due to receive it on the next batch.
What are my options, what are your thoughts?
Just my 2 cents.
1) I agree with Dave re: make it take longer than it needs. Most black/grey listing is the the recipient mail server deciding there is an odd amount of traffic compared to usual. Yahoo is terrible for this and given the sheer volume of domains they deal with, it can get annoying.
2) Using multiple IP addresses doesn't hurt. Extra IPs are usually free, and if you are using exim, you can easily configure it to use each IP based on e.g. EHLO, or send to domain etc etc. This means that there is less volume coming from any one IP.
3) Keep a track of bounce backs. Some providers will keep a track of the number of attempts to deliver mail to a non-existent address, the more you try, the more suspicious it looks. Easiest way to keep a track of them in my experience is to use VERPs and get PHP to process the incoming data.
4) Set up reverse DNS. Some hosts will just ignore the mail if you have not got it set up for the sender. I believe AOL is an example.
Just a few pointers from my experience.
Your first approach is a pretty good idea. As long as you're ok with it taking about 3 minutes to send all the emails (which it should be, if you're only sending a weekly newsletter or the like). To reduce the chance of getting marked spam, lower the threshold or make it take longer. Beyond that, it might be a good idea to move into a hosted mass mailing service.
The act of sending 7000 [or more] emails is not going to get you marked as a spammer ~assuming~ that the mail you are sending is:
not spam
contains complete & accurate header information
The only problem you might get is clogging up your mail queue, but you r volume is still pretty low. I would say for scalability - throttle your delivery over about an hour or so.
and consider ceejayoz's suggestion "MailChimp" it's free up to 5000 emails/month and has an excellent API that you can literally integrate into your app in a couple of hours.
-sean
I'm about ready to unveil a "coming soon" page and one thing I need is a way for users to enter their email address for me to email once the site goes live. What is the best way to do this?
Should I store the emails in a DB and then run a PHP script to email them from my web host? Should I have the emails just kind of be collected and then emailed manually by me either locally or from the server? Is there another way I should do it?
Also, are there certain web-hosts that restrain the amount of emails you can send out, thus causing a problem for mass email.
The only "unknown" currently is how large of a response I'll get...only time will tell.
Thanks guys!
Store it in the database - Yes. It's no unlike storing any other piece of data.
Mass email at once - No
Mass email manually (or individually mail manually) - No
Do some web hosts limit how many you can send at once? - Yes
I don't know the "best" way but I know a really good way. We have built several mass emailing programs and the technique we incorporated was a throttling technique whereby we had a script that ran every three minutes and sent 20 emails at a time.
It keeps the server from choking and the mail queue from exceeding any of our hosts' mail limits.
Store the email addresses in a database and write a batch job that mail merges them into your message and sends them out as needed.
There are dozens of methods you can use for this kind of problem, but unfortunately there's no real way to pinpoint a solution for you, since there are a lot of variables.
If you only get 3 responses, then you might want to just manually email the users from your email client using BCC:. You've already got all the tools you need to do that, and setting up a script might be a waste of time. This isn't really a great long-term solution however.
For most moderate-sized web sites you would store the emails in a database, and use a mailing script to send them out. PHPMailer is a good tool to help with getting mail sent, and you can manage a decent amount of email addresses manually. Managing addresses manually can be a bit of a pain however, as you have to deal with unsubscribe links, script timeouts, bounced email, etc.
If you get up the tens of thousands of email addresses, you may want to start looking at a third-party mailing software or service that can do threaded sending. Looping through and sending email to 50,000 people via a PHP script can be slow, and take hours.
Basically you have to weigh the difficulty, time and cost of each method versus how much flexibility and power you think you're going to need.
To store emails in a db, and sending them out using a cronjob is a good way to solve the problem, if you have constrains regarding the amount of emails you're allowed to send within a period of time, you can handle it by keeping track of how many emails you've sent every time the cronjob runs.
Also, there are open source products to do this, such as phplist http://www.phplist.com/
Like Eric mentioned above, another option is to use a commercial service to manage your mailing list.
I use Mailchimp (http://www.mailchimp.com). They give you everything you need (signup forms, email templates, etc.), and are completely free unless your list grows to more than 500 subscribers.
Writing the emails to a database or a text/log file are both fine ways to store the emails.
Depending on how many emails you receive, you may want to write a program (PHP works) to send a separate email to each person. Don't send a mass email from your regular email client with everyone on a big To: line.
There are also commercial programs that manage mailing to lists of people (probably open source ones, too). Most of those commercial ones offer a free trial period.
What is the most proper way to sending email of minimal 1000 or more in PHP? Any reliable email queuing technique that is capable to handle that?
You could just insert your emails into a Mail Queue database table, and have a separate process check the queue and batch send a certain number at once.
There's a tested solution for that: PEAR Mail_Queue
Works fine for me.
as mercutio suggested, i would insert a new record into a mail queue table for each email waiting to be sent and then use a separate process (like a CRON) to check the table periodically for any queued items.
if any emails are queued (and the email is not customised for each recipient) i would then group the emails by domain and send blocks together to reduce the total number of emails that have to be sent, i.e. if you have 1000 emails queued and 250 are to gmail accounts i would send the 250 in 25 blocks of 10 (remember to Bcc recipients to avoid them seeing each other).
to actually send the mail i would use PEAR mail over php's mail() function
after sending the email either delete record(s) from the queue or change a status flag to show it was sent and loop - i would also add a counter to keep track of emails that failed to send and remove them after x failed attempts
to overcome timeout issues i would either,(depending on the situation)
- set the set_time_limit() to x seconds and keep track of the script execution time (killing the script after (x-1) seconds)
- call the script from the command line to avoid timeouts
- set a limit to the number of emails the script could send in one execution
Sure, the database table might be an idea. But what about sending 1000 e-mails with a 2mb attachment? you'd have to take that into account as well. I had the problem myself, and I eventually resorted to writing the e-mail to the database, and the files to the filesystem. The e-mail script I used then read the database records, and tried to fetch the attachments to send.
Are you sure you need do this mail queuing yourself?
Just deliver all mail to the local machine's mail transfer agent (sendmail...) and let that take care of the queuing and sending. After all, that's what was designed for.
In other words: don't worry about it!
I created Emailqueue, which is a server that allows you to add emails to a queue so your app get relieved of the stress of the mailing, and also provides useful additional options, like the ability to program emails to be sent in the future, or setting per-email sending priorities. I think this might very well be what you're searching for.
Emailqueue is available here: https://github.com/tin-cat/emailqueue
And there is also a Docker version that allows you to set up a working Emailqueue server in just a few minutes, here: https://github.com/tin-cat/emailqueue-docker
I've generally relied on a hack.
I have a database list of email addresses and then use a meta-redirect to self with an increasing 'offset' parameter that specifies which row in the database I am up to. Server redirects cause problems because browsers assume that the time taken indicates an infinite loop.