I have an issue. I have situation where I need to send around 3000 emails per request using SMTP. However, only 30-40 reaches destination.
Do you have any idea what can be a problem and how to solve it. as server side script I am using PHP.
How the big e-mail service providers (Constant Contact, WhatCounts, etc.) do mass e-mails is to put the "campaign" in a queue and send it at a later time. They have dedicated, high-performance delivery software for looking through the queue for new campaigns to send and then send them out at rates exceeding 50,000 messages per minute. Anything you might do in PHP won't even compare.
If you are trying to send from your local computer, that won't work. DNSRBL lookups will identify your computer as being on a "DUN" (Dial-Up Network) and will block the message. Most PHP scripts also have a timeout of 30 seconds in a web server environment but running via cron a PHP script can run as long as it needs to.
You shouldn't send mass e-mails from your main e-mail server either. That's a nice, fast way to get on global blacklists so that you can't send regular e-mail to common hosts (e.g. Hotmail, GMail, etc). The big e-mail service providers have dedicated staff whose job it is to remove themselves from global blacklists. That's a full time job. You are better off paying for the service (don't forget to set up SPF records correctly if you do go this route).
Warnings and advice aside, to answer the question, use a cron job for your PHP script and put the e-mails to be sent out in a queue.
If your looking to inboxing as many emails as possible and you are not sending spam, and prefer to use your own smtp, check them out.
Related
We have an internal message board. If a user posts a note then all other users that have access to the thread are sent an email alert. So if there are 20 users on a thread then you could see bursts of 100s of emails over only a couple minutes.
Currently the email alerts are sent with the PHP mail() function. One time we got some local relay alerts and I think our server IP was blacklisted for a short period of time. And I think these negative consequences happened because of an email alert burst that uses mail().
Is the solution simply to use anything other than mail(), or is there more to it? How can I reproduce/test the local relays and blacklisting so that I am sure the issue is solved?
If you're sending large amounts of email the best thing you can do is use a third party email provider. There are several that offer API/SMTP services and will not be blacklisted because they work on a reputation system. Most offer composer packages you can drop in and configure relatively quickly.
SendGrid / sendgrid-api - We use this to send around 2 million emails a month without issue.
Mailgun / mailgun-php
Mandrill / mandrill-api-php
Each have their pros and cons and some offer free emails to a certain limit.
If this isn't an option you should create a dedicated email server that has all the correct DKIM/SPF records that over time will become reputable over time however it'll still slow down when you're sending many emails as it'll be limited to 1 email at a time and if it encounters a slow connection it can take a while to complete. If email is important a third party provider is definitely the best way to go.
I am creating a system, where a list of thousands of emails will be sent periodically, I know that the mail() function in PHP is quite heavy, specially if calling it too many times simultaneously.
Roughly the way my system works , is that I create queue of emails in MySQL and send them in batches of 25 using mail(), removing from the table the top 25 sent. And I wait 2 seconds between each set of 25.
Is this too much effort for the server or I can push it a bit further?
Lets say 50 per second? Or there is a better way of sending many emails in less time without sacrificing Server performance.
I have a dedicated server without any mail() call limit.
There are other factors to consider besides performance, but the short answer is: there are better options. Amazon SES and MailChimp are the two I know about have heard positive feedback on.
Look at j08691's answer regarding the performance, but other issues with using mail() for this purpose include:
Scalability (you will hit a wall that no SMTP server can handle eventually, and you're already thinking about it)
Integrity - You are much more likely to get flagged as spammy when rolling your own mass mailer, especially using mail as it uses the local sendmail by design.
Cost/Benefit and ROI - the reliable mass-mailers get it right and for a competitive rate. At some point you are paying yourself less per hour to maintain your mail server when it crashes, getting off of black lists, writing the email layout by hand, general upkeep, etc etc than you would paying for the mass mailer service.
Overall, the big issue is that you have to do all the work yourself and you're likely to get flagged as SPAM for the benefit of not paying for a service that will be able to send hundreds of emails a second versus a hundred a minute when PHP isn't busy doing everything else it handles for your web app.
Personal anecdote (not an endorsement for SES, just mass-mailers) : We had a client that sent 100k+ emails per campaign, with 1 - 3 campaigns per day minimum. They started complaining that the clients were getting emails about "daily deals" 2 days late. It wasn't because the Mailer library was slow (even this app avoided using plain mail), it was that it couldn't be sure to send all of the emails for every campaign before the email was irrelevant. We switched them over to SES (with some optimizing on our end, but not much), and they could clear a campaign in under an hour.
From the PHP manual:
Note:
It is worth noting that the mail() function is not suitable for larger
volumes of email in a loop. This function opens and closes an SMTP
socket for each email, which is not very efficient.
For the sending of large amounts of email, see the » PEAR::Mail, and »
PEAR::Mail_Queue packages.
Try using PHPMailer. i used it to send about 100 mails everyday without any problem
I have over 4,000 users who subscribed to weekly updates via email. To send an email, I simply use:
mail($to,$subject,$message,$headers);
Which works perfectly fine. My question is: how many emails can you send at once? Can I send all 4,000 at once by stepping through array of emails inside a loop? Or will that cause a time-out situation, wjere only part of the list receives emails?
Generally, your limit is PHP's memory allocation and script timeout, both of which can be increased. A bigger limiting factor is spam firewalls, which typically limit the number of emails that can come from a server and to another domain. Bulk email services use intervals to limit the amount of mail sent to any given domain during a given time period.
PHP has no limit, and you can do what you say and run it in a tight loop.
On linux it is sendmail, or a sendmail clone. If it starts to get overloaded it should start queueing your mails, and delivering them when it can. For Windows you have a socket connection, and I suppose the mail server might start refusing connections if it got overloaded.
Unless you are in a hurry to send them all, consider a sleep(2) between each call to mail(). If all goes smoothly then try sleep(1) the next time. Then usleep(500000) next time, etc. (UPDATE: following Matt's suggestion, put set_time_limit(10) inside the loop to avoid running out of CPU time.)
If you want them sent out even quicker you round-robin with multiple local relays. (On windows you could handle this by setting the SMTP php.ini setting; on linux you'd use on of the libraries that sends email over a socket.) Or consider a professional mail broadcasting service, where they take care of all these issues for you.
EDIT Just saw your comment about running it from your Mac. I'd assumed a server situation, with a real mail server with a global IP. If your Mac is behind a firewall you are most likely set to forward all your email to your ISPs mail server. Your ISP may well get upset about receiving 4000 emails from you in a short period of time. (E.g. they may assume your machine has been compromised by a virus and shut you down.)
Also, on shared hosting, agreeing not to send out lots of email is often part of the contract. So be careful there too.
It does not depend on php, but it depends on your sendmail server capability and your hosting provider support on the limit of emails that you can send with in a span of time..! PHP only puts your email in the sendmail queue...!
Considering that loop will be sequential you can send them all at once by looping. It's more mailserver dependent, rather than PHP.
Although if you're using WAMP, that might cause a problem, because behavior of mail() under windows is different. Instead of talking to an MTA it tries to use sockets to deliver mail.
Giving exact numbers is not possible.Its depends on your mail server & maximum execution time.
Best practice to send large number of email is "cron job"
I'm developing a site that sends email notifications to users after certain events/user actions (tagged in a photo, added as a friend, etc.)
From my experience, sending emails from a web server can be slow. For instance when creating a new user I will send an email to confirm/welcome the account. However, there is short pause sometimes waiting for the page to load while it finishes sending the email.
My fear is that, if I ever have a problem connecting to a mail server or the mail server is overloaded, it will bottleneck my application/page loads
So my question is, what is the best way to send emails from a web application that sends emails on consistent basis?
Should I create a queue system, by storing email content in a database and then run scheduled cron jobs to process all the undelivered emails?
Or should I just send the email directly after a user completes an action/event (creating a member account, tagged in a photo, etc...)?
No. Ask your sysadmin to configure mail server. Correctly configured mail server should be faster then storing messages on database.
Use local sending to avoid network delays.
[mail function]
sendmail_path = /usr/sbin/sendmail
in php.ini
It would work even if mail server is down (message will be queued).
Set up a mail relay on the local machine. That way you do not have to bother with cron and database storage of the mails, yet still prevent the overhead of immediate network traffic for every mail. On the downside you'll have to manage an additional daemon and you have to make sure you do not create an open relay.
Yes, set up a queue system by creating a mysql table that holds all necessary information and set a cron job to run every x minutes to select the emails from the queue and send them (don't forget to set a LIMIT for your sql query that fetches the emails for sending).
Db-solution you mention is fine
Simple solution would be to send mail as final action on script (when html has been sent),
Or configure php to use a less bogged down smtp,
http://email.about.com/od/emailprogrammingtips/qt/Configure_PHP_to_Use_a_Remote_SMTP_Server_for_Sending_Mail.htm
regards,
//t
As you said, you won't be sending emails synchronously with your web requests/responses - which is a good idea as you don't want your user experience blocking on something like connecting to a mail server. Even if it's a local mail server, it's always a good idea to make it async and decouple the two. In other words, your mails will be pushed into a queue like mechanism and sent async.
In this case, the advantage of persisting your to-be-sent messages in a database (or some other store) - as opposed to an in memory queue - is that you have more recoverability in case your webserver fails. You can always process persisted emails later, but if you have them in memory, any failure of your webserver process would mean losing the queued mails.
so i have a couple of sites on paid shared hosting, my host limits the mail to 300 per hour.
One of my sites has more than 500 subscribers.
My question is how can i send the newsletter to all my subscribers? is there a way or a script that i can use to send the first 300 users the email and after an hour to send the rest...?
i've also considered making a gmail account to send the newsletters via smtp. Do you guys know the limit of free gmail smtp?
You shouldn't circumvent the restrictions placed by your host. I would suggest you pace your sends, and record your last-sent-id, picking up from there in your next hour. That, or you can place sufficiently sleep-time between sends to allow the entire thing to go out at a rate of about 300/hr.
Thank You for all your Reply guys... it really helped me find a solution for this inconvenience. I personally can't afford VPS hosting nor pay extra for an external for a mail server...
Considering Jonathan Solution and William's Comments i ended up developing a small php application based on XML to send different batch to 250 recipients each with a GAP of 65 minutes.
So the way it works, by default it only enables the first batch link to be clicked and send the newsletters to the first batch of users and recording the exact time this was sent in a XML file.
Then using the XML file info the next link registers that the batch before it was sent and starts a count down of 65 minutes with the time on the XML as reference.
So a Script will not be running for hours and the browser could be safely close since all the info required is in the XML file.
This may sound Simple but is a complex and efficient app that dynamically adapts to growth (new subscribers) as it queries the master Table on a database using the sql LIMIT clause to make the different batches. So it doesn't required maintenance.
If anyone is interested on the source code feel free to contact me # admin#thechozenfew.net
Google Mail does have limits, see:
Sending limits In an effort to fight
spam and prevent abuse, Google will
temporarily disable your account if
you send messages to more than 500
recipients or if you send a large
number of undeliverable messages. If
you use a POP or IMAP client
(Microsoft Outlook or Apple Mail,
e.g.), you may only send a message to
100 people at a time. Your account
should be re-enabled within 24 hours.
Source: http://mail.google.com/support/bin/answer.py?hl=en&answer=22839
To get around the problem, you could create a queue table in your db with a list of all the users you're sending the newsletter to. Then send e-mails out in bulks (500 example). Remove the e-mails from the queue table as they're sent out. You could use a cron (if on linux and host allows) to run a PHP script every hour that sends e-mails based off the queue.
I'd seek a place just to park your MX (not sure of Google's limits, but that could be a start). Its very common for mailing list managers to queue mails to fit within sending limits. I.e. a cron job queries a database, picks up 250 emails to send and sends them out.
The problem lies when you have 10,000 subscribers and need to send non-automated emails from the same MX. I.e., if your limit is consumed getting out a newsletter, what happens to your ability to reply to your own e-mail?
A lot of companies offer MX only hosting, I'd go with one of them and move the whole business of sending the list over there. Or, just get yourself a VPS (its going to be about the same monthly price).