My company has a website built with PHP. We use the built-in PHP email functionality to send thousands of emails to subscribers on a daily basis.
This is a terrible idea. It chokes out our server, and takes hours to complete the whole batch.
Now I've looked at mass mailing services like MailChimp (which would replace our current system of sending the same email to many people), but what I think I'd really like to do is to set up a somewhat-sophisticated notification system.
Rather than send a mass email to each person each time something important happens, I'd like clients to be able to customize the rate and content of the emails that they receive.
Even using this new idea, we're talking about A LOT of emails being sent.
So my question is very specific: I have a rough idea of how to build the system internally, but what is the best way to send out all of these emails?
Bullet points to consider:
Sometimes emails' contents are identical across recipients, but many of them will be customized per-user (they choose what they get notified about, and sometimes it is aggregated).
I want a system that is not going to choke the server, and will complete in a decent amount of time. I don't mind going with a third-party service (even a paid one) if that is what it is going to take.
The system should hook into PHP easily, or the API or whatever should be relatively easy for me to call from your typical web server.
We have a dedicated server and full control over it (so we can install apps, services, whatever).
Any kind of detailed tracking information (opens, clicks, etc) is a huge plus.
These emails are sometimes time-sensitive (so can't take all day to send).
Thoughts? Tips? Point me in the right direction?
EDIT
To clarify:
I can do these on my own:
maintain user list
handle email content generation based on user preferences
And need something else (app, third-party service, w/e) to:
accept email content and addresses and actually send the emails out
provide tracking data (opens, clicks, etc). The more detail the better.
I'm leaning towards a third-party service, since I'm not sure any app can avoid choking the server when sending thousands of emails (though I wouldn't consider myself an email expert so I could be wrong).
We use the built-in PHP email functionality to send thousands of emails to subscribers on a daily basis.
This is a terrible idea. It chokes out our server, and takes hours to complete the whole batch.
Why do you think that your problems are anything to do with the built-in PHP email function? It's a very thin wrapper around 'mail' or a simple SMTP client depending on what platform you are running on.
When you say it chokes your server - do you mean your email server? Your web server? something else?
There's nowhere near enough information here to make a proper diagnosis but it looks like the problems are of your own making - sure, there are lots of people out there who promise to sort all your problems for you if only you buy their latest product/service. But there's a very good chance that this isn't going to solve your current problems.
Can you tell us:
what OS the PHP is running on
how you invoke the code to create the emails
what the mail config in the php.ini file is
what type of MTA are you using? On what OS?
how is youe MTA copnfigured - does it deliver directly or via a smart relay?
which server is getting "choked"?
What anti-spam measures do you have in place for outgoing mail?
Then tell us what you've done to diagnose the fault and why you think its specifically on sending mails.
C.
I'd recommend using the third party mailing service Silverpop, or something like it. We've used them for a few years and have been fairly satisfied. They already have relationships with the major email clients (AOL, Yahoo!, Gmail, etc.) and they do a good job of telling you if the stuff you're sending is likely to be classified as SPAM.
They have a fairly extensive API that uses XML HTTP/HTTPS requests that can tie in to existing systems. You can use it to remotely trigger emails, schedule mailings, customize email contents, set up, manage and query huge lists of recipients, run batch processes, etc.
It isn't a perfect service, but compared to a lot of others out there, they do pretty well. I have had very few complaints about them thus far.
I usually got around this by having a mail "sending" function that dumped the emails into a queue (database table) with a job that ran every couple of minutes, grabbed the next x emails in the queue, sent those out and marked them as succeeded. That's the simple bones of it. You can then add on handling for email failures, returned mail, etc in version 2.
Use Google AppEngine if you are worried about scalability & customization: it provides an email API and you can interface anything to it provided it is through an HTTP interface.
Of course, this is just a suggestion - disregard if this doesn't fit.
Quite possibly not ideal, but if you're looking at large scale transmission there are commercial solutions such as Port 25's PowerMTA that can be set up to effectively transmit the contents of a given folder.
As such, you'd simply use PHP to create the personalised MIME formatted raw data for each outbound email and place these in a temporary directory prior to transmission. (I've written such a system in the past and you'd be surprised at the PHP's ability to grind out the emails, even with quite complex text & HTML emails with images as inline-attachments, etc.) Once you were ready to transmit, you'd then move the files en-masse to the PowerMTA monitored folder and it would take care of the transmission.
Depending on how you look at it the benefit/problem with such a solution is that you'll need to build trust relationships with people such as AOL, MSN/Hotmail, etc. to ensure that your mail server isn't blacklisted due to user's reporting email as SPAM. (That said, this will likely be a factor with any DIY solution.)
Why not keep your PHP system and use a third party SMTP service? Some reliable companies offer the use of e-mailing-only machines at reasonable prices, e.g. Dewahost who I am planning to use.
Also see the question Third Party Email Senders and my answer there.
Check out Campaign Enterprise for a possible in-house solution.
One of my friend uses http://www.tailoredmail.com but i havn't used it personally
I know this is an older question, but I just wanted to suggest SendGrid which is essentially an "Email Server as a Service" allowing you to send emails with cost per email.
Related
When an order is placed, the order details are sent via email to the customer and seller but it takes too long to send the email due to which the order placement takes time. What should be done to make the process faster?
I guess you should implement a Queue, for example [https://laravel.com/docs/5.2/queues]
Install a local mail server (I'd recommend postfix) and send to it from PHPMailer using SMTP. That is the fastest way of getting a message on its way. It means that your script does not have to deal with DNS lookups, remote connections, authentication, and it hands off control to the mail server that deals with all those things asynchronously, including queuing, retrying, bouncing etc.
You can implement queuing in PHP, but it's really the wrong tool for the job. Regardless of the implementation, it will be much, much slower and more error-prone than using a mail server, and more to the point, it's software that you will need to write, test and maintain.
Background: I run a site which is basically free. It has a sizeable number of email subscribers but because currently there's no pricing model I cannot afford to use Mailchimp to overcome their 2K subscriber limit.
I'm thinking to send it out myself. If I've got an array of ~12000 emails and I use PHP to send out the newsletter, from the server what will the impact be on the server? Could response time for the actual site users (seeing as the website is hosted on the same server) be impacted? Could the server crash? How long might this task take approximately?
in my experience, PHP list would be your best option for this as it's the most popular open source email managment system. Depending on your host, they may have limits on the number of outgoing emails that you send so it's best to check with them first and then configure it so that it is inline with their rules. It takes some configuring, but once you're up and running, it's very easy to use.
In saying that - after managing much more sizeable lists for a good 5-6 years, I've now migrated them all over to Campaign Monitor.
All in all, PHP List will definitely do what you need.
I am working on a business marketing website. The criteria is that we have to send around 80,000 (say) newsletters per day to different email addresses. But my hosting company allowed me to send only 10,000 (say) emails from one IP per day. So to solve this issue my client provided me with 8 different IPs. I am having following questions:
I don't know how to send each and every newsletter from different IPs, means first from first IP, second from second IP and so on?
As I'm a PHP developer and not an expert at server end, can I accomplish this through my PHP script only or do I have to write a shell script to run at my server?
Is there any algorithm to accomplish such type of problem?
Had similiar problem couple years ago. I resosolved it in simple way. Create table in SQL, or create a class and serialize it. Class or SQL have to contain mails addresses (if You have some kind of volatile content then also mail content), do a class to send mail in certain order, and do object of this class on each ip. That depends what kind of config You have on server. I had a easy way cause all of IPs had its own folder with content, so I just put there code to do object, and just redirected couple of times the website, since it had to be done by web. If You have can have it by CLI there is pretty chance that You could do it by include or similar.
This way is pretty lame, I know it, but didn't want to do something more sophisticated at that time. Later on I wrote a class to manage mail connections via SMTP, so I could chose with mail from witch account would be send. That is better way, but not all servers could support it.
Another way is to do a cron job and do baskets of mail to send portions over the time. (this way was most common on servers that I was repairing).
Another way is to do a bouncing effect on servers with IP. Probably You could do also some shell scripting and invoke it via php script.
Well, I think that there are better ways, but it really depends on server config.
PS. Forgotten to mention, that code can be invoked by AJAX.
I'm thinking about how to handle sending large amounts of email from my web applications, and whether there are any best practices for doing so. StackOverflow is already labeling this as 'subjective', which it may be to an extent, but I need to know the most successful way to implement this system, or whether any standardized approach exists.
In my webapp, there are users who are heads of groups of 1 to 10,000 users. This user must be able to email send a message to all of these users through my system. Therefore, my system is responsible for sending up to 10,000 emails to individual users for each group head.
As far as I can tell, there is no rate limit in GMail for sending messages to individuals (although there is a 500 recipient max).
Right now, my current setup is:
When a message is sent through the system, it enters an email queue.
A cron script grabs messages from the queue every few minutes, and sends out those emails.
All email is taking place through GMail's SMTP server.
The actual application doing the mailing is PHPMailer.
This setup, as the user base grows, will probably not suffice. The questions I have are:
Should I be using a local SMTP server instead?
Should I use a mail binary on the local machine instead? I this case, I could probably skip the queue altogether?
Is there an accepted way to do this?
Thanks!
Google App Engine
I would write this in Google app engine (python) because:
It scales well.
It has a good email api.
It has a taskqueue with a good api to access it.
Because python is a real nice language.
It is (relatively) cheap.
PHP
If I would implement it in PHP I would
Find yourself a good SMTP server which allows you to sent this volume of mails because Gmail will not allow you to sent this kind of volume. I am for sure that this will cost you some money.
Find yourself a decent PHP email library to sent messages with like for example PHPMailer like you said.
Use a message queue like for example beanstalkd to put email messages in queue and sent email asynchronously. First because with this the user will have snappier page load. Second with a message queue like beanstalkd you can regulate speed of sending better which will prevent from overloading your pc with work. You will need to have ssh access to the server to compile(install) beanstalkd. You can find beanstalkd at beanstalkd
You would also need ssh access to run a PHP script in the background which will process the message queue. You can find a beanstalkd-client at php beanstalkd-client
from php/apache/webpage
This is the page from which you will sent messages out to user. From this page you will sent message to beanstalkd by coding something in the lines of this:
// register Pheanstalk class loader
require_once('pheanstalk_init.php');
$pheanstalk = new Pheanstalk('127.0.0.1');
$message = ""; // This would contain your message
$pheanstalk->put(json_encode($message);
You have to put messages in message queue using put command
From long running PHP script in background:
The code would look something like this:
// register Pheanstalk class loader
require_once('pheanstalk_init.php');
$pheanstalk = new Pheanstalk('127.0.0.1');
while(true) {
$job = $pheanstalk->reserve();
$email = json_decode($job->getData());
// Sent email using PHP mailer.
$pheanstalk->delete($job);
}
Like I am saying it is possible with both PHP and Google app engine but I would go for app engine because it is easier to implement.
With an email count as "high" as 10.000 a day, I wouldn't rely on GMail (or any other) SMTP. Not that they can't handle it, obviously they can handle A LOT more. But they possibly don't want to.
Having a local SMTP server is IMO the way to go :
It's pretty easy to set up (just DON'T let people use it without a strong authent scheme)
Most modern MTA handle sending queues very well
You won't have to deal with GMail (or others) deciding to block your account someday for quota reasons
Gmail and Google Apps limits you to around 500 emails a day. I'm not sure how that combines with the 500 recipient max, but if you want to send 10 000 emails you'll probably want to find another mail server. I personally use a local server or the ISP or datacenter's SMTP.
If you are sending that many emails, I would recommend using the queue so the user's isn't sitting there waiting for the email to be sent.
Be very careful that your domain doesn't get blacklisted as a spam domain. If it does, you can expect most of your emails to be blocked, support, sales, etc. Which could in turn be very expensive.
You may instead want to use a service like AWeber. Not only are they setup to handle these amounts of emails, but they can probably give you more metrics than you can implement on your own.
I'm not sure if it's publishe anywhere, but from experience I can tell you that Gmail will put a fifteen minute or so freeze on your account if you start sending hundreds of messages at a time. This happened to me last week. I think you should host your own SMTP server. Using the mail() function often will put your mail in someone's spam folder.
Just install Postfix on the local machine, or a machine on the same LAN for maximum access speeds. Make sure it is well secured from the outside, and quickly accessible from the inside.
Then code your PHP script to directly inject the emails to the Postfix queue. That shall dramatically increase the processing speed of mail delivery.
1) How can a linux server be configured such that it can receive any emails sent to account#domain.com.
2) How can these emails be accessed and displayed by a PHP script?
I'm building a simple web mail script, so i want to receive emails only for the registered email accounts and classify their emails and show them.
Needs also to have the ability to show attachments.
Honestly, you're better off looking at one of the webmail systems already out there and modifying it to fit your needs rather than writing something from scratch.
The email RFCs are a fun read (no, no they're not) and attachment handling has caused many developers to take up heavy drinking. Don't get me wrong, it can be done. Its just a lot of work and not something that you want to take on unless you have a bunch of time to dig deep into it.
However, if you wanted to write something quick and simple, I suggest setting up a linux box with an Imap or pop server and then accessing that mailbox with the appropriate PHP / Pear libraries. That will be the quickest way to get going.
But I would still recommend using one of the off the shelf solutions that are out there and hacking it up to fit your needs.
The mail server itself can be setup to receive email with any number of MTA's (sendmail, postfix, exim, etc). You'll want SPAM protection and Virus scanning as well (again there are free packages in any distribution for this). Be warned, running your own mail server is NOT a job for the faint of heart these days.
Once that's in place I would probably setup IMAP and use PHP's IMAP functions to retrieve and parse the messages. Viewing attachments will be a lot harder if you want them readable in the browser, but easy to mark that they are there and make them downloadable.
There is likely a class already written that does most of this for you (excepting the setup of course).
Update: In fact there is an old article at evolt that appears to cover this.
You have a lot of work in front of you - don't expect someone to just post some code here.
Start by reading the POP3, SMTP, and IMAP RFCs. Then go find a commercial/open source solution.