I am developing email queue module for my php application. So all emails(user notifications, password reminder ...) will be put into the queue and will send by cron process based on the priority of emails. But i will have news letter module soon. so my question is either to keep newsletter in seperated queue or can be used the centralized queue since i have priority attribute for all emails ?
Thanks.
Word of caution: Do you have experience building email senders? It's a hairy adventure and you're almost always better off outsourcing the task. Email deliverability is not easy or predictable.
You can stick with one queue, but be sure to have the ability to specify which IP address a particular email can be sent from. You'll want to have different IP addresses for sending newsletters, signups, invoices, etc. And even further, you'll want to have an IP for sending newsletters to trusted addresses and untrusted addresses.
If you can do them with the same module, I'd consider that preferable since there's less code to worry about.
The only potential problem I can see is the differing nature of the two email types. User notifications and password reminders would tend to have one recipient. Newsletters would be emailed to all of your users at once.
If this doesn't cause a problem (and you can't see any other problems), I'd stick with the one-mailer-to-rule-them-all approach.
as Gary Richardson mentions, email deliverability can be tricky.
This is not an ad. But I highly recommend PostMarkApp.com. I am not related to that company in any other way than as a paying customer (well, my company pays).
They have a mail delivery system and a API that you can use from your PHP scripts. You just submit your mails to their queue, and they will do the sending and handle spam reports, bouncing, etc. And their API allows you to know which of your messages bounced and handle them.
This service is the equivalent of CampaignMonitor for raw email sending. BTW, you cannot send newsletters through PostMarkApp, they only allow one-to-one emails (like your user notifications and password reminders).
A few years ago I started building my own email delivery script, until the hosting company mentioned there was a maximum of 500 emails per day. Then I used a 'newsletter' delivery system, with some hacky work-arounds to make it do what I wanted. It was a mess.
Until recently, we were also using another custom-made mail-delivery script but, as Gary Richardson said, you need to take into account IPs, bounces, etc. I find the postmark thing so exciting (in a geeky way) it's embarrassing :D
Having said that, once you are outsourcing the actual delivery of your mails, you could have one single mail queue for your system, using your priority levels. In this way, your application would add mails to your own mailing queue, and your mailing system would deliver it to the outsourced platform. This is preferred to trying to send the email straight away during a page refresh after a user presses submit or similar.
PS: If anybody knows of any service similar to postmark, please let me know!
Give SendGrid a look. Seems to be working great for us after recently realizing that handling email is not worth the dev time.
Related
I am building a program and need to send multiple emails based on certain events. I have coded using the PHP mail function but was advised against its use.
Can somebody explain the difference between the workings of the PHP mail function vs using commercial services like Sendgrid, Mandrill etc? Why would one use these services when I can very well use the free PHP mail function to send mails?
Bulk mailing and campaign management is a complex organizational and technological challenge but will not be helped by most opensource php lib for emails (which simply add html formatting and support for attachments).
They won't handle IPR, SPF, spam trap avoidance, bounce handling, throttling, subscription management and more. Which good bulk emaillers provide.
If you know what you are doing, have full admin rights on your system and NS records and time to write the code, then, yes, you could build the same service - but its not just comparing ants to elephants - its comparing a spanner to an orange.
You cannot send bulk emails in one go using this function.
For example, if you want to send emails to 100k users, you will have to iterate over the emails ids. Yes, you can use the cc though but what if you have a template where in every email the user name will be added like:
Dear User abc
You will have to send one by one. You can't achieve this burst email shoot feature in mail function unless you use multiple processes.
MailQueue is good but I have used it, it does the same things, saves the emails into the database and dequeues the emails and send them one by one.
Again we have swift mailer too.
Again, this is on your requirement. If you don't need to send hundreds of notifications that are time sensitive, you don't need these tools then.
comparing mail() to commercial services is like comparing ant to elephant. mail() is good for some fast notifications from the server if you dont need lots of customisation, but if you want to do something like email campaigns you would have to do a lot of additional server setup and build tons of custom functionallity arround your mail() function. Commercial services are good for campaigns but not notifications (waste of money imho). Try using some opensource php lib for emails, that will have a lot of that custom functionality sorted for you.
First of all, by profession I am a part of a global ESP "Pepipost" which helps brands in delivering the business critical triggered and transactional emails to their customers at the most .
But, the intend of myself here answering this question is not to promote an ESP over PHP mail function. I will share you some facts which we observed during our research at Pepipost Labs.
PHP mail function is just a basic method/the core function to generate the email. But, it cannot deliver your emails at a scale, neither it can help in better deliverability, both of these are key to any businesses.
With the evolution of more sophisticated technologies and algorithms, ISPs (Gmail, yahoo, Outlook and others) and Spam Filters are getting more and more smarter. Before reaching the mailbox, each email passes through all these filters where the content, reputation and other parameters are scanned. Hence, its very important to have good reputed IPs and envelope domains.
Once the email reaches the recipient server either it will get deliver, throttle or bounce. In case of Bounce, Spam Complaint, Unsubscribe, the recipient server will communicate your server, and in that case you need to handle all these properly else the reputation of your domain and IP will goes down.
In case of throttling, another set of complex deliver logics required.
In short, if you are planning to send few hundred emails, then PHP mail function might be ok to use, but to scale further it's important to use a good ESP (Email Service Provider).
Mail() function works very simply on *nix systems, it just executes local command sendmail to send message. It does have disadvantages like:
do not work in many cheap hosting environments
You do not have any information if message is accepted by sending MTA, it could be rejected or quarantined or stay forever in local queue
in some environments it could generate garbled headers because of wrong EOL characters (\r\r\n instead of \r\n)
I think You should use open source library like PHPMailer or Swift Mailer, because it has many more functionality besides simply sending mail.
Some advantages of these libraries:
You could easily send HTML mail with attachments and images
You could send messages through external SMTP services
they have DKIM and S/MIME signing support
PS. I do not use commercial services like Sendgrid, Mandrill, thus can't compare.
I need to send about ~20k emails to potential clients and need the best way to avoid:
server crash
being put in spammers list
I've searched a few forums and people said it's best to send mails in 1k packages
but my question is what delay between sending each 1k should I set? I mean 1k/day or 1k/hour or 1k/min etc.
I recently had to build system to do so, here is the solution we chose :
1- The management system setup emails and store them in a database.
2- We setup a no-reply account on our mail server to get the bounces back.
3- We build a small code over the great mailing library swiftmailer (swiftmailer.org).
4- We run the script we build through a cron and send mails in packages of 50 per minute (hotmail will most likely flag email as spam if too much emails are sent from the same adresse/server in the same minute). We store each swiftId i the mail database
5- At the runtime of the mail sending script we check for bounces or other errors in our no-reply mailbox and flag our messages in our mail table accordingly (status : 1 - success, 2 - invalid email, 3 - bounces)
6- We sync back our data (upon user accessing the section of the system) in the main system when they have a status and that the last change was made at least 10 minutes in the past to limit the change of states in the system. (the synchronisation script can change the status of an item in the main system, but the 10 minutes delay reduce the risk of changes)
I think your 1K thing is a red herring. There have been a number of similar questions asked with good answers, have a look though these:
How do you make sure email you send programmatically is not automatically marked as spam?
Avoid being blocked by web mail companies for mass/bulk emailing?
Sending solicited mass email
Note that the primary requirement to avoid being your mail being marked as spam is NOT TO SEND SPAM. Your description sounds a lot like spam, in which case no amount of clever tricks will help you.
Providing you are not actually sending spam, a professional organisation such as http://www.campaignmonitor.com/ or http://www.mailchimp.com/ is probably cheaper than the amount of time you will require to do this properly.
Your best bet is to farm it out to specialists.
There's a whole load of subtleties about mass mailing that are easy to get wrong.
Best case is that your server gets put on an email blacklist - meaning all your mails from now on get binned as spam. Worst case is fines and/or jail time, depending n your local laws and the laws where the recipients live.
When it comes down to being put in spammers lists, the question is, how many (very) similar mails from the same sender reach a certain host. If you flood yahoo-mail-accounts with thousands of mails within a short period of time, you can be sure to be on their list shortly.
What you can do: pay a professional service to send out the mails for you. They usually have contracts with the big providers so you don't end up on anyones blacklist (will cost you money, though).If you're sending them out from your server, make sure the "reverse DNS lookup" for your server is activated - I don't know if that's checked any more but it gave me a hard time once.
If your mails don't need to be sent out at a certain time, I'd send the mails in very small packages, e.g. not more than one mail per second, not too much in a row. Then wait for a certain amount of time and repeat until all mails have been sent.A company I used to work for would send out not more than 200 and then wait half an hour before the program proceeded. As far as I know, we never ended up on some blacklist.
The spam list has been well answered, but as far as not bombing out your server, the answer is generally the same. Use a service that does this for you.
If you absolutely must send this volume of e-mail on your own, you want to do it with some sort of background service. This can be written in anything (provided that you don't simply buy one of the many pieces of software off the shelf), but ideally it should be multi-threaded.
Your management application should not be sending these e-mails... only queuing them up in a database or something.
Again though, why reinvent the wheel? You'll save yourself a lot of time by going with something off-the-shelf.
I work for a small marketing company that has asked me to develop a PHP application to send emails for their company without getting blacklisted or emails showing up as spam. I have a fairly good understanding of PHP (OOP) but I do not understand how to build this program. They need approximately 500,000 to 1,000,000 emails sent per month. They are ready to buy ip addresses, SMTP servers etc but I need to code the program to work. So basically what I'm asking is what is the breakdown of the different parts of an application like this needs and how it can function without being blacklisted or considered spam. I apologize if i confused anyone on this topic, its just that i used the mail() function numerous times and many of the mail keeps going to spam. This is a legitimate pharmaceutical company as far as i know, they just contacted me and asked me if i could do this so thats why im here to see if it is possible or not.
I know there are companies out there that can do this but they charge from $500 upwards and we cannot afford this. If there are specific equipment that needs to be bought we are ready to buy it to offset long term costs.
Any help is appreciated, thank you!
You're company is willing to buy hardware, but not pay $500 to have a company do it? For doing large mailings, you can use SwiftMailer. It has certain features to support bulk mailings and is written in PHP.
However, not getting blacklisted or put in the spam folder has nothing to do with hardware or software. You need a valid domain and valid emails. Put "V1agra" in an email an it probably won't get received by anyone, not matter what you are using.
The best way to be not be considered as spam and not being blacklisted is to send legitimate emails.
There is no problem sending a lot of mails from the same IP if no recipient mark it as "spam" at the end. To achieve this, a lot of things have to be taken into consideration:
Having good lists with opt-in is not an option.
Track bounces (both soft and hard), so you don't send unnecessary emails (don't send again emails to hard bounces, try again sometimes for the soft bounces and remove them if successful).
Run spam tests before sending bulk (SpamAssasin, etc.) (no advertising, but Litmus does that)
In fact, if you do everything good, your IP can end up with a good sending reputation and more mails will get to the inbox. Also be aware that when sending bulk, not everybody gets the email at the end. Some of them are "lost" by different filters in the middle of the process.
If you never sent mail from an IP, try to be progressive and do not send 1 millions the same month. I advice you to be progressive on how many mail send 1 IP.
Look at what specialized companies do. For example, you can learn a lot from Campaign Monitor and MailChimp. They have really great blog and resources to look at. Follow their blog, keep informed.
A company I work with has had success using an Open Source (LGLP license) project called PHPMailer for a similar function -- maybe this would be a good start for you?
EDIT: PHPMailer is great software but I spoke too quickly. I meant to say phplist.
If you are sending legit emails you should look into SPF.
Only validates an IP as valid sender of emails from a certain domain. Does not validate content or ensure safe passage through spamfilters.
But since spam is illegal in many countries, this validation means they know who to sue/fine.
well, you may simply use mail() but no amount of PHP is going to help you get past spam filters. They use blacklists, statistical filters, human-filters, neural networks and all kinds of technologies to determine if someone is sending spam.
1.000.000 emails per month sounds like spam to me, so most likely you'll be tagged as "spammer" :)
New to the forum but love the comments. I'm tech savvy but not when it comes to programming, coding etc. So - I guess that makes me tech stupid for the moment.
Basically - I want to create a mass email program that will allow us to send bulk emails to our customers who signed an email slip. Currently - we're at 100,000 or so in our database but we expect to see this rise to 200,000 at some point.
What is the best script language to use for this?
Is PHP going to be handle this many emails?
What is the most emails that I can send in any one batch for it to still be "safe" so that the emails arrive to the proper recipients?
What is the term when you send directly from the server?
We currently have 3 servers and I'm assuming they are capable of doing this no problem...but I haven't spoken to our webhosting company just yet.
Lastly - I put together a powerpoint...that gives specs - is there anyone here that would be interested in a little side work to do the project? Really interested in the feedback. Thanks.
In general and based off what you said -- do not create your own mass email program.
Use a service, such as MailChimp or iContact.
Here's a small list of some of the things they do for you:
Manage the list
Handle unsubscriptions
Handle soft bounced emails
Handle hard bounced emails
Handle white listing (You have to pay third party companies $$$s a month to be on these lists)
Give you an idea of how likely and spammy your email is
How fast your email can be sent out (AOL will put you on their blacklist if you send out too fast)
Handle Analytics (such as how many were opened, when, where, what links were clicked on, linking to Google Analytics)
Handle multiple lists and variable replacement (such as "Dear _____")
Nice, easy-to-use interface.
Again, that's a small list in the scheme of things. Those were off the top of my head without too much thought.
EDIT
A couple big advantages I didn't mention:
a) Support
b) A community
c) Constant improvements
d) No server-management
Step 1. Acquire a real email server.
Step 2. Utilize said email server by having your scripts send the mail to it for delivery, and not directly.
Step 3. Throttle the mail server as necessary
Step 4. Monitor the non-delivery reports and remove addresses as necessary...
It depends on how you do this. You can:
1) loop and send 100 000 individual mail
2) Send 1 email with 100 000 people in BCC
or mix both. php can handle it well, but it's a huge task on the mail server.
I'm about ready to unveil a "coming soon" page and one thing I need is a way for users to enter their email address for me to email once the site goes live. What is the best way to do this?
Should I store the emails in a DB and then run a PHP script to email them from my web host? Should I have the emails just kind of be collected and then emailed manually by me either locally or from the server? Is there another way I should do it?
Also, are there certain web-hosts that restrain the amount of emails you can send out, thus causing a problem for mass email.
The only "unknown" currently is how large of a response I'll get...only time will tell.
Thanks guys!
Store it in the database - Yes. It's no unlike storing any other piece of data.
Mass email at once - No
Mass email manually (or individually mail manually) - No
Do some web hosts limit how many you can send at once? - Yes
I don't know the "best" way but I know a really good way. We have built several mass emailing programs and the technique we incorporated was a throttling technique whereby we had a script that ran every three minutes and sent 20 emails at a time.
It keeps the server from choking and the mail queue from exceeding any of our hosts' mail limits.
Store the email addresses in a database and write a batch job that mail merges them into your message and sends them out as needed.
There are dozens of methods you can use for this kind of problem, but unfortunately there's no real way to pinpoint a solution for you, since there are a lot of variables.
If you only get 3 responses, then you might want to just manually email the users from your email client using BCC:. You've already got all the tools you need to do that, and setting up a script might be a waste of time. This isn't really a great long-term solution however.
For most moderate-sized web sites you would store the emails in a database, and use a mailing script to send them out. PHPMailer is a good tool to help with getting mail sent, and you can manage a decent amount of email addresses manually. Managing addresses manually can be a bit of a pain however, as you have to deal with unsubscribe links, script timeouts, bounced email, etc.
If you get up the tens of thousands of email addresses, you may want to start looking at a third-party mailing software or service that can do threaded sending. Looping through and sending email to 50,000 people via a PHP script can be slow, and take hours.
Basically you have to weigh the difficulty, time and cost of each method versus how much flexibility and power you think you're going to need.
To store emails in a db, and sending them out using a cronjob is a good way to solve the problem, if you have constrains regarding the amount of emails you're allowed to send within a period of time, you can handle it by keeping track of how many emails you've sent every time the cronjob runs.
Also, there are open source products to do this, such as phplist http://www.phplist.com/
Like Eric mentioned above, another option is to use a commercial service to manage your mailing list.
I use Mailchimp (http://www.mailchimp.com). They give you everything you need (signup forms, email templates, etc.), and are completely free unless your list grows to more than 500 subscribers.
Writing the emails to a database or a text/log file are both fine ways to store the emails.
Depending on how many emails you receive, you may want to write a program (PHP works) to send a separate email to each person. Don't send a mass email from your regular email client with everyone on a big To: line.
There are also commercial programs that manage mailing to lists of people (probably open source ones, too). Most of those commercial ones offer a free trial period.