I plan to send an excess amount of emails daily to my website users. I am using php. Will such a task affect the performance of my website? What can be done to prevent issues?
Note: Emails are sent with users consent and scheduled by them. No spamming.
Instead of sending the email directly, why not create a database table to store the email details. and just write to that table each time you need to send an email.
Then have a scheduled task that occasionally queries this table and do the mail sending.
That way, you do not need to worry about the php page timing out. and in future, you can also pull the schedule task out and run it on a separate server if it uses too much resources.
Some off the top of my head:
If your on shared hosting, you may get shut down for attempted spamming
The server IP may get labelled as a spam source if your sending email to random people
Send the emails in a cronjob rather than in real time, that way you can do the email sending at non-peak times
My suggestion and the way we handle this is to utilize a message queue, we currently use Zend Frameworks Package, then from a cron job retrieve the queue and send the email. This enables us to increase performance and avoid delays in displaying the page to the user. Hope that helps!
Related
I'm creating a site that sends out a daily news email to about 800 users, at a time they can specify. My problem is that my script takes a long time to run and times out, so I'm looking for some advice on how I could be approaching this better.
Current approach:
Users are placed in a 'mailing queue' database table with their ID, receive time, and a sent flag.
I'm then running a CRON script every minute which does the following:
Grab all from mailing queue with a 'receive time' less than or equal to now, that haven't sent
Loop through the users, joining a preferences table to get their chosen categories (up to about 30 per user).
For each category, find the latest 3 articles
Prepare an HTML email with this content using PHPMailer
PHPMailer is using Mailgun SMTP to avoid overloading my SMTP server
Send mail to user, mark as sent in database
My observations so far are:
When testing the script by running in-browser, it runs incredibly slowly for a few minutes then times out (without sending any emails).
When running every minute via CRON, it sends way over the number of emails (about 1400) over the course of 40 minutes, I guess because the script is overlapping itself and the sent flag is not reliably updated.
The majority of users are set to receive their email at the same time, so I'm doing 'worst case scenario' testing on this basis
Questions
Is my script far too heavy, by querying the database and generating the HTML email content for each user on the fly? I'm wondering if it would be better to generate the content ahead of time and store against the user in the mailing queue.
Would a queue manager like Beanstalkd help? I've had a look into it, but am struggling to see how to implement into my routine.
Ultimately I need the emails to be sent reliably to each user at the time they expect.
Any advice much appreciated!
You can do this in PHP, but you probably shouldn't. You're trying to build a mail server when there are much better ways of doing this, which mainly involve using a mail server.
Sending high volumes of email during page loads is not workable – it can be troublesome even for single messages, yet many still try. Approach it like this:
Store your list in a database.
When you want to send, generate a record representing each message to send (essentially a copy of the list).
Have a daemon (a long-running task) or cron job that sends the messages in chunks.
Create messages one at a time and submit them to a local mail server.
Use DKIM signatures.
As each message is sent, mark them as sent in the database, but you need to be very aware of how database transactions and locks work for this to work safely and avoid duplicates – do this right and overlapping processes work just fine.
You can generate messages as fast as you like, and your mail server will deal with queuing, onward delivery, retries, bounces.
Use VERP addressing, and feed bounces into a bounce handler (be warned, writing these is not fun!), and have that prevent sending to bouncing addresses in future.
This approach works well – it is exactly how my own system ([Smartmessages.net](https://info.smartmessages.net/ , which is built in PHP) works, and I can sustain over 200 messages per second using multiple message generators running in parallel (database transactions FTW!).
If you find all this a bit too much (it is very difficult), you're probably better off using a commercial sending service (like my own) or hosting a DIY solution, such as Mailcoach by my good friends at Spatie. Either of these would work well, and your list is pretty small – I'm often handling lists of over 100,000.
I started managing a website with a member list of over 50,000 registered members.
I am having this problem:
Each time i try sending the emails, the page runs for several minutes (more than 20 minutes) and eventually ends with up 'Internal Server Error'.
Now, i do not know how many mails that were sent or if any was sent at all.
How do i send the emails:
I select all emails from the database, run a loop and send one after the other. I chose this option because:
i do not want to add several emails in 'To' field, so people's privacy will be maintained.
Adding the emails in Bcc will make it appear as a spam.
Please how can i handle this.
Thank You
All you need to do is break the job into manageable chunks, instead of reading in all 50,000 emails and trying to send them all.
Read in only X emails from the database at a time.
Send that batch of X emails individually (If X = 10, you will send 10 separate emails).
Mark each email as "sent" as soon as each email is sent, otherwise, if there is an error, you may want to increment "send attempts" on that particular email.
After the X emails have been sent (or attempted), update your statistics to indicate that X emails have been sent.
Go back to 1.
I suggest starting with X = 1
I did this by making a script that calls itself with the id of the user it last processed.
E.g. sendmail.php will then call sendmail.php?id=1 which will then call sendmail.php?id=2 and so on.
That will fix your timeout issue.
Within the script you can echo out the results of the mail command. If its false, then you could echo out an error.
I also added a field in my user table for mail send date and time, so i could have a confirmation of where the script got to incase the browser died.
Your sql for getting the next user to send email to would then be something along the lines of
Select useremail from users where sendmaildatetime is not null limit 1
Assuming your default for the sendmaildatetime column is null.
Sorry if im not detailed enough, im writing this from my iphone and its a PITA :)
#hamlin11 response is right, if you want to manage such a big email sending you MUST do it in small chunks and manage this task as an asynchronous task.
To Manage asynchronous tasks you'll have a lot of way, a subprocess check on all your requests that some async jobs are waiting, a separate cron job, a call on a cron.php script from a crontab, even psynnott answer, with an external script relaunching himself at the end.
But you could also use the right tool for the right task, if you have some control on the system under your website. Send one simple email to a mailing list manager that would do the job for you. This would only imply you create the right user list in the mailing list manager. External mailing list manager are for example mailman or sympa. These tools contains robots that you can talk with to feed mailing list recipients.#psynnott answer can be seen as an external PHP script performing very simple mailing list manager tasks. If you want to alter content of the email depending on some user parameters you'll quite certainly have to write your own separate process managing the tasks.
But you could also search for web services handling this job for you. Spam management is a hard job and managing an official big mailer is an hard job as well, but this is not free, of course.
I need to send about ~20k emails to potential clients and need the best way to avoid:
server crash
being put in spammers list
I've searched a few forums and people said it's best to send mails in 1k packages
but my question is what delay between sending each 1k should I set? I mean 1k/day or 1k/hour or 1k/min etc.
I recently had to build system to do so, here is the solution we chose :
1- The management system setup emails and store them in a database.
2- We setup a no-reply account on our mail server to get the bounces back.
3- We build a small code over the great mailing library swiftmailer (swiftmailer.org).
4- We run the script we build through a cron and send mails in packages of 50 per minute (hotmail will most likely flag email as spam if too much emails are sent from the same adresse/server in the same minute). We store each swiftId i the mail database
5- At the runtime of the mail sending script we check for bounces or other errors in our no-reply mailbox and flag our messages in our mail table accordingly (status : 1 - success, 2 - invalid email, 3 - bounces)
6- We sync back our data (upon user accessing the section of the system) in the main system when they have a status and that the last change was made at least 10 minutes in the past to limit the change of states in the system. (the synchronisation script can change the status of an item in the main system, but the 10 minutes delay reduce the risk of changes)
I think your 1K thing is a red herring. There have been a number of similar questions asked with good answers, have a look though these:
How do you make sure email you send programmatically is not automatically marked as spam?
Avoid being blocked by web mail companies for mass/bulk emailing?
Sending solicited mass email
Note that the primary requirement to avoid being your mail being marked as spam is NOT TO SEND SPAM. Your description sounds a lot like spam, in which case no amount of clever tricks will help you.
Providing you are not actually sending spam, a professional organisation such as http://www.campaignmonitor.com/ or http://www.mailchimp.com/ is probably cheaper than the amount of time you will require to do this properly.
Your best bet is to farm it out to specialists.
There's a whole load of subtleties about mass mailing that are easy to get wrong.
Best case is that your server gets put on an email blacklist - meaning all your mails from now on get binned as spam. Worst case is fines and/or jail time, depending n your local laws and the laws where the recipients live.
When it comes down to being put in spammers lists, the question is, how many (very) similar mails from the same sender reach a certain host. If you flood yahoo-mail-accounts with thousands of mails within a short period of time, you can be sure to be on their list shortly.
What you can do: pay a professional service to send out the mails for you. They usually have contracts with the big providers so you don't end up on anyones blacklist (will cost you money, though).If you're sending them out from your server, make sure the "reverse DNS lookup" for your server is activated - I don't know if that's checked any more but it gave me a hard time once.
If your mails don't need to be sent out at a certain time, I'd send the mails in very small packages, e.g. not more than one mail per second, not too much in a row. Then wait for a certain amount of time and repeat until all mails have been sent.A company I used to work for would send out not more than 200 and then wait half an hour before the program proceeded. As far as I know, we never ended up on some blacklist.
The spam list has been well answered, but as far as not bombing out your server, the answer is generally the same. Use a service that does this for you.
If you absolutely must send this volume of e-mail on your own, you want to do it with some sort of background service. This can be written in anything (provided that you don't simply buy one of the many pieces of software off the shelf), but ideally it should be multi-threaded.
Your management application should not be sending these e-mails... only queuing them up in a database or something.
Again though, why reinvent the wheel? You'll save yourself a lot of time by going with something off-the-shelf.
Okay, so here's my problem:
I have a list of members on a website, and periodically one of the admins my site (who are not very web or tech savvy) will send a newsletter to the memberlist.
My current memberlist is well over 800 individuals long.
So, I wrote an email script that sends the email to the full memberlist, with the members listed in the Bcc header.
However, I've discovered that my host server has a limit of 300 emails per hour, which I apparently exceed even though the members are listed in the Bcc field. (I wasn't previously aware that the behaviour of Bcc was to send separate emails for each name on the list...)
After some thought, I've come to the conclusion that my only solution is to have my script send only the email to only the first 300 emails, wait an hour, and send a second email to the next three hundred, wait another hour, and so on until I've sent the email to the whole member list.
Looking around on the internet, I've seen some other solutions people have come up with for delaying emails in PHP. Sleep() is obviously not an option, because I can't just leave the script open and running for 3 or four hours. I've seen some people suggest cron jobs, but I'm not sure how feasible it would be to create three new cron jobs every time I send an email, use them once, and then delete them afterward.
The final (and what I think is the smartest) solution I've seen, is to have a table in my database to temporarily store the emails to be delayed and sent later, and then create a cron job that checks this sql table every hour or so, compares the timestamp of the row to the current timestamp, and then sends the email if an hour has passed.
So I'm asking you all which method you would recommend. Is there an easier solution that I've completely looked over (aside from getting a different hosting plan. ha!), or is there a cleaner way to do it than the database / cron job approach?
tl;dr: I have >800 emails to send in an hour on a server that limits me to 300/hr. Using PHP, find a way to get around this problem in a way that the person sending the email needs only to click "send."
You could send this into a gearman queue and then have a gearman worker with the appropriate sleep calls. See http://gearman.org/ and http://php.net/gearman
Sounds like you need to setup a batching function that pulls from a pool of messages to be sent and processes X everytime its run during cron. Then you would have a table that tracks messages that were sent and to who, so you can keep track of who has received emails.
I would recommend that you create a queue, and process X number of items from the queue each time you need to send email. The sender of the messages just places the email in the queue and your processing code picks up the item sending the maximum number of items in that period. Occasionally you will have failures and use of the queue will allow simple recovery. You only remove items from the queue when they are processed.
You can use a simple database table to act as the queue but you may prefer to use a specialist queuing solution.
Another recommendation would be to look into external email services like Strongmail. These will help you send more email per hour.
I'm about ready to unveil a "coming soon" page and one thing I need is a way for users to enter their email address for me to email once the site goes live. What is the best way to do this?
Should I store the emails in a DB and then run a PHP script to email them from my web host? Should I have the emails just kind of be collected and then emailed manually by me either locally or from the server? Is there another way I should do it?
Also, are there certain web-hosts that restrain the amount of emails you can send out, thus causing a problem for mass email.
The only "unknown" currently is how large of a response I'll get...only time will tell.
Thanks guys!
Store it in the database - Yes. It's no unlike storing any other piece of data.
Mass email at once - No
Mass email manually (or individually mail manually) - No
Do some web hosts limit how many you can send at once? - Yes
I don't know the "best" way but I know a really good way. We have built several mass emailing programs and the technique we incorporated was a throttling technique whereby we had a script that ran every three minutes and sent 20 emails at a time.
It keeps the server from choking and the mail queue from exceeding any of our hosts' mail limits.
Store the email addresses in a database and write a batch job that mail merges them into your message and sends them out as needed.
There are dozens of methods you can use for this kind of problem, but unfortunately there's no real way to pinpoint a solution for you, since there are a lot of variables.
If you only get 3 responses, then you might want to just manually email the users from your email client using BCC:. You've already got all the tools you need to do that, and setting up a script might be a waste of time. This isn't really a great long-term solution however.
For most moderate-sized web sites you would store the emails in a database, and use a mailing script to send them out. PHPMailer is a good tool to help with getting mail sent, and you can manage a decent amount of email addresses manually. Managing addresses manually can be a bit of a pain however, as you have to deal with unsubscribe links, script timeouts, bounced email, etc.
If you get up the tens of thousands of email addresses, you may want to start looking at a third-party mailing software or service that can do threaded sending. Looping through and sending email to 50,000 people via a PHP script can be slow, and take hours.
Basically you have to weigh the difficulty, time and cost of each method versus how much flexibility and power you think you're going to need.
To store emails in a db, and sending them out using a cronjob is a good way to solve the problem, if you have constrains regarding the amount of emails you're allowed to send within a period of time, you can handle it by keeping track of how many emails you've sent every time the cronjob runs.
Also, there are open source products to do this, such as phplist http://www.phplist.com/
Like Eric mentioned above, another option is to use a commercial service to manage your mailing list.
I use Mailchimp (http://www.mailchimp.com). They give you everything you need (signup forms, email templates, etc.), and are completely free unless your list grows to more than 500 subscribers.
Writing the emails to a database or a text/log file are both fine ways to store the emails.
Depending on how many emails you receive, you may want to write a program (PHP works) to send a separate email to each person. Don't send a mass email from your regular email client with everyone on a big To: line.
There are also commercial programs that manage mailing to lists of people (probably open source ones, too). Most of those commercial ones offer a free trial period.