I started managing a website with a member list of over 50,000 registered members.
I am having this problem:
Each time i try sending the emails, the page runs for several minutes (more than 20 minutes) and eventually ends with up 'Internal Server Error'.
Now, i do not know how many mails that were sent or if any was sent at all.
How do i send the emails:
I select all emails from the database, run a loop and send one after the other. I chose this option because:
i do not want to add several emails in 'To' field, so people's privacy will be maintained.
Adding the emails in Bcc will make it appear as a spam.
Please how can i handle this.
Thank You
All you need to do is break the job into manageable chunks, instead of reading in all 50,000 emails and trying to send them all.
Read in only X emails from the database at a time.
Send that batch of X emails individually (If X = 10, you will send 10 separate emails).
Mark each email as "sent" as soon as each email is sent, otherwise, if there is an error, you may want to increment "send attempts" on that particular email.
After the X emails have been sent (or attempted), update your statistics to indicate that X emails have been sent.
Go back to 1.
I suggest starting with X = 1
I did this by making a script that calls itself with the id of the user it last processed.
E.g. sendmail.php will then call sendmail.php?id=1 which will then call sendmail.php?id=2 and so on.
That will fix your timeout issue.
Within the script you can echo out the results of the mail command. If its false, then you could echo out an error.
I also added a field in my user table for mail send date and time, so i could have a confirmation of where the script got to incase the browser died.
Your sql for getting the next user to send email to would then be something along the lines of
Select useremail from users where sendmaildatetime is not null limit 1
Assuming your default for the sendmaildatetime column is null.
Sorry if im not detailed enough, im writing this from my iphone and its a PITA :)
#hamlin11 response is right, if you want to manage such a big email sending you MUST do it in small chunks and manage this task as an asynchronous task.
To Manage asynchronous tasks you'll have a lot of way, a subprocess check on all your requests that some async jobs are waiting, a separate cron job, a call on a cron.php script from a crontab, even psynnott answer, with an external script relaunching himself at the end.
But you could also use the right tool for the right task, if you have some control on the system under your website. Send one simple email to a mailing list manager that would do the job for you. This would only imply you create the right user list in the mailing list manager. External mailing list manager are for example mailman or sympa. These tools contains robots that you can talk with to feed mailing list recipients.#psynnott answer can be seen as an external PHP script performing very simple mailing list manager tasks. If you want to alter content of the email depending on some user parameters you'll quite certainly have to write your own separate process managing the tasks.
But you could also search for web services handling this job for you. Spam management is a hard job and managing an official big mailer is an hard job as well, but this is not free, of course.
Related
I'm creating a site that sends out a daily news email to about 800 users, at a time they can specify. My problem is that my script takes a long time to run and times out, so I'm looking for some advice on how I could be approaching this better.
Current approach:
Users are placed in a 'mailing queue' database table with their ID, receive time, and a sent flag.
I'm then running a CRON script every minute which does the following:
Grab all from mailing queue with a 'receive time' less than or equal to now, that haven't sent
Loop through the users, joining a preferences table to get their chosen categories (up to about 30 per user).
For each category, find the latest 3 articles
Prepare an HTML email with this content using PHPMailer
PHPMailer is using Mailgun SMTP to avoid overloading my SMTP server
Send mail to user, mark as sent in database
My observations so far are:
When testing the script by running in-browser, it runs incredibly slowly for a few minutes then times out (without sending any emails).
When running every minute via CRON, it sends way over the number of emails (about 1400) over the course of 40 minutes, I guess because the script is overlapping itself and the sent flag is not reliably updated.
The majority of users are set to receive their email at the same time, so I'm doing 'worst case scenario' testing on this basis
Questions
Is my script far too heavy, by querying the database and generating the HTML email content for each user on the fly? I'm wondering if it would be better to generate the content ahead of time and store against the user in the mailing queue.
Would a queue manager like Beanstalkd help? I've had a look into it, but am struggling to see how to implement into my routine.
Ultimately I need the emails to be sent reliably to each user at the time they expect.
Any advice much appreciated!
You can do this in PHP, but you probably shouldn't. You're trying to build a mail server when there are much better ways of doing this, which mainly involve using a mail server.
Sending high volumes of email during page loads is not workable – it can be troublesome even for single messages, yet many still try. Approach it like this:
Store your list in a database.
When you want to send, generate a record representing each message to send (essentially a copy of the list).
Have a daemon (a long-running task) or cron job that sends the messages in chunks.
Create messages one at a time and submit them to a local mail server.
Use DKIM signatures.
As each message is sent, mark them as sent in the database, but you need to be very aware of how database transactions and locks work for this to work safely and avoid duplicates – do this right and overlapping processes work just fine.
You can generate messages as fast as you like, and your mail server will deal with queuing, onward delivery, retries, bounces.
Use VERP addressing, and feed bounces into a bounce handler (be warned, writing these is not fun!), and have that prevent sending to bouncing addresses in future.
This approach works well – it is exactly how my own system ([Smartmessages.net](https://info.smartmessages.net/ , which is built in PHP) works, and I can sustain over 200 messages per second using multiple message generators running in parallel (database transactions FTW!).
If you find all this a bit too much (it is very difficult), you're probably better off using a commercial sending service (like my own) or hosting a DIY solution, such as Mailcoach by my good friends at Spatie. Either of these would work well, and your list is pretty small – I'm often handling lists of over 100,000.
I have a big amount of users that agreed to recive daily newsletter. Content of newsletter is being auto-generated, so the only thing to do is to set up a cron job which would send e-mails.
However, if there is e.g. 10.000 users, such cron job would kill my server. What can be done to solve this problem?
Is sleep(1) after sending 100 e-mails enough? (and of course setting execution time limit to 1 day)
Have a look into http://php.net/manual/en/function.mail.php
Note:
It is worth noting that the mail() function is not suitable for larger
volumes of email in a loop. This function opens and closes an SMTP
socket for each email, which is not very efficient.
For the sending of large amounts of email, see the » PEAR::Mail, and »
PEAR::Mail_Queue packages.
So simply use the Mail_Queue package... which takes every mail and then simply works through them.
I have made a system for sending emails for project few months ago so I done the following:
In database, I have 3 tables:
users
user_emails (some users have multiple emails)
email_campaign (this is table in which I store data temporary while sending campaign and on finish I truncate everything in it)
And when I start sending campaign I insert row in email_campaign table for every user that I finished sending email to.
This way if error occur before campaign finishes, I know where to continue and know to whom I sent email and to who need to send email.
Practicaly I was able to send 45.000 emails during 2 hours. Without server overload.
I use sleep() on every 100 emails like you wanted to do.
Also I send campaigns at 2 am in the morning when my server load is the lowest.
You could also configure your email server to send limited ammount of emails per hour.
This would slow sending but it will reduce server load.
I plan to send an excess amount of emails daily to my website users. I am using php. Will such a task affect the performance of my website? What can be done to prevent issues?
Note: Emails are sent with users consent and scheduled by them. No spamming.
Instead of sending the email directly, why not create a database table to store the email details. and just write to that table each time you need to send an email.
Then have a scheduled task that occasionally queries this table and do the mail sending.
That way, you do not need to worry about the php page timing out. and in future, you can also pull the schedule task out and run it on a separate server if it uses too much resources.
Some off the top of my head:
If your on shared hosting, you may get shut down for attempted spamming
The server IP may get labelled as a spam source if your sending email to random people
Send the emails in a cronjob rather than in real time, that way you can do the email sending at non-peak times
My suggestion and the way we handle this is to utilize a message queue, we currently use Zend Frameworks Package, then from a cron job retrieve the queue and send the email. This enables us to increase performance and avoid delays in displaying the page to the user. Hope that helps!
Okay, so here's my problem:
I have a list of members on a website, and periodically one of the admins my site (who are not very web or tech savvy) will send a newsletter to the memberlist.
My current memberlist is well over 800 individuals long.
So, I wrote an email script that sends the email to the full memberlist, with the members listed in the Bcc header.
However, I've discovered that my host server has a limit of 300 emails per hour, which I apparently exceed even though the members are listed in the Bcc field. (I wasn't previously aware that the behaviour of Bcc was to send separate emails for each name on the list...)
After some thought, I've come to the conclusion that my only solution is to have my script send only the email to only the first 300 emails, wait an hour, and send a second email to the next three hundred, wait another hour, and so on until I've sent the email to the whole member list.
Looking around on the internet, I've seen some other solutions people have come up with for delaying emails in PHP. Sleep() is obviously not an option, because I can't just leave the script open and running for 3 or four hours. I've seen some people suggest cron jobs, but I'm not sure how feasible it would be to create three new cron jobs every time I send an email, use them once, and then delete them afterward.
The final (and what I think is the smartest) solution I've seen, is to have a table in my database to temporarily store the emails to be delayed and sent later, and then create a cron job that checks this sql table every hour or so, compares the timestamp of the row to the current timestamp, and then sends the email if an hour has passed.
So I'm asking you all which method you would recommend. Is there an easier solution that I've completely looked over (aside from getting a different hosting plan. ha!), or is there a cleaner way to do it than the database / cron job approach?
tl;dr: I have >800 emails to send in an hour on a server that limits me to 300/hr. Using PHP, find a way to get around this problem in a way that the person sending the email needs only to click "send."
You could send this into a gearman queue and then have a gearman worker with the appropriate sleep calls. See http://gearman.org/ and http://php.net/gearman
Sounds like you need to setup a batching function that pulls from a pool of messages to be sent and processes X everytime its run during cron. Then you would have a table that tracks messages that were sent and to who, so you can keep track of who has received emails.
I would recommend that you create a queue, and process X number of items from the queue each time you need to send email. The sender of the messages just places the email in the queue and your processing code picks up the item sending the maximum number of items in that period. Occasionally you will have failures and use of the queue will allow simple recovery. You only remove items from the queue when they are processed.
You can use a simple database table to act as the queue but you may prefer to use a specialist queuing solution.
Another recommendation would be to look into external email services like Strongmail. These will help you send more email per hour.
I have a list of around 5,000 to 10,000 (individual user supplied) email addresses from people all over the world, each associated with their username and language codes. I also have a single message translated into the different languages of the users that I want to email. Now, I would like to send a single plain text email to each of the address, with the actual text of the email varying based on the user language, and personalised with the username of the person I'm emailing.
Because of the personalised requirement, and the fact that they will only be emailed once (per year or two with a overlapped but different user list), formal mailing list is probably (& preferably) out. 3rd party bulk email service is also out.
Ignoring programming time, what is the least manually time consuming way to do this in (preferably) PHP? (I am writing the script(s), but not necessarily the person that end up "pressing the button" to send it.) The ideal result is the person sending only having to type a single command to run the script (supplying the email list) and all the email will be sent with no more user intervention. This mean I am looking to avoid things like setting up cron jobs to run the script repeatedly until the email list is exhausted.
When this was done before a year ago, I wrote a PHP script that simply read in the email list line by line processing the username, email address, and language code and build the desired email text out of that before supplying it to PHPMailer to send individually. The problem I had was the script timing out and me not knowing where it got up to so that I can trim the email list at the right place to start again. I ended up manually splitting up the 1 email list into several sub-list that was short enough so that the script doesn't time out. How do I either avoid timing out, or keep track of where the script is up to email address wise so that it can be restarted manually and no person is sent emails more than once?
What other issues are there to take into account, such as avoiding blacklisting etc.?
You need to read about the function set_time_limit and maybe ini_set('memory_limit', xxMB') for memory;
You can run a php cli script from your web process, which forks (pcntl_fork) then kills the parent (the parent cli process exits). The webserver thread running the script can then continue with other code or exit.
Now the cli child process (A) can fork (and become the parent) and monitor a child (B) which sends out the emails. If the child B dies, the now parent A process can fork again and the new child continues where the previous left off.
You have to keep track of who you sent email to in a file/db/shared memory or through paired sockets (socket_create_pair) with the parent process.
I hope you get the idea.
PHP's not the right tool for the job here - you'll want something that runs independent of a web server (although potentially triggered by it) or you'll certainly run into timeouts.
For example, you could have PHP system() out to another script (Perl? Python?), which reads job information from a database then forks into the background to do its work.
You said no Cron scripts - do you not have access to Cron on your server? Or is your concern not wanting to manually set up the Cron job every time a mail job has to be sent out?
You might want to look at http://pear.php.net/package/Mail_Queue - it's something I've been using recently. You can have your user-called script dump all the emails at once into the mail queue. This is much faster than actually sending the emails on the fly since you won't have to deal with communication between servers, but rather just your database.
Then you have a Cron script that, every few minutes, calls the Mail_Queue::sendMailsInQueue command with a limit on the number of emails it sends per Cron call. You won't have to mess with the Cron script if that's your concern - if the queue is empty, it'll just exit. Your email will then trickle out of the server at a reasonable pace per call. This will prevent timeouts due to the limit on emails per cron call. It will also help avoid trouble with other mail servers, who might not be happy to suddenly be hammered by a lot of emails from you at once.