I have a cron which generates the whole mail info and puts in a database table using $mail_queue->put(.....) with an option to delete emails after they're sent.
Here's where I need a little help: What is the best way to send the emails after I have the above info? Running the $mail_queue->sendMailsInQueue() right away, using other cron job(s) or something else?
The server limit by the way is 100 emails / minute. Currently the last csv diff for Mail_Queue is not applied (currently working with the support on that), so I can't use the "delay" option.
I had an idea to use the $seconds_to_send option, but it's calculated on the base of create_time field, which isn't a big help, but it's also an option.
Any ideas and suggestions would be really appreciated.
Personally, I would go the cron way because it gives less opportunity for failure. Say your mail server stops responding or for some other reason becomes unavailable. Or what if your entire network goes offline for a few hours, but the servers are still generating emails. I say use the queue.
As for the delay thing, just have a service/cronjob pick up the queue every sixty seconds, pop 100 emails and send them, then quit. You might get a queue of emails to be sent but that's going to happen no matter what system you choose. The queue will empty during off-peak hours, anyways.
use two scripts. one for populating your mail_queue table with the emails you need to send and the second script to send those emails in chunks of 90 mails at a go. set the second script to be activated about every 2 minutes or so.
you could also just upgrade your hosting plan ;-)
why you dont loop through 100 emails and sleep for 60seconds. this costs you no servertime, the only downside your script runs a little longer.
Related
I'm creating a site that sends out a daily news email to about 800 users, at a time they can specify. My problem is that my script takes a long time to run and times out, so I'm looking for some advice on how I could be approaching this better.
Current approach:
Users are placed in a 'mailing queue' database table with their ID, receive time, and a sent flag.
I'm then running a CRON script every minute which does the following:
Grab all from mailing queue with a 'receive time' less than or equal to now, that haven't sent
Loop through the users, joining a preferences table to get their chosen categories (up to about 30 per user).
For each category, find the latest 3 articles
Prepare an HTML email with this content using PHPMailer
PHPMailer is using Mailgun SMTP to avoid overloading my SMTP server
Send mail to user, mark as sent in database
My observations so far are:
When testing the script by running in-browser, it runs incredibly slowly for a few minutes then times out (without sending any emails).
When running every minute via CRON, it sends way over the number of emails (about 1400) over the course of 40 minutes, I guess because the script is overlapping itself and the sent flag is not reliably updated.
The majority of users are set to receive their email at the same time, so I'm doing 'worst case scenario' testing on this basis
Questions
Is my script far too heavy, by querying the database and generating the HTML email content for each user on the fly? I'm wondering if it would be better to generate the content ahead of time and store against the user in the mailing queue.
Would a queue manager like Beanstalkd help? I've had a look into it, but am struggling to see how to implement into my routine.
Ultimately I need the emails to be sent reliably to each user at the time they expect.
Any advice much appreciated!
You can do this in PHP, but you probably shouldn't. You're trying to build a mail server when there are much better ways of doing this, which mainly involve using a mail server.
Sending high volumes of email during page loads is not workable – it can be troublesome even for single messages, yet many still try. Approach it like this:
Store your list in a database.
When you want to send, generate a record representing each message to send (essentially a copy of the list).
Have a daemon (a long-running task) or cron job that sends the messages in chunks.
Create messages one at a time and submit them to a local mail server.
Use DKIM signatures.
As each message is sent, mark them as sent in the database, but you need to be very aware of how database transactions and locks work for this to work safely and avoid duplicates – do this right and overlapping processes work just fine.
You can generate messages as fast as you like, and your mail server will deal with queuing, onward delivery, retries, bounces.
Use VERP addressing, and feed bounces into a bounce handler (be warned, writing these is not fun!), and have that prevent sending to bouncing addresses in future.
This approach works well – it is exactly how my own system ([Smartmessages.net](https://info.smartmessages.net/ , which is built in PHP) works, and I can sustain over 200 messages per second using multiple message generators running in parallel (database transactions FTW!).
If you find all this a bit too much (it is very difficult), you're probably better off using a commercial sending service (like my own) or hosting a DIY solution, such as Mailcoach by my good friends at Spatie. Either of these would work well, and your list is pretty small – I'm often handling lists of over 100,000.
Hoping someone has some suggestions or work-arounds.
I recently wrote an email parser to work for mail forwarded through a pipe in CPanel and have the details entered in a mySQL table.
The client now wants historical mail importing to this database and has approx 50 Gmail accounts with some having over 20,000 messages in.
I have been able to get up to around 7000 emails out of 25000 on the 'test' account I'm using. To try and work out why I've stripped it down so there's a very simple php imap script running just to dump the a couple of parts of the email in to a cell in a table and was going to worry about parsing it after, but the same thing happens. It hits around the 6000-7000 email mark and stops - no errors or anything, it just stops.
I've just found out that Gmail has a bandwidth limit for IMAP of 750MB per hour and doing the math it looks quite likely I'm hitting that and that's why it's stopping.
First question - anyone know how I can check if that's the reason?
Secondly - assuming it is that can anyone think of a best solution?
I considered doing it in blocks of a few thousand emails but this would mean a lot of manual intervention because of the number of accounts and it would take a LONG time to download them all which I don't have.
The second (and so far best) idea I've had is to use one of Gmail's recommended methods to migrate all the mail to an email address created on the client's Cpanel and use the inbuilt pipe feature to forward the mail through my script in the same way I'm doing with their new mail. (I'm not 100% sure the migration will trigger CPanel to apply the pipe rule, I'd have to test).
Appreciate any input.
thanks in advance.
I don't have an answer to your first question.
But as far as your second question goes you don't necessarily have to re-run your program manually if you split the task up into a couple thousand emails. You could either use sleep to stop the program for an hour after x number of emails have been pulled over, or you could setup a cron job that runs once per hour and you would just have to save a pointer to the last email you processed in a database or temp file so you know where to pick up and start again next cron job run.
Okay, so here's my problem:
I have a list of members on a website, and periodically one of the admins my site (who are not very web or tech savvy) will send a newsletter to the memberlist.
My current memberlist is well over 800 individuals long.
So, I wrote an email script that sends the email to the full memberlist, with the members listed in the Bcc header.
However, I've discovered that my host server has a limit of 300 emails per hour, which I apparently exceed even though the members are listed in the Bcc field. (I wasn't previously aware that the behaviour of Bcc was to send separate emails for each name on the list...)
After some thought, I've come to the conclusion that my only solution is to have my script send only the email to only the first 300 emails, wait an hour, and send a second email to the next three hundred, wait another hour, and so on until I've sent the email to the whole member list.
Looking around on the internet, I've seen some other solutions people have come up with for delaying emails in PHP. Sleep() is obviously not an option, because I can't just leave the script open and running for 3 or four hours. I've seen some people suggest cron jobs, but I'm not sure how feasible it would be to create three new cron jobs every time I send an email, use them once, and then delete them afterward.
The final (and what I think is the smartest) solution I've seen, is to have a table in my database to temporarily store the emails to be delayed and sent later, and then create a cron job that checks this sql table every hour or so, compares the timestamp of the row to the current timestamp, and then sends the email if an hour has passed.
So I'm asking you all which method you would recommend. Is there an easier solution that I've completely looked over (aside from getting a different hosting plan. ha!), or is there a cleaner way to do it than the database / cron job approach?
tl;dr: I have >800 emails to send in an hour on a server that limits me to 300/hr. Using PHP, find a way to get around this problem in a way that the person sending the email needs only to click "send."
You could send this into a gearman queue and then have a gearman worker with the appropriate sleep calls. See http://gearman.org/ and http://php.net/gearman
Sounds like you need to setup a batching function that pulls from a pool of messages to be sent and processes X everytime its run during cron. Then you would have a table that tracks messages that were sent and to who, so you can keep track of who has received emails.
I would recommend that you create a queue, and process X number of items from the queue each time you need to send email. The sender of the messages just places the email in the queue and your processing code picks up the item sending the maximum number of items in that period. Occasionally you will have failures and use of the queue will allow simple recovery. You only remove items from the queue when they are processed.
You can use a simple database table to act as the queue but you may prefer to use a specialist queuing solution.
Another recommendation would be to look into external email services like Strongmail. These will help you send more email per hour.
How would i send an email, to say 3000 recipients - with a Max 500 emails / hours on my dedicated IP? So far my thought is to send each email every 9 seconds, this would come to about 450 emails an hour... but how could i do this?
My plan for the sending of the emails would be the following...
$emails = ARRAY OF EMAILS, MYSQL RESULT
for($emails){
mail($subject,$row[email],$headers);
}
This wont work, wrong kind of statement but this concept anyway....
What I would do is :
create a PHP script that is launched via cron once per hour
this script only sends 450 e-mails, at its own speed
when the 450 mails are sent, the script dies
and some time later, it is re-launched, by cron, to send 450 other mails.
The trick is : you have to know which mails where already sent.
Ordering the mails by id in your DB, or something like that, and using limit, would be OK, I suppose
If you want to sleep for a while between mails, use the sleep function ; something between 2 and 5 seconds would probably be OK, to be sure you script the chunk of 450 mails is finished before the script is re-launched by cron.
And, thinking about it :
You should put some logging stuff in place : if someone complains, saying he received 10 emails, it could help you find out why.
I wouldn't use the mail function : there are plenty of other possibilities, using libraries that are well-tested and provide lots of functionnalities, already developped : don't re-invent the wheel ;-)
Here are a couple of libraries I can think about :
Swift Mailer
Zend_Mail
PHPMailer
PEAR::Mail
Rmail
Store You messages for sending in a database, mark messages which are sent.
In a cron job select some of them that are not sent, and process them.
The frequency of the cron job determines the speed of sending the emails.
SwiftMailer does it this for you:
http://swiftmailer.org/docs/antiflood-plugin
http://swiftmailer.org/docs/throttler-plugin
You could use this very handy Timer class to do the heavy lifting for you (start, stop and get the elapsed time within your loop, etc): PHPClasses: Timer.php.
Thanks for all the answers!
The best way i found was actually to simply sleep() between calls using the sleep() as i tested 400 mails, this took 17 seconds :)
It is unlikely the user will send more than the 450 limit ... but if they do i have an if statement before the while() ends checking if there are more than 450 rows, if so it will sleep between each... this works without fiddly databases :)
Thanks!
I have to made a page which will send Email to Newsletter subscribers. There is more then 14000 subscriber. I want to use php mail() function to send Email to them. But I'm afraid that it will not be able to send email to all subscribers for php 30sec max_execution_time limit. Its not possible to test how much Email can be sent by sending Test Email to subscribers. So I want to know how much Email can be sent with mail() function in 30 second max_execution_time limit.
Will be very helpful if you can answer me.
also another question - Is mysql execution time is also count in php?
Apache version 2.2.13 (Unix)
PHP version 5.2.11
The php max_execution_time setting is customizable. 30 seconds is the default but you can set it to 0 seconds for no execution time limit at all. Use set_time_limit().
set_time_limit(0);
If you do this, you should be able to send all your email.
Please be careful about sending more than one email to the same mail server per second. You don't want to get blacklisted.
You should run this from a cronjob or spawn a background task or use something else better suited to batch jobs.
You might get 14000 emails out in 30sec if your mailserver is fast enough, but what happens when you get a few more subscribers and it stops working properly?
Perhaps you can set a flag in the database for each user, then reset the flag as their email is sent by a background task. That will help to avoid duplicates and so on if there is a problem with the mail server.
That depends on so many variables that a single answer is not possible. Factors include:
The speed of the CPU
The bandwidth available from the sending system to the MTA
The MTA's capacity to accept emails
The only way to find out is to try it.
I had this exact problem a while back on one of my projects. The solution is to isolate the email sending from the actual site.
I coded a small class that would be called to send an email. It would be passed a templated email, which it would then store into the database in a mail queue. On the back end, I had a cron job that called a mailer script every X seconds. Script looks at the database queue for emails, grabs X number from the queue to attempt to send (ordered by timestamp in), then would attempt delivery. Assuming no errors were thrown, script would mark the message as sent. Next step would be to purge all emails from the queue that were sent and older than X days (kept for logging).
Hope that is helpful.
Seriously, if you want to send the same mail to ten people from your normal mailapp, do you normally create ten identical mails or do you just send the mail once adding the recipients to the send list?
Edit: If the answer is "I send it once", I think you should look in that direction here as well (it is even described how to send to multiple recipients at http://www.php.net/mail)