Putting aside the disdain from junk marketing I need to send around 15,000 emails to customers. My coworker has tried to send them through php mail loop but obviously it gets stuck quickly. Is there a conventional (i.e. through PHP script) to accomplish this swiftly? If not, how do you suggest I do this (maybe through exec) without too much overhead?
Thanks!
I've used PEAR's Mail_Queue to queue up 200,000+ mails at a time. Populating a database is easy and quick, even with customised content, and then a fairly simple script sends around 250 a time - if the load average isn't too high. Then it loops around and sends the next batch.
You won't send them any faster than is usually possible, but it will do it without any problems.
The tutorial gives you almost everything you need - just loop around th 'send_messages.php' script (from the command line is better) until the database queue is empty.
You could look at using something like Gearman to create a queue system as recommended here. Another option would be to look at a paid service like Amazon's Simple Email Service (SES)
No matter how you implement immediate delivery: it'll be a lengthy process that's always subject to interruptions and you can't afford restarting the delivery and sending the same message twice to 5,000 customers.
I believe that a reliable system must use queues. The main script simply adds recipients to a queue and then you have a secondary process that picks items from the queue, get them sent and finally mark them as sent. This secondary process can be launched manually (maybe from the command line) or via cron tab.
I've never used but I have this in my bookmarks: http://ledscripts.com/free/php/phpledmailer
Are you running it through CGI or as a script on the command line? It's best to run it as a script on the command line.
If you say that it gets stuck, try running set_time_limit(0); to avoid PHP quitting from the execution being too long.
Related
I was wondering if there is a way to run a PHP loop in order to send a few hundred emails to subscribers in background. My goal is to format the newsletter, click send and then close the browser or change page. Of course, the actual process of sending the e-mail will be running in background and would not be interrupted by the browser closing.
I know this can be made with a cron job reading from a queue saved in MySQL or text file, but this way, even if there is no queue for a long period, the cron will always be running, looking for the queue...
I've seen this funcionality in a script called Pommo (https://github.com/soonick/poMMo) but can't seem to understand how it's done.
Does anyone have an idea for this?
I was going to add a comment to your question, but them I didn't have enough space there to format and give the example.
Here is an idea I believe might work:
1 - Load all the emails you want to send to a database or file.
2 - From your web application click on the button to send emails. This will submit an Ajax request to the server. You can define the number of emails you want to send within a certain timeframe. Remember that most hosts have limits on number of emails you can send every hour.
3 - create a php script that will receive the Ajax request and send all the emails within the parameters you define.
4 - I believe you can kill your web browser because the PHP script will run through the whole list and will not return until it finishes sending all the emails.
The above might work, however I would never do it this way. I would use a cronjob as stated above. Your cronjob would only have to check if there are emails to send or not. This is not resource intensive.
If you decide to implement the ideas above, please make sure you let us know. I am curious if that would work.
Good luck!
I know this can be made with a cron job reading from a queue saved in
MySQL or text file, but this way, even if there is no queue for a long
period, the cron will always be running, looking for the queue...
That pretty much beats the purpose of Cron. You should create a job that runs, say, every 15 minutes and checks the queue for mails that need to be sent. If there are no mails, let the script die, it'll run again in 15 minutes.
If there are mails to be sent, update the rows to indicate that you're processing them before you start sending, so a run taking more than 15 minutes won't cause another script instance to send the same mails.
You need a queue system. There is e.g. Beanstalkd for linux, which you would feed things with php.
I have a mailing script which loops through approx 900 subscribers (in database), building an individual email for each subscriber and sending via SMTP. This script is fired manually via the browser, however the number of records involved have resulted in the script starting to time out part way through. In tests, the error is almost always Fatal Error max execution time exceeded (although one time there was an error related to mail() and SMTP). I'm thinking that I should probably be running this type of script from the command line, however, the script still needs to be triggered manually (via a CMS admin user) - does anyone have any suggestions for a good way to handle this?
I think you can set the timeout for a PHP script from within the script.
However, I'd rather use CRON+PHP-CLI based solution. The PHP code remains the same whether you use CGI or CLI, however, in CLI mode there is no implicit time limit.
Tweeking the run time limit will only temporarily defer the problem.
You didn't say what OS this is running on - certainly if you've got a local MTA that may help, but the right solution would be to spawn a seperate process. Note that there are a number of pitfalls in this approach - see this post for details of how to do it correctly on Unix type systems.
set_time_limit (0); should do the trick.
There is not much to say about more efficient way, unless we see your code....
work with cronjobs and every time the cron executes the script it can send a small amount of mails.
For example the Typo3 mailer engine is invoked every 5 minutes and will check if any mails need to be sent.
So the user doesnt manually trigger the mailing script. The user only marks a newsletter which should be send by the mailer engine.
I built an email alert for my users (now are only 2,000)
so every night a crontab execute a php script that query the mysql to find matches with user's saved search. it's a classified website in my case, but i would like to learn in case i had to build something for bigger clients
my concerns are:
what happen if my user grow x10 or
x100 times? is the server going to
crash? there any tip you can suggest
on manage something like that?
there is any way to protect my file
cron/nightly_script.php to be
executed form outside calling it in
the url of the browser? consider
tham im using a string in crontab
like:
lynx [absolute url/script.php]
what about the email blast? for each
query if the query has results the
script sends an email, so it means a
blast of emails...is it going to be
considered spam automatically and
then i could blacklisted?
thanks!!!
what happen if my user grow x10 or
x100 times? is the server going to
crash? there any tip you can suggest
on manage something like that?
Your server could crash/get slow as hell because of extensive memory/cpu usage. You should use a message queue like redis/beanstalkd/gearmand to throttle your email alerts. My preference goes out to redis. use the blocking pop/push with predis library which support blocking pop/push.
there is any way to protect my file
cron/nightly_script.php to be executed
form outside calling it in the url of
the browser? consider tham im using a
string in crontab like:
Don't use cron if you want to scale. Instead create couple of daemons.
1 to schedule sending messages(this part could also be cron) to message queue,
1 to process messages send to message queue.
Daemons don't need to be spawned each time and spawning processes is (relative) expensive. Second your script should not call any URL anymore but instead call the PHP scripts directly(CLI).
what about the email blast? for each
query if the query has results the
script sends an email, so it means a
blast of emails...is it going to be
considered spam automatically and then
i could blacklisted?
When using a message queue you can throttle yourself!
Well, you should probably modify your script so that you can spread the load. For example, you can have the cron run 4+ times a day and each time it does a percentage of the user base, instead of doing them all once a day.
You can take it out of the web server target path and put the cron somewhere that i not accessible externally. It could be executed like this: php /location/of/script.php
I guess it will vary depending on who you send it to, but you should consider how often you send this notice.
Number 1: Monitor the server, watch the load and the time it takes to run. It shouldn't crash it but you may find you get to the point where the load is to high and requests for web pages start to slow down.
But one thing to watch is PHP's memory garbage can be odd sometimes, so watch memory usage of the cron job. If it gets to high PHP will crash.
If it starts to get to much there are lots of solutions; there is no need to have the web server and the email sending on the same machine for instance. As long as they can access the same DB, set up a 2nd server just for email sending. This is what cloud computing is perfect for, hire a 2nd server 4 hours a night (or whatever) and turn it off the rest of the time.
That's just one suggestion ... there are many solutions and it really depends on your situation.
As for number 2, the best solution would be to move the script outside the document root so it's not accessible from a browser and call it directly
php [location/script.php]
If you can't do that, I would do a IP check and only allow it to be called from localhost IP.
You could also build in safe checks: store when the last time you sent a email to a particular user was and check that before sending another. This would protect against crontab problems as well as hackers.
I have developed a web application where students across the country come and register for some academic purpose. The users are expected to be around 100k within next year.
I need to send all of these people periodic mails. The web app is developed using codeigniter. The php script can run for 3000 seconds. But still the app is unable to send mails to more that 100 users.
The machine I run is in cloud and has got 256MB ram. I used the free -m command to check the memory usage but that doesnt seem to be a problem. Everything works fine for 10-20 mails.
What would be the best solutions? Is there any way I can transfer this job to some other app/program/shell script ?
If you cannot use some external service for your emails I would just setup a cronjob that sends a couple of emails every n seconds. Its pretty cumbersome to send a lot of emails with php as you have discovered. But the cronjob solution works everytime as far as I know.
So you have a list of emails/addresses and a cronjob that iterates that list and sends the emails.
Sure you can send the emails yourself from a server, but that is only half the battle.
If you are sending bulk emails, as opposed to the transactional type, it's best to use a third party service that is already whitelisted on mail servers. The primary reason being, you might get blacklisted by the major mail servers as a spammer. If this happens, you will have to work with them individually to get removed from the blacklists.
Also if you are operating in the United States you should be familiar with CAN-SPAM: http://business.ftc.gov/documents/bus61-can-spam-act-Compliance-Guide-for-Business
MailChimp is a viable candidate for this. Serving mail is a time-consuming task, and sending it up to 100k email addresses will be an arduous task for your server.
They provide an very capable PHP API.
https://developer.mailchimp.com/
It is very appropriate to get this out of your web server threads and into something that runs standalone. Typically for stuff like this, I have tables in the DB where the appropriate information is written to from the web site, so when I am ready to e-mail, something on the backend can assemble the e-mails and send them out. If you are sending out 100,000 e-mails, you are going to want something multithreaded.
It might be good in this case to use one of the many off-the-shelf tools for this, rather than reinventing the wheel. We use an old version of Campaign Enterprise here, and I am able to throw queries at it which I can use to pull data from my web DB directly, over ODBC. That may or may not work well for you, considering you are in the cloud.
Edit: You can also write a PHP script to do this and call PHP from the shell. Perhaps you can get around your timeout limit this way? (This is assuming you are referring to some service-level timeout. If you are talking about the regular PHP timeout, this can worked around with set_time_limit().)
You might be able to do with using pcntl_fork or creating a daemon process.
Fork: By using the fork process you could batch the emails into groups and send them out. each batch could be in it's own fork child process
Daemon: By using a Daemon you could create a batch of emails and send them to be processed by the daemon. a daemon could run multiple batches at once.
I am creating a web application using zend, here I create an interface from where user-A can send email to more than one user(s) & it works excellent but it slow the execution time because of which user-A wait too much for the "acknowledged response" ( which will show after the emails have sent. )
In Java there are "Threads" by which we can perform that task (send emails) & it does not slow the rest application.
Is there any technique in PHP/Zend just like in Java by which we can divide our tasks which could take much time eg: sending emails.
EDIT (thanks #Efazati, there seems to be new development in this direction)
http://php.net/manual/en/book.pthreads.php
Caution: (from here on the bottom):
pthreads was, and is, an experiment with pretty good results. Any of its limitations or features may change at any time; [...]
/EDIT
No threads in PHP!
The workaround is to store jobs in a queue (say rows in a table with the emails) and have a cronjob call your php script at a given interval (say 2 minutes) and poll for jobs. When jobs present fetch a few (depending on your php's install timeout) and send emails.
The main idea to defer execution:
main script adds jobs in the queue
cron script sends them in tiny slices
Gotchas:
make sure u don't send an email without deleting from queue (worst case would be if a user rescieves some spam at 2 mins interval ...)
make sure you don't delete a job without executing it first ...
handle bouncing email using a score algorithm
You could look into using multiple processes, such as with fork. The communication between them wouldn't be as simple as with threads (but then, it won't come with all of its pitfalls either), but if you're just sending emails, it might not be necessary to communicate much, if at all.
Watch out for doing forks on an Apache process. You may get some behaviors that you are not expecting. If you are looking to do any kind of asynchronous execution it should be via some kind of queuing mechanism. Gearman is one. Zend Server Job Queue is another. I have some demo code at Do you queue? Introduction to the Zend Server Job Queue. Cron can be used, but you'll have the problem of depending on your cron scheduler to run tasks whereas asynchronous computing often needs to be run immediately. Using a queuing system allows you to do that without threading.
There is a Threading extension being developed based on PThreads that looks promising at https://github.com/krakjoe/pthreads
There is pcntl, which allows you to create sub-processes, but php doesn't work very well for this kind of architecture. You're probably better off creating a long-running script (a daemon) and spawning multiple of them.
As of PHP there are no threads in it. However for php, you can have a look at this roundabout way
http://www.alternateinterior.com/2007/05/multi-threading-strategies-in-php.html
You may want to use a queue system for your email sending and send the email from another system which supports threads. PHP is just a tool and you should the tool that is best fitted for the job.
PHP doesn't include threading as part of the language, there are some methods that can emulate it but they aren't foolproof.
This Google search shows a few potential workarounds