Hoping someone has some suggestions or work-arounds.
I recently wrote an email parser to work for mail forwarded through a pipe in CPanel and have the details entered in a mySQL table.
The client now wants historical mail importing to this database and has approx 50 Gmail accounts with some having over 20,000 messages in.
I have been able to get up to around 7000 emails out of 25000 on the 'test' account I'm using. To try and work out why I've stripped it down so there's a very simple php imap script running just to dump the a couple of parts of the email in to a cell in a table and was going to worry about parsing it after, but the same thing happens. It hits around the 6000-7000 email mark and stops - no errors or anything, it just stops.
I've just found out that Gmail has a bandwidth limit for IMAP of 750MB per hour and doing the math it looks quite likely I'm hitting that and that's why it's stopping.
First question - anyone know how I can check if that's the reason?
Secondly - assuming it is that can anyone think of a best solution?
I considered doing it in blocks of a few thousand emails but this would mean a lot of manual intervention because of the number of accounts and it would take a LONG time to download them all which I don't have.
The second (and so far best) idea I've had is to use one of Gmail's recommended methods to migrate all the mail to an email address created on the client's Cpanel and use the inbuilt pipe feature to forward the mail through my script in the same way I'm doing with their new mail. (I'm not 100% sure the migration will trigger CPanel to apply the pipe rule, I'd have to test).
Appreciate any input.
thanks in advance.
I don't have an answer to your first question.
But as far as your second question goes you don't necessarily have to re-run your program manually if you split the task up into a couple thousand emails. You could either use sleep to stop the program for an hour after x number of emails have been pulled over, or you could setup a cron job that runs once per hour and you would just have to save a pointer to the last email you processed in a database or temp file so you know where to pick up and start again next cron job run.
Related
I'm creating a site that sends out a daily news email to about 800 users, at a time they can specify. My problem is that my script takes a long time to run and times out, so I'm looking for some advice on how I could be approaching this better.
Current approach:
Users are placed in a 'mailing queue' database table with their ID, receive time, and a sent flag.
I'm then running a CRON script every minute which does the following:
Grab all from mailing queue with a 'receive time' less than or equal to now, that haven't sent
Loop through the users, joining a preferences table to get their chosen categories (up to about 30 per user).
For each category, find the latest 3 articles
Prepare an HTML email with this content using PHPMailer
PHPMailer is using Mailgun SMTP to avoid overloading my SMTP server
Send mail to user, mark as sent in database
My observations so far are:
When testing the script by running in-browser, it runs incredibly slowly for a few minutes then times out (without sending any emails).
When running every minute via CRON, it sends way over the number of emails (about 1400) over the course of 40 minutes, I guess because the script is overlapping itself and the sent flag is not reliably updated.
The majority of users are set to receive their email at the same time, so I'm doing 'worst case scenario' testing on this basis
Questions
Is my script far too heavy, by querying the database and generating the HTML email content for each user on the fly? I'm wondering if it would be better to generate the content ahead of time and store against the user in the mailing queue.
Would a queue manager like Beanstalkd help? I've had a look into it, but am struggling to see how to implement into my routine.
Ultimately I need the emails to be sent reliably to each user at the time they expect.
Any advice much appreciated!
You can do this in PHP, but you probably shouldn't. You're trying to build a mail server when there are much better ways of doing this, which mainly involve using a mail server.
Sending high volumes of email during page loads is not workable – it can be troublesome even for single messages, yet many still try. Approach it like this:
Store your list in a database.
When you want to send, generate a record representing each message to send (essentially a copy of the list).
Have a daemon (a long-running task) or cron job that sends the messages in chunks.
Create messages one at a time and submit them to a local mail server.
Use DKIM signatures.
As each message is sent, mark them as sent in the database, but you need to be very aware of how database transactions and locks work for this to work safely and avoid duplicates – do this right and overlapping processes work just fine.
You can generate messages as fast as you like, and your mail server will deal with queuing, onward delivery, retries, bounces.
Use VERP addressing, and feed bounces into a bounce handler (be warned, writing these is not fun!), and have that prevent sending to bouncing addresses in future.
This approach works well – it is exactly how my own system ([Smartmessages.net](https://info.smartmessages.net/ , which is built in PHP) works, and I can sustain over 200 messages per second using multiple message generators running in parallel (database transactions FTW!).
If you find all this a bit too much (it is very difficult), you're probably better off using a commercial sending service (like my own) or hosting a DIY solution, such as Mailcoach by my good friends at Spatie. Either of these would work well, and your list is pretty small – I'm often handling lists of over 100,000.
I have been working on this subject for a long time now and to be honest I am in need of some help if you guys have some time to answer to my problem ;)
I have a few k of emails in my database (around 2k and coming up...) and I would like to send two newsletters a week ... what would be the best way to achieve that ?
So far here is what I am doing :
- a script php (with meta tag refresh set to 600) which sends every 10 minutes an email to 50 people (using basic mail() function). It runs for 6 hours to get to the 2k emails. I am using the less expensive shared server at 1and1 (2EUR a month :D).
1- Is this way to proceed a good way ? I heard when you put people in BCC you go to SPAM directly ... Also many people do not even receive the emails ... no idea why. So far it is not really conclusive.
2- Do you think it would worth taking the dedicated server and send 1 email to each people thats to say 2k different emails ? Or is it a bad move ?
3- How do people proceed when sending newsletters ? I'm sure I look ridiculous with 2k emails and though I have difficulties in making my way through this ;)
Any help welcome.
Cheers.
G/
maximize time between connections to same mail servers
maximize number of servers you use
send to one person at a time
include images, text and html version, unsubscribe link
Check what mail servers respond (like IP blocking notice with link to unblock)
And yes, use cron
Bit of a long shot, but is it possible to count how many emails a given SMTP account has sent in an hour?
Say john#example.com sends emails via Thunderbird and a PHP script.
Can that PHP script first check somehow how many emails have been sent that hour?
Obviously the PHP script can easilly keep a tally of how many emails it sends, but no way of knowing if John has been sending too many from his email client, or another script...
Many thanks.
If SMTP server does not have such statistics, then it is not possible to get the number.
Why are you asking, is there some authority that counts e-mails and says No when you reach the limit? If so, then ask that authority.
You can try to analyze /var/log/mail.log file. Usually it keeps all the info you need. For example, you can write a script, that gathers all data from mail.log, analyzes it, and puts it into some database table. This script could be run by cron every hour, or so (depending when you need the info)
Short answer is no (there's a lot wrong with Konstantin's reply, but criticizing it probably isn't going to help). What is the problem you are trying to solve and why do you think a PHP script is the right solution?
C.
I have a list of around 5,000 to 10,000 (individual user supplied) email addresses from people all over the world, each associated with their username and language codes. I also have a single message translated into the different languages of the users that I want to email. Now, I would like to send a single plain text email to each of the address, with the actual text of the email varying based on the user language, and personalised with the username of the person I'm emailing.
Because of the personalised requirement, and the fact that they will only be emailed once (per year or two with a overlapped but different user list), formal mailing list is probably (& preferably) out. 3rd party bulk email service is also out.
Ignoring programming time, what is the least manually time consuming way to do this in (preferably) PHP? (I am writing the script(s), but not necessarily the person that end up "pressing the button" to send it.) The ideal result is the person sending only having to type a single command to run the script (supplying the email list) and all the email will be sent with no more user intervention. This mean I am looking to avoid things like setting up cron jobs to run the script repeatedly until the email list is exhausted.
When this was done before a year ago, I wrote a PHP script that simply read in the email list line by line processing the username, email address, and language code and build the desired email text out of that before supplying it to PHPMailer to send individually. The problem I had was the script timing out and me not knowing where it got up to so that I can trim the email list at the right place to start again. I ended up manually splitting up the 1 email list into several sub-list that was short enough so that the script doesn't time out. How do I either avoid timing out, or keep track of where the script is up to email address wise so that it can be restarted manually and no person is sent emails more than once?
What other issues are there to take into account, such as avoiding blacklisting etc.?
You need to read about the function set_time_limit and maybe ini_set('memory_limit', xxMB') for memory;
You can run a php cli script from your web process, which forks (pcntl_fork) then kills the parent (the parent cli process exits). The webserver thread running the script can then continue with other code or exit.
Now the cli child process (A) can fork (and become the parent) and monitor a child (B) which sends out the emails. If the child B dies, the now parent A process can fork again and the new child continues where the previous left off.
You have to keep track of who you sent email to in a file/db/shared memory or through paired sockets (socket_create_pair) with the parent process.
I hope you get the idea.
PHP's not the right tool for the job here - you'll want something that runs independent of a web server (although potentially triggered by it) or you'll certainly run into timeouts.
For example, you could have PHP system() out to another script (Perl? Python?), which reads job information from a database then forks into the background to do its work.
You said no Cron scripts - do you not have access to Cron on your server? Or is your concern not wanting to manually set up the Cron job every time a mail job has to be sent out?
You might want to look at http://pear.php.net/package/Mail_Queue - it's something I've been using recently. You can have your user-called script dump all the emails at once into the mail queue. This is much faster than actually sending the emails on the fly since you won't have to deal with communication between servers, but rather just your database.
Then you have a Cron script that, every few minutes, calls the Mail_Queue::sendMailsInQueue command with a limit on the number of emails it sends per Cron call. You won't have to mess with the Cron script if that's your concern - if the queue is empty, it'll just exit. Your email will then trickle out of the server at a reasonable pace per call. This will prevent timeouts due to the limit on emails per cron call. It will also help avoid trouble with other mail servers, who might not be happy to suddenly be hammered by a lot of emails from you at once.
I have a cron which generates the whole mail info and puts in a database table using $mail_queue->put(.....) with an option to delete emails after they're sent.
Here's where I need a little help: What is the best way to send the emails after I have the above info? Running the $mail_queue->sendMailsInQueue() right away, using other cron job(s) or something else?
The server limit by the way is 100 emails / minute. Currently the last csv diff for Mail_Queue is not applied (currently working with the support on that), so I can't use the "delay" option.
I had an idea to use the $seconds_to_send option, but it's calculated on the base of create_time field, which isn't a big help, but it's also an option.
Any ideas and suggestions would be really appreciated.
Personally, I would go the cron way because it gives less opportunity for failure. Say your mail server stops responding or for some other reason becomes unavailable. Or what if your entire network goes offline for a few hours, but the servers are still generating emails. I say use the queue.
As for the delay thing, just have a service/cronjob pick up the queue every sixty seconds, pop 100 emails and send them, then quit. You might get a queue of emails to be sent but that's going to happen no matter what system you choose. The queue will empty during off-peak hours, anyways.
use two scripts. one for populating your mail_queue table with the emails you need to send and the second script to send those emails in chunks of 90 mails at a go. set the second script to be activated about every 2 minutes or so.
you could also just upgrade your hosting plan ;-)
why you dont loop through 100 emails and sleep for 60seconds. this costs you no servertime, the only downside your script runs a little longer.