Background: I run a site which is basically free. It has a sizeable number of email subscribers but because currently there's no pricing model I cannot afford to use Mailchimp to overcome their 2K subscriber limit.
I'm thinking to send it out myself. If I've got an array of ~12000 emails and I use PHP to send out the newsletter, from the server what will the impact be on the server? Could response time for the actual site users (seeing as the website is hosted on the same server) be impacted? Could the server crash? How long might this task take approximately?
in my experience, PHP list would be your best option for this as it's the most popular open source email managment system. Depending on your host, they may have limits on the number of outgoing emails that you send so it's best to check with them first and then configure it so that it is inline with their rules. It takes some configuring, but once you're up and running, it's very easy to use.
In saying that - after managing much more sizeable lists for a good 5-6 years, I've now migrated them all over to Campaign Monitor.
All in all, PHP List will definitely do what you need.
Related
In our application we need to connect to the user email inbox and fetch all of messages.
The application will run in a dedicated server by Ubuntu OS.
I have ran some test in my pc by using imap extension (imap_open, imap_fetchheader, imap_fetchbody, ...).
There are multiple problems in using this approach:
1- The connection and fetching speed is very slow.
2- The php script will time out in case of many messages exists in inbox.
3- The application is multiuser and the number of requests is high.
My search results to solve above problems:
1- We have to rent a static IP from google or other email servers for imap connections !
2- We have to use PHP CLI to fetch orders from database and fetch inbox messages.
3- Multithreading in php.
Summary:
I want to write a bot in php to Connect to mail servers everyday. What connection and fetching way do you suggest? (library, method, language and etc). Thanks
**UPDATE : The code that I had ran to fetch 30 first messages:
$mbox = imap_open('{imap.gmail.com:993/imap/ssl/novalidate-cert}', 'myemailaddress', 'mypassword');
$count = imap_num_msg($mbox);
for($i = $count; $i >= $count - 30; $i--){
$header = imap_fetchheader($mbox, $i);
/* process header & body */
}
imap_close($mbox);
This code works well but when I change imap_fetchheader to imap_fetchbody it takes more time.
Update :
Based on the arkascha answer, I worked on my design and architecture and I found out the low speed reason is in the connections to the mail server. Because this is a I/O bounded task, I cached the imap connection to each mail server and the hole speed improved but not so much.
Sorry, but your search results are simply wrong.
you do not need a static IP address to fetch messages from an imap server. And certainly not one "rented from google". Why should that be required?
I see no reasons why the cli variant of php should be better suited here. In contrary, it is less efficient, since it has a much bigger startup load bacause a process must be spawned for each and every request.
why should you need multithreading? What benefits should it offer for this situation? Multithreading is only of interest for interactive stuff where responsiveness might be an issue. And even then this can be the wrong idea.
I myself implemented an imap client a few times and ran into no such problems. You certainly can implement a robust and efficient solution based on the phps imap extension. Speed of connection and transfer depend on many details, I doubt that php imap implementation is the cause of your problems. There might be additional issues which lead you to your statements, but you do not specify such in your question.
In general you should never fetch a huge list of data in one go and then process it. That is simply really bad design which cannot scale in any way. Instead you should always process lists in a sequential strategy: fetch a single data unit and process it. Then head on to the next. That way your memory footprint stays small, you prevent hitting php limits. This also enables you to process the whole list in smaller chunks (or even one message at a time) which prevents you from hitting phps execution time limit or a timeout on client side. These are general successful implementation patterns which are the base of a robust processing implementation which also apply to email messages.
To answer your bottom line question:
nothing speaks against using phps imap extension
use a cron job for this and do it on a regular base, not just once a day
invest into planning a clean architecture before starting the implementation
I might be biased but I'd use Zend Framework with Zend_Mail_Storage classes: http://framework.zend.com/manual/1.7/en/zend.mail.read.html You get a couple of options like switching to directly accessing mbox or maildir files if possible or getting closer to the underlying protocol and hack something in the zend_mail_protocol_imap to improve speed.
I am working on a business marketing website. The criteria is that we have to send around 80,000 (say) newsletters per day to different email addresses. But my hosting company allowed me to send only 10,000 (say) emails from one IP per day. So to solve this issue my client provided me with 8 different IPs. I am having following questions:
I don't know how to send each and every newsletter from different IPs, means first from first IP, second from second IP and so on?
As I'm a PHP developer and not an expert at server end, can I accomplish this through my PHP script only or do I have to write a shell script to run at my server?
Is there any algorithm to accomplish such type of problem?
Had similiar problem couple years ago. I resosolved it in simple way. Create table in SQL, or create a class and serialize it. Class or SQL have to contain mails addresses (if You have some kind of volatile content then also mail content), do a class to send mail in certain order, and do object of this class on each ip. That depends what kind of config You have on server. I had a easy way cause all of IPs had its own folder with content, so I just put there code to do object, and just redirected couple of times the website, since it had to be done by web. If You have can have it by CLI there is pretty chance that You could do it by include or similar.
This way is pretty lame, I know it, but didn't want to do something more sophisticated at that time. Later on I wrote a class to manage mail connections via SMTP, so I could chose with mail from witch account would be send. That is better way, but not all servers could support it.
Another way is to do a cron job and do baskets of mail to send portions over the time. (this way was most common on servers that I was repairing).
Another way is to do a bouncing effect on servers with IP. Probably You could do also some shell scripting and invoke it via php script.
Well, I think that there are better ways, but it really depends on server config.
PS. Forgotten to mention, that code can be invoked by AJAX.
I'm thinking about how to handle sending large amounts of email from my web applications, and whether there are any best practices for doing so. StackOverflow is already labeling this as 'subjective', which it may be to an extent, but I need to know the most successful way to implement this system, or whether any standardized approach exists.
In my webapp, there are users who are heads of groups of 1 to 10,000 users. This user must be able to email send a message to all of these users through my system. Therefore, my system is responsible for sending up to 10,000 emails to individual users for each group head.
As far as I can tell, there is no rate limit in GMail for sending messages to individuals (although there is a 500 recipient max).
Right now, my current setup is:
When a message is sent through the system, it enters an email queue.
A cron script grabs messages from the queue every few minutes, and sends out those emails.
All email is taking place through GMail's SMTP server.
The actual application doing the mailing is PHPMailer.
This setup, as the user base grows, will probably not suffice. The questions I have are:
Should I be using a local SMTP server instead?
Should I use a mail binary on the local machine instead? I this case, I could probably skip the queue altogether?
Is there an accepted way to do this?
Thanks!
Google App Engine
I would write this in Google app engine (python) because:
It scales well.
It has a good email api.
It has a taskqueue with a good api to access it.
Because python is a real nice language.
It is (relatively) cheap.
PHP
If I would implement it in PHP I would
Find yourself a good SMTP server which allows you to sent this volume of mails because Gmail will not allow you to sent this kind of volume. I am for sure that this will cost you some money.
Find yourself a decent PHP email library to sent messages with like for example PHPMailer like you said.
Use a message queue like for example beanstalkd to put email messages in queue and sent email asynchronously. First because with this the user will have snappier page load. Second with a message queue like beanstalkd you can regulate speed of sending better which will prevent from overloading your pc with work. You will need to have ssh access to the server to compile(install) beanstalkd. You can find beanstalkd at beanstalkd
You would also need ssh access to run a PHP script in the background which will process the message queue. You can find a beanstalkd-client at php beanstalkd-client
from php/apache/webpage
This is the page from which you will sent messages out to user. From this page you will sent message to beanstalkd by coding something in the lines of this:
// register Pheanstalk class loader
require_once('pheanstalk_init.php');
$pheanstalk = new Pheanstalk('127.0.0.1');
$message = ""; // This would contain your message
$pheanstalk->put(json_encode($message);
You have to put messages in message queue using put command
From long running PHP script in background:
The code would look something like this:
// register Pheanstalk class loader
require_once('pheanstalk_init.php');
$pheanstalk = new Pheanstalk('127.0.0.1');
while(true) {
$job = $pheanstalk->reserve();
$email = json_decode($job->getData());
// Sent email using PHP mailer.
$pheanstalk->delete($job);
}
Like I am saying it is possible with both PHP and Google app engine but I would go for app engine because it is easier to implement.
With an email count as "high" as 10.000 a day, I wouldn't rely on GMail (or any other) SMTP. Not that they can't handle it, obviously they can handle A LOT more. But they possibly don't want to.
Having a local SMTP server is IMO the way to go :
It's pretty easy to set up (just DON'T let people use it without a strong authent scheme)
Most modern MTA handle sending queues very well
You won't have to deal with GMail (or others) deciding to block your account someday for quota reasons
Gmail and Google Apps limits you to around 500 emails a day. I'm not sure how that combines with the 500 recipient max, but if you want to send 10 000 emails you'll probably want to find another mail server. I personally use a local server or the ISP or datacenter's SMTP.
If you are sending that many emails, I would recommend using the queue so the user's isn't sitting there waiting for the email to be sent.
Be very careful that your domain doesn't get blacklisted as a spam domain. If it does, you can expect most of your emails to be blocked, support, sales, etc. Which could in turn be very expensive.
You may instead want to use a service like AWeber. Not only are they setup to handle these amounts of emails, but they can probably give you more metrics than you can implement on your own.
I'm not sure if it's publishe anywhere, but from experience I can tell you that Gmail will put a fifteen minute or so freeze on your account if you start sending hundreds of messages at a time. This happened to me last week. I think you should host your own SMTP server. Using the mail() function often will put your mail in someone's spam folder.
Just install Postfix on the local machine, or a machine on the same LAN for maximum access speeds. Make sure it is well secured from the outside, and quickly accessible from the inside.
Then code your PHP script to directly inject the emails to the Postfix queue. That shall dramatically increase the processing speed of mail delivery.
My company has a website built with PHP. We use the built-in PHP email functionality to send thousands of emails to subscribers on a daily basis.
This is a terrible idea. It chokes out our server, and takes hours to complete the whole batch.
Now I've looked at mass mailing services like MailChimp (which would replace our current system of sending the same email to many people), but what I think I'd really like to do is to set up a somewhat-sophisticated notification system.
Rather than send a mass email to each person each time something important happens, I'd like clients to be able to customize the rate and content of the emails that they receive.
Even using this new idea, we're talking about A LOT of emails being sent.
So my question is very specific: I have a rough idea of how to build the system internally, but what is the best way to send out all of these emails?
Bullet points to consider:
Sometimes emails' contents are identical across recipients, but many of them will be customized per-user (they choose what they get notified about, and sometimes it is aggregated).
I want a system that is not going to choke the server, and will complete in a decent amount of time. I don't mind going with a third-party service (even a paid one) if that is what it is going to take.
The system should hook into PHP easily, or the API or whatever should be relatively easy for me to call from your typical web server.
We have a dedicated server and full control over it (so we can install apps, services, whatever).
Any kind of detailed tracking information (opens, clicks, etc) is a huge plus.
These emails are sometimes time-sensitive (so can't take all day to send).
Thoughts? Tips? Point me in the right direction?
EDIT
To clarify:
I can do these on my own:
maintain user list
handle email content generation based on user preferences
And need something else (app, third-party service, w/e) to:
accept email content and addresses and actually send the emails out
provide tracking data (opens, clicks, etc). The more detail the better.
I'm leaning towards a third-party service, since I'm not sure any app can avoid choking the server when sending thousands of emails (though I wouldn't consider myself an email expert so I could be wrong).
We use the built-in PHP email functionality to send thousands of emails to subscribers on a daily basis.
This is a terrible idea. It chokes out our server, and takes hours to complete the whole batch.
Why do you think that your problems are anything to do with the built-in PHP email function? It's a very thin wrapper around 'mail' or a simple SMTP client depending on what platform you are running on.
When you say it chokes your server - do you mean your email server? Your web server? something else?
There's nowhere near enough information here to make a proper diagnosis but it looks like the problems are of your own making - sure, there are lots of people out there who promise to sort all your problems for you if only you buy their latest product/service. But there's a very good chance that this isn't going to solve your current problems.
Can you tell us:
what OS the PHP is running on
how you invoke the code to create the emails
what the mail config in the php.ini file is
what type of MTA are you using? On what OS?
how is youe MTA copnfigured - does it deliver directly or via a smart relay?
which server is getting "choked"?
What anti-spam measures do you have in place for outgoing mail?
Then tell us what you've done to diagnose the fault and why you think its specifically on sending mails.
C.
I'd recommend using the third party mailing service Silverpop, or something like it. We've used them for a few years and have been fairly satisfied. They already have relationships with the major email clients (AOL, Yahoo!, Gmail, etc.) and they do a good job of telling you if the stuff you're sending is likely to be classified as SPAM.
They have a fairly extensive API that uses XML HTTP/HTTPS requests that can tie in to existing systems. You can use it to remotely trigger emails, schedule mailings, customize email contents, set up, manage and query huge lists of recipients, run batch processes, etc.
It isn't a perfect service, but compared to a lot of others out there, they do pretty well. I have had very few complaints about them thus far.
I usually got around this by having a mail "sending" function that dumped the emails into a queue (database table) with a job that ran every couple of minutes, grabbed the next x emails in the queue, sent those out and marked them as succeeded. That's the simple bones of it. You can then add on handling for email failures, returned mail, etc in version 2.
Use Google AppEngine if you are worried about scalability & customization: it provides an email API and you can interface anything to it provided it is through an HTTP interface.
Of course, this is just a suggestion - disregard if this doesn't fit.
Quite possibly not ideal, but if you're looking at large scale transmission there are commercial solutions such as Port 25's PowerMTA that can be set up to effectively transmit the contents of a given folder.
As such, you'd simply use PHP to create the personalised MIME formatted raw data for each outbound email and place these in a temporary directory prior to transmission. (I've written such a system in the past and you'd be surprised at the PHP's ability to grind out the emails, even with quite complex text & HTML emails with images as inline-attachments, etc.) Once you were ready to transmit, you'd then move the files en-masse to the PowerMTA monitored folder and it would take care of the transmission.
Depending on how you look at it the benefit/problem with such a solution is that you'll need to build trust relationships with people such as AOL, MSN/Hotmail, etc. to ensure that your mail server isn't blacklisted due to user's reporting email as SPAM. (That said, this will likely be a factor with any DIY solution.)
Why not keep your PHP system and use a third party SMTP service? Some reliable companies offer the use of e-mailing-only machines at reasonable prices, e.g. Dewahost who I am planning to use.
Also see the question Third Party Email Senders and my answer there.
Check out Campaign Enterprise for a possible in-house solution.
One of my friend uses http://www.tailoredmail.com but i havn't used it personally
I know this is an older question, but I just wanted to suggest SendGrid which is essentially an "Email Server as a Service" allowing you to send emails with cost per email.
I have a php script that does some processing (creates remittance advice PDFs, self-billing invoices, Sage CSV file etc...) and at the end outputs a screen with a form, in which the names and e-mail addresses of the people paid appear. User makes a selection of names by clicking check boxes and then there is a Send button which sends out emails with remittance advices and self-billing invoices attached. This all works nicely, BUT they now decided that when they click the Send button, they would like the e-mails to be sent NOT straight-away, but at 6.00pm.
Is it possible to set the dispatch time of the message in the SMTP header? Can the MS Exchange server be configured so that e-mails from a particular sender will be held until a certain time before they get sent? IT Support dept. claim it used to be possible in the old days of dial-up connections when it was simply cheaper to send stuff at night... but that this functionality was removed. Is this true? I have no idea how hard is the task at hand. It seems very straightforward and I guess it really is a task for the IT support guys to handle. But maybe I am wrong?
If this can not be set up at the Exchange server side, how could I go about achieving the requested functionality? And, no, this is not an exact duplicate of this question. I had a look at that but it didn't seem to answer my questions. Any help greatly appreciated!
Edit
Apache running on MS Windows Server 2003. Database is Oracle 10g. There will be no CRON set up. The email queue table would need to store all attachments too. I would like to avoid doing this at all cost. No way to specify dispatch time in header?
You could save those emails into a database. And then create a cronjob PHP file that executes every few minutes to check if there are emails to be sent in the "queue" database.
Here's a tutorial on something like this.
Windows (non-server editions, at least) has a "Scheduled Tasks" control panel item similar to CRON. Just from looking at it briefly, you can probably write a PHP script which sends your mail "now," but run it using the CLI at whatever time you want to send the mail using Scheduled Tasks.
Actually you can do this. If you are using an exchange 2003 server, you can set up a different SMTP connector. Under the delivery options there is a way to change it from send always any scheudle you pick.
Here is a nice tutorial for setting one up.
link text
One thing you will need to do is when you create the new SMTP connector, set it up with a different port to listen on. That way you can send to that one instead of the standard connector.