in php5 web app, getting email address from user and sticking it into database, but want to send the email without user waiting for the email to complete. using gmail, sometimes takes a few seconds.
thinking about fopen( "http://self.com/mailer.php", "r" ), where mailer.php would do the actual mailing. will this work?
There are two approaches that have worked well for me in the past.
Database Email Queue
Create an "email_queue" table in your database or whatever else you are using for persistent storage. Each entry contains everything you will need in order to send an email (e.g. Subject, recipient, sender, body, etc...) and a "sent" flag field. To send an email you add an entry to this table using a helper class.
Then you create a cli php script that reads from the queue table filtering for entries that have the sent flag set to 0, send the email and finally set the sent flag to 1. This script needs to run in a cron job, but since you probably want to run it more often than one minute, you can use something like Frequent-cron. The scheduled task should only run in one web server in case you have many (and assuming your persistent storage is shared).
Local email queue
I've also had some success configuring a local postfix server on each web server to deliver mail using an external smtp service. The goal here is to get postfix to accept the email as fast as possible (because it is local) and then deliver it via the external SMTP server in a different process. Postfix will act as an intermediary queue.
I personally like the first solution because it gives your app more information about email delivery for statistical analysis' record keeping, etc..
Hope that helps.
Related
Background of the project :
I am working on a project which goal is to keep track of all emails received in a mailbox.
The app connects to the mailbox and store every emails as files.
The user must have the possibility to use his mailbox regardless of the app behavior.
I am using the pop protocol at the moment i am writing this.
How it should work :
Let's say there are 10 unread mails in the mailbox, not stored by the app yet.
Case 1 : The user connects first : he reads, deletes others, tags them as spam or whatever : the app must still store each of the 10emails as files.
Case 2 : The app connects first, store the 10 emails and do something to never store them again, but the when the user will connect to his mailbox, he should still see the 10 emails as unread, and do whatever he wants with them.
How it works at the moment.. :
The function does retrieve every unread emails, and stores them as files (also downloads any attachments).
The first case works very well, i'm using outlook with my mailbox, and whatever i am doing with my emails, the app will still see the emails as unread, find them, and finally do the job.
The second case, however, seems to not work as intended : after storing the emails, i am using the imap_delete to mark these emails for deletion, and then, when everything is finished, i use imap_close with the CL_EXPUNGE flag to delete them, so that the app will not store them again next time...
The problem is that, this also deletes the emails from the server...meaning that in some cases, although the emails will have been stored, the user will never see them in his mailbox...
Question Time
Is there a way, in PHP, to delete emails only for my app, so it does not store the same emails several times, instead of deleting them from the server ?
I know outlook makes it possible, because if not, it would not work in the first case either...
Is there any solution to this ?
Is there a flag or a function that would do it correctly in php with the pop protocol ? Or is it possible using imap protocol and imap functions in php ?
Your problem is the IMAP protocol. While using IMAP you're always working with the mails stored at the server, so if you mark them as read, you're marking them read at the server so any further connections (using IMAP) won't see the unread messages, also if you delete them you're deleting it from server so no further visibility (either with POP - if never connected before, of course - or IMAP).
You should switch your connection protocol in order to use POP3, doing it so you download a copy of the mail to your machine (actually to the machine executing the PHP script) and work with a local copy.
See this question for more details.
I assume I will need to point MX records at my server (LAMP), -- what processes the incoming e-mail message?
Are there any existing PHP libraries to do this?
You don't want to use PHP as a mail server. You've got two options:
Set up a classic email server (postfix, sendmail, exim, etc) that delivers new messages to a local mailbox. Use IMAP or POP to access that mailbox from PHP, and pull messages out of it. Alternatively, this same method may be used with (virtually) any remote mail service as well, thus relieving you of the duty of administering a mail server. (Which you'll likely find to be not worth it for the sake of one mailbox.) This method would usually be run via cron every few minutes, so you're not going to get "instant" activation if that's a requirement.
Set up a classic email server (postfix, sendmail, exim, etc) and use procmail or similar to intercept messages at delivery time, and pipe them to a PHP script. This method will fire the script the instant the email arrives, so you'll have no lag time like in #1. However, it's more difficult to configure (especially if you haven't maintained a mail server before) and won't work with most external hosted email services.
Use a pipe alias to receive the emails.
I would recommend you to do processing in Perl (python is also ok, but Perl has very similar syntax to PHP), which is much more suitable for the task. You can also find a lot of libraries through CPAN there.
http://search.cpan.org/~rjbs/Email-Simple-2.100/lib/Email/Simple.pm
I'm thinking about how to handle sending large amounts of email from my web applications, and whether there are any best practices for doing so. StackOverflow is already labeling this as 'subjective', which it may be to an extent, but I need to know the most successful way to implement this system, or whether any standardized approach exists.
In my webapp, there are users who are heads of groups of 1 to 10,000 users. This user must be able to email send a message to all of these users through my system. Therefore, my system is responsible for sending up to 10,000 emails to individual users for each group head.
As far as I can tell, there is no rate limit in GMail for sending messages to individuals (although there is a 500 recipient max).
Right now, my current setup is:
When a message is sent through the system, it enters an email queue.
A cron script grabs messages from the queue every few minutes, and sends out those emails.
All email is taking place through GMail's SMTP server.
The actual application doing the mailing is PHPMailer.
This setup, as the user base grows, will probably not suffice. The questions I have are:
Should I be using a local SMTP server instead?
Should I use a mail binary on the local machine instead? I this case, I could probably skip the queue altogether?
Is there an accepted way to do this?
Thanks!
Google App Engine
I would write this in Google app engine (python) because:
It scales well.
It has a good email api.
It has a taskqueue with a good api to access it.
Because python is a real nice language.
It is (relatively) cheap.
PHP
If I would implement it in PHP I would
Find yourself a good SMTP server which allows you to sent this volume of mails because Gmail will not allow you to sent this kind of volume. I am for sure that this will cost you some money.
Find yourself a decent PHP email library to sent messages with like for example PHPMailer like you said.
Use a message queue like for example beanstalkd to put email messages in queue and sent email asynchronously. First because with this the user will have snappier page load. Second with a message queue like beanstalkd you can regulate speed of sending better which will prevent from overloading your pc with work. You will need to have ssh access to the server to compile(install) beanstalkd. You can find beanstalkd at beanstalkd
You would also need ssh access to run a PHP script in the background which will process the message queue. You can find a beanstalkd-client at php beanstalkd-client
from php/apache/webpage
This is the page from which you will sent messages out to user. From this page you will sent message to beanstalkd by coding something in the lines of this:
// register Pheanstalk class loader
require_once('pheanstalk_init.php');
$pheanstalk = new Pheanstalk('127.0.0.1');
$message = ""; // This would contain your message
$pheanstalk->put(json_encode($message);
You have to put messages in message queue using put command
From long running PHP script in background:
The code would look something like this:
// register Pheanstalk class loader
require_once('pheanstalk_init.php');
$pheanstalk = new Pheanstalk('127.0.0.1');
while(true) {
$job = $pheanstalk->reserve();
$email = json_decode($job->getData());
// Sent email using PHP mailer.
$pheanstalk->delete($job);
}
Like I am saying it is possible with both PHP and Google app engine but I would go for app engine because it is easier to implement.
With an email count as "high" as 10.000 a day, I wouldn't rely on GMail (or any other) SMTP. Not that they can't handle it, obviously they can handle A LOT more. But they possibly don't want to.
Having a local SMTP server is IMO the way to go :
It's pretty easy to set up (just DON'T let people use it without a strong authent scheme)
Most modern MTA handle sending queues very well
You won't have to deal with GMail (or others) deciding to block your account someday for quota reasons
Gmail and Google Apps limits you to around 500 emails a day. I'm not sure how that combines with the 500 recipient max, but if you want to send 10 000 emails you'll probably want to find another mail server. I personally use a local server or the ISP or datacenter's SMTP.
If you are sending that many emails, I would recommend using the queue so the user's isn't sitting there waiting for the email to be sent.
Be very careful that your domain doesn't get blacklisted as a spam domain. If it does, you can expect most of your emails to be blocked, support, sales, etc. Which could in turn be very expensive.
You may instead want to use a service like AWeber. Not only are they setup to handle these amounts of emails, but they can probably give you more metrics than you can implement on your own.
I'm not sure if it's publishe anywhere, but from experience I can tell you that Gmail will put a fifteen minute or so freeze on your account if you start sending hundreds of messages at a time. This happened to me last week. I think you should host your own SMTP server. Using the mail() function often will put your mail in someone's spam folder.
Just install Postfix on the local machine, or a machine on the same LAN for maximum access speeds. Make sure it is well secured from the outside, and quickly accessible from the inside.
Then code your PHP script to directly inject the emails to the Postfix queue. That shall dramatically increase the processing speed of mail delivery.
Is there a way to configure a server or a script to execute a php script when an email is received?
In theory this could be extended to other protocols, such as XMPP or SMS, etc.
What I have in mind is a user could send a message to checking-in#example.org and this would trigger a script which would then do whatever needed to be done, either irrelevant to the message (an automated message that gets sent whenever some other even occurs, like a server having issues) or related to the message (like it could store the subject in a database that other users could view as an RSS feed).
I know that most list-serve software have a means of sending commands (like unsubscribe), but I'm not sure how complicated the process is and if it is feasible to have something like this on a server-script level.
Would this need to occur at the IMAP/SMTP level, or could it be done closer to the script or HTTP server?
Just to give some context, one of the things I'm considering is a message-based clock in server for one of my work sites. Users could IM/email/text that they are at their scheduled location and this could trigger a script that would update a DB, rather then send the managers a direct message they need to log. This would just be one option (a web-based method is already in the works), but we want to make it as platform/protocol independent as possible. But if we can achieve that, we can look at implementing it for many other day-to-day needs.
Another way of asking might be: is there a way to have "discovery" of users from a server-script app or does something need to be doing a constant check to relevant sources to see such changes?
If you control your own machine you can configure it to pipe the e-mail through a filter. Simplest would be to setup a new account and setup a .forward or alias
The following examples are for Sendmail, but (all?) Unix e-mail programs offer a similar service.
Setting up an alias (option 1)
Look in the directory /etc on your server for your alias file. Add the line:
script: "|/path/toyourscript/pipe.php"
Using a .forward file (option 2)
Create a .forward file in your main home directory.
"|/path/toyourscript/pipe.php"
or:
myemail#example.com,"|/path/toyourscript/pipe.php"
If you are on shared hosting then most hosting providers also provide the possiblity to "pipe" e-mails received to a particular account through a script instead of storing them in a mailbox. Check the CPanel setup.
Use a cron job to check for emails through a pop3 interface. If an "interesting" mail is found, run your php script.
Note that the cron script can be in PHP too.
As jldupont said, it is easy to do ith php itself, simple reading the smtp continuosly with a cronjob.
Otherwise, you could use a daemon, in python for example.
Both ways allow you to do an important thing: check if the sender of the email is a your user (you have the users in a db, right?), or a spambot (nomatter what kind of anti-spam you use, soon or later, checking-in#example.org will be full of spam)
I have a php script that does some processing (creates remittance advice PDFs, self-billing invoices, Sage CSV file etc...) and at the end outputs a screen with a form, in which the names and e-mail addresses of the people paid appear. User makes a selection of names by clicking check boxes and then there is a Send button which sends out emails with remittance advices and self-billing invoices attached. This all works nicely, BUT they now decided that when they click the Send button, they would like the e-mails to be sent NOT straight-away, but at 6.00pm.
Is it possible to set the dispatch time of the message in the SMTP header? Can the MS Exchange server be configured so that e-mails from a particular sender will be held until a certain time before they get sent? IT Support dept. claim it used to be possible in the old days of dial-up connections when it was simply cheaper to send stuff at night... but that this functionality was removed. Is this true? I have no idea how hard is the task at hand. It seems very straightforward and I guess it really is a task for the IT support guys to handle. But maybe I am wrong?
If this can not be set up at the Exchange server side, how could I go about achieving the requested functionality? And, no, this is not an exact duplicate of this question. I had a look at that but it didn't seem to answer my questions. Any help greatly appreciated!
Edit
Apache running on MS Windows Server 2003. Database is Oracle 10g. There will be no CRON set up. The email queue table would need to store all attachments too. I would like to avoid doing this at all cost. No way to specify dispatch time in header?
You could save those emails into a database. And then create a cronjob PHP file that executes every few minutes to check if there are emails to be sent in the "queue" database.
Here's a tutorial on something like this.
Windows (non-server editions, at least) has a "Scheduled Tasks" control panel item similar to CRON. Just from looking at it briefly, you can probably write a PHP script which sends your mail "now," but run it using the CLI at whatever time you want to send the mail using Scheduled Tasks.
Actually you can do this. If you are using an exchange 2003 server, you can set up a different SMTP connector. Under the delivery options there is a way to change it from send always any scheudle you pick.
Here is a nice tutorial for setting one up.
link text
One thing you will need to do is when you create the new SMTP connector, set it up with a different port to listen on. That way you can send to that one instead of the standard connector.