I'm in the process of writing a small php-cli script that will loop over my personal inbox and then send me an SMS via a gateway.
The question I have is:
As will have the script launch via cron every 10 minutes, if there is an email sitting in my inbox that is not read before the next script launch then I will receive 2 sms.
Does any one (pseudocode will do) have any idea what the best practice would be in php5 to ensure only 1 SMS is sent?
What I am currently learning towards is storing the message ID in a sqlite DB and flagging a field whether an SMS has been sent or not - but wondering if there is an easier way?
You shouldn't need to store anything. If I understand you correctly, you simply want your program to send you a message if a new, unread email has arrived within the last 10 minutes. In that case, could you not do the following?
foreach($inbox[] as $message){
if(($message.isUnread) && ($message.receiveTime > NOW() - 10 minutes)){
$sendMessage = TRUE;
}
}
if($sendMessage){
sendMessage();
}
I don't know how you're able to access the data, so the above is unlikely to be usable as-is. Assuming you can filter your inbox's contents by read status and date arrived, however, shouldn't that work in principle?
What I am currently learning towards is storing the message ID in a sqlite DB and flagging a field whether an SMS has been sent or not
This is one of the most simple and straightforward ways to do this. Regardless of storage method, you'll have to keep the list of seen-but-unread message IDs somewhere. May as well be in a simple, self-contained SQLite file. I suppose you could just keep them in a list in a flat file. That is by definition more simple, but is it really easier?
I don't see you getting away without persisting something somewhere.
That something should be of ever-incrementing nature - message ID maybe, timestamp for sure. As long as you can look at it and determine if new messages have arrived since.
If all you wish to communicate with the SMS is that there are unread messages, I don't see the need to persist more then a single ID or a timestamp.
After a few hours sleep (up since 4am watching Australia vs Germany, World Cup) I've come up with an even better method, a bit more complex then just using a sqlite DB but has some extra benefits.
I'm using imap_mail to send the SMS to the gateway, and storing the message_id that the SMS is notifying to as a custom "X-Relationship:" header in the outgoing email and then copying the message to my Sent Items.
Then the notifier simply scans my Sent Items for the presence of the X-Relationship header within my Sent Items.
This what I have a record (Sent Item) of the email sent to the SMS Gateway - seems to be working well.
Related
I'm creating a site that sends out a daily news email to about 800 users, at a time they can specify. My problem is that my script takes a long time to run and times out, so I'm looking for some advice on how I could be approaching this better.
Current approach:
Users are placed in a 'mailing queue' database table with their ID, receive time, and a sent flag.
I'm then running a CRON script every minute which does the following:
Grab all from mailing queue with a 'receive time' less than or equal to now, that haven't sent
Loop through the users, joining a preferences table to get their chosen categories (up to about 30 per user).
For each category, find the latest 3 articles
Prepare an HTML email with this content using PHPMailer
PHPMailer is using Mailgun SMTP to avoid overloading my SMTP server
Send mail to user, mark as sent in database
My observations so far are:
When testing the script by running in-browser, it runs incredibly slowly for a few minutes then times out (without sending any emails).
When running every minute via CRON, it sends way over the number of emails (about 1400) over the course of 40 minutes, I guess because the script is overlapping itself and the sent flag is not reliably updated.
The majority of users are set to receive their email at the same time, so I'm doing 'worst case scenario' testing on this basis
Questions
Is my script far too heavy, by querying the database and generating the HTML email content for each user on the fly? I'm wondering if it would be better to generate the content ahead of time and store against the user in the mailing queue.
Would a queue manager like Beanstalkd help? I've had a look into it, but am struggling to see how to implement into my routine.
Ultimately I need the emails to be sent reliably to each user at the time they expect.
Any advice much appreciated!
You can do this in PHP, but you probably shouldn't. You're trying to build a mail server when there are much better ways of doing this, which mainly involve using a mail server.
Sending high volumes of email during page loads is not workable – it can be troublesome even for single messages, yet many still try. Approach it like this:
Store your list in a database.
When you want to send, generate a record representing each message to send (essentially a copy of the list).
Have a daemon (a long-running task) or cron job that sends the messages in chunks.
Create messages one at a time and submit them to a local mail server.
Use DKIM signatures.
As each message is sent, mark them as sent in the database, but you need to be very aware of how database transactions and locks work for this to work safely and avoid duplicates – do this right and overlapping processes work just fine.
You can generate messages as fast as you like, and your mail server will deal with queuing, onward delivery, retries, bounces.
Use VERP addressing, and feed bounces into a bounce handler (be warned, writing these is not fun!), and have that prevent sending to bouncing addresses in future.
This approach works well – it is exactly how my own system ([Smartmessages.net](https://info.smartmessages.net/ , which is built in PHP) works, and I can sustain over 200 messages per second using multiple message generators running in parallel (database transactions FTW!).
If you find all this a bit too much (it is very difficult), you're probably better off using a commercial sending service (like my own) or hosting a DIY solution, such as Mailcoach by my good friends at Spatie. Either of these would work well, and your list is pretty small – I'm often handling lists of over 100,000.
Hoping someone has some suggestions or work-arounds.
I recently wrote an email parser to work for mail forwarded through a pipe in CPanel and have the details entered in a mySQL table.
The client now wants historical mail importing to this database and has approx 50 Gmail accounts with some having over 20,000 messages in.
I have been able to get up to around 7000 emails out of 25000 on the 'test' account I'm using. To try and work out why I've stripped it down so there's a very simple php imap script running just to dump the a couple of parts of the email in to a cell in a table and was going to worry about parsing it after, but the same thing happens. It hits around the 6000-7000 email mark and stops - no errors or anything, it just stops.
I've just found out that Gmail has a bandwidth limit for IMAP of 750MB per hour and doing the math it looks quite likely I'm hitting that and that's why it's stopping.
First question - anyone know how I can check if that's the reason?
Secondly - assuming it is that can anyone think of a best solution?
I considered doing it in blocks of a few thousand emails but this would mean a lot of manual intervention because of the number of accounts and it would take a LONG time to download them all which I don't have.
The second (and so far best) idea I've had is to use one of Gmail's recommended methods to migrate all the mail to an email address created on the client's Cpanel and use the inbuilt pipe feature to forward the mail through my script in the same way I'm doing with their new mail. (I'm not 100% sure the migration will trigger CPanel to apply the pipe rule, I'd have to test).
Appreciate any input.
thanks in advance.
I don't have an answer to your first question.
But as far as your second question goes you don't necessarily have to re-run your program manually if you split the task up into a couple thousand emails. You could either use sleep to stop the program for an hour after x number of emails have been pulled over, or you could setup a cron job that runs once per hour and you would just have to save a pointer to the last email you processed in a database or temp file so you know where to pick up and start again next cron job run.
I started managing a website with a member list of over 50,000 registered members.
I am having this problem:
Each time i try sending the emails, the page runs for several minutes (more than 20 minutes) and eventually ends with up 'Internal Server Error'.
Now, i do not know how many mails that were sent or if any was sent at all.
How do i send the emails:
I select all emails from the database, run a loop and send one after the other. I chose this option because:
i do not want to add several emails in 'To' field, so people's privacy will be maintained.
Adding the emails in Bcc will make it appear as a spam.
Please how can i handle this.
Thank You
All you need to do is break the job into manageable chunks, instead of reading in all 50,000 emails and trying to send them all.
Read in only X emails from the database at a time.
Send that batch of X emails individually (If X = 10, you will send 10 separate emails).
Mark each email as "sent" as soon as each email is sent, otherwise, if there is an error, you may want to increment "send attempts" on that particular email.
After the X emails have been sent (or attempted), update your statistics to indicate that X emails have been sent.
Go back to 1.
I suggest starting with X = 1
I did this by making a script that calls itself with the id of the user it last processed.
E.g. sendmail.php will then call sendmail.php?id=1 which will then call sendmail.php?id=2 and so on.
That will fix your timeout issue.
Within the script you can echo out the results of the mail command. If its false, then you could echo out an error.
I also added a field in my user table for mail send date and time, so i could have a confirmation of where the script got to incase the browser died.
Your sql for getting the next user to send email to would then be something along the lines of
Select useremail from users where sendmaildatetime is not null limit 1
Assuming your default for the sendmaildatetime column is null.
Sorry if im not detailed enough, im writing this from my iphone and its a PITA :)
#hamlin11 response is right, if you want to manage such a big email sending you MUST do it in small chunks and manage this task as an asynchronous task.
To Manage asynchronous tasks you'll have a lot of way, a subprocess check on all your requests that some async jobs are waiting, a separate cron job, a call on a cron.php script from a crontab, even psynnott answer, with an external script relaunching himself at the end.
But you could also use the right tool for the right task, if you have some control on the system under your website. Send one simple email to a mailing list manager that would do the job for you. This would only imply you create the right user list in the mailing list manager. External mailing list manager are for example mailman or sympa. These tools contains robots that you can talk with to feed mailing list recipients.#psynnott answer can be seen as an external PHP script performing very simple mailing list manager tasks. If you want to alter content of the email depending on some user parameters you'll quite certainly have to write your own separate process managing the tasks.
But you could also search for web services handling this job for you. Spam management is a hard job and managing an official big mailer is an hard job as well, but this is not free, of course.
We send a lot of emails to a lot of our users (ranges in 20k+ per day). One of the major problems we face are invalid or dead emails - sometimes our users delete their accounts, change their email address without updating their profile, or our email database builders simply caught emails that are invalid or no longer active. These unresolved returned status messages keep not only piling up on our webmaster account, but also waste precious server resources and flag us as spam more often because of the repeated attempts.
Now, while our mail server is set up to keep trying to send an email to an address that returns "temporarily unavailable", I would like to be able to receive status messages into PHP immediately after sending. For example, when my Sender class sends an email, I would like to know the status message that was returned - whether the email is no longer active, or the server does not exist, or the email was simply moved to another address.
Naturally, I want to be able to receive a status message of a delayed email as well. So if an email does not get sent because the recipient email address is temporarily unavailable, I would like to get the "temporarily unavailable" message back into Php, but I would also like the real one to get passed back once the sending succeeds (for example, if the email does go through 2 days later).
Is there a library that would help me achieve this? What are the most common approaches to this problem, if any?
Like most questions about PHP and mail this is mainly about the MTA.
Bulk emailing is a science in its own right (OK, so more like a black art) and at these kinds of volumes you need to up your game a lot if you want reasonable rates of deliverability.
But back to the question.
An awful lot of this is about how you configure your mail server. AFAIK, most MTAs will only send back a failure message when the message is removed from the queue (e.g. after the last attempt at delivery). Which gives 2 options for tracking each attempt:
1) parse the log files
2) set the number of attempts to 1 (and optionally handle re-queueing yourself).
Given that a message can fail to be delivered after it has successfully departed from your server, it makes a lot of sense to use delivery status notifications (i.e. bounce emails) to track the progress of messages - so using option 2 avoids having to build different code for handling different scenarios.
Without knowing what OS this is running on, nor which MTA, it's impossible to give more specific recommendations.
symcbean's answer gives many theorical inputs and several means to handle your case.
Besides, maybe you could have a look on how other libs or builtin functions work. For instance, you can have a look at :
PHPList
Extended PHP Mailer
i used PHPList some times ago but it was already a reliable solution. I don't know the PHP Mailer class but i may be worth a try (or a least a look on how they deal with similar issues).
Regards,
Max
I am currently running a PHP script that queries the database and then sends text messages (via carrier's email-to-sms gateway).
In other words, I have the phone numbers stored in the DB as:
1234567890#txt.att.net
1236547890#vtext.com // etc...
My method for sending these is using the Swift Mailer PHP Library. I am sending the message using "SMTP", and using the SMTP account from my Host Provider to send it out.
Typically, using this SMTP account, I will send no more than 30-40 text messages in 1 minute.
Here's my question:
I have noticed that when I send these text messages, many times it appears to work fine, however, lately it appears that some users receive the text messages maybe an hour (or several) later.
I understand that once it leaves my server, it is up to the carrier to decide when to send it. However, could it possible that something in my script causes the issue?
For example, I have noticed that in some parts of the DB, a user's number will be displayed as:
1234567890 //notice, no '#vtext.com' or whatever
Could issues like this cause the delay of other emails/texts?
Finally, does anyone know of any script or way to build a script that can query a cell phone number and determine who the carrier is?
Many thanks on this!
This question can only be answered if you provide us the code, what you have done. I can give you a tip. Try to send the message with the previous timestamp so that your carrier server processes them first.