I have two bots on my server but since 4 days ago both of them stopped working.
I checked the script on the other server and I'm pretty sure there is nothing wrong with it. also I talked to my hosting provider and seems that there is nothing wrong with it too.
What's going wrong?
Update : "I'm using webhooks"
When sending messages inside a particular chat, avoid sending more than one message per second. We may allow short bursts that go over this limit, but eventually you'll begin receiving 429 errors.
If you're sending bulk notifications to multiple users, the API will not allow more than 30 messages per second or so. Consider spreading out notifications over large intervals of 8—12 hours for best results.
Also note that your bot will not be able to send more than 20 messages per minute to the same group.
Related
I'm creating a site that sends out a daily news email to about 800 users, at a time they can specify. My problem is that my script takes a long time to run and times out, so I'm looking for some advice on how I could be approaching this better.
Current approach:
Users are placed in a 'mailing queue' database table with their ID, receive time, and a sent flag.
I'm then running a CRON script every minute which does the following:
Grab all from mailing queue with a 'receive time' less than or equal to now, that haven't sent
Loop through the users, joining a preferences table to get their chosen categories (up to about 30 per user).
For each category, find the latest 3 articles
Prepare an HTML email with this content using PHPMailer
PHPMailer is using Mailgun SMTP to avoid overloading my SMTP server
Send mail to user, mark as sent in database
My observations so far are:
When testing the script by running in-browser, it runs incredibly slowly for a few minutes then times out (without sending any emails).
When running every minute via CRON, it sends way over the number of emails (about 1400) over the course of 40 minutes, I guess because the script is overlapping itself and the sent flag is not reliably updated.
The majority of users are set to receive their email at the same time, so I'm doing 'worst case scenario' testing on this basis
Questions
Is my script far too heavy, by querying the database and generating the HTML email content for each user on the fly? I'm wondering if it would be better to generate the content ahead of time and store against the user in the mailing queue.
Would a queue manager like Beanstalkd help? I've had a look into it, but am struggling to see how to implement into my routine.
Ultimately I need the emails to be sent reliably to each user at the time they expect.
Any advice much appreciated!
You can do this in PHP, but you probably shouldn't. You're trying to build a mail server when there are much better ways of doing this, which mainly involve using a mail server.
Sending high volumes of email during page loads is not workable – it can be troublesome even for single messages, yet many still try. Approach it like this:
Store your list in a database.
When you want to send, generate a record representing each message to send (essentially a copy of the list).
Have a daemon (a long-running task) or cron job that sends the messages in chunks.
Create messages one at a time and submit them to a local mail server.
Use DKIM signatures.
As each message is sent, mark them as sent in the database, but you need to be very aware of how database transactions and locks work for this to work safely and avoid duplicates – do this right and overlapping processes work just fine.
You can generate messages as fast as you like, and your mail server will deal with queuing, onward delivery, retries, bounces.
Use VERP addressing, and feed bounces into a bounce handler (be warned, writing these is not fun!), and have that prevent sending to bouncing addresses in future.
This approach works well – it is exactly how my own system ([Smartmessages.net](https://info.smartmessages.net/ , which is built in PHP) works, and I can sustain over 200 messages per second using multiple message generators running in parallel (database transactions FTW!).
If you find all this a bit too much (it is very difficult), you're probably better off using a commercial sending service (like my own) or hosting a DIY solution, such as Mailcoach by my good friends at Spatie. Either of these would work well, and your list is pretty small – I'm often handling lists of over 100,000.
we send messages with Amazon SES using SES API. Our send rate is now 90 messages per second. But we are getting throttling exception even when we don't reach this limit but just trying to approach it.
Right now we can steadily send at the rate of 30 messages per second. The question is how to send faster.
Let me dive into some more details and clarify the question.
It might take from 0.3 to 3 seconds for a single API send request to complete. This is why if we would send messages sequentially we would hardly get 1 message per second speed.
Luckily we can send messages in parallel and this is what we are doing.
For each thread we check that it does not send more then allowed number of messages per second. For example if we have 40 threads then we don't allow each thread to send more then 2 messages per second. Yes, this is not optimal.
We register every message send and the time when the API request has completed (when we got the response from API). This allows to get some statistics.
When we restrict the sending limit to be less then allowed limit (like 60 instead of 90) everything works fine.
When we try to send at maximum limit we start getting throttling errors. Like when have speed of 80 requests per second reached we start getting exceptions.
This allows me to put the questions:
Q: How to send messages with highest allowed speed?
Let's start with another question - 'How does SES calculate the number of messages to check send rate?'
Let me guess. When we submit a new request they look at the number of requests submitted during last second from current moment and if this number is less then our limit the request is accepted.
But wait. If we have 40 threads and each thread can not send more then 2 messages per second then we can never reach the limit. But we do get the exceptions.
Research
There is a great blog post on Amazon SES blog about handling the limits. We are trying to adopt this approach but have not succeed yet.
We are using PHP in our application and PHP SES SDK.
I guess this is quite common task but for some reason I'm not lucky to find the complete solution.
Any help or ideas would be greatly appreciated. Thank you.
The key take away is:
A variety of factors can affect your send rate, e.g. message size,
network performance or Amazon SES availability.
Based of what you've said it seems like you're using a bit of fuzzy logic to try and calculate how many messages you're sending. This won't be perfect so if your AWS limit is 90p/s then getting setting your code lower, e/g to 60p/s makes sense (once again this all depends on how accurate your estimates are).
You should consider the other approaches as you've mentioned, such as "Exponential backoff" as described in the link you provided.
Another thing you can consider is taking advantage of a queue, like SQS. This way you can pick tasks off the list as quick as possible, and if you're a little too quick you can always back off and then jump back on the queue as soon as possible.
Hoping someone has some suggestions or work-arounds.
I recently wrote an email parser to work for mail forwarded through a pipe in CPanel and have the details entered in a mySQL table.
The client now wants historical mail importing to this database and has approx 50 Gmail accounts with some having over 20,000 messages in.
I have been able to get up to around 7000 emails out of 25000 on the 'test' account I'm using. To try and work out why I've stripped it down so there's a very simple php imap script running just to dump the a couple of parts of the email in to a cell in a table and was going to worry about parsing it after, but the same thing happens. It hits around the 6000-7000 email mark and stops - no errors or anything, it just stops.
I've just found out that Gmail has a bandwidth limit for IMAP of 750MB per hour and doing the math it looks quite likely I'm hitting that and that's why it's stopping.
First question - anyone know how I can check if that's the reason?
Secondly - assuming it is that can anyone think of a best solution?
I considered doing it in blocks of a few thousand emails but this would mean a lot of manual intervention because of the number of accounts and it would take a LONG time to download them all which I don't have.
The second (and so far best) idea I've had is to use one of Gmail's recommended methods to migrate all the mail to an email address created on the client's Cpanel and use the inbuilt pipe feature to forward the mail through my script in the same way I'm doing with their new mail. (I'm not 100% sure the migration will trigger CPanel to apply the pipe rule, I'd have to test).
Appreciate any input.
thanks in advance.
I don't have an answer to your first question.
But as far as your second question goes you don't necessarily have to re-run your program manually if you split the task up into a couple thousand emails. You could either use sleep to stop the program for an hour after x number of emails have been pulled over, or you could setup a cron job that runs once per hour and you would just have to save a pointer to the last email you processed in a database or temp file so you know where to pick up and start again next cron job run.
I'm using a homemade PHP script with PHPmailer for sending email, currently I just do one select statement in MYSQL and then run a 100-150 email send request from PHPmailer via a for loop.
my problem is that I have to port this script to a 7000 user mailing list, and I know it's not going to hold up since sending to 150 takes about 8 secs of execution time - also sending 7000 emails might mark me as a spammer or I won't be able to handle all responces.
I thought about splitting the 7000 into groups of like 200-500 and sending an email to each group every day until sending everyone - to do that i'll have to write a PHP script that handles the limits and remembers when & who on the list hasn't received that email yet and is due to receive it on the next batch.
What are my options, what are your thoughts?
Just my 2 cents.
1) I agree with Dave re: make it take longer than it needs. Most black/grey listing is the the recipient mail server deciding there is an odd amount of traffic compared to usual. Yahoo is terrible for this and given the sheer volume of domains they deal with, it can get annoying.
2) Using multiple IP addresses doesn't hurt. Extra IPs are usually free, and if you are using exim, you can easily configure it to use each IP based on e.g. EHLO, or send to domain etc etc. This means that there is less volume coming from any one IP.
3) Keep a track of bounce backs. Some providers will keep a track of the number of attempts to deliver mail to a non-existent address, the more you try, the more suspicious it looks. Easiest way to keep a track of them in my experience is to use VERPs and get PHP to process the incoming data.
4) Set up reverse DNS. Some hosts will just ignore the mail if you have not got it set up for the sender. I believe AOL is an example.
Just a few pointers from my experience.
Your first approach is a pretty good idea. As long as you're ok with it taking about 3 minutes to send all the emails (which it should be, if you're only sending a weekly newsletter or the like). To reduce the chance of getting marked spam, lower the threshold or make it take longer. Beyond that, it might be a good idea to move into a hosted mass mailing service.
The act of sending 7000 [or more] emails is not going to get you marked as a spammer ~assuming~ that the mail you are sending is:
not spam
contains complete & accurate header information
The only problem you might get is clogging up your mail queue, but you r volume is still pretty low. I would say for scalability - throttle your delivery over about an hour or so.
and consider ceejayoz's suggestion "MailChimp" it's free up to 5000 emails/month and has an excellent API that you can literally integrate into your app in a couple of hours.
-sean
Okay, so here's my problem:
I have a list of members on a website, and periodically one of the admins my site (who are not very web or tech savvy) will send a newsletter to the memberlist.
My current memberlist is well over 800 individuals long.
So, I wrote an email script that sends the email to the full memberlist, with the members listed in the Bcc header.
However, I've discovered that my host server has a limit of 300 emails per hour, which I apparently exceed even though the members are listed in the Bcc field. (I wasn't previously aware that the behaviour of Bcc was to send separate emails for each name on the list...)
After some thought, I've come to the conclusion that my only solution is to have my script send only the email to only the first 300 emails, wait an hour, and send a second email to the next three hundred, wait another hour, and so on until I've sent the email to the whole member list.
Looking around on the internet, I've seen some other solutions people have come up with for delaying emails in PHP. Sleep() is obviously not an option, because I can't just leave the script open and running for 3 or four hours. I've seen some people suggest cron jobs, but I'm not sure how feasible it would be to create three new cron jobs every time I send an email, use them once, and then delete them afterward.
The final (and what I think is the smartest) solution I've seen, is to have a table in my database to temporarily store the emails to be delayed and sent later, and then create a cron job that checks this sql table every hour or so, compares the timestamp of the row to the current timestamp, and then sends the email if an hour has passed.
So I'm asking you all which method you would recommend. Is there an easier solution that I've completely looked over (aside from getting a different hosting plan. ha!), or is there a cleaner way to do it than the database / cron job approach?
tl;dr: I have >800 emails to send in an hour on a server that limits me to 300/hr. Using PHP, find a way to get around this problem in a way that the person sending the email needs only to click "send."
You could send this into a gearman queue and then have a gearman worker with the appropriate sleep calls. See http://gearman.org/ and http://php.net/gearman
Sounds like you need to setup a batching function that pulls from a pool of messages to be sent and processes X everytime its run during cron. Then you would have a table that tracks messages that were sent and to who, so you can keep track of who has received emails.
I would recommend that you create a queue, and process X number of items from the queue each time you need to send email. The sender of the messages just places the email in the queue and your processing code picks up the item sending the maximum number of items in that period. Occasionally you will have failures and use of the queue will allow simple recovery. You only remove items from the queue when they are processed.
You can use a simple database table to act as the queue but you may prefer to use a specialist queuing solution.
Another recommendation would be to look into external email services like Strongmail. These will help you send more email per hour.