I'm in the midst of coding a throttling code in php for outgoing emails.
I want to know what algo is good for throttling?
Can any help to comment on the below design?
Specs:
The emails are stored in MySQL.
PHP will retrieve the data and use PHPmail to send.
The plan so far:
Grab the email column from the database.
Take out the domain, IDs of the records will be paired.
Check the domain (mx & a records).
Update the valid domains in a table.
Sort the domains in stacks of uniques.
Pop each stack once, using the ID to search for the record and send the mail.
Goto next stack, repeat from first stack when done.
Thanks in advance for any help!
Jase
In the name of all the gods why?
This is really bad architecture - PHP is a very flexible tool which you could potentially use to implement such a system - but its not the right tool. You should be using PHP to implement the logic of your application - dealing with the peculiarities of the underlying technologies and varying implementations is best delegated to the tools which are specifically designed for the process - in this case the MTA.
This is particularly true when there are well-written off-the-shelf solutions to the problem e.g. milter-limit
And before anyone points out that not everyone has access to reconfigure their MTA - in most cases these people won't have access to run a daemon in PHP
Swift Mailer is a great library to use for sending email from your PHP script. It comes with a Throttler plugin that should do what you want.
Related
I am working on a business marketing website. The criteria is that we have to send around 80,000 (say) newsletters per day to different email addresses. But my hosting company allowed me to send only 10,000 (say) emails from one IP per day. So to solve this issue my client provided me with 8 different IPs. I am having following questions:
I don't know how to send each and every newsletter from different IPs, means first from first IP, second from second IP and so on?
As I'm a PHP developer and not an expert at server end, can I accomplish this through my PHP script only or do I have to write a shell script to run at my server?
Is there any algorithm to accomplish such type of problem?
Had similiar problem couple years ago. I resosolved it in simple way. Create table in SQL, or create a class and serialize it. Class or SQL have to contain mails addresses (if You have some kind of volatile content then also mail content), do a class to send mail in certain order, and do object of this class on each ip. That depends what kind of config You have on server. I had a easy way cause all of IPs had its own folder with content, so I just put there code to do object, and just redirected couple of times the website, since it had to be done by web. If You have can have it by CLI there is pretty chance that You could do it by include or similar.
This way is pretty lame, I know it, but didn't want to do something more sophisticated at that time. Later on I wrote a class to manage mail connections via SMTP, so I could chose with mail from witch account would be send. That is better way, but not all servers could support it.
Another way is to do a cron job and do baskets of mail to send portions over the time. (this way was most common on servers that I was repairing).
Another way is to do a bouncing effect on servers with IP. Probably You could do also some shell scripting and invoke it via php script.
Well, I think that there are better ways, but it really depends on server config.
PS. Forgotten to mention, that code can be invoked by AJAX.
I have a hobby wesbite written in PHP and I like to know if there is a problem with it (database errors, an update broke something, etc.) I have a simple notification system which sends me an email if there is a problem and that would be enough for me. Unfortunately, the mail sending feature of the hosting provider is not very reliable. Usually it works, but there are periods when it simply swallows the mails and doesn't send anything.
Is there some other reliable method for notification of the maintainer in case of an error? It's a hobby site, so I'm looking for something simple. Not an industrial strength solution, but something more reliable than email. How do you monitor your hobby sites?
I tagged the question with PHP, because the site is written in it, but I'm also interested in generic suggestions, not just in concrete PHP solutions.
EDIT: the question is about the mechanism of active notification. I want to be notified when something happens. If PHP email is not reliable then what are the other possibilites of notification?
EDIT2: two examples to illustrate what kind of solutions I'm thinking of:
Store the errors and provide a page listing the latest errors (maybe password protected) which would be polled from my computer which could pop up some window if there is an error. It can work, but it works only if I'm at my home computer.
Use google calendar api to insert an event into it when an error occurs, and google calendar will send me an email reliably. It may work, though it's cumbersome.
some other idea?
Are you looking only for email based alerting systems? If not, you should try Notifo. You can use their API to push notifications and it'll be instantly sent to your phone.
PHP has an error_log function for returning errors in various ways, either via email to an admin, to the servers log file or to an external file. I assume that you could merely substitute this functionality for your mailto when you find an error:
http://php.net/manual/en/function.error-log.php
I've run into the issues you've mentioned with my hobby project as well. When I started I was using GoDaddy who's mail relay was pretty unreliable for delivering mail in a timely fashion.
Two things I'd suggest:
For sending email messages with higher reliablity, check out Postmark. Its a paid solution, but the rates are pretty reasonable and it comes with PHP classes you can hook your code up to fairly easily.
For custom error handling, check out PHP's set_error_handler(). Its a good way to have custom code execute on error conditions on your site. From the documentation:
set_error_handler — Sets a user-defined error handler function.
This function can be used for defining your own way of handling errors during runtime, for example in applications in which you need to do cleanup of data/files when a critical error happens, or when you need to trigger an error under certain conditions (using trigger_error()).
Maybe give Airbrake (formerly Hoptoad) a try. This is a commercial service, but they have a basic free plan (tiny little link at the bottom of the pricing page), and the tool looks pretty cool. It's focused on Ruby on Rails but according to their site has plugins for various other frameworks and languages, inlcuding PHP.
http://airbrakeapp.com/pages/home
We have a system set-up that polls specific pages on our important websites every now and then and checks for certain strings. Would something like that be viable to you?
How to make php client to client, like the chat way ? One client connects and sends something to the other client and only he recieve not all clients.
Your Question?
If I understand correctly you want one-to-one(private) messaging.
Socket Programming using PHP
You need to learn Socket programming with PHP. You could start by studying this tutorial. This has scaling problems written all over it, because PHP does not have non blocking IO, proper thread model. I advise you to just use it for fun little projects.
Non blocking IO using PHP
You could try and use PHP-MIO. I have not yet tried this, but I guess it might scale. But then again from Apache(PHP) side you will have the same problems. But when using this from both sides it could work...
long-polling(blocking IO) using PHP
P.S: got bored so I have not completely tested this ;)
Download
Below I present two solutions(prototypes) which do NOT scale. One solution uses Redis pubsub. For this you need to install(compile) redis. For this you a POSIX OS is desired, although some people have ported it to Windows. You can also use the free redistogo.com instance. The Redis solution is the prefered solution. I have put everything in an archive which you can download from here.
I also give a solution which uses named pipes. This solution does not require you to use Redis, but instead this approach needs access to the file system. I believe that this approach should also work on Windows(You have to change the filename to windows-style). I would like for somebody to try this out :). I can not test this anymore, because I have completely switched to POSIX OS(Ubuntu) a long time ago.
Requirements
At least PHP 5.3 and preferable a POSIX OS, redis.
How to use
To use both solutions you need to open two browsers(Browser A/B). I assume you are using localhost for development and that you can access files from http://localhost/6646733.
point browser A to http://localhost/6646733/redis?me=somebodyelse&to=alfred you should replace redis with pipe when trying out named pipes.
Point browser B to http://localhost/6646733/redis?me=alfred&to=somebodyelse
In browser A type a message into the textarea, which will be sent to browser B.
In browser B read the message just sent from browser A
Solutions not using PHP
The solutions below scale.
Pusher(Hosted)
With for example the hosted solution Pusher you can do chat/messaging without the scaling nightmare. Pusher even is generous to provide a free plan. But be aware that the cheap plans do NOT offer SSL so the messages can be intercepted. You should never sent private information over the wire, when not using SSL. Users/developers have provided a nice little library to use Pusher from PHP. The problem with this approach is that you are not in control, but pusher is, but then again you don't have to worry about any details.
Socket.io(open-source)
I really like socket.io, but there are off course a lot of other solutions like for example tornado. You could use Redis to efficiently communicate between PHP and the other solution(socket.io).
I don't fully understand what you are trying to do, but you can use some kind of database and have it store messages that is sent to each user, and then have your client page refresh the chat part with a AJAX kind of query to update the chat. It will then behave simular to the new Facebook chat, where all messages are stored even the ones sent in normal chat and mail. So clients can mail and chat each other at all times, when they are online it will show in their chat and when thy are offline it will show up in their inbox. But this might not be what you are trying to do.
For implementing best Chat application, use jabber server and write clients using js/flex
http://en.wikipedia.org/wiki/Extensible_Messaging_and_Presence_Protocol
If it's not like chat but you want to send messages over without a server, you need both clients be server as well. A server will listen on a port for incomming connections. Write a daemon that spawns a new thread each time a client connects. Within this thread you handle the messaging.
Client A opens a connection to the server (Client B) and they can talk to each other. Or you let Client A become server and let Client B connect.
My company has a website built with PHP. We use the built-in PHP email functionality to send thousands of emails to subscribers on a daily basis.
This is a terrible idea. It chokes out our server, and takes hours to complete the whole batch.
Now I've looked at mass mailing services like MailChimp (which would replace our current system of sending the same email to many people), but what I think I'd really like to do is to set up a somewhat-sophisticated notification system.
Rather than send a mass email to each person each time something important happens, I'd like clients to be able to customize the rate and content of the emails that they receive.
Even using this new idea, we're talking about A LOT of emails being sent.
So my question is very specific: I have a rough idea of how to build the system internally, but what is the best way to send out all of these emails?
Bullet points to consider:
Sometimes emails' contents are identical across recipients, but many of them will be customized per-user (they choose what they get notified about, and sometimes it is aggregated).
I want a system that is not going to choke the server, and will complete in a decent amount of time. I don't mind going with a third-party service (even a paid one) if that is what it is going to take.
The system should hook into PHP easily, or the API or whatever should be relatively easy for me to call from your typical web server.
We have a dedicated server and full control over it (so we can install apps, services, whatever).
Any kind of detailed tracking information (opens, clicks, etc) is a huge plus.
These emails are sometimes time-sensitive (so can't take all day to send).
Thoughts? Tips? Point me in the right direction?
EDIT
To clarify:
I can do these on my own:
maintain user list
handle email content generation based on user preferences
And need something else (app, third-party service, w/e) to:
accept email content and addresses and actually send the emails out
provide tracking data (opens, clicks, etc). The more detail the better.
I'm leaning towards a third-party service, since I'm not sure any app can avoid choking the server when sending thousands of emails (though I wouldn't consider myself an email expert so I could be wrong).
We use the built-in PHP email functionality to send thousands of emails to subscribers on a daily basis.
This is a terrible idea. It chokes out our server, and takes hours to complete the whole batch.
Why do you think that your problems are anything to do with the built-in PHP email function? It's a very thin wrapper around 'mail' or a simple SMTP client depending on what platform you are running on.
When you say it chokes your server - do you mean your email server? Your web server? something else?
There's nowhere near enough information here to make a proper diagnosis but it looks like the problems are of your own making - sure, there are lots of people out there who promise to sort all your problems for you if only you buy their latest product/service. But there's a very good chance that this isn't going to solve your current problems.
Can you tell us:
what OS the PHP is running on
how you invoke the code to create the emails
what the mail config in the php.ini file is
what type of MTA are you using? On what OS?
how is youe MTA copnfigured - does it deliver directly or via a smart relay?
which server is getting "choked"?
What anti-spam measures do you have in place for outgoing mail?
Then tell us what you've done to diagnose the fault and why you think its specifically on sending mails.
C.
I'd recommend using the third party mailing service Silverpop, or something like it. We've used them for a few years and have been fairly satisfied. They already have relationships with the major email clients (AOL, Yahoo!, Gmail, etc.) and they do a good job of telling you if the stuff you're sending is likely to be classified as SPAM.
They have a fairly extensive API that uses XML HTTP/HTTPS requests that can tie in to existing systems. You can use it to remotely trigger emails, schedule mailings, customize email contents, set up, manage and query huge lists of recipients, run batch processes, etc.
It isn't a perfect service, but compared to a lot of others out there, they do pretty well. I have had very few complaints about them thus far.
I usually got around this by having a mail "sending" function that dumped the emails into a queue (database table) with a job that ran every couple of minutes, grabbed the next x emails in the queue, sent those out and marked them as succeeded. That's the simple bones of it. You can then add on handling for email failures, returned mail, etc in version 2.
Use Google AppEngine if you are worried about scalability & customization: it provides an email API and you can interface anything to it provided it is through an HTTP interface.
Of course, this is just a suggestion - disregard if this doesn't fit.
Quite possibly not ideal, but if you're looking at large scale transmission there are commercial solutions such as Port 25's PowerMTA that can be set up to effectively transmit the contents of a given folder.
As such, you'd simply use PHP to create the personalised MIME formatted raw data for each outbound email and place these in a temporary directory prior to transmission. (I've written such a system in the past and you'd be surprised at the PHP's ability to grind out the emails, even with quite complex text & HTML emails with images as inline-attachments, etc.) Once you were ready to transmit, you'd then move the files en-masse to the PowerMTA monitored folder and it would take care of the transmission.
Depending on how you look at it the benefit/problem with such a solution is that you'll need to build trust relationships with people such as AOL, MSN/Hotmail, etc. to ensure that your mail server isn't blacklisted due to user's reporting email as SPAM. (That said, this will likely be a factor with any DIY solution.)
Why not keep your PHP system and use a third party SMTP service? Some reliable companies offer the use of e-mailing-only machines at reasonable prices, e.g. Dewahost who I am planning to use.
Also see the question Third Party Email Senders and my answer there.
Check out Campaign Enterprise for a possible in-house solution.
One of my friend uses http://www.tailoredmail.com but i havn't used it personally
I know this is an older question, but I just wanted to suggest SendGrid which is essentially an "Email Server as a Service" allowing you to send emails with cost per email.
1) How can a linux server be configured such that it can receive any emails sent to account#domain.com.
2) How can these emails be accessed and displayed by a PHP script?
I'm building a simple web mail script, so i want to receive emails only for the registered email accounts and classify their emails and show them.
Needs also to have the ability to show attachments.
Honestly, you're better off looking at one of the webmail systems already out there and modifying it to fit your needs rather than writing something from scratch.
The email RFCs are a fun read (no, no they're not) and attachment handling has caused many developers to take up heavy drinking. Don't get me wrong, it can be done. Its just a lot of work and not something that you want to take on unless you have a bunch of time to dig deep into it.
However, if you wanted to write something quick and simple, I suggest setting up a linux box with an Imap or pop server and then accessing that mailbox with the appropriate PHP / Pear libraries. That will be the quickest way to get going.
But I would still recommend using one of the off the shelf solutions that are out there and hacking it up to fit your needs.
The mail server itself can be setup to receive email with any number of MTA's (sendmail, postfix, exim, etc). You'll want SPAM protection and Virus scanning as well (again there are free packages in any distribution for this). Be warned, running your own mail server is NOT a job for the faint of heart these days.
Once that's in place I would probably setup IMAP and use PHP's IMAP functions to retrieve and parse the messages. Viewing attachments will be a lot harder if you want them readable in the browser, but easy to mark that they are there and make them downloadable.
There is likely a class already written that does most of this for you (excepting the setup of course).
Update: In fact there is an old article at evolt that appears to cover this.
You have a lot of work in front of you - don't expect someone to just post some code here.
Start by reading the POP3, SMTP, and IMAP RFCs. Then go find a commercial/open source solution.