Like in title, how reliable is this check.
https://github.com/symfony/symfony/blob/3.0/src/Symfony/Component/Validator/Constraints/EmailValidator.php#L139-L142
Every single server in the world have their MX record exposed to the world?
Or is there a possibility that there exists a server that hides their MX record and even if email will be valid, check will fail.
UPDATE:
I have already checked Symfony documentation and source.
I know and I've tested that only emails domain is checked, not the user part.
I just don't know how reliable it is. Is it always possible to check servers MX records.
This validator only check if the DNS user in the email is valid for example if you set this email fail#google.com it will be validated even this email doesn't exit.
http://symfony.com/doc/current/reference/constraints/Email.html#checkmx
The checkMX option is done using PHP's checkdnsrr function, along with the checkHost option. You can also set a strict option and include the egulias/email-validator library for tighter restrictions. Using those options should be completely sufficient and reliable in determining if an email address is valid.
From experience, It seems like a good idea in theory and in practice will work most of the time but there will be occasions where the MX lookup fails due to network issues etc, and then the email will return as invalid, this then causes users who have inserted their correct email to become frustrated.
This in turn then causes developers to have to spend time looking at what went wrong.
Additionally, if a user is putting in a fake email, all they need do is use a real domain and fake user (like fbjdsbafjkbsdjafj #gmail.com) so it is of limited usefulness.
For 99.9% of cases it will be sufficient to check for email well-formedness without mx lookup using an established email checking library, (rather that roll-your-own)
Related
I have to make the copy of my existing site for testing purposes at home. I have got the permission from the company.
But in the database I have the more than 10,000 customer records with email.
I don't want to accidentally send any emails to them while i make some mess with site during various tests.
whats the best way to avoid that
I do need the email functionality for testing other stuff
The most idiot-proof method you can use is often the best for these things as we all have those days where anything that can go wrong will. It's best to be careful, even borderline paranoid, when a mistake could really ruin your day.
Here's a few methods that might work:
Impotent Configuration by Default
The absolutely safest system is to keep the SMTP server configuration for the production server on the production server and only on the production server. Your development copy would have some other SMTP configuration, like to a testing GMail SMTP account. Usually GMail limits you to 500 emails per day on ordinary accounts so you'll quickly hit this limit if you really screw things up somehow.
Replace Customer Emails in Database
Another thing to consider is scrubbing all of the customer emails in your database, removing them and replacing them with mytestaccount+0000#gmail.com and mytestaccount0001#gmail.com if you care to actually receive and inspect them, taking advantage of the fact that the + and subsequent content is ignored for delivery to GMail, effectively giving you unlimited potential email addresses.
As an example:
UPDATE customers SET email=CONCAT('mytestaccount+', customer.id, '#gmail.com')
You'll have to customize this to be whatever email address you want. One advantage to doing this is that you won't have a valuable list of customer email addresses sitting on your development drive and any associated back-ups of it. To be thorough you should probably scramble the hashed passwords as well just so the database is basically worthless to potential hackers. Too many times passwords get scraped from backups that aren't secured properly.
Render Customer Emails Undeliverable
The next best approach is to add ".test" to the end of every email in the system you don't want to send so that it will hard bounce instead of going to someone's inbox.
This is basically a one-liner:
UPDATE customers SET email=CONCAT(email, '.test')
Over-Ride Email at Delivery Time
You can always include some conditional logic like where you will deliberately substitute the recipient of the email message. This can be risky because there's a chance that you might disable that switch by accident, though, so as always, be careful.
In practice this looks something like:
if ($i_should_not_spam_customer_accounts_accidentally)
{
$mail->to = "nobody#nowhere"
}
Use an API Driven Service
Some Mail Service Providers have an API that can help you when testing email messages. I'm a co-founder at PostageApp and the service was designed so you can send messages using an API key that's specifically configured to receive but not deliver emails. Other services like MailGun can be used in a similar fashion.
No Single Point of Failure
It isn't a good feeling being one logical test away from tragedy, though. You should make sure there's several things that have to go wrong before you have a fiasco.
If you don't want to change your code or the data in the database, and if you are using postfix in your local machine, you can rewrite all the outgoing mail to your addres. More info: http://www.postfix.org/ADDRESS_REWRITING_README.html
Change All the Email Address, For Example email#domain.com to email#domain.com. After the test completes you can replace # with #.
Sometimes users misspelled their email domain and hence they enter wrong email address.
Eg. abc#gmial.com rather than abc#gmail.com
Has anybody thought about this before? Can anybody suggest how to handle this type of mistakes?
It didn't exist when this question was asked, but I recommend MailCheck which auto-suggests corrections to entered emails. It's used successfully by large companies.
Can anybody suggest how to handle this type of mistakes?
You would usually send a confirmation E-Mail to the address given, and proceed only if a link in that E-Mail has been clicked.
There is no other good way to deal with this - it's impossible to tell for sure whether gmial.com is a typo or not, seeing as it's a valid domain.
Create a list of common email domain names:
hotmail.com
gmail.com
googlemail.com
... etc
When a user enters an email address, take the domain name of the entered address and take the Levenstein distance between your list. If the distance is 1 (or maybe up to 2) then ask the user to confirm that's the email address they meant.
In my opinion it is bordering on impossible to come up with a generic solution for the generic case.
That being said, the most common typo is to interchange two adajcent letters.
So you might want to check for character content for the largest sites gmail, yahoo and what have you; Based on that suggest an alternative spelling if the original does not match gmail etc.
Do not assume the user is at fault, suggest alternatives if it looks suspicious compared to common names. A white-list was mentioned in another reply.
Use confirmation mails if you need to know you can get a reply from this address.
You cannot assume the spelling you find is in error, that is what confirmation mails are for.
Make it very non-obtrusive (ajax springs to mind).
In our forms we're using a combination of techniques. While bad data can still slip through, the chances are vastly reduced.
First is to do a simple formatting regex that is commonly available - just be sure it's RFC-compliant. If this fails, it's good to offer the user a confirmation form at this point, because they may catch other errors for you while fixing this problem.
The next part is to check the TLD part of the domain. Since all TLDs can be known, these are relatively easy to scan for misspellings using some regex tests. Just keep a list of all current TLDs in a table somewhere and update it form time to time as needed (mind you, this list can get complex when dealing with international TLDs. If you're only dealing with US traffic, the rules are much easier, and that's something else you can filter out. For example, if you're selling a service only available in the US, it would make sense to filter out international emails at form submission time. We are, so this works for us).
Third is to do something like what #npclaudiu suggested - scan for common misspellings of big-name mail hosts (gmail, hotmail, yahoo, etc) in the domain part and if a possible hit is detected, offer a confirmation form to the user. (You entered someone#hptmail.com, did you mean hotmail.com?)
If you get through those steps, then you can do the MX lookup suggested by #symcbean.
Finally, if all of that succeeds, there is a method (but I've not yet tested it) for communicating with the remote SMTP host to see if the mailbox exists. We're about to begin testing this ourselves. I found the how-to for such here:
http://www.webdigi.co.uk/blog/2009/how-to-check-if-an-email-address-exists-without-sending-an-email/
The funny thing is that the url does exist http://www.gmial.com
In fact it would be very difficult for you to know if it's a mistake or just a "strange" domain. Look at the Google API's because when you type something wrong in Google they propose you "did you mean...."
good luck
Arnaud
You can not provide this functionality in a way that you auto correct the misspelled email domain names, because the name which you are assuming to be invalid, would be valid. you should expect anything to be entered as a email address domain name.
I would suggest, if you are creating a signup form, you provide user with a dropdown having all possible domain names which you are aware of so that he can make a selection from that.
Hope this helps.
You could create a list of popular e-mail domains (gmail.com, yahoo.com, ymail.com, etc) in your db and validate the e-mail address that the user inputs against this list, and if the domain resembles with one of these domains, you should show a warning and allow the user to correct it if necessary, not auto correct it. And to compare the domain entered with the domains in your list, you might use an algorithm like the the one used in the soundex function in SQL Server, that matches words based on if one word sounds like the second.
Edit: you can find more details the SOUNDEX function here.
As mentioned before, it is not a good idea to automatically assume that someone has mistyped an email. A better approach would be to implement a little javascript function that checks if the domain of the email was possibly mistyped and alert the user instead of assuming they were wrong from the start.
Give me a minute to create a little mockup.
EDIT: OK, so maybe it was more than a minute. Take a look at http://jsbin.com/iyaxuq/8/edit and see for yourself how javascript can help prevent common typing errors. Try emails like: test#gmail.cmo, another#yhaoo.com, loser#htomali.ocm (typo of hotmail), and me#aol.com.
Note: I used a lazy regex to validate the email. Don't rely on it (or for that matter, most regexes) for a real app.
Trying to automate correction of bad data is a very dangerous practice. Ultimately, only the user can provide the correct data. However there are strict rules about formatting an email address - a regex check can be run in javascript (or using the preg functions with the same regex syntax) - but note that there are a lot of bad examples on the internet of regexes claiming to solve the problem.
This should be a fairly complete implementation of an RFC2822 ADDR_SPEC validator:
/[a-z0-9!#$%&'*+/=?^_`{|}~-]+(?:\.[a-z0-9!#$%&'*+/=?^_`{|}~-]+)*#(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?/gi
However in practice I find this to be adequate:
/^[a-z0-9\._%+!$&*=^|~#%'`?{}/\-]+#([a-z0-9\-]+\.){1,}([a-z]{2,22})$/gi
Then, serverside, you can do an MX lookup to verify that the domain provided not only meets the formatting requirements but exists as an email receiving site.
This does not prove that the named mailbox exists at that site, nor that it is accepting emails - ultimately you'd need to send an email to that address including a click back link / password to establish whether the email address is valid.
Update
While, as the top voted answer here says, the best way to validate an ADDR_SPEC is to send a token to the address to be submitted back via the web, this is not of much help if the data is not coming from the person whom controls the mailbox, and the action is dissociated from the primary interaction even when they do. A further consideration is that an email address which is valid today might not be tomorrow.
Using a regex (and an MX lookup) is still a good idea to provide immediate feedback to the user, but for a complete solution you also need to monitor the bounces.
Is there an API (preferably PHP based) or another means of checking a use-inputted email address against a list of known email honeypots or other email-address related spam stopping techniques?
Context: I'm working on a system to handle contacts for our clients. It'll eventually interface with Verticalresponse or similar. I want to check all incoming contact email addresses to be sure they're legit and not a purchased list.
You may not find such a database, or a PHP interface. Project Honeypot alone has 62,782,527 trap addresses on their monitor. That's 62 million addresses.
Anyone can make a spam trap. Check for example these references to get a picture of the futility of detecting honeypot addresses.
What is a honeypot
Project Honeypot home page
So what can you do to check that your customers' lists are legitimate? Use an check list for evaluating list contents. If you see anything like removethis, or any such common strings fooling spam harvesters, the list you're looking is probably not a good one. Also, a DNS check for the existence of any records for the domain is also a good way to see to which addresses it is possible to try to deliver mail. The DNS check won't tell you if your client has bought the list or not, but it will at least allow you to disregard those recipients.
Scenario:
I have a contact form on my web app, it gets alot of spam.
I am validating the format of email addresses loosely i.e. ^.+#.+\..+$
I am using a spam filtering service (defensio) but the spam scores returned are overlapping with valid messages. At a threshold of 0.4 some spam gets through and some customer's questions are wrongly thrown in a log and an error displayed.
All of the spam messages use fake email addresses e.g. zxmzxm#ywduasm.com
Dedicated PHP5 Linux server in US, mysql, logging spam only, emailing the non spam messages (not stored).
Proposal:
Use php's checkdnsrr(preg_replace(/^.+?#/, '', $_POST['email']), 'MX') to check the email domain resolves to a valid address, log to file, then redirect with an error for messages that don't resolve, proceed to the spam filter service as before for addresses that do resolve according to checkdnsrr().
I have read (and i am sceptical about this myself) that you should never leave this type of validation up to remote lookups, but why?
Aside from connectivity issues, where i will have bigger problems than a contact form anyway, is checkdnsrr going to encounter false positives/negatives?
Would there be some address types that wont resolve? gov addresses? ip email addresses?
Do i need to escape the hostname i pass to checkdnsrr()?
Solution:
A combination of all three answers (wish i could accept more than one as a compound answer).
I am using:
$email_domain = preg_replace('/^.+?#/', '', $email).'.';
if(!checkdnsrr($email_domain, 'MX') && !checkdnsrr($email_domain, 'A')){
//validation error
}
All spam is being logged and rotated.
With a view to upgrading to a job queue at a later date.
Some comments were made about asking the mail server for the user to verify, i felt this would be too much traffic and might get my server banned or into trouble in some way, and this is only to cut out most of the emails that were being bounced back due to invalid server addresses.
http://en.wikipedia.org/wiki/Fqdn
and
RFC2821
The lookup first attempts to locate an MX record associated with the name.
If a CNAME record is found instead, the resulting name is processed as if
it were the initial name.
If no MX records are found, but an A RR is found, the A RR is treated as
if it was associated with an implicit MX RR, with a preference of 0,
pointing to that host. If one or more MX RRs are found for a given
name, SMTP systems MUST NOT utilize any A RRs associated with that
name unless they are located using the MX RRs; the "implicit MX" rule
above applies only if there are no MX records present. If MX records
are present, but none of them are usable, this situation MUST be
reported as an error.
Many thanks to all (especially ZoogieZork for the A record fallback tip)
I see no harm doing a MX lookup with checkdnsrr() and I also don't see how false positives may appear. You don't need to escape the hostname, in fact you can use this technique and take it a little further by talking to the MTA and testing if the user exists at a given host (however this technique may and probably will get you some false positives in some hosts).
DNS lookups can be slow at times, depending on network traffic & congestion, so that's something to be aware of.
If I were in your shoes, I'd test it out and see how it goes. For a week or so, log all emails to a database or log file and include a field to indicate if it would be marked as spam or legitimate email. After the week is over, take a look at the results and see if it's performing as you would expect.
Taking this logging/testing approach gives you the flexibility to test it out and not worry about loosing customer emails.
I've gotten into the habit of adding an extra field to my forms that is hidden with CSS, if it's filled in I assume it's being submitted by a spam bot. I also make sure to use a name like "url" or "website_url" something that looks like a legitimate field name to a spam bot. Add a label for the field that says something like "Don't fill out this field" so if someone's browser doesn't render it correctly, they will know not to fill out the spam field. So far it's working very well for me.
function mxrecordValidate($email){
list($user, $domain) = explode('#', $email);
$arr= dns_get_record($domain,DNS_MX);
if($arr[0]['host']==$domain&&!empty($arr[0]['target'])){
return $arr[0]['target'];
}
}
$email= 'user#radiffmail.com';
if(mxrecordValidate($email)) {
echo('This MX records exists; I will accept this email as valid.');
}
else {
echo('No MX record exists; Invalid email.');
}
//The Code *https://davidwalsh.name/php-email-validator*
function domain_exists($email, $record = 'MX'){
list($user, $domain) = explode('#', $email);
return checkdnsrr($domain, $record);
}
if(domain_exists('user#davidwalsh.name')) {
echo('This MX records exists; I will accept this email as valid.');
} else {
echo('No MX record exists; Invalid email.');
}
An MX Lookup is only part of the picture, if you want to ensure the email address is itself valid, then you need to attempt to send an email to that account.
The other possible scenario is, someone can be simply using hijacked email accounts from a compromised machine anyway. Of course, that is probably a little bit less likely to occur, but it still does.
There are email address validation libraries out there that do this, simply search for email validation.
All of this can be done asynchronously. I have this setup on my site in which case the email is saved in the database (for auditing purposes), a job queued, then when the job comes time to execute, any additional validation is performed at that point in time. It offloads the heavy lifting to another thread.
To the user, it appears as if the email was sent already, it was (it's in the database), and can be viewed internally, but the actual email won't get mailed out until that job executes which can be immediately or some set amount of time depending on the server load.
Walter
I am getting more and more spam emails recently. I already validate my email using regular expression, all the emails must be something like this: xxx#xxxx.xxx
But the problem i have now is, there are alot spammers, type hsdjsdhgf#gmail.com, iluvhahahahah#yahoo.com, these emails are not existed, because i tried to send email to them.
How to avoid those email?
You're barking up the wrong tree. The better way to stop spam is by filtering them out in other ways from the form. If you are not a fan of CAPTCHAs like reCAPTCHA, you should look into what is known as "Honeypots". Essentially, add an extra field to your form with a common name like "email" and hide it with CSS. Mark it as "leave blank" for anyone browsing with styles disabled. If the field has a value in it, it is most likely a bot, so throw the submission out. Voila. They work really well for your average mid-size website that spammers don't really care to specifically set out to beat. Check out this related question.
All things considered, I like the honeypots because it is usually enough to deter 99% of your spamming while not making your average user have to do anything. This is important.
To quickly answer your original question: the only way to verify the email is valid is to actually send out an email to that address and see if it bounces. It is generally not worth the hassle, however.
If they are signing up for a newsletter or something that you will email them more than once I like to send a confirmation email to them which they must click a link to verify that it is a real email. If they dont confirm after a few days then you know you can delete it. If they do confirm then you know that it is a real person. You will still get bounce backs when you first email them but it will eliminate reoccurring bounce backs.
I also like CAPTCHAs or if you want something simpler ask them to enter the first letter of the title of your website (or some other word that will always remain the same on the page), it works for smaller to mid sized sites but is mostly effective.
Short answer:
Use a CAPTCHA.
Long answer:
Well, you could do an MX lookup, where you find the server responsible for delivering email for their given domain, then query the mail-server to see if the address is valid. In a perfect world, this would be the ideal way to validate email addresses.
Unfortunately, as an anti-spam measure, most mail servers these days will respond to such a query with either all positives or all negatives.
This leaves us with really only one practical solution: CAPTCHAs.