Folks,
I have a PHP-based site (using the QCubed framework); as a part of the site, I have a daemon that's sending out several thousand emails a day (no i'm not a spammer, everything is opt-in :)). Emails are sent through a custom framework component; that component serves as an SMTP client. I'm using a paid SMTP gateway from DNSExit.com to get the emails actually delivered.
Those emails are simple HTML-based emails; they really have just simple links inside.
My issue is that these links sometimes (not consistently!) get scrambled during transition. Tags somehow get mixed up, and some links are non-functional in the email. The issue happens on a small percentage of all sent emails; it is not consistent (i.e. the same exact source message HTML may or may not cause the scrambling in transition).
Have any of you seen this? Any thoughts on how to troubleshoot?
Is it possible that you are using temp files to create the emails (or at minimum to create the variable content)? I did something vaguely similar once upon a time. The email text was generated and written to a temp file based on the exact time in seconds. Unfortunately, when sending thousands per day, we were hitting the same second more than once (since there are only 86k seconds available). That might explain a) the small error rate and b) the apparent randomness. For troubleshooting, I'd just see if the error rate increases with the number of emails and go from there.
I ran into a similar problem on a server running sendmail.
I was creating and testing an html email that would one day be mass mailed (opt-in, of course). I had myself a template for the email that was easy for any html programmer to read, but as such was heavy on the whitespace to line everything up correctly. I thought to myself, if this is going to be mass emailed, after the template is rendered, I think I will minimize the whitespace in the file to save on space! So I created a brilliant regular expression to rid any unnecessary to send whitespace from the rendered email.
Upon sending the email to myself, I opened the email and was baffled when I saw that some of the css and html were not showing up correctly, when my previous emails prior to my regexp were. By looking at the original message I noticed that every once in a while, an exclamation mark (!) was appearing seemingly randomly throughout the message, thus breaking any css and html that came in its random path.
Turns out that sendmail doesn't like it if a line in your email gets too long without a line break. When the line does get too long, sendmail will insert an exclamation mark followed by a line break right then and there, just to confuse and confound you.
Why did it not just choose a space between words to line break? Why insert the exclamation mark? Questions I'm afraid, without answers.
My solution?
sudo apt-get remove sendmail
sudo apt-get install exim4
I was having other problems with sendmail like it taking a full 60 seconds to send an email and exim4 just worked and I have never had to think about it again.
If your mail server is using sendmail, this very well could be the problem, if not, thank you for letting me share my story with you. I needed to vent.
When you're sending email you should encode it so every line in the message body is not longer then 76 characters. You could use base64 for this but most systems use the
quoted-printable encoding for text because it generates smaller messages.
Base64 is usually only used for binary data.
The problem is that HTML is not compatible with email. That is why I created Mail Markup Language.
HTML was created to operate with the HTTP protocol as those two technologies were invented by the same person at about the same time. The difference is that HTTP is a single session one way transfer from a server to a client. That never changes as the HTML document always originates on a server, is sent to a requesting client, and once the transfer completes the connection between the client and server is dropped.
Email does not behave in such a way. In email a communication originates at a client, is sent to one or more email serves, and then terminates at a distant client. The biggest difference, however, is that the document does not die with finality of a single transmission instance as is the case with a document transfer over HTTP. A document sent in SMTP can be replied to, forwarded, or copied to multiple unrequested users. This one difference is profound when consideration for an email thread is considered.
The problem is that SMTP and HTTP are different as demonstrated in the prior two paragraphs. This differences is compounded in that SMTP and HTTP have radically different formatting methods for the creation of header data. HTML has header data that is intended to be compatible with the headers of HTTP transmissions and offer no compliance to SMTP transmissions. The HTML headers also do not account for the complexity of an email thread.
The problem is exemplified when email software corrupts a HTML document to add formatting changes necessary to fit the conforming demands of that software and to also write header data directly into the document. This exemplification becomes extremely pronounced when an HTML email becomes an email thread. Since the HTML header data has no method to account for the complexities of an email thread there is no way to supply relevant presentation definitions from a stylesheet that survive the transfer of the document. Each time a HTML document, or a document with HTML formatting, is sent from one email software to another the document is corrupted and each email software device corrupts the prior corruption. Email processing software may refer to either an email client, which certainly will corrupt a document, or an email server, that may only likely corrupt an email document.
The solution to the problem is to create a markup language convention that recognizes the requirements of email header data directly. Those requirements are defined in RFC 5321 for the SMTP protocol and RFC 5322 for the client processing. The only way to properly extend this solution to account for the complexities of an email thread are to provide a convention for a multi-agent DOM.
Paragraphs deleted due to technical inaccuracy and difference between the term multi-agent DOM and the nature of an invented feature not mentioned here even prior to the edit.
EDIT: a multi-agent DOM applies some degree of hierarchy, which may not be necessary to represent an email thread.
Had 2 problems with email data - usually "?" symbol somehow got inside some words, another was UTF and title related. First got "fixed" by changing hosting provider (so it was mail-server related) second one got fixed by changing PHPmailer library.
Try to specify how exactly data is scrambled.
Have you any special attributes in your links? May be title attribute with not escaped quotes inside?
Something like this: Link
Related
This may be a bit of a noob question, sorry.
There aren't any security holes in this kind of code are there? I've been using it everywhere but wanted to make sure I'm not leaving vulnerabilities around.
$body = print_r($_POST, true);
mail($to, $subject, $body, $headers, "-f $from_address");
No, it's not safe.
But you'll probably get away with it as you need other badly set up systems to let a hack through.
Details
The "normal" security issues with email are well known: always vet anything going into the header to prevent header injection; most simply by removing new line characters (or rejecting if any are sent: means someone's hacking/testing). This isn't a concern you've raised.
However there have been hacks in the past with SMTP injection: you can terminate the SMTP message and start a new one with judicious use of a full stop (periods) and %0D %0A (CRLF) and MAIL TO: commands. An SMTP message ends with a full stop on its own: rather simple to add. Look here: http://projects.webappsec.org/w/page/13246948/Mail%20Command%20Injection and https://www.owasp.org/index.php/Testing_for_IMAP/SMTP_Injection_(OTG-INPVAL-012)
There are defenses against this: e.g. SMTP servers can be configured to not allow chaining and will treat all as one. So HOPEFULLY you are safe. But to be sure, see if you can strip out . on its own line, and convert %0D %0A (CRLF) into %0A to help remove the risk (but check that last one: your results may vary and it may not be suitable).
For completeness, there are also XSS injection possibilities (i.e. including links) but that's harmless unless you follow a link. Web clients may automatically convert plain text links into clickable links: I can't think how this can be exploited but there's always a way: so vet for links if you're concerned.
Adding an attachment to the existing email (as suggested in one answer) is only possible if you are sending a multi-part email and the user/hacker knows the border separator; so randomly generate border separators or send plain text (not multi-part) to avoid this.
There's potentially the ability to run Javascript if you have exactly the right circumstances. As with attachments, if you're sending a multi-part message, you can create an HTML part with the border separators; so randomly change your border separator and vet for HTML tags. If you're using a web client that allows javascript sent as plain text to run, then you need a new web client (I really doubt there is one).
Those are what I can think of quickly.
Simplest solution: use a library. Well known libraries are designed to avoid these issues. You might also get more advice by checking the source code and how they cope. Swiftmailer is my current library of choice.
You should take good care of $headers and $from_address as those can easily be exploited for Email Injection attacks.
Furthermore, if you print_r a variable that contains HTML, you'll send a string with HTML via email. Although the email may be sent as plain text, it's still up to the email client how those strings are interpreted and displayed. As such, you cannot simply assume that there is no security hole.
As already shown in the comments, you can never trust any user input. Some possibilities of exploits could be:
Embed a link to a phishing site
Add an attachment to the e-mail with a Trojan
Create a 1GB large e-mail
The company I work for provides bulk-mailing functionality to our clients [double opt-in, not spam, I promise] and we get a figurative ton of reports back via Feedback Loops from AOL, Comcast, Yahoo, etc. These are generally from people that signed up, don't want it anymore, have been conditioned to not click 'Unsubscribe' links, [because "that's how the spammers get you"] and simply mark all the messages as spam.
Now, these FBL emails follow a specific format where the message is multipart, there are one or two text parts, and then the original message is attached, usually with all recipient information stripped out. This attached email is also multipart and contains the unsubscribe link, but the section in the attached email the link occurs in is quoted-printable encoded and the link is longer than what quoted-printable allows for in a line, so it get munged. Occasionally the section seems to get base64-encoded, I think it happens if the client is using a fancy language like chinese/japanese/etc.
What I need is a mime/multipart data parser that can give me these parts. PHP has oh so helpfully not implemented any form of multipart parser that I can find outside of what's internal to either their horrid IMAP functions, or internal to PHP itself which processes multipart form data.
Does anyone know of something I can use for this short of having to write my own? I had found one script, but it relies on old PECL functionality that relies on a custom-compilation of PHP which is not an option for this server.
TL;DR: PHP's imap_* functions will parse the parts of the message received from the server, but I need to parse the parts of an email attached to the email downloaded from the server.
This guy's script is ugly as sin, but it gets the job done:
http://www.phpclasses.org/package/3169-PHP-Decode-MIME-e-mail-messages.html
I've searched the web on this and I can find two solutions:
Use a URL-shortner.
Use brackets < > to allow insertion of whitespace. This only applies to plain-text emails.
I'm sending HTML emails that contains sensitive information, like password recovery links and auto-authentication links. These secure links are, by their nature, quite long. Easily longer than 70 characters or whatever arbitrary limit is imposed on e-mail bodies.
Obviously I can't use any form of URL-shortning because it would circumvent any secure cryptography technique I've used for the links.
I suppose the largest issue is that the links aren't wrapped nicely by email clients. The nice word-wrapping is supported in CSS3 technique, so there is no way MS Outlook will ever support this.
How do I make it so the links are not visually disruptive and possibly easy to copy-paste in case of email client malfunction?
Example
If the link doesn't seem to work, try copy-pasting it to your browser:
hxxp://development/#auth/YoxOntz0Oj0ExOiJfcmFuG9tbmVzcyI7czoxOTI6kJsYkV0qMGMuZ1pFZVQ5YkRQNDZPR200Unl60dGlpNlhZZC9QcEVeH0lvV2NVVlpWWFcwWlF0VjRyc0p4akYzclJ0GTkJmSlgxco0aEtDS0FuTlBVSXAxUVhCcGdRNGpHMVl5UGZMRFVacDVSQ1BqcU0tKYlNxZ0FqYXpjTkNqTS9LV29xSk4ydGtyeFpNdV0c2VLMERUbEYwc08xUWU5aXR0GOXl0bVFpcjlXeGZjUE100S0o1L1FcmQ4MmhOdm5LUSI7fQ
Additional notes:
I do not want my links to expire. If I want them to expire at all I'd have a time frame of months, not hours. These links are not just password recovery links.
Have you tried the Microsoft proprietary word-break:break-all; ?
<td style=“word-break:break-all;”>
This worked best for me (best compatibility across vendors):
<p style="word-break:break-all;">
<font style="word-break:break-all;">hxxp://really_long_link</font>
</p>
Tested on: MS Office 2007/2010, outlook.com, hotmail.com, gmail.com, yahoo.com (yahoo did not display nicely)
While this was edited to include <p>...</p> I would highly discourage the use of paragraph tags in HTML email if spacing is impotant since email clients interpret these differently.
In Html emails - use html
If you are sending html emails, what's wrong with:
... reset your password ...
If the links are getting mangled when you send html emails - the problem is how you're sending emails, not the recipient's email client.
Use a shorter hash
Is a super long hash (is it a hash, or is it encrypted?) really necessary? Using any hash that is long enough to not be brute-forced before it expires aught to be sufficient. Asking users to copy and paste a string that is obviously going to wrap isn't going to help the user, it's just pushing a development problem onto the user.
I was wondering if anyone here with experience of PEAR mail or PEAR mail queue could help me out with this.
I am working on creating a bulk mailing service using PEAR and am adding X-headers to give information on where and when people signed up.
So am am trying to create a X-header similar to this:
X-Subscription: Subscribed on 2010/09/01, via web form, by 92.8.196.121 from http://mydomain.com/signup.htm
However after I pass the headers to PEAR mail mime and queue they are formatted with a line break at certain points so they end up looking like this:
X-Subscription: Subscribed on 2010/09/01, via web form, by 92.8.196.121 from
http://mydomain.com/signup.htm
I have tested this by creating a few different headers and the line break always comes after a certain amount of characters but I cannot seem to find any code in PEAR which would cause this.
Does anyone here have any experience of this? Or know of a way I could fix this?
Thanks for looking
The "issue" of headers being split onto multiple lines is correct behavior according to RFC 822, section "3.1.1. LONG HEADER FIELDS":
For convenience, the field-body portion of this conceptual
entity can be split into a multiple-line representation; this
is called "folding". The general rule is that wherever there
may be linear-white-space (NOT simply LWSP-chars), a CRLF
immediately followed by AT LEAST one LWSP-char may instead be
inserted.
As described in What is the email subject length limit?, RFC 2822 suggests to keep a line length of 78.
I am looking for a solution that will enable me to connect to a mailbox, obtain an email, apply specific modifications to the email body (for example, change the content), and then forward the newly modified email to a new email address.
The trick is that such modification must not destroy the format and headers of the original email and I must not lose any attachments that were in the original email.
The sort of manipulation that will be performed will need to be done by an external process that knows the logic of my application.
The solution I am looking for can be an external software that can invoke some API for processing the content of the emails, or even API by itself that my code will invoke.
Our solution is currently based on PHP, but any other solution is also acceptable.
I started working with the Zend Mail library but I am running into problem having to understand the inner-workings of email formats. I wouldn't want to start messing around with the mime objects in the email format. I only want to alter the textual content of the message and keep the rest untouched.
http://php.net/manual/en/book.imap.php - functions that let you manipulate email systems.
What mail server are you using? In qmail its easy to process any incomming email. You can put any script in any language to process the lines of the email.
If you have IMAP access to your server you can use the php IMAP lib. http://www.php.net/manual/en/book.imap.php
I wrote a library as part of a larger open source app that may help you a bit. Its an object orientated wrapper around the PHP imap functions and can be found at google code.
Unfortunately this doesn't do exactly what you want. What in the message are you trying to change? I may be possible to just grab a raw version and specifically search out what you want to change, ignoring the whole mimetype processing altogether, and then just send the whole message along again.
Resending the email is simple enough, and this (small tutorial)* on sending email with attachments can refresh you on the basics (although most of what is in there you can skip as the attachments and mimetypes will already be built).
* I can't post the link because my reputation isn't high enough for two links in a single post, so I'll add it in a comment.