Random errors appear in alert emails - how could this happen? - php

I'm the webmaster and lead developer for a set of websites providing assessment services. I have no background in server administration and am ashamed to say that being a sysadmin is pretty intimidating to me; I'm comfortable with basic command line, but I'm frequently overwhelmed by all of the myriad configurations needed to get a standard LAMP stack working with several subdomains and an email server running.
I've finally reached something that thoroughly stumps me. Let me first list our basic server configuration:
VPS running Ubuntu 12.04
Standard Apache, PHP, Mysql, BIND services
1 main domain + 4 subdomain websites configured in the above services, two of which make extensive use of mysql databases having 300-400 tables each
Citadel email server (a choice I'm increasingly come to see as a mistake)
Swiftmailer (for sending automated emails from PHP)
Some of our pages (relating to account and assessment management) send alert emails to the test taker, coach, and/or us, when a specific action is taken. The text for these emails is fetched from a "boilerplate" table in the mysql db. Then in PHP, a couple "placeholder" tags are swapped out for test taker name, date, etc. The PHP code then uses Swift Mailer to send the prepared text as an email, with our local Citadel email server, to a given set of (usually external) addresses. We are CC'd on most of these emails, to provide us with a second record of all of the automated messages we've sent.
(Feel free to skip over this) Here's the gist of my code that fetches and processes the text, and sends it as an email:
$subject = $bp['email_report_title'];
$subject = str_replace("[NAME]", $name, $subject);
$subject = str_replace("[INSTRUMENT_NAME]", $instrument_name, $subject);
$body = $bp['email_report'];
$body = str_replace("[NAME]", $name, $body);
$body = str_replace("[INSTRUMENT_NAME]", $instrument_name, $body);
$body = str_replace("[INSTRUMENT_BASE]", $instrument_base, $body);
sendEmail($email_to, $email_from, $email_cc, $subject, $body);
In a nutshell, here's my problem: the emails we see (as a result of the above events) sometimes have typos appear in them -- inconsistently. Here are a couple images as example, showing a set of auto emails generated & sent on the same day using the same PHP code drawing from the same (unaltered) string in our database. These are screenshots from our internal (CC'd) account, viewed in Mac Mail; but based on test taker complaints, all email recipients appear to be seeing the same text. In the first image, note the differences in spacing in the titles.
In the second & third images, note the incorrect link (two periods at the end). I have checked the PHP code and the mysql source text, and can find absolutely no reason that the login link would be changed to have two periods in that spot. EDIT: Based on commenter request, I've copied the full-length email text below the images.
[name removed]'s LSUA report has been completed. To view the report, click on the following link:
https://www.devtestservice.org/security/logIn..php
Once you have logged in, click the LSUA icon, and follow the link that says "View your report".
The commenting on our assessments is done by highly trained human beings who occasionally make mistakes. If you notice any inconsistencies or errors in your report, please let us know and we will make prompt repairs. If you have any problems logging in or viewing your report, please contact us using the following link:
https://www.devtestservice.org/contactus.php
Lectica, Inc.
Northampton, MA 01060
Our mission is to develop standardized, formative, and diagnostic developmental assessments of the knowledge and skills required to meet the challenges of the 21st century. Our aims are threefold: (1) to build engaging, educative, and feedback-rich developmental assessments and learning resources for K-18 students and their teachers, (2) to create equally rich assessments that diagnose learning needs and support the development of adults (in the workplace and beyond), and (3) to build (and share) knowledge about learning and its role in the future of society.
This message contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended solely for the use of the addressee(s) named above. Any disclosure, distribution, copying or use of the information by others is strictly prohibited. If you have received this message in error, please advise the sender by immediate reply and delete the original message and any and all attachments. Thank you.
Sorry for the long background explanation. But my question is simple: How could these typos happen? At what point in the workflow could a (verified consistent) mysql data cell, fetched and processed by (straightforward) PHP code, plopped into a standard PHP -> SMTP mailer plugin, lead to the creation of an email that appears to have random typos?
I'd love any ideas you have, and I'm glad to provide more detailed code if needed. Thanks in advance.
EDIT: Thanks to Eggyal for pointing out that only the longer subject lines have the spaces removed. This pattern holds true across other older emails too. Also, I've provided the full text of one email alert above; note the '..php' at the end of the link. All emails have the same signature line, and all alert emails of the same type have the same length body (plus or minus 15 characters for different instrument names and test taker names).

I have experienced very similar behavior in emails when using plain php's mail() function and postfix smtp server. It took me weeks to figure it out why HTML in those emails is randomly corrupted: it was long lines in the email source, exactly as #eggyal writes in the comments.
RFC2822 states that lines should be no more than 78 chars. To comply, SMTP server inserts newlines at selected places, however this can easily corrupt space-sensitive content. Check the Content-Transfer-Encoding header in your emails, I assume it will be either 7bit, 8bit or quoted printable.
Try changing it to base64 - that's the easy way. I'm not familiar with Swift Mailer but I guess this could work:
$transferEncoding = $message->getHeaders()->get('Content-Transfer-Encoding');
$transferEncoding->setValue('base64');
Alternatively, you can try to wrap lines manually before handing over to SMTP server.

Related

Need to parse unfiltered url querys via PHP, can't deal with '&' signs

first of all: I'm not a professional php developer, I'm just able to scramble together a few bits of code to make basic stuff work.
I also don't know how to properly name all that stuff, so let me give you some background:
A client of ours wants to send an email containing a link to a vcard qr-code, that contains contact details of their clients to every client they have (between 100-5000 clients possibly).
They don't want to do this by hand so they asked me for an automation solution.
They have a marketing tool that allows them to send mass emails containing the customers details.
So entering 'Hello %name%' in the emails text will be replaced with each customers name in their respective email.
So I found an api that does what they need, it takes url query strings to generate a vcard qr-code (as png file).
I made a script that takes custom query strings, some of them are being sent to said api, some of them are just being embed on the page, so people can print the qr-code including some extra information they need.
The whole system will be used as a "entry-ticket-system" for an event they are hosting, that being said, no high requirements, those tickets are free, our customer wants to use the v-card qr code so they can scan the "tickets" on entry and verify who was actually there, to target them with marketing campaigns later on.
Now to get to the technical part:
This is the api we are using:
https://qrcode.tec-it.com/en/VCard
The way we are currently using it, is by displaying the qr-code in html (<img src="https://qrcode.tec-it.com/API/QRCode?data=BEGIN%3aVCARD%0d%0aVERSION%3a2.1%0d%0aN%3aJohn+Doe%0d%0aTEL%3bHOME%3bVOICE%3a555-555-5555%0d%0aTEL%3bWORK%3bVOICE%3a666-666-6666%0d%0aEMAIL%3aemail%40example.com%0d%0aORG%3aTEC-IT%0d%0aURL%3ahttps%3a%2f%2fwww.example.com%0d%0aEND%3aVCARD&backcolor=%23ffffff" />) and then entering the url query strings as a variable.
This looks something like this:
$fname = filter_var($_GET['fname'], FILTER_SANITIZE_STRING);
$lname = filter_var($_GET['lname'], FILTER_SANITIZE_STRING);
$email = filter_var($_GET['email'], FILTER_SANITIZE_EMAIL);
$fullname = $fname . ' ' . $lname;
echo '<br><img src="https://qrcode.tec-it.com/API/QRCode?data=BEGIN%3aVCARD%0d%0aVERSION%3a2.1%0d%0aN%3a'.$fullname.'%0d%0aEMAIL%3a'.$email. (...)
Now we come to the problem: The email service our customer is using is only able to fill in basic strings, which we cannot modify. So if a client's email would theoretically contain a '&' sign, it would break up the url by indicating a new url query string. We cannot encode them, nor can we do http push.
We can't properly access the data either, we could only export everyting into a csv, modify them and re-import all of their customers (last solution)
Is there any solution for this, so we can accept basically any character within the url. A friend of mine suggested to use a json in the url, however json just has different characters that would break everything.
Everything else does work fine, it's just the '&' sign I'm currently aware of.
PS: I do know that the php code isn't proper code, very low quality. This is okay, the customer is aware of that, they just wanted the cheapest possible solution so they don't have to do it by hand, they have been informed about possible bugs and the very much existing security flaws.
I hope someone can give me a push in the right direction, how could one possibly handle this?
Thanks in advance and apologies for low quality code and possibly unclear explanations, I'm trying my best.

Website Form to SMS

I would like to add a "Text Me" form to my website that will allow users to type their Name, Phone Number, and a Message that i receive on my phone via SMS. I would prefer something easy and free, perhaps a code snippet. Does anybody have any suggestions for a "Text Me" form or a service that provides one?
A quick google-search reveals (at least for German) customers: lox24.eu - they have a VERY simple way for sending SMS:
https://www.lox24.eu/API/httpsms.php?konto=<yourId>&password=<yourPassword>&service=<yourTariff>&text=<SMSText>&from=<yourSender>&to=<receivingNumber>
All you have to do, would be creating a short text from the $_POST variables of your form and then do a HTTP-request (preferrably cURL) from within the PHP-page that receives the form-data.
You might try to google for "websms api" to find a SMS-gateway in your country that provides the services you need.
HTH
^KMP
I recommend If This Then That. You can hit a simple web backend, supplying a JSON payload, and IFTTT will then send an SMS to your cell with the payload text, or whatever text you want.
Maker channel: https://ifttt.com/maker
SMS channel: https://ifttt.com/sms
IFTTT will abstract away the specifics of how the text is sent (doesn't matter what carrier, what country, etc.) and it will (probably) break the text into properly sized chunks. If not, you might have to do that yourself, though - just use substr() to divide the text payload into chunks of 160 characters and hit the backend as many times as necessary for the message to be sent.
You can use raw cURL to hit the backend, but I find that Guzzle is a great PHP library for making web requests of any kind.
A lot of mobile carriers in the US (not sure about abroad) will have a text equivalent email. For example, AT&T will text you if you send an email to yourmobilenumber#txt.att.net. Its free.

Send Text Message From PHP Phone Number

I'm a bit confused here and due to the keywords involved Google has not been much help. I want to send text messages from my website to users with cellphones via SMS. Sounds simple enough using things like 1234567890#carrier.com with php mail scripts right? Well one thing I've encountered before is that sometimes it sends an MMS(multimedia message) instead of a regular text message.
What I am trying to find out(using keywords in Google, send text message phone number php) is how to get a phone number for your website? I've seen sites that actually have a phone number linked to the site that you can receive messages from.
Example Facebook uses 32665 to send and receive messages on.
How do you accomplish this?
Is it any way possible using the 1234567890#carrier.com with php mail to strictly only send a text message and not multimedia?
Example php mail script using a phone number:
<?PHP
mail("5558349324#vtext.com", "", "Test SMS", "From: RRPowered <test#rrpowered.com>rn");
?>
Also thanks to any help ahead of time, because I am really really confused at this point.
UPDATE: I just tried using the regular PHP mail functions to a phone number and it appears to cut the messages off at around 140 characters(I believe that is standard message size).
UPDATE 2: Before anyone mentions API like Text Magic, I am aware of those things, BUT I am wanting to build my own code instead of integrating someone elses API(if possible).

PHP parsing gmail mail reply

I am trying to parse GMail emails, but have one problem: how do I know which message a reply corresponds to?
I tried sorting email by subject. For example, if a message has the subject "hi Jack", then all messages with subject "Re: hi Jack" are a reply to this mail.
But what do I do if I have many emails with the same subject? How do I know which email they are replies to?
Do emails perhaps have a unique code for what the reply goes to? Maybe there is an ID or something like that to know what the children of a message are(?).
Threading by subject is not a good idea because there may be as you noticed several different threads based on identical subjects.
You need to examine 3 headers in the message to make threading (or other kind of grouping) possible:
Message-ID: contains unique message identifier (what you call "unique code") in a string surrounded by < and > characters e.g. <123456#User1PC> Most MUAs will create identifiers in above form or something similar to that. This header should be generated when a new message is sent.
In-Reply-To: contains a message this particular reply is related to e.g. <789abcd#User2PC>. This header should be copied from Message-ID it replies to.
References: contains list of recent references to messages in this "thread". The format is similar to above except they are separated e.g. <123456#User1PC> <789abcd#User2PC> It is there so that you can use it to locate message in the thread.
If one message has been replied or posted a few days later it might be hard to locate it without list of references. Usually list of references is trimmed by mail clients to a reasonable size. By reasonable, I mean, trimming it enough to be able to locate message in a thread but keep the header under reasonable size (not having too many references). For example it may contain 5-10 references which is more than enough usually to connect it to other messages. References: are also useful in case if original message (first one) has been deleted so even without it, you can still utilize References: list to build a threaded (grouped) messages.
So, in order to thread messages, you would need to read all of them, and then sort threads based on the information you can extract from above headers.
If references or message ids are not in form you can recognize (e.g. <example#something> you can bail out by not threading these messages and displaying them as unthreaded. So generic algorithm for threading/locating might look something like this:
Take first message ID
Examine nearby (by date) messages to see if one of them contains message ID in its references list or in-reply-to - if there are none - you can't group it so keep it as standalone message.
Group messages somehow, perhaps based on Date:, or Received: header
Place this message into "Done" list so you don't need to examine it further (or related references)
Continue until you can't find any more references and then move to next message which is not already in "Done" list and repeat steps until you process entire message list.
It will probably take you a while to get this done properly but now at least you have a starting point to look into.

Extract email message itself from all its prior messages and meta data (Sendgrid Parse API/PHP)?

I'm using Sendgrid and their Parse API to send/receive email. The Parse API allows one's web app to receive email as a $_POST but the problem is that in the $_POST I want to be able to extract the message itself from its prior messages and meta data that get chained along.
To show you what I mean in the following picture, i'd just like to capture the text, "trying sending from 12373 to 12373 from GMAIL" and not all the junk below it. If that is not possible, does anyone have any suggestions on how to parse the email body ($_POST['text']) such that I can separate out the message itself?
The problem is see is that depending on the email client (gmail, outlook, etc.), It's not clear to me that the date information, in this case: "On Wed, Jan 23, 2013...", will allows follow the message itself. If all email client's put the date beneath the message, then it would seem I could design a fancy regex to look for a line break followed by a date or something. Thoughts?
You have a couple of options:
1) Insert a token that splits the emails
You could do something like --- reply above this line --- and then cut out everything below that token.
2) Use an email reply parsing library
There is a really good one done by github, but it's in ruby. There's a php port though that might be good for what you need:
Fully working code:
<?php
require_once 'application/third_party/EmailReplyParser-master/src/autoload.php';
$email = new \EmailReplyParser\Email();
$reply = $email->read($_POST['text']);
$message=$reply[0]->getContent();
$message=preg_replace('~On(.*?)wrote:(.*?)$~si', '', $message);
//Last line is needed for some email clients, e.g., some university e-mails: foo#bar.edu but not Gmail or Hotmail, to get rid of "On Jan 23...wrote:"
//This failure to remove "On Jan 23...wrote:" is a known issue and is documented in their README
?>
There's simply no guaranteed way to parse quoted message threads from an email message, so you won't find a regex or any other code that will work in all cases. There's no standard to define formatting of replies, and as you've already observed different mail clients use different conventions. Many, in fact, will allow the user to edit the quoted text. Also, users can paste in unrelated messages, with or without headers, resulting in a mix-and-match of formats.
If you can record and keep the history of all messages as they are sent and received, then you can (usually, but not always) use the In-Reply-To header (see RFC-5322) to locate the previous message by matching it's Message-ID header, and do a diff on the body and remove duplicate text runs. It's apparent that some email systems do this to improve their presentations, but I'm not aware of any available open source code.
// cut quoted text, https://regex101.com/r/xO8nI1/5
$message = preg_replace('/(On\s.*<\n){0,1}(.*\n(\n){0,1}((^>+\s?.*$)+\n?)+)/mi', '', $message);
How about replies in languages other than English? We came out with solution to add marker, but instead of translating it for every email (based on user's language) we put some invisible characters into it (zero width space U+200B , to be precise). Basing on "On..." regexp it's error prone, it can easily cut some email content.

Categories