Aparently, encoding japanese emails is somewhat challenging, which I am slowly discovering myself. In case there are any experts (even those with limited experience will do), can I please have some guidelines as to how to do it, how to test it and how to verify it?
Bear in mind that I've never set foot anywhere near Japan, it is simply that the product I'm developing is used there, among other places.
What (I think) I know so far is following:
- Japanese emails should be encoded in ISO-2022-JP, Japanese JIS codepage 50220 or possibly SHIFT_JIS codepage 932
- Email transfer encoding should be set to Base64 for plain text and 7Bit for Html
- Email subject should be encoded separately to start with "=?ISO-2022-JP?B?" (don't know what this is supposed to mean). I've tried encoding the subject with
"=?ISO-2022-JP?B?" + Convert.ToBase64String(Encoding.Unicode.GetBytes(subject))
which basically gives the encoded string as expected but it doesn't get presented as any japanese text in an email program
- I've tested in Outlook 2003, Outlook Express and GMail
Any help would be greatly appreciated
Ok, so to post a short update, thanks to the two helpful answers, I've managed to get the right format and encoding. Now, Outlook gives something that resembles the correct subject:
=?iso-2022-jp?B?6 Japanese test に各々の視点で語ってもらった。 6相当の防水?=
However, the exact same email in Outlook Express gives subject like this:
=?iso-2022-jp?B?6 Japanese test 縺ｫ蜷・・・隕也せ縺ｧ隱槭▲縺ｦ繧ゅｉ縺｣縺溘・ 6逶ｸ蠖薙・髦ｲ豌ｴ?=
Furthermore, when viewed in the Inbox view in Outlook Express, the email subject is even more weird, like this:
=?iso-2022-jp?B?6 Japanese test ??????????????? 6???????=
Gmail seems to be working in the similar fashion to Outlook, which looks correct.
I just can't get my head around this one.
I've been dealing with Japanese encodings for almost 20 years and so I can sympathize with your difficulties. Websites that I've worked on send hundreds of emails daily to Japanese customers so I can share with you what's worked for us.
First of all, do not use Shift-JIS. I personally receive tons of Japanese emails and almost never are they encoded using Shift-JIS. I think an old (circa Win 98?) version of Outlook Express encoded outgoing mail using Shift-JIS, but nowadays you just don't see it.
As you've figured out, you need to use ISO-2022-JP as your encoding for at least anything that goes in the mail header. This includes the Subject, To line, and CC line. UTF-8 will also work in most cases, but it will not work on Yahoo Japan mail, and as you can imagine, many Japanese users use Yahoo Japan mail.
You can use UTF-8 in the body of the email, but it is recommended that you base64 encode the UTF-8 encoded Japanese text and put that in the body instead of raw UTF-8 text. However, in practice, I believe that raw UTF-8 text will work fine these days, for the body of the email.
As I alluded to above, you need to at least test on Outlook (Exchange), Outlook Express (IMAP/POP3), and Yahoo Japan web mail. Yahoo Japan is the trickiest because I believe they use EUC for the encoding of their web pages, and so you need to follow the correct standards for your emails or they won't work (ISO-2022-JP is the standard for sending Japanese emails).
Also, your subject line should not exceed 75 characters per line. That is, 75 characters after you've encoded in ISO-2022-JP and base64, not 75 characters before conversion. If you exceed 75 characters, you need to break your encoded subject into multiple lines, starting with "=?iso-2022-jp?B?" and ending with "?=" on each line. If you don't do this, your subject might get truncated (depending on the email reader, and also the content of your subject text). According to RFC 2047:
"An 'encoded-word' may not be more than 75 characters long, including 'charset', 'encoding', 'encoded-text', and delimiters. If it is desirable to encode more text than will fit in an 'encoded-word' of 75 characters, multiple 'encoded-word's (separated by CRLF SPACE) may be used."
Here's some sample PHP code to encode the subject:
// Convert Japanese subject to ISO-2022-JP (JIS is essentially ISO-2022-JP)
$subject = mb_convert_encoding ($subject, "JIS", "SJIS");
// Now, base64 encode the subject
$subject = base64_encode ($subject);
// Add the encoding markers to the subject
$subject = "=?iso-2022-jp?B?" . $subject . "?=";
// Now, $subject can be placed as-is into the raw mail header.
See RFC 2047 for a complete description of how to encode your email header.
Check http://en.wikipedia.org/wiki/MIME#Encoded-Word for a description on how to encode header fields in MIME-compliant messages. You seem to be missing a “?=” at the end of your subject.
=?ISO-2022-JP?B?TEXTTEXT...
ISO_2022-JP means that string is encoded in ISO-2022-JP codepage (eg. not Unicode)
B means that string is bese64 encoded
In your example, you should just supply your string in ISO-2022-JP instead of Unicode.
I have some experience composing and sending email in japanese...Normally you have to beware what encoding used for operating system and how you store your japanese strings!
My Mail objects are normally encoded as follows:
string s = "V‚µ‚¢ŠwK–#‚Ì‚²’ñˆÄ"; // Our japanese are shift-jis encoded, so it appears like garbled
MailMessage message = new MailMessage();
message.BodyEncoding = Encoding.GetEncoding("iso-2022-jp");
message.SubjectEncoding = Encoding.GetEncoding("iso-2022-jp");
message.Subject = s.ToEncoding(Encoding.GetEncoding("Shift-Jis")); // Change the encoding to whatever your source is
message.Body = s.ToEncoding(Encoding.GetEncoding("Shift-Jis")); // Change the encoding to whatever your source is
Then i have an extension method to which does the conversion for me:
public static string ToEncoding(this string s, Encoding targetEncoding)
{
return s == null ? null : targetEncoding.GetString(Encoding.GetEncoding(1252).GetBytes(s)); //1252 is the windows OS codepage
}
something like this should get the job done in python:
#!/usr/bin/python
# -*- mode: python; coding: utf-8 -*-
import smtplib
from email.MIMEText import MIMEText
from email.Header import Header
from email.Utils import formatdate
def send_from_gmail( from_addr, to_addr, subject, body, password, encoding="iso-2022-jp" ):
msg = MIMEText(body.encode(encoding), 'plain', encoding)
msg['Subject'] = Header(subject.encode(encoding), encoding)
msg['From'] = from_addr
msg['To'] = to_addr
msg['Date'] = formatdate()
s = smtplib.SMTP('smtp.gmail.com', 587)
s.ehlo(); s.starttls(); s.ehlo()
s.login(from_addr, password)
s.sendmail(from_addr, to_addr, msg.as_string())
s.close()
return "Sent mail to: %s" % to_addr
if __name__ == "__main__":
import sys
for n,item in enumerate(sys.argv):
sys.argv[n] = sys.argv[n].decode("utf8")
if len(sys.argv)==6:
print send_from_gmail( sys.argv[1], sys.argv[2], sys.argv[3], sys.argv[4], sys.argv[5] )
elif len(sys.argv)==7:
print send_from_gmail( sys.argv[1], sys.argv[2], sys.argv[3], sys.argv[4], sys.argv[5], encoding=sys.argv[6] )
else:
raise "SYNTAX: %s <from_addr> <to_addr> <subject> <body> <password> [encoding]"
**blatantly stolen/adapted from:
http://mtokyo.blog9.fc2.com/blog-entry-127.html
First of all you should be using:
Encoding.GetEncoding("ISO-2022-JP")
to convert your subject line into bytes that will be processed by Convert.ToBase64String().
=?ISO-2022-JP?B?TEXTTEXT...?= tells the receiving mail client which encoding was used on the sender's side to convert japanese "letters" into a byte stream.
Currently you're using UTF-16 to encode, but specifying ISO-2022-JP to decode. These are obviously two different encodings, I guess, just like ISO-8859-1 is different from Unicode (most extended western-europe chars are represented by one byte in ISO-XXX, but two bytes in Unicode).
I'm not sure what you mean about UTF-8 being second-class citizen. As long as the receiving mail client understands UTF-8 and is able to convert it to the current japanese locale, everything is fine.
<?php
function sendMail($to, $subject, $body, $from_email,$from_name)
{
$headers = "MIME-Version: 1.0 \n" ;
$headers .= "From: " .
"".mb_encode_mimeheader (mb_convert_encoding($from_name,"ISO-2022-JP","AUTO")) ."" .
"<".$from_email."> \n";
$headers .= "Reply-To: " .
"".mb_encode_mimeheader (mb_convert_encoding($from_name,"ISO-2022-JP","AUTO")) ."" .
"<".$from_email."> \n";
$headers .= "Content-Type: text/plain;charset=ISO-2022-JP \n";
/* Convert body to same encoding as stated
in Content-Type header above */
$body = mb_convert_encoding($body, "ISO-2022-JP","AUTO");
/* Mail, optional parameters. */
$sendmail_params = "-f$from_email";
mb_language("ja");
$subject = mb_convert_encoding($subject, "ISO-2022-JP","AUTO");
$subject = mb_encode_mimeheader($subject);
$result = mail($to, $subject, $body, $headers, $sendmail_params);
return $result;
}
Introduction of Japanese encoding to e-mail happened at JUNET(UUCP based nation-wide network) in early 90's.
At that time, RFC1468 was defined.
If you follow RFC1468 in plain text mail, there would be no problem.
If you want to handle html mail, RFC1468 is useless except for header parts.
Here's what I use to send Japanese emails. Subject line looks fine in Outlook 2010, gmail and on iPhone.
Encoding encoding = Encoding.GetEncoding("iso-2022-jp");
byte[] bytes = encoding.GetBytes(subject);
string uuEncoded = Convert.ToBase64String(bytes);
subject = "=?iso-2022-jp?B?" + uuEncoded + "?=";
// not sure this is actually necessary...
mailMessage.SubjectEncoding = Encoding.GetEncoding("iso-2022-jp");
I'm using the mail function on php to send a confirmation email. It actually works, but just for some mails. For example, if you recieve that on a gmail it's fine, but on a college mail it appears ? instead of special characters.
The problem is that that mail includes a validation link,which looks like www.myweb.com/confirmation.php?passkey=(passkey) and if the mail and the special characters aren't send properly this link is also wrong (the = doesn't appear).
I've already tried writting
iconv_set_encoding("internal_encoding", "UTF-8");
on top of the mail function, but it doesn't work. What can it be?
Thanks for your help!
Try utf-8 encoding in header
For example:
$headers .= "Content-Type: text/html; charset=utf-8\r\n";
Character sets are driving me round the bend!
My database is utf8_general_ci and the tables within it are utf8_unicode_ci.
All my PHP pages have
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
in the head.
When I type a euro symbol (€) in the PHP page and input it to the database it appears in phpMyAdmin as €
When I use a PHP page to copy it from the database, it reappears as €
So far so good, but when I write am html email using PHP, no matter what I do it comes out in the email as €
I've tried specifying the character set in the html email but it doesn't work. Probably because email clients take their character set from the mail server and not the headers.
I've also got issues with accented letters and the like, but they are being written into the text so I could overcome them by using é for é, and so on. Messy, but not impossible.
Surely there's a better way!
MY SOLUTION (SORT OF)
Thanks to all who contributed. I have tried all ways to specify the character set to utf-8 (in the mail headers, in the html head, and both) but the message still arrives in iso. So it seems the information I got from elsewhere was right: character set is defined by the server.
I have had to settle for typing things like é each time I want é into the fixed text, which is cumbersome but at least it works.
For the euro symbol, I have wrapped the variable in the htmlentities function. It works, but I will have to remember to do it with any other variables taken from the database if I encounter similar problems in similar files. It would have been easier to wrap the function around the entire html body but that doesn't work, presumably because it does funny things with the .
Check the encoding in your text editor. Crazy things can happen if this is wrong!
For MySQL, see mysql_set_charset.
Emails actually get their character set from the Content-Type header, not from the email server. Make sure you set this header to an appropriate value, such as Content-Type: text/html; charset="UTF-8". See also this question.
And keep in mind that for HTML, you can use $encoded = htmlentities( $string, ENT_QUOTES, 'utf-8' ) so that all characters which have HTML character entity equivalents are translated into these entities.
I guess that the correct answer for you is just setting utf-8 charset for the e-mail:
$headers = "MIME-Version: 1.0\r\n";
$headers.= "From: =?utf-8?b?".base64_encode($from_name)."?= <".$from_a.">\r\n";
$headers.= "Content-Type: text/plain;charset=utf-8\r\n";
$headers.= "Reply-To: $reply\r\n";
$headers.= "X-Mailer: PHP/" . phpversion();
mail($to, $s, $body, $headers);
If you open the email source in your client (ctrl+u in thunderbird). You will see a Content-Type header. This should be something like:
Content-Type: text/html; charset=utf-8
If your email contains multiple parts your need to add that header to html part.
Header values need to be encoded separately (each line).
Subject: =?utf-8?B?...?=
For the html content you can just use htmlentities() but this will not work for the headers or a text email.
I'm trying to send these characters through PHP:
áéíóúüchlñÁÉÍÓÚÜCLÑ
They show up in the received email like this:
áéÃóúüchlñÃÃÃÃÃ
I tried htmlentities but without success:
$newsubject = htmlentities($subject, ENT_COMPAT, "UTF-8");
mail($notes,$newsubject,$message,$headers);
Does anybody have an idea what I could try?
Thanks
I think, you need to use MIME (Multipurpose Internet Mail Extensions).
Add your mail headers the following:
MIME-Version: 1.0
Content-Type: text/plain;charset=utf-8
You are attempting to send them as UTF-8 but your PHP is handling them as latin-1.
Call utf8_encode on the input string to treat it as UTF-8 again.
EDIT: Misread the question. Add a header to the email you're sending:
Content-Type: text/plain; charset=utf-8
Your character set is wrong on the characters themselves. Try this: (Windows) Copy and paste those characters from a "UTF-8 character set" site back into your application. Make sure your doucument is UTF-8, and BOM Signature disabled.
For some time now I've had the problem of some of my users getting =0A=0A instead of new lines in emails I send to them via PHP. Correspondence via email client works well, but PHP generated emails always look like this with some users (a minority). Googling revealed no decent results, all search results seem to be connected with outlook somehow - and it is unacceptable to think that all outlook users would suffer from this problem. Does anyone know a correct way of handling this and avoiding these new line encoding issues?
Edit: FYI I'm using Zend's Mailer class.
Thanks
Edit 2:
Changing the encoding type did not work. I encoded the headers to base64, and the body to 64, got garbled stuff. Then I tried with base64 headers, and did base64_decode(base64_decode($body)) on the body, and that was fine on the user's "CNR Server but not in the inbox" whatever that means. When I tried mb_convert_encoding to base64, I got the encoded string instead of the body again, so no use.
What else can I try? Zend Mailer only supports Quoted Printable and Base64 header encoding. Not sure what to do to the body for it to match the quoted printable encoding...
The email body has been encoded using quoted-printable - but the mime type declared in the email is text/html (or text/plain or undefined).
How you make the encoding of the body of the email match the mime header is up to you.