I have long enough string (5000+ chars), and I need to check if it is in English.
After brief web search I found several solutions:
using of PEAR Text_LanguageDetect (it looks attractive but I'm still avoiding solutions which I don't understand how thet works)
check letters frequency (I made a function below with some comments)
check the string for national charecters (like č, ß and so on)
check the string for markers like 'is', 'the' or anything
So the function is the following:
function is_english($str){
// Most used English chars frequencies
$chars = array(
array('e',12.702),
array('t', 9.056),
array('a', 8.167),
array('o', 7.507),
array('i', 6.966),
array('n', 6.749),
array('s', 6.327),
array('h', 6.094),
array('r', 5.987),
);
$str = strtolower($str);
$sum = 0;
foreach($chars as $key=>$char){
$i = substr_count($str,$char[0]);
$i = 100*$i/strlen($str); // Normalization
$i = $i/$char[1];
$sum += $i;
}
$avg = $sum/count($chars);
// Calculation of mean square value
$value = 0;
foreach($chars as $char)
$value += pow($char[2]-$avg,2);
// Average value
$value = $value / count($chars);
return $value;
}
Generally this function estimates the chars frequency and compares it with given pattern. Result should be closer to 0 as the frequency closer the pattern.
Unfortunately it working not as good: mostly I could consider that results 0.05 and lower is English and higher is not. But there are many English strings have high values and many foreign (in my case mostly German) - low.
I can't implement Third solution yet as I wasn't able to find any comprehensive chars set - foreign language markers.
The forth looks attractive but I can not figure out which marker is best to be used.
Any thoughts?
PS After some discussion Zod proposed that this question is duplicate to question Regular expression to match non-English characters?, which answers only in part. So I'd like to keep this question independent.
I think the fourth solution might be your best bet, but I would expand it to include a wider dictionary.
You can find some comprehensive lists at: https://en.wikipedia.org/wiki/Most_common_words_in_English
With your current implementation, you will suffer some setbacks because many languages use the standard latin alphabet. Even languages that go beyond the standard latin alphabet typically use primarily "English-compliant characters," so to speak. For example, the sentence "Ich bin lustig" is German, but uses only latin alphabetic characters. Likewise, "Jeg er glad" is Danish, but uses only latin alphabetic characters. Of course, in a string of 5000+ characters, you will probably see some non-latin characters, but that is not guaranteed. Additionally, but focusing solely on character frequency, you might find that foreign languages which utilize the latin alphabet typically have similar character occurrence frequencies, thus rendering your existing solution ineffective.
By using an english dictionary to find occurrences of English words, you would be able to look over a string and determine exactly how many of the words are English, and from there, calculate a frequency of the number of words that are English. (With a higher percentage indicating the sentence is probably English.)
The following is a potential solution:
<?php
$testString = "Some long string of text that you would like to test.";
// Words from: https://en.wikipedia.org/wiki/Most_common_words_in_English
$common_english_words = array('time', 'person', 'year', 'way', 'day', 'thing', 'man', 'world', 'life', 'hand', 'part', 'child', 'eye', 'woman', 'place', 'work', 'week', 'case', 'point', 'government', 'company', 'number', 'group', 'problem', 'fact', 'be', 'have', 'do', 'say', 'get', 'make', 'go', 'know', 'take', 'see', 'come', 'think', 'look', 'want', 'give', 'use', 'find', 'tell', 'ask', 'seem', 'feel', 'try', 'leave', 'call', 'good', 'new', 'first', 'last', 'long', 'great', 'little', 'own', 'other', 'old', 'right', 'big', 'high', 'different', 'small', 'large', 'next', 'early', 'young', 'important', 'few', 'public', 'bad', 'same', 'able', 'to', 'of', 'in', 'for', 'on', 'with', 'at', 'by', 'from', 'up', 'about', 'into', 'over', 'after', 'beneath', 'under', 'above', 'the', 'and', 'a', 'that', 'i', 'it', 'not', 'he', 'as', 'you', 'this', 'but', 'his', 'they', 'her', 'she', 'or', 'an', 'will', 'my', 'one', 'all', 'would', 'there', 'their', 'I', 'we', 'what', 'so', 'out', 'if', 'who', 'which', 'me', 'when', 'can', 'like', 'no', 'just', 'him', 'people', 'your', 'some', 'could', 'them', 'than', 'then', 'now', 'only', 'its', 'also', 'back', 'two', 'how', 'our', 'well', 'even', 'because', 'any', 'these', 'most', 'us');
/* you might also consider replacing "'s" with ' ', because 's is common in English
as a contraction and simply removing the single quote could throw off the frequency. */
$transformedTest = preg_replace('#\s+#', ' ', preg_replace("#[^a-zA-Z'\s]#", ' ', strtolower($testString)));
$splitTest = explode(' ', $transformedTest);
$matchCount = 0;
for($i=0;$i<count($splitTest);$i++){
if(in_array($splitTest[$i], $common_english_words))
$matchCount++;
}
echo "raw count: $matchCount\n<br>\nPercent: " . ($matchCount/count($common_english_words))*100 . "%\n<br>\n";
if(($matchCount/count($common_english_words)) > 0.5){
echo "More than half of the test string is English. Text is likely English.";
}else{
echo "Text is likely a foreign language.";
}
?>
You can see an example here which includes two sample strings to test (one which is German, and one which is English): https://ideone.com/lfYcs2
In the IDEOne code, when running it on the English string, you will see that the result is roughly 69.3% matching with the common English words. When running it on the German, the match percentage is only 4.57% matching with the common English words.
This problem is called language detection and is not trivial to solve with a single function. I suggest you use LanguageDetector from github.
i would go with the fourth solution and try to also search for not englisch. For Example if you find "the" then high posibility for english. If you find "el" or "la" then the posibility is high for spanish. I would search for "der","die"and "das" then it is very posible that it is German.
I'm searching for a function that can transform or convert any characters (from all languages) to English characters.
I saw mb_convert_encoding, iconv('UTF-8', 'ASCII//TRANSLIT', $str) , ....
I probably miss something...
Example : I want to go from '后杨庄' to 'hou yang zhuang'
but iconv echo nothing ;( and mb_convert echo 后杨庄.
Is someone knows how to better use mb_convert or iconv ?
or is there a better PHP function that can achieve this ?
Thanks to answer
you mean romanize (right?), because translation is one thing, and convert encoding another, and romanization a third one
Korean Syllable PHP or JavaScript Romanizer (or any other alternative?)
I finally found Transliterator plugin for PHP
And with combination with iconv I can remove all accents from all languages
$transliterator = Transliterator::create( "Any-Latin; [:Punctuation:] Remove; NFD; NFC;" );
$jStringAccentsRemoved = $transliterator->transliterate($jStringToRemoveAccents);
$jStringAccentsRemoved = iconv('UTF-8', 'us-ascii//TRANSLIT', $jStringAccentsRemoved);
* IMPROVED CODE (but PECL intl and PECL translit required) *
$tranliterare_Parameters = "Any-Latin; NFD; NFC; NFKD; NFKC; ";
$transliterator = Transliterator::create( $tranliterare_Parameters );
$stringToClean = $transliterator->transliterate($stringToClean);
$stringToClean = transliterate( $stringToClean, array( 'han_transliterate', 'diacritical_remove' ), 'utf-8', 'utf-8');
$stringToClean = preg_replace('/\p{Mn}/u', '', Normalizer::normalize($stringToClean, Normalizer::FORM_KD));
// this extra portion is from http://d3s.mff.cuni.cz/~holub/sw/phpaccents/
$unwanted_array = array(
'œ' => 'oe' , 'æ' => 'ae' , 'đ' => 'd' , 'ð' => 'd' , 'ø' => 'o' , 'ł' => 'l' , 'ß' => 'ss' ,
'Œ' => 'OE' , 'Æ' => 'AE' , 'Đ' => 'D' , 'Ð' => 'D' , 'Ø' => 'O' , 'Ł' => 'L' ,
'Ə' => 'E', 'ə' => 'e', 'Ǝ' => 'E', 'ə' => 'e','ı' => 'i','I' => 'i',
'`' => '' , '?' => '' , ' ' => ' ' , '"' => '' , "'" => '' ,
'À'=>'A','Á'=>'A','Â'=>'A','Ã'=>'A','Ä'=>'A','Å'=>'A','Ç'=>'C','È'=>'E',
'É'=>'E','Ê'=>'E','Ë'=>'E','Ì'=>'I','Í'=>'I','Î'=>'I','Ï'=>'I','Ñ'=>'N',
'Ò'=>'O','Ó'=>'O','Ô'=>'O','Õ'=>'O','Ö'=>'O','Ø'=>'O','Ù'=>'U','Ú'=>'U',
'Û'=>'U','Ü'=>'U','Ý'=>'Y','à'=>'a','á'=>'a','â'=>'a','ã'=>'a','ä'=>'a',
'å'=>'a','ç'=>'c','è'=>'e','é'=>'e','ê'=>'e','ë'=>'e','ì'=>'i','í'=>'i',
'î'=>'i','ï'=>'i','ñ'=>'n','ò'=>'o','ó'=>'o','ô'=>'o','õ'=>'o','ö'=>'o',
'ø'=>'o','ù'=>'u','ú'=>'u','û'=>'u','ü'=>'u','ý'=>'y','ÿ'=>'y','Ā'=>'A',
'ā'=>'a','Ă'=>'A','ă'=>'a','Ą'=>'A','ą'=>'a','Ć'=>'C','ć'=>'c','Ĉ'=>'C',
'ĉ'=>'c','Ċ'=>'C','ċ'=>'c','Č'=>'C','č'=>'c','Ď'=>'D','ď'=>'d','Đ'=>'D',
'đ'=>'d','Ē'=>'E','ē'=>'e','Ĕ'=>'E','ĕ'=>'e','Ė'=>'E','ė'=>'e','Ę'=>'E',
'ę'=>'e','Ě'=>'E','ě'=>'e','Ĝ'=>'G','ĝ'=>'g','Ğ'=>'G','ğ'=>'g','Ġ'=>'G',
'ġ'=>'g','Ģ'=>'G','ģ'=>'g','Ĥ'=>'H','ĥ'=>'h','Ħ'=>'H','ħ'=>'h','Ĩ'=>'I',
'ĩ'=>'i','Ī'=>'I','ī'=>'i','Ĭ'=>'I','ĭ'=>'i','Į'=>'I','į'=>'i','İ'=>'I',
'ı'=>'i','Ĵ'=>'J','ĵ'=>'j','Ķ'=>'K','ķ'=>'k','Ĺ'=>'L','ĺ'=>'l','Ļ'=>'L',
'ļ'=>'l','Ľ'=>'L','ľ'=>'l','Ŀ'=>'L','ŀ'=>'l','Ł'=>'L','ł'=>'l','Ń'=>'N',
'ń'=>'n','Ņ'=>'N','ņ'=>'n','Ň'=>'N','ň'=>'n','ʼn'=>'n','Ō'=>'O','ō'=>'o',
'Ŏ'=>'O','ŏ'=>'o','Ő'=>'O','ő'=>'o','Ŕ'=>'R','ŕ'=>'r','Ŗ'=>'R','ŗ'=>'r',
'Ř'=>'R','ř'=>'r','Ś'=>'S','ś'=>'s','Ŝ'=>'S','ŝ'=>'s','Ş'=>'S','ş'=>'s',
'Š'=>'S','š'=>'s','Ţ'=>'T','ţ'=>'t','Ť'=>'T','ť'=>'t','Ŧ'=>'T','ŧ'=>'t',
'Ũ'=>'U','ũ'=>'u','Ū'=>'U','ū'=>'u','Ŭ'=>'U','ŭ'=>'u','Ů'=>'U','ů'=>'u',
'Ű'=>'U','ű'=>'u','Ų'=>'U','ų'=>'u','Ŵ'=>'W','ŵ'=>'w','Ŷ'=>'Y','ŷ'=>'y',
'Ÿ'=>'Y','Ź'=>'Z','ź'=>'z','Ż'=>'Z','ż'=>'z','Ž'=>'Z','ž'=>'z','ƀ'=>'b',
'Ɓ'=>'B','Ƃ'=>'B','ƃ'=>'b','Ƈ'=>'C','ƈ'=>'c','Ɗ'=>'D','Ƌ'=>'D','ƌ'=>'d',
'Ƒ'=>'F','ƒ'=>'f','Ɠ'=>'G','Ɨ'=>'I','Ƙ'=>'K','ƙ'=>'k','ƚ'=>'l','Ɲ'=>'N',
'ƞ'=>'n','Ɵ'=>'O','Ơ'=>'O','ơ'=>'o','Ƥ'=>'P','ƥ'=>'p','ƫ'=>'t','Ƭ'=>'T',
'ƭ'=>'t','Ʈ'=>'T','Ư'=>'U','ư'=>'u','Ʋ'=>'V','Ƴ'=>'Y','ƴ'=>'y','Ƶ'=>'Z',
'ƶ'=>'z','Dž'=>'D','Lj'=>'L','Nj'=>'N','Ǎ'=>'A','ǎ'=>'a','Ǐ'=>'I','ǐ'=>'i',
'Ǒ'=>'O','ǒ'=>'o','Ǔ'=>'U','ǔ'=>'u','Ǖ'=>'U','ǖ'=>'u','Ǘ'=>'U','ǘ'=>'u',
'Ǚ'=>'U','ǚ'=>'u','Ǜ'=>'U','ǜ'=>'u','Ǟ'=>'A','ǟ'=>'a','Ǡ'=>'A','ǡ'=>'a',
'Ǥ'=>'G','ǥ'=>'g','Ǧ'=>'G','ǧ'=>'g','Ǩ'=>'K','ǩ'=>'k','Ǫ'=>'O','ǫ'=>'o',
'Ǭ'=>'O','ǭ'=>'o','ǰ'=>'j','Dz'=>'D','Ǵ'=>'G','ǵ'=>'g','Ǹ'=>'N','ǹ'=>'n',
'Ǻ'=>'A','ǻ'=>'a','Ǿ'=>'O','ǿ'=>'o','Ȁ'=>'A','ȁ'=>'a','Ȃ'=>'A','ȃ'=>'a',
'Ȅ'=>'E','ȅ'=>'e','Ȇ'=>'E','ȇ'=>'e','Ȉ'=>'I','ȉ'=>'i','Ȋ'=>'I','ȋ'=>'i',
'Ȍ'=>'O','ȍ'=>'o','Ȏ'=>'O','ȏ'=>'o','Ȑ'=>'R','ȑ'=>'r','Ȓ'=>'R','ȓ'=>'r',
'Ȕ'=>'U','ȕ'=>'u','Ȗ'=>'U','ȗ'=>'u','Ș'=>'S','ș'=>'s','Ț'=>'T','ț'=>'t',
'Ȟ'=>'H','ȟ'=>'h','Ƞ'=>'N','ȡ'=>'d','Ȥ'=>'Z','ȥ'=>'z','Ȧ'=>'A','ȧ'=>'a',
'Ȩ'=>'E','ȩ'=>'e','Ȫ'=>'O','ȫ'=>'o','Ȭ'=>'O','ȭ'=>'o','Ȯ'=>'O','ȯ'=>'o',
'Ȱ'=>'O','ȱ'=>'o','Ȳ'=>'Y','ȳ'=>'y','ȴ'=>'l','ȵ'=>'n','ȶ'=>'t','ȷ'=>'j',
'Ⱥ'=>'A','Ȼ'=>'C','ȼ'=>'c','Ƚ'=>'L','Ⱦ'=>'T','ȿ'=>'s','ɀ'=>'z','Ƀ'=>'B',
'Ʉ'=>'U','Ɇ'=>'E','ɇ'=>'e','Ɉ'=>'J','ɉ'=>'j','ɋ'=>'q','Ɍ'=>'R','ɍ'=>'r',
'Ɏ'=>'Y','ɏ'=>'y','ɓ'=>'b','ɕ'=>'c','ɖ'=>'d','ɗ'=>'d','ɟ'=>'j','ɠ'=>'g',
'ɦ'=>'h','ɨ'=>'i','ɫ'=>'l','ɬ'=>'l','ɭ'=>'l','ɱ'=>'m','ɲ'=>'n','ɳ'=>'n',
'ɵ'=>'o','ɼ'=>'r','ɽ'=>'r','ɾ'=>'r','ʂ'=>'s','ʄ'=>'j','ʈ'=>'t','ʉ'=>'u',
'ʋ'=>'v','ʐ'=>'z','ʑ'=>'z','ʝ'=>'j','ʠ'=>'q','ͣ'=>'a','ͤ'=>'e','ͥ'=>'i',
'ͦ'=>'o','ͧ'=>'u','ͨ'=>'c','ͩ'=>'d','ͪ'=>'h','ͫ'=>'m','ͬ'=>'r','ͭ'=>'t',
'ͮ'=>'v','ͯ'=>'x','ᵢ'=>'i','ᵣ'=>'r','ᵤ'=>'u','ᵥ'=>'v','ᵬ'=>'b','ᵭ'=>'d',
'ᵮ'=>'f','ᵯ'=>'m','ᵰ'=>'n','ᵱ'=>'p','ᵲ'=>'r','ᵳ'=>'r','ᵴ'=>'s','ᵵ'=>'t',
'ᵶ'=>'z','ᵻ'=>'i','ᵽ'=>'p','ᵾ'=>'u','ᶀ'=>'b','ᶁ'=>'d','ᶂ'=>'f','ᶃ'=>'g',
'ᶄ'=>'k','ᶅ'=>'l','ᶆ'=>'m','ᶇ'=>'n','ᶈ'=>'p','ᶉ'=>'r','ᶊ'=>'s','ᶌ'=>'v',
'ᶍ'=>'x','ᶎ'=>'z','ᶏ'=>'a','ᶑ'=>'d','ᶒ'=>'e','ᶖ'=>'i','ᶙ'=>'u','᷊'=>'r',
'ᷗ'=>'c','ᷚ'=>'g','ᷜ'=>'k','ᷝ'=>'l','ᷠ'=>'n','ᷣ'=>'r','ᷤ'=>'s','ᷦ'=>'z',
'Ḁ'=>'A','ḁ'=>'a','Ḃ'=>'B','ḃ'=>'b','Ḅ'=>'B','ḅ'=>'b','Ḇ'=>'B','ḇ'=>'b',
'Ḉ'=>'C','ḉ'=>'c','Ḋ'=>'D','ḋ'=>'d','Ḍ'=>'D','ḍ'=>'d','Ḏ'=>'D','ḏ'=>'d',
'Ḑ'=>'D','ḑ'=>'d','Ḓ'=>'D','ḓ'=>'d','Ḕ'=>'E','ḕ'=>'e','Ḗ'=>'E','ḗ'=>'e',
'Ḙ'=>'E','ḙ'=>'e','Ḛ'=>'E','ḛ'=>'e','Ḝ'=>'E','ḝ'=>'e','Ḟ'=>'F','ḟ'=>'f',
'Ḡ'=>'G','ḡ'=>'g','Ḣ'=>'H','ḣ'=>'h','Ḥ'=>'H','ḥ'=>'h','Ḧ'=>'H','ḧ'=>'h',
'Ḩ'=>'H','ḩ'=>'h','Ḫ'=>'H','ḫ'=>'h','Ḭ'=>'I','ḭ'=>'i','Ḯ'=>'I','ḯ'=>'i',
'Ḱ'=>'K','ḱ'=>'k','Ḳ'=>'K','ḳ'=>'k','Ḵ'=>'K','ḵ'=>'k','Ḷ'=>'L','ḷ'=>'l',
'Ḹ'=>'L','ḹ'=>'l','Ḻ'=>'L','ḻ'=>'l','Ḽ'=>'L','ḽ'=>'l','Ḿ'=>'M','ḿ'=>'m',
'Ṁ'=>'M','ṁ'=>'m','Ṃ'=>'M','ṃ'=>'m','Ṅ'=>'N','ṅ'=>'n','Ṇ'=>'N','ṇ'=>'n',
'Ṉ'=>'N','ṉ'=>'n','Ṋ'=>'N','ṋ'=>'n','Ṍ'=>'O','ṍ'=>'o','Ṏ'=>'O','ṏ'=>'o',
'Ṑ'=>'O','ṑ'=>'o','Ṓ'=>'O','ṓ'=>'o','Ṕ'=>'P','ṕ'=>'p','Ṗ'=>'P','ṗ'=>'p',
'Ṙ'=>'R','ṙ'=>'r','Ṛ'=>'R','ṛ'=>'r','Ṝ'=>'R','ṝ'=>'r','Ṟ'=>'R','ṟ'=>'r',
'Ṡ'=>'S','ṡ'=>'s','Ṣ'=>'S','ṣ'=>'s','Ṥ'=>'S','ṥ'=>'s','Ṧ'=>'S','ṧ'=>'s',
'Ṩ'=>'S','ṩ'=>'s','Ṫ'=>'T','ṫ'=>'t','Ṭ'=>'T','ṭ'=>'t','Ṯ'=>'T','ṯ'=>'t',
'Ṱ'=>'T','ṱ'=>'t','Ṳ'=>'U','ṳ'=>'u','Ṵ'=>'U','ṵ'=>'u','Ṷ'=>'U','ṷ'=>'u',
'Ṹ'=>'U','ṹ'=>'u','Ṻ'=>'U','ṻ'=>'u','Ṽ'=>'V','ṽ'=>'v','Ṿ'=>'V','ṿ'=>'v',
'Ẁ'=>'W','ẁ'=>'w','Ẃ'=>'W','ẃ'=>'w','Ẅ'=>'W','ẅ'=>'w','Ẇ'=>'W','ẇ'=>'w',
'Ẉ'=>'W','ẉ'=>'w','Ẋ'=>'X','ẋ'=>'x','Ẍ'=>'X','ẍ'=>'x','Ẏ'=>'Y','ẏ'=>'y',
'Ẑ'=>'Z','ẑ'=>'z','Ẓ'=>'Z','ẓ'=>'z','Ẕ'=>'Z','ẕ'=>'z','ẖ'=>'h','ẗ'=>'t',
'ẘ'=>'w','ẙ'=>'y','ẚ'=>'a','Ạ'=>'A','ạ'=>'a','Ả'=>'A','ả'=>'a','Ấ'=>'A',
'ấ'=>'a','Ầ'=>'A','ầ'=>'a','Ẩ'=>'A','ẩ'=>'a','Ẫ'=>'A','ẫ'=>'a','Ậ'=>'A',
'ậ'=>'a','Ắ'=>'A','ắ'=>'a','Ằ'=>'A','ằ'=>'a','Ẳ'=>'A','ẳ'=>'a','Ẵ'=>'A',
'ẵ'=>'a','Ặ'=>'A','ặ'=>'a','Ẹ'=>'E','ẹ'=>'e','Ẻ'=>'E','ẻ'=>'e','Ẽ'=>'E',
'ẽ'=>'e','Ế'=>'E','ế'=>'e','Ề'=>'E','ề'=>'e','Ể'=>'E','ể'=>'e','Ễ'=>'E',
'ễ'=>'e','Ệ'=>'E','ệ'=>'e','Ỉ'=>'I','ỉ'=>'i','Ị'=>'I','ị'=>'i','Ọ'=>'O',
'ọ'=>'o','Ỏ'=>'O','ỏ'=>'o','Ố'=>'O','ố'=>'o','Ồ'=>'O','ồ'=>'o','Ổ'=>'O',
'ổ'=>'o','Ỗ'=>'O','ỗ'=>'o','Ộ'=>'O','ộ'=>'o','Ớ'=>'O','ớ'=>'o','Ờ'=>'O',
'ờ'=>'o','Ở'=>'O','ở'=>'o','Ỡ'=>'O','ỡ'=>'o','Ợ'=>'O','ợ'=>'o','Ụ'=>'U',
'ụ'=>'u','Ủ'=>'U','ủ'=>'u','Ứ'=>'U','ứ'=>'u','Ừ'=>'U','ừ'=>'u','Ử'=>'U',
'ử'=>'u','Ữ'=>'U','ữ'=>'u','Ự'=>'U','ự'=>'u','Ỳ'=>'Y','ỳ'=>'y','Ỵ'=>'Y',
'ỵ'=>'y','Ỷ'=>'Y','ỷ'=>'y','Ỹ'=>'Y','ỹ'=>'y','Ỿ'=>'Y','ỿ'=>'y','ⁱ'=>'i',
'ⁿ'=>'n','ₐ'=>'a','ₑ'=>'e','ₒ'=>'o','ₓ'=>'x','⒜'=>'a','⒝'=>'b','⒞'=>'c',
'⒟'=>'d','⒠'=>'e','⒡'=>'f','⒢'=>'g','⒣'=>'h','⒤'=>'i','⒥'=>'j','⒦'=>'k',
'⒧'=>'l','⒨'=>'m','⒩'=>'n','⒪'=>'o','⒫'=>'p','⒬'=>'q','⒭'=>'r','⒮'=>'s',
'⒯'=>'t','⒰'=>'u','⒱'=>'v','⒲'=>'w','⒳'=>'x','⒴'=>'y','⒵'=>'z','Ⓐ'=>'A',
'Ⓑ'=>'B','Ⓒ'=>'C','Ⓓ'=>'D','Ⓔ'=>'E','Ⓕ'=>'F','Ⓖ'=>'G','Ⓗ'=>'H','Ⓘ'=>'I',
'Ⓙ'=>'J','Ⓚ'=>'K','Ⓛ'=>'L','Ⓜ'=>'M','Ⓝ'=>'N','Ⓞ'=>'O','Ⓟ'=>'P','Ⓠ'=>'Q',
'Ⓡ'=>'R','Ⓢ'=>'S','Ⓣ'=>'T','Ⓤ'=>'U','Ⓥ'=>'V','Ⓦ'=>'W','Ⓧ'=>'X','Ⓨ'=>'Y',
'Ⓩ'=>'Z','ⓐ'=>'a','ⓑ'=>'b','ⓒ'=>'c','ⓓ'=>'d','ⓔ'=>'e','ⓕ'=>'f','ⓖ'=>'g',
'ⓗ'=>'h','ⓘ'=>'i','ⓙ'=>'j','ⓚ'=>'k','ⓛ'=>'l','ⓜ'=>'m','ⓝ'=>'n','ⓞ'=>'o',
'ⓟ'=>'p','ⓠ'=>'q','ⓡ'=>'r','ⓢ'=>'s','ⓣ'=>'t','ⓤ'=>'u','ⓥ'=>'v','ⓦ'=>'w',
'ⓧ'=>'x','ⓨ'=>'y','ⓩ'=>'z','Ⱡ'=>'L','ⱡ'=>'l','Ɫ'=>'L','Ᵽ'=>'P','Ɽ'=>'R',
'ⱥ'=>'a','ⱦ'=>'t','Ⱨ'=>'H','ⱨ'=>'h','Ⱪ'=>'K','ⱪ'=>'k','Ⱬ'=>'Z','ⱬ'=>'z',
'Ɱ'=>'M','ⱱ'=>'v','Ⱳ'=>'W','ⱳ'=>'w','ⱴ'=>'v','ⱸ'=>'e','ⱺ'=>'o','ⱼ'=>'j',
'Ꝁ'=>'K','ꝁ'=>'k','Ꝃ'=>'K','ꝃ'=>'k','Ꝅ'=>'K','ꝅ'=>'k','Ꝉ'=>'L','ꝉ'=>'l',
'Ꝋ'=>'O','ꝋ'=>'o','Ꝍ'=>'O','ꝍ'=>'o','Ꝑ'=>'P','ꝑ'=>'p','Ꝓ'=>'P','ꝓ'=>'p',
'Ꝕ'=>'P','ꝕ'=>'p','Ꝗ'=>'Q','ꝗ'=>'q','Ꝙ'=>'Q','ꝙ'=>'q','Ꝛ'=>'R','ꝛ'=>'r',
'Ꝟ'=>'V','ꝟ'=>'v','Ａ'=>'A','Ｂ'=>'B','Ｃ'=>'C','Ｄ'=>'D','Ｅ'=>'E','Ｆ'=>'F',
'Ｇ'=>'G','Ｈ'=>'H','Ｉ'=>'I','Ｊ'=>'J','Ｋ'=>'K','Ｌ'=>'L','Ｍ'=>'M','Ｎ'=>'N',
'Ｏ'=>'O','Ｐ'=>'P','Ｑ'=>'Q','Ｒ'=>'R','Ｓ'=>'S','Ｔ'=>'T','Ｕ'=>'U','Ｖ'=>'V',
'Ｗ'=>'W','Ｘ'=>'X','Ｙ'=>'Y','Ｚ'=>'Z','ａ'=>'a','ｂ'=>'b','ｃ'=>'c','ｄ'=>'d',
'ｅ'=>'e','ｆ'=>'f','ｇ'=>'g','ｈ'=>'h','ｉ'=>'i','ｊ'=>'j','ｋ'=>'k','ｌ'=>'l',
'ｍ'=>'m','ｎ'=>'n','ｏ'=>'o','ｐ'=>'p','ｑ'=>'q','ｒ'=>'r','ｓ'=>'s','ｔ'=>'t',
'ｕ'=>'u','ｖ'=>'v','ｗ'=>'w','ｘ'=>'x','ｙ'=>'y','ｚ'=>'z'
);
$stringToClean = strtr( $stringToClean , $unwanted_array );
The result :
FROM
后杨庄 • องค์การโทรศัพท์ร่อนพิบูลย àéç öî-ïüùç ËÀÌ --- ÀØėÿᾜὨζὅБю ---
Ⱥⱥ Ƀƀ Ȼȼ Đđ Ɇɇ Ǥǥ Ħħ Ɨɨ Ɉɉ Ꝁꝁ Łł Øø Ᵽᵽ Ɍɍ Ŧŧ Ʉʉ Ɏɏ Ƶƶ
TO
hou yang zhuang • xngkhkar thorsaphth rxnphibuly aec oi-iuuc EAi --- AOeyEiOzhoBu ---\
Aa Bb Cc Dd Ee GG Hh Ii Jj Kk Ll Oo Pp Rr Tt Uu Yy Zz
Is there a PHP function to convert named HTML entities into their respective numeric HTML entities?
For example:
$str = "Oggi è un bel giorno";
echo entities_to_unicode($str); // Oggi è un bel giorno
Thanks in advance, have a nice day!
You are looking for a simple translation function from named HTML entities to their numeric counterpart.
This can be done by using a translation table (that is an array) and the string translation function (strtr):
$translated = strtr($string, $HTML401NamedToNumeric);
This works for $string being UTF-8 encoded or of a single-byte character set.
Example array for the HTML 4.01 named entities as specified by W3C as used above follows. It contains 252 entities. If you want to support XHTML, then there is one more (I put it at the end):
$HTML401NamedToNumeric = array(
' ' => ' ', # no-break space = non-breaking space, U+00A0 ISOnum
'¡' => '¡', # inverted exclamation mark, U+00A1 ISOnum
'¢' => '¢', # cent sign, U+00A2 ISOnum
'£' => '£', # pound sign, U+00A3 ISOnum
'¤' => '¤', # currency sign, U+00A4 ISOnum
'¥' => '¥', # yen sign = yuan sign, U+00A5 ISOnum
'¦' => '¦', # broken bar = broken vertical bar, U+00A6 ISOnum
'§' => '§', # section sign, U+00A7 ISOnum
'¨' => '¨', # diaeresis = spacing diaeresis, U+00A8 ISOdia
'©' => '©', # copyright sign, U+00A9 ISOnum
'ª' => 'ª', # feminine ordinal indicator, U+00AA ISOnum
'«' => '«', # left-pointing double angle quotation mark = left pointing guillemet, U+00AB ISOnum
'¬' => '¬', # not sign, U+00AC ISOnum
'' => '', # soft hyphen = discretionary hyphen, U+00AD ISOnum
'®' => '®', # registered sign = registered trade mark sign, U+00AE ISOnum
'¯' => '¯', # macron = spacing macron = overline = APL overbar, U+00AF ISOdia
'°' => '°', # degree sign, U+00B0 ISOnum
'±' => '±', # plus-minus sign = plus-or-minus sign, U+00B1 ISOnum
'²' => '²', # superscript two = superscript digit two = squared, U+00B2 ISOnum
'³' => '³', # superscript three = superscript digit three = cubed, U+00B3 ISOnum
'´' => '´', # acute accent = spacing acute, U+00B4 ISOdia
'µ' => 'µ', # micro sign, U+00B5 ISOnum
'¶' => '¶', # pilcrow sign = paragraph sign, U+00B6 ISOnum
'·' => '·', # middle dot = Georgian comma = Greek middle dot, U+00B7 ISOnum
'¸' => '¸', # cedilla = spacing cedilla, U+00B8 ISOdia
'¹' => '¹', # superscript one = superscript digit one, U+00B9 ISOnum
'º' => 'º', # masculine ordinal indicator, U+00BA ISOnum
'»' => '»', # right-pointing double angle quotation mark = right pointing guillemet, U+00BB ISOnum
'¼' => '¼', # vulgar fraction one quarter = fraction one quarter, U+00BC ISOnum
'½' => '½', # vulgar fraction one half = fraction one half, U+00BD ISOnum
'¾' => '¾', # vulgar fraction three quarters = fraction three quarters, U+00BE ISOnum
'¿' => '¿', # inverted question mark = turned question mark, U+00BF ISOnum
'À' => 'À', # latin capital letter A with grave = latin capital letter A grave, U+00C0 ISOlat1
'Á' => 'Á', # latin capital letter A with acute, U+00C1 ISOlat1
'Â' => 'Â', # latin capital letter A with circumflex, U+00C2 ISOlat1
'Ã' => 'Ã', # latin capital letter A with tilde, U+00C3 ISOlat1
'Ä' => 'Ä', # latin capital letter A with diaeresis, U+00C4 ISOlat1
'Å' => 'Å', # latin capital letter A with ring above = latin capital letter A ring, U+00C5 ISOlat1
'Æ' => 'Æ', # latin capital letter AE = latin capital ligature AE, U+00C6 ISOlat1
'Ç' => 'Ç', # latin capital letter C with cedilla, U+00C7 ISOlat1
'È' => 'È', # latin capital letter E with grave, U+00C8 ISOlat1
'É' => 'É', # latin capital letter E with acute, U+00C9 ISOlat1
'Ê' => 'Ê', # latin capital letter E with circumflex, U+00CA ISOlat1
'Ë' => 'Ë', # latin capital letter E with diaeresis, U+00CB ISOlat1
'Ì' => 'Ì', # latin capital letter I with grave, U+00CC ISOlat1
'Í' => 'Í', # latin capital letter I with acute, U+00CD ISOlat1
'Î' => 'Î', # latin capital letter I with circumflex, U+00CE ISOlat1
'Ï' => 'Ï', # latin capital letter I with diaeresis, U+00CF ISOlat1
'Ð' => 'Ð', # latin capital letter ETH, U+00D0 ISOlat1
'Ñ' => 'Ñ', # latin capital letter N with tilde, U+00D1 ISOlat1
'Ò' => 'Ò', # latin capital letter O with grave, U+00D2 ISOlat1
'Ó' => 'Ó', # latin capital letter O with acute, U+00D3 ISOlat1
'Ô' => 'Ô', # latin capital letter O with circumflex, U+00D4 ISOlat1
'Õ' => 'Õ', # latin capital letter O with tilde, U+00D5 ISOlat1
'Ö' => 'Ö', # latin capital letter O with diaeresis, U+00D6 ISOlat1
'×' => '×', # multiplication sign, U+00D7 ISOnum
'Ø' => 'Ø', # latin capital letter O with stroke = latin capital letter O slash, U+00D8 ISOlat1
'Ù' => 'Ù', # latin capital letter U with grave, U+00D9 ISOlat1
'Ú' => 'Ú', # latin capital letter U with acute, U+00DA ISOlat1
'Û' => 'Û', # latin capital letter U with circumflex, U+00DB ISOlat1
'Ü' => 'Ü', # latin capital letter U with diaeresis, U+00DC ISOlat1
'Ý' => 'Ý', # latin capital letter Y with acute, U+00DD ISOlat1
'Þ' => 'Þ', # latin capital letter THORN, U+00DE ISOlat1
'ß' => 'ß', # latin small letter sharp s = ess-zed, U+00DF ISOlat1
'à' => 'à', # latin small letter a with grave = latin small letter a grave, U+00E0 ISOlat1
'á' => 'á', # latin small letter a with acute, U+00E1 ISOlat1
'â' => 'â', # latin small letter a with circumflex, U+00E2 ISOlat1
'ã' => 'ã', # latin small letter a with tilde, U+00E3 ISOlat1
'ä' => 'ä', # latin small letter a with diaeresis, U+00E4 ISOlat1
'å' => 'å', # latin small letter a with ring above = latin small letter a ring, U+00E5 ISOlat1
'æ' => 'æ', # latin small letter ae = latin small ligature ae, U+00E6 ISOlat1
'ç' => 'ç', # latin small letter c with cedilla, U+00E7 ISOlat1
'è' => 'è', # latin small letter e with grave, U+00E8 ISOlat1
'é' => 'é', # latin small letter e with acute, U+00E9 ISOlat1
'ê' => 'ê', # latin small letter e with circumflex, U+00EA ISOlat1
'ë' => 'ë', # latin small letter e with diaeresis, U+00EB ISOlat1
'ì' => 'ì', # latin small letter i with grave, U+00EC ISOlat1
'í' => 'í', # latin small letter i with acute, U+00ED ISOlat1
'î' => 'î', # latin small letter i with circumflex, U+00EE ISOlat1
'ï' => 'ï', # latin small letter i with diaeresis, U+00EF ISOlat1
'ð' => 'ð', # latin small letter eth, U+00F0 ISOlat1
'ñ' => 'ñ', # latin small letter n with tilde, U+00F1 ISOlat1
'ò' => 'ò', # latin small letter o with grave, U+00F2 ISOlat1
'ó' => 'ó', # latin small letter o with acute, U+00F3 ISOlat1
'ô' => 'ô', # latin small letter o with circumflex, U+00F4 ISOlat1
'õ' => 'õ', # latin small letter o with tilde, U+00F5 ISOlat1
'ö' => 'ö', # latin small letter o with diaeresis, U+00F6 ISOlat1
'÷' => '÷', # division sign, U+00F7 ISOnum
'ø' => 'ø', # latin small letter o with stroke, = latin small letter o slash, U+00F8 ISOlat1
'ù' => 'ù', # latin small letter u with grave, U+00F9 ISOlat1
'ú' => 'ú', # latin small letter u with acute, U+00FA ISOlat1
'û' => 'û', # latin small letter u with circumflex, U+00FB ISOlat1
'ü' => 'ü', # latin small letter u with diaeresis, U+00FC ISOlat1
'ý' => 'ý', # latin small letter y with acute, U+00FD ISOlat1
'þ' => 'þ', # latin small letter thorn, U+00FE ISOlat1
'ÿ' => 'ÿ', # latin small letter y with diaeresis, U+00FF ISOlat1
'ƒ' => 'ƒ', # latin small f with hook = function = florin, U+0192 ISOtech
'Α' => 'Α', # greek capital letter alpha, U+0391
'Β' => 'Β', # greek capital letter beta, U+0392
'Γ' => 'Γ', # greek capital letter gamma, U+0393 ISOgrk3
'Δ' => 'Δ', # greek capital letter delta, U+0394 ISOgrk3
'Ε' => 'Ε', # greek capital letter epsilon, U+0395
'Ζ' => 'Ζ', # greek capital letter zeta, U+0396
'Η' => 'Η', # greek capital letter eta, U+0397
'Θ' => 'Θ', # greek capital letter theta, U+0398 ISOgrk3
'Ι' => 'Ι', # greek capital letter iota, U+0399
'Κ' => 'Κ', # greek capital letter kappa, U+039A
'Λ' => 'Λ', # greek capital letter lambda, U+039B ISOgrk3
'Μ' => 'Μ', # greek capital letter mu, U+039C
'Ν' => 'Ν', # greek capital letter nu, U+039D
'Ξ' => 'Ξ', # greek capital letter xi, U+039E ISOgrk3
'Ο' => 'Ο', # greek capital letter omicron, U+039F
'Π' => 'Π', # greek capital letter pi, U+03A0 ISOgrk3
'Ρ' => 'Ρ', # greek capital letter rho, U+03A1
'Σ' => 'Σ', # greek capital letter sigma, U+03A3 ISOgrk3
'Τ' => 'Τ', # greek capital letter tau, U+03A4
'Υ' => 'Υ', # greek capital letter upsilon, U+03A5 ISOgrk3
'Φ' => 'Φ', # greek capital letter phi, U+03A6 ISOgrk3
'Χ' => 'Χ', # greek capital letter chi, U+03A7
'Ψ' => 'Ψ', # greek capital letter psi, U+03A8 ISOgrk3
'Ω' => 'Ω', # greek capital letter omega, U+03A9 ISOgrk3
'α' => 'α', # greek small letter alpha, U+03B1 ISOgrk3
'β' => 'β', # greek small letter beta, U+03B2 ISOgrk3
'γ' => 'γ', # greek small letter gamma, U+03B3 ISOgrk3
'δ' => 'δ', # greek small letter delta, U+03B4 ISOgrk3
'ε' => 'ε', # greek small letter epsilon, U+03B5 ISOgrk3
'ζ' => 'ζ', # greek small letter zeta, U+03B6 ISOgrk3
'η' => 'η', # greek small letter eta, U+03B7 ISOgrk3
'θ' => 'θ', # greek small letter theta, U+03B8 ISOgrk3
'ι' => 'ι', # greek small letter iota, U+03B9 ISOgrk3
'κ' => 'κ', # greek small letter kappa, U+03BA ISOgrk3
'λ' => 'λ', # greek small letter lambda, U+03BB ISOgrk3
'μ' => 'μ', # greek small letter mu, U+03BC ISOgrk3
'ν' => 'ν', # greek small letter nu, U+03BD ISOgrk3
'ξ' => 'ξ', # greek small letter xi, U+03BE ISOgrk3
'ο' => 'ο', # greek small letter omicron, U+03BF NEW
'π' => 'π', # greek small letter pi, U+03C0 ISOgrk3
'ρ' => 'ρ', # greek small letter rho, U+03C1 ISOgrk3
'ς' => 'ς', # greek small letter final sigma, U+03C2 ISOgrk3
'σ' => 'σ', # greek small letter sigma, U+03C3 ISOgrk3
'τ' => 'τ', # greek small letter tau, U+03C4 ISOgrk3
'υ' => 'υ', # greek small letter upsilon, U+03C5 ISOgrk3
'φ' => 'φ', # greek small letter phi, U+03C6 ISOgrk3
'χ' => 'χ', # greek small letter chi, U+03C7 ISOgrk3
'ψ' => 'ψ', # greek small letter psi, U+03C8 ISOgrk3
'ω' => 'ω', # greek small letter omega, U+03C9 ISOgrk3
'ϑ' => 'ϑ', # greek small letter theta symbol, U+03D1 NEW
'ϒ' => 'ϒ', # greek upsilon with hook symbol, U+03D2 NEW
'ϖ' => 'ϖ', # greek pi symbol, U+03D6 ISOgrk3
'•' => '•', # bullet = black small circle, U+2022 ISOpub
'…' => '…', # horizontal ellipsis = three dot leader, U+2026 ISOpub
'′' => '′', # prime = minutes = feet, U+2032 ISOtech
'″' => '″', # double prime = seconds = inches, U+2033 ISOtech
'‾' => '‾', # overline = spacing overscore, U+203E NEW
'⁄' => '⁄', # fraction slash, U+2044 NEW
'℘' => '℘', # script capital P = power set = Weierstrass p, U+2118 ISOamso
'ℑ' => 'ℑ', # blackletter capital I = imaginary part, U+2111 ISOamso
'ℜ' => 'ℜ', # blackletter capital R = real part symbol, U+211C ISOamso
'™' => '™', # trade mark sign, U+2122 ISOnum
'ℵ' => 'ℵ', # alef symbol = first transfinite cardinal, U+2135 NEW
'←' => '←', # leftwards arrow, U+2190 ISOnum
'↑' => '↑', # upwards arrow, U+2191 ISOnum
'→' => '→', # rightwards arrow, U+2192 ISOnum
'↓' => '↓', # downwards arrow, U+2193 ISOnum
'↔' => '↔', # left right arrow, U+2194 ISOamsa
'↵' => '↵', # downwards arrow with corner leftwards = carriage return, U+21B5 NEW
'⇐' => '⇐', # leftwards double arrow, U+21D0 ISOtech
'⇑' => '⇑', # upwards double arrow, U+21D1 ISOamsa
'⇒' => '⇒', # rightwards double arrow, U+21D2 ISOtech
'⇓' => '⇓', # downwards double arrow, U+21D3 ISOamsa
'⇔' => '⇔', # left right double arrow, U+21D4 ISOamsa
'∀' => '∀', # for all, U+2200 ISOtech
'∂' => '∂', # partial differential, U+2202 ISOtech
'∃' => '∃', # there exists, U+2203 ISOtech
'∅' => '∅', # empty set = null set = diameter, U+2205 ISOamso
'∇' => '∇', # nabla = backward difference, U+2207 ISOtech
'∈' => '∈', # element of, U+2208 ISOtech
'∉' => '∉', # not an element of, U+2209 ISOtech
'∋' => '∋', # contains as member, U+220B ISOtech
'∏' => '∏', # n-ary product = product sign, U+220F ISOamsb
'∑' => '∑', # n-ary sumation, U+2211 ISOamsb
'−' => '−', # minus sign, U+2212 ISOtech
'∗' => '∗', # asterisk operator, U+2217 ISOtech
'√' => '√', # square root = radical sign, U+221A ISOtech
'∝' => '∝', # proportional to, U+221D ISOtech
'∞' => '∞', # infinity, U+221E ISOtech
'∠' => '∠', # angle, U+2220 ISOamso
'∧' => '∧', # logical and = wedge, U+2227 ISOtech
'∨' => '∨', # logical or = vee, U+2228 ISOtech
'∩' => '∩', # intersection = cap, U+2229 ISOtech
'∪' => '∪', # union = cup, U+222A ISOtech
'∫' => '∫', # integral, U+222B ISOtech
'∴' => '∴', # therefore, U+2234 ISOtech
'∼' => '∼', # tilde operator = varies with = similar to, U+223C ISOtech
'≅' => '≅', # approximately equal to, U+2245 ISOtech
'≈' => '≈', # almost equal to = asymptotic to, U+2248 ISOamsr
'≠' => '≠', # not equal to, U+2260 ISOtech
'≡' => '≡', # identical to, U+2261 ISOtech
'≤' => '≤', # less-than or equal to, U+2264 ISOtech
'≥' => '≥', # greater-than or equal to, U+2265 ISOtech
'⊂' => '⊂', # subset of, U+2282 ISOtech
'⊃' => '⊃', # superset of, U+2283 ISOtech
'⊄' => '⊄', # not a subset of, U+2284 ISOamsn
'⊆' => '⊆', # subset of or equal to, U+2286 ISOtech
'⊇' => '⊇', # superset of or equal to, U+2287 ISOtech
'⊕' => '⊕', # circled plus = direct sum, U+2295 ISOamsb
'⊗' => '⊗', # circled times = vector product, U+2297 ISOamsb
'⊥' => '⊥', # up tack = orthogonal to = perpendicular, U+22A5 ISOtech
'⋅' => '⋅', # dot operator, U+22C5 ISOamsb
'⌈' => '⌈', # left ceiling = apl upstile, U+2308 ISOamsc
'⌉' => '⌉', # right ceiling, U+2309 ISOamsc
'⌊' => '⌊', # left floor = apl downstile, U+230A ISOamsc
'⌋' => '⌋', # right floor, U+230B ISOamsc
'〈' => '〈', # left-pointing angle bracket = bra, U+2329 ISOtech
'〉' => '〉', # right-pointing angle bracket = ket, U+232A ISOtech
'◊' => '◊', # lozenge, U+25CA ISOpub
'♠' => '♠', # black spade suit, U+2660 ISOpub
'♣' => '♣', # black club suit = shamrock, U+2663 ISOpub
'♥' => '♥', # black heart suit = valentine, U+2665 ISOpub
'♦' => '♦', # black diamond suit, U+2666 ISOpub
'"' => '"', # quotation mark = APL quote, U+0022 ISOnum
'&' => '&', # ampersand, U+0026 ISOnum
'<' => '<', # less-than sign, U+003C ISOnum
'>' => '>', # greater-than sign, U+003E ISOnum
'Œ' => 'Œ', # latin capital ligature OE, U+0152 ISOlat2
'œ' => 'œ', # latin small ligature oe, U+0153 ISOlat2
'Š' => 'Š', # latin capital letter S with caron, U+0160 ISOlat2
'š' => 'š', # latin small letter s with caron, U+0161 ISOlat2
'Ÿ' => 'Ÿ', # latin capital letter Y with diaeresis, U+0178 ISOlat2
'ˆ' => 'ˆ', # modifier letter circumflex accent, U+02C6 ISOpub
'˜' => '˜', # small tilde, U+02DC ISOdia
' ' => ' ', # en space, U+2002 ISOpub
' ' => ' ', # em space, U+2003 ISOpub
' ' => ' ', # thin space, U+2009 ISOpub
'' => '', # zero width non-joiner, U+200C NEW RFC 2070
'' => '', # zero width joiner, U+200D NEW RFC 2070
'' => '', # left-to-right mark, U+200E NEW RFC 2070
'' => '', # right-to-left mark, U+200F NEW RFC 2070
'–' => '–', # en dash, U+2013 ISOpub
'—' => '—', # em dash, U+2014 ISOpub
'‘' => '‘', # left single quotation mark, U+2018 ISOnum
'’' => '’', # right single quotation mark, U+2019 ISOnum
'‚' => '‚', # single low-9 quotation mark, U+201A NEW
'“' => '“', # left double quotation mark, U+201C ISOnum
'”' => '”', # right double quotation mark, U+201D ISOnum
'„' => '„', # double low-9 quotation mark, U+201E NEW
'†' => '†', # dagger, U+2020 ISOpub
'‡' => '‡', # double dagger, U+2021 ISOpub
'‰' => '‰', # per mille sign, U+2030 ISOtech
'‹' => '‹', # single left-pointing angle quotation mark, U+2039 ISO proposed
'›' => '›', # single right-pointing angle quotation mark, U+203A ISO proposed
'€' => '€', # euro sign, U+20AC NEW
);
And the one for XHTML:
''' => ''', # apostrophe = APL quote, U+0027 ISOnum
This solution is based on the code from php.net:
function entities_to_unicode($str) {
$str = html_entity_decode($str, ENT_QUOTES, 'UTF-8');
$str = preg_replace_callback("/(&#[0-9]+;)/", function($m) { return mb_convert_encoding($m[1], "UTF-8", "HTML-ENTITIES"); }, $str);
return $str;
}
$str = 'Oggi è un bel giorno';
echo entities_to_unicode($str);
This one
does not require enumeration of entities in user's code,
works on HTML code containing named entities (rude applying html_entity_decode to the whole string messes up < and > (converted < and >) and HTML tag start/end):
Here it is
function htmlent2xml($s) {
return preg_replace_callback("/(&[a-zA-Z][a-zA-Z0-9]*;)/",function($m){
$c = html_entity_decode($m[0],ENT_HTML5,"UTF-8");
# return htmlentities($c,ENT_XML1,"UTF-8"); -- see update below
$convmap = array(0x80, 0xffff, 0, 0xffff);
return mb_encode_numericentity($c, $convmap, 'UTF-8');
},$s);
}
upd: Following Carlos's remark, the proposed solution is fixed using this answer to get the required numeric entities.
echo preg_replace('/[^!-%\x27-;=?-~ ]/e', '"&#".ord("$0").";"', html_entity_decode($str))
First use html_entity_decode to get the unencoded version of your source code. If necessary, set the third parameter (encoding) to the proper value.
Then use utf8_encode on that source code.
$source_code_without_entities = html_entity_decode($source_code_with_entities);
$utf8_source_code = utf8_encode($source_code_without_entities);
The answer #hakre provided is the only one that really works to solve the proposed problem. Interestingly all the other answers, INCLUDING THE ONE THAT WAS ACCEPTED, do not work. Incidentally, the accepted answer doesn't really do anything! The others at least did something, but it was wrong. People who tried to answer, seem not to have understood that what the author wants is to convert named entities into their numerical correspondents.
Below is my contribution, based on a comment from the PHP documentation (https://www.php.net/manual/pt_BR/function.htmlentities.php#106535)
function xmlentities($aString) {
$validChars = "A-Z0-9a-z\s_-";
$twoChars = null;
return preg_replace_callback("/[^$validChars]/"
// Utilizar use(&$twoChars) faz com que $twoChars seja visível dentro da
// função anônima. É necessário usar o "&" se se pretende alterar o
// valor desta variável
,function ($aMatches) use(&$twoChars) {
$oneChar = $aMatches[0];
switch($oneChar) {
// Realiza substituições diretas. No caso, substitui as entidades que o
// XML reconhece. Eu poderia ter usado uma função do próprio PHP para
// isso, mas resolvi não usar porque são só 5 caracteres a substituir
case "'": return "'";
case '"': return """;
case '&': return "&";
case '<': return "<";
case '>': return ">";
// Caso não seja uma entidade reconhecida pelo xml, tratamentos
// especiais são necessários para identificar estamos lidadando com
// caracteres ISO-8859-1 ou UTF-8
default:
// A tabela UTF-8 estende de forma compatível a tabela ASCII. Os
// primeiros 127 caracteres tem 1 byte e todos os demais tem dois bytes.
// Os caracteres UTF-8 com 2 bytes começam com C2 (194) e seguem a
// sequência até chegar em CF (207). A condição abaixo detecta a
// existência de um destes bytes, que identificam um caractere UTF-8.
// Neste caso, se deve acumular ele numa variável com o intuito de,
// posteriormente realizar a conversão de dois bytes e obter um único
// byte ISO-8859-1. Nesta primeira condição, há apenas o acúmulo na
// variável. Nada é retornado
if (194 <= ord($oneChar) && ord($oneChar) <= 207) {
$twoChars = $oneChar;
return;
// Caso $twoChars contenha um valor, é porque em um passo anterior ele
// foi preenchido com o primeiro caractere de um par UTF-8. Neste caso
// devemos concatenar o segundo para, convertê-los para ISO-8859-1 e
// atribuir null à variável de controle ($twoChars). Em seguida,
// retornamos a saída formatada com o ordinal do caractere na tabela
// ISO-8859-1
} else if ($twoChars) {
$twoChars .= $oneChar;
$ansiChar = utf8_decode($twoChars);
$twoChars = null;
return "&#" . str_pad(ord($ansiChar), 3, "0", STR_PAD_LEFT) . ";";
// Caso a string informada no argumento $aString da função já esteja
// codificada em ISO-88959-1, todos os seus caracteres terão 1 byte e
// neste caso, basta formatar diretamente este byte
} else {
return "&#" . str_pad(ord($oneChar), 3, "0", STR_PAD_LEFT) . ";";
}
}
}
,$aString);
}
My version comes with comments (use Google Translator) and is capable of handling only "raw" strings, without entities (& xxx;), so to use it, if your string has named entities, first convert it to its raw form:
$text = "Oggi è un bel giorno";
$text = html_entity_decode($text,ENT_QUOTES || ENT_HTML5,"UTF-8");
$text = xmlentities($text);
echo($text); // Output = Oggi è un bel giorno