Create keywords from text string - encoding issue for cyrillic - php

Question

I was searching for a code that can automatically create keywords from text string. The code below works fine with latin characters but if I try to use it with russian(cyrillic) - the result is just commas with white spaces in between.
I am not sure what the problem is, but I believe it has to do with the encoding. I have tried to encode the string with this
$string = preg_replace('/\s\s+/i', '', mb_strtolower(utf8_encode($string)));
but no luck. The result is the same.
Also, I included only russian words in the $stopwords var
$stopWords = array('у','ни','на','да','и','нет','были','уже','лет','в','что','их','до','сих','пор','из','но','бы','чтобы','вы','была','было','мы','к','от','с','так','как','не','есть','ее','нам','для','вас','о','них','без');
Here is the original code
function extractCommonWords($string){
$stopWords = array('i','a','about','an','and','are','as','at','be','by','com','de','en','for','from','how','in','is','it','la','of','on','or','that','the','this','to','was','what','when','where','who','will','with','und','the','www');
$string = preg_replace('/\s\s+/i', '', $string); // replace whitespace
$string = trim($string); // trim the string
$string = preg_replace('/[^a-zA-Z0-9 -]/', '', $string); // only take alphanumerical characters, but keep the spaces and dashes too…
$string = strtolower($string); // make it lowercase
preg_match_all('/\b.*?\b/i', $string, $matchWords);
$matchWords = $matchWords[0];
foreach ( $matchWords as $key=>$item ) {
if ( $item == '' || in_array(strtolower($item), $stopWords) || strlen($item) <= 3 ) {
unset($matchWords[$key]);
}
}
$wordCountArr = array();
if ( is_array($matchWords) ) {
foreach ( $matchWords as $key => $val ) {
$val = strtolower($val);
if ( isset($wordCountArr[$val]) ) {
$wordCountArr[$val]++;
} else {
$wordCountArr[$val] = 1;
}
}
}
arsort($wordCountArr);
$wordCountArr = array_slice($wordCountArr, 0, 10);
return $wordCountArr;}
Any help would be appreciated.

Answer 1

Cyrillic characters are matched by the following regex and removed from the string:
$string = preg_replace('/[^a-zA-Z0-9 -]/', '', $string);
To avoid it, match word characters with \w. But you need to use the u (PCRE_UTF8) modifier for \w and friends to match letters from other scripts (including Cyrillic).
$string = preg_replace('/[^\w -]+/u', '', $string);
One more thing, the regex /\b.*?\b/i is wrong for several reasons. But if you think about it, it could match the same boundary twice.
Anyway, I believe you don't need to remove unwanted characters form your string. You're only trying to extract words. You could simply use 1 regex to match them.
Regex to match words (with Unicode Scripts, including Cyrillic)
preg_match_all('/\w+/u', $string, $matchWords);
Code
function extractCommonWords($string){
$stopWords = array('i','a','about','an','and','are','as','at','be','by','com','de','en','for','from','how','in','is','it','la','of','on','or','that','the','this','to','was','what','when','where','who','will','with','und','the','www');
$string = strtolower($string);
preg_match_all('/\w+/u', $string, $matchWords);
$matchWords = $matchWords[0];
foreach ( $matchWords as $key=>$item ) {
if ( $item == '' || in_array(strtolower($item), $stopWords) || strlen($item) <= 3 ) {
unset($matchWords[$key]);
}
}
$wordCountArr = array();
if ( is_array($matchWords) ) {
foreach ( $matchWords as $key => $val ) {
$val = mb_strtolower($val);
if ( isset($wordCountArr[$val]) ) {
$wordCountArr[$val]++;
} else {
$wordCountArr[$val] = 1;
}
}
}
arsort($wordCountArr);
$wordCountArr = array_slice($wordCountArr, 0, 10);
return $wordCountArr;
}
//for testing purposes
$sc = 'Hello there нам для!';
$result = extractCommonWords($sc);
print_r($result);
*Disclaimer: I didn't even read the rest of the code. I'm simply pointing out what was wrong
rextester demo