Spell checking UTF-8 text with HunSpellChecker class - php

Question

I'm trying to spell check strings using the HunSpellChecker class (see https://web.archive.org/web/20130311163032/http://www.phpkode.com/source/s/php-spell-checker/php-spell-checker/HunSpellChecker.class.php) and the hunspell spelling engine. The relevant function is copied here:
public function checkSpelling ($text, $locale, $suggestions = true) {
$text = trim($text);
if ($this->textIsHtml == true) {
$text = strtr($text, "\n", ' ');
} elseif ($text == "") {
$this->spellingWarnings[] = array(self::SPELLING_WARNING__TEXT_EMPTY=>"Text empty");
return false;
}
$descspec = array(
0=>array('pipe', 'r'),
1=>array('pipe', 'w'),
2=>array('pipe', 'w')
);
$pipes = array();
$cmd = $this->hunspellPath;
$cmd .= ($this->textIsHtml) ? " -H ":"";
$cmd .= " -d ".dirname(__FILE__)."/dictionaries/hunspell/".$locale;
$process = proc_open($cmd, $descspec, $pipes);
if (!is_resource($process)) {
$this->spellingError[] = array(self::SPELLING_ERROR__INTERNAL_ERROR=>"Hunspell process could not be created.");
return false;
}
fwrite($pipes[0], $text);
fclose($pipes[0]);
$out = '';
while (!feof($pipes[1])) {
$out .= fread($pipes[1], 4096);
}
fclose($pipes[1]);
// check for errors
$err = '';
while (!feof($pipes[2])) {
$err .= fread($pipes[2], 4096);
}
if ($err != '') {
$this->spellingError[] = array(self::SPELLING_ERROR__INTERNAL_ERROR=>"Spell checking error: ".$err);
fclose($pipes[2]);
return false;
}
fclose($pipes[2]);
proc_close($process);
if (strlen($out) === 0) {
$this->spellingError[] = array(self::SPELLING_WARNING__EMPTY_RESULT=>"Empty result");
return false;
}
return $this->parseHunspellOutput(explode("\n", $out), $locale, $suggestions);
}
It works fine with ASCII strings, but I must check strings in different languages, which have accented characters (necessário, segurança, etc) or are in non-Latin alphabets (Greek, Arabic, etc.).
The problem in those cases is that non-ASCII words are segmented incorrectly and the "misspelled" word sent to Hunspell is in fact a substring rather than the full word (necess, seguran).
I tried to track where the issue happens, and I assume it must be in line 072 of the class linked above, when the string is converted into a resource (or somewhere after that). Line 072 contains:
fwrite($pipes[0], $text);
The class is not commented so I'm not really sure what's going on there.
Has anyone dealt with similar issues, or could someone provide any help?
That class is included in file examples/HunspellBased.php (package downloaded from http://titirit.users.phpclasses.org/package/5597-PHP-Check-spelling-of-text-and-get-fix-suggestions.html). I tried to use Enchant, but I didn't manage to make it work at all.
Thank you!
Cheers, Manuel

Answer 1

I think your issue is either HTML entities, or a problem with your dictionary files.
Trying your example with the Portuguese dictionary downloaded from Mozilla add-ons, I can reproduce your problem only when using HTML-encoded entities. i.e. segurança is fine, but segurança get's segmented as you say.
I don't think this is an issue with the class. All the class does is pipe the text to the command line program. You can eliminate the PHP class as an issue by using the program directly as follows:
Change working directory to the place you have your dictionaries, php-spell-checker/dictionaries/hunspell according to your code above. Prepare a text file containing the accented words you want to test and then do:
hunspell -l -d pt-PT test.text
or for HTML:
hunspell -l -d pt-PT -H test.html
Where pt_PT represents the name of the Portuguese dictionary file pair, namely pt-PT.aff and pt–PT.dic
No output means no errors. If you get partials words like "necess" only when using HTML entities, then this is your issue. If not, then it's either some other kind of string-encoding issue, or an issue with the dictionary you're using.
I suspect this is a limitation of hunspell's HTML parser - that it ignores HTML tags and other punctuating entities, but won't include and decode a word with an entity in the middle.
The only way around this (assuming HTML is your issue) is do your own pre-processing before sending HTML to the spellcheck. PHP's html_entity_decode will convert ç -> ç so you could try calling that on every string. Ideally though you'd parse the HTML DOM and pull out only the text nodes.
If HTML is not your issue, check that the strings are valid UTF-8.
Failing that try another dictionary file. The one I grabbed from Mozilla works fine with plain text. Just rename the .xpi file to .gzip, expand it using whatever decompress software you have, then copy the .dic and .aff files to your dictionary folder.

Answer 2

I think you can add After :
$cmd = $this->hunspellPath;
$cmd .= ($this->textIsHtml) ? " -H ":"";
$cmd .= " -d ".dirname(__FILE__)."/dictionaries/hunspell/".$locale;
Add
$cmd .= " -i UTF-8";