Alphanumeric regex not working with non-roman characters - php

Question

I want to only have alphanumeric characters [a-f0-9] in a string. To achieve this, I have:
$text = preg_replace("/[^[:alnum:]]/u", '', $text);
Works fine in this case:
$text = 'hello?world'; // becomes 'helloworld'
The problem is that it doesn't seem to work for other languages, for example:
$text = '日本国'; // becomes '日本国'
That should be empty!
Ideone demo. What am I doing wrong here?

Answer 1

To be more clear, by default [:alnum:] contains [a-zA-Z0-9] (letters and digits from the ASCII range 0-127).
But if you use the u modifier, this class is extended to all UNICODE letters and digits.
The u modifier:
changes the way the subject string (and the pattern) is read (code point by code point instead of byte by byte)
extends several* character classes to UNICODE characters (*as a counter example, the \h character class doesn't change.)
It's possible to separate these two behaviors with commands at the start of the pattern:
(*UTF) at the start of the pattern informs that the subject and the pattern have to be read as utf (utf-8 in php) encoded strings (and not byte by byte).
(*UCP) extends the character classes.
(see several tests here:)
So instead of the u modifier, you can write your pattern this way:
$str = preg_replace('~(*UTF)[^[:alnum:]]+~', '', $str);
You can also choose to not use the [:alnum:] class at all and to be more explicit:
$str = preg_replace('~[^a-z0-9]+~ui', '', $str);
Since there is no predefined character class in the pattern, the (*UCP) part of the u modifier doesn't change anything.
Obviously, as noted in comments, it's also possible to ignore the fact that your subject string may contain characters out of the ASCII range, and read this string byte by byte with:
$str = preg_replace('~[^[:alnum:]]+~', '', $str);
// or
$str = preg_replace('~[^a-z0-9]+~i', '', $str);
and it will work too, but IMO it's less rigorous.