Issue with file_get_contents encoding - php

Question

I'm getting file_get_contents(uri) and getting back a Json that I'm unable to encode.
I tried several encodings and str_replace but I don't quite understand what the issue is.
This is the start of my json with file_get_contents:
string(67702) "��{"localidades"
I know it's finding unknown characters and that's what the ? are for, but I don't understand how to solve it.
I've tried this but to no avail
if(substr($s, 0, 2) == chr(0xFF).chr(0xFE)){
return substr($s,3);
}
else{
return $s;
}
}
This is xxd | head from terminal
00000000: fffe 7b00 2200 6c00 6f00 6300 6100 6c00 ..{.".l.o.c.a.l.
00000010: 6900 6400 6100 6400 6500 7300 2200 3a00 i.d.a.d.e.s.".:.
00000020: 2000 5b00 7b00 2200 6900 6400 4c00 6f00 .[.{.".i.d.L.o.
00000030: 6300 6100 6c00 6900 6400 6100 6400 2200 c.a.l.i.d.a.d.".
00000040: 3a00 2000 3300 2c00 2200 6c00 6f00 6300 :. .3.,.".l.o.c.
00000050: 6100 6c00 6900 6400 6100 6400 2200 3a00 a.l.i.d.a.d.".:.
00000060: 2000 2200 4200 7500 6500 6e00 6f00 7300 .".B.u.e.n.o.s.
00000070: 2000 4100 6900 7200 6500 7300 2200 2c00 .A.i.r.e.s.".,.
00000080: 2200 6900 6400 5000 7200 6f00 7600 6900 ".i.d.P.r.o.v.i.
00000090: 6e00 6300 6900 6100 2200 3a00 2000 2200 n.c.i.a.".:. .".

Answer 1

What you have there is UTF-16LE in which each codepoint is encoded as at least two bytes, even "basic ASCII". The first two bytes of the document are the Byte Order Mark [BOM] that declares in what byte-order [endian] those codepoints are encoded
$input = "\xff\xfe{\x00}\x00"; // UTF-16-LE with BOM
function convert_utf16($input, $charset=NULL) {
// if your data has no BOM you must explicitly define the charset.
if( is_null($charset) ) {
$bom = substr($input, 0, 2);
switch($bom) {
case "\xff\xfe":
$charset = "UTF-16LE";
break;
case "\xfe\xff":
$charset = "UTF-16BE";
break;
default:
throw new \Exception("No encoding specified, and no BOM detected");
break;
}
$input = substr($input, 2);
}
return mb_convert_encoding($input, "UTF-8", $charset);
}
$output = convert_utf16($input);
var_dump(
$output,
bin2hex($output),
json_decode($output, true)
);
Output:
string(2) "{}"
string(4) "7b7d"
array(0) {
}
It's also worth noting that using anything other than UTF-8 to encode JSON makes it invalid JSON, and you should tell whoever is giving you this data to fix their app.

Answer 2

What you are getting is UTF-16 LE. The fffe at the beginning is called a BOM. You can use iconv:
$data = iconv( 'UTF-16', 'UTF-8', $data);
And now you have a UTF-8 with BOM. Which i think will work with json_decode, because PHP seems to handle it. Still, if you want to remove the BOM, which you should (see #Sammitch comment), you can use this one as well:
$data = preg_replace("/^pack('H*','EFBBBF')/", '', $data);
I recreated a part of your file and i get this:
$data = file_get_contents('/var/www/html/utf16le.json');
$data = preg_replace("/^pack('H*','EFBBBF')/", '', iconv( 'UTF-16', 'UTF-8', $data));
print_r(json_decode($data));
Output:
stdClass Object
(
[localidades] => Array
(
[0] => stdClass Object
(
[idLocalidad] => 3
[localidad] => Buenos Aires
)
)
)
And from xxd:

Answer 3

The file you try to process is encoded in UTF-16, which isn’t natively supported by PHP. So, in order to process it, you’ll have to remove BOM header first (first two bytes) and then convert encoding to UTF-8 using iconv or mbstring.