Unicode error when loading a json_encoded PHP array from memcached - php

Question

I need to transfer a PHP associative array to further processing using python. The python code however using pylibmc is unable to load the string from memcached, throwing this error:
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe0 in position 32: invalid continuation byte
I wrote a little tester. The PHP code to create the memcached data:
<?php
$mc = new Memcached();
$mc->addServer('localhost', 11211);
$data = array();
for ( $i = 0; $i < 100; $i++) {
$index = "ti" . $i;
$data += [$index => "test string $i"];
}
$mc->delete('test');
$mc->add('test', json_encode($data), 60);
$reverse = $mc->get('test');
echo "$reverse\n"; // prints {"ti0":"test string 0" ...... "ti99":"test string 99"} as expected
$reverse_array = json_decode($reverse, true);
echo $reverse_array['ti10'] . "\n";
//prints 'test string 10' as expected
?>
so this works fine writing to memcached from PHP and reading it back.
On the python side, this is the code I use to read it in:
#!/usr/bin/python
import pylibmc
import json
mc = pylibmc.Client(["127.0.0.1"], binary=True, behaviors={"cas": True, "tcp_nodelay": True,"ketama": True})
temp = json.loads(mc.get("test"))
When running the python code, this is the output I get:
Traceback (most recent call last):
File "./mctest.py", line 7, in <module>
temp = json.loads(mc.get("test")))
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe0 in position 32: invalid continuation byte
If I create a non-associative array in PHP and share that through memcached, things work fine.
Two further options I've tried:
adding utf8_encode to make sure it's properly encoded:
$mc->add('test', utf8_encode(json_encode($data)), 60);
adding JSON_UNESCAPED_UNICODE to the json_encode function:
$mc->add('test', json_encode($data, JSON_UNESCAPED_UNICODE), 60);
both result in identical outcomes on the python side.
Bit at a loss here - any ideas welcome!

Answer 1

While trying to determine the encoding of the resulting string retrieved from memcached via pymemcache, it occurred to me that the string didn't look like any known encoding, I confirmed this using chardet as well as cchardet.
After some more digging at the PHP end, I discovered that the PHP memcached module adulterates the strings it saves to memcached by compressing the data!
Solution was to add this line to the /etc/php/7.2/cli/conf.d/25-memcached.ini file:
memcached.compression_threshold=9999999999
Now the data comes into python as it should!