So I've got some data that's been compressed with PHP's gzcompress method:
http://us2.php.net/manual/en/function.gzcompress.php
How can I decode this data from node.js??
I've tried "compress", "zlib" and several other node compression libraries, but none of them seem to reckognize the data. For example, zlib just gives me "Error: incorrect header check"
Answer: Turns out that "zlib" is the way to go. We had an additional issue with binary data from memcache. If you have binary data in a node.js Buffer object and you call toString() instead of .toString('binary'), it get's all kinds of scrambled as stuff is escaped or escape sequences are interpreted or whatever. Unfortunately, all the memcache plugins I've tried to date assume string data from memcache, and are not disciplined about handling it properly.
Best ZLIB module I've found:
https://github.com/kkaefer/node-zlib
// first run "npm install zlib", then...
var zlib = require('zlib');
var gz = zlib.deflate(new Buffer("Hello World", 'binary')); // also another 'Buffer'
console.log(zlib.inflate(gz).toString('binary'));
FYI, this question is VERY similar to a related question about Java:
PHP's gzuncompress function in Java?
Stealing from another post (Which compression method to use in PHP?)
gzencode() uses the fully self-contained gzip format, same as the gzip command line tool
gzcompress() uses the raw ZLIB format. It is similar to gzencode but has different header data, etc. I think it was intended for streaming.
gzdeflate() uses the raw DEFLATE algorithm on its own, which is the basis for both the other formats.
Thus, "zlib" would be the correct choice. This is NOT cross-compatible with gzip.
Try https://github.com/kkaefer/node-zlib
php:
<?php
$data = 'HelloWorld';
$gzcompress = gzcompress($data);
$gzcompress_base64_encode = base64_encode($gzcompress);
echo "Compressing: {$data}\n";
echo $gzcompress."\n";
echo $gzcompress_base64_encode."\n";
echo "--- inverse ---\n";
echo gzuncompress(base64_decode($gzcompress_base64_encode))."\n";
nodejs:
const zlib = require('zlib');
var data = 'HelloWorld';
var z1 = zlib.deflateSync( Buffer.from(data));
var gzcompress = z1.toString();
var gzcompress_base64_encode = z1.toString('base64');
console.log('Compressing '+data);
console.log(gzcompress);
console.log(gzcompress_base64_encode);
console.log('--- inverse ---');
console.log(zlib.inflateSync(Buffer.from(gzcompress_base64_encode,'base64')).toString());
Related
When I compress a string in PHP with encoding ZLIB_ENCODING_DEFLATE and output the hex data, I can convert this back to the original string using zlib deflate() in a c++ project.
Per the example here ( https://www.php.net/manual/en/function.zlib-encode.php ) :
<?php
$str = 'hello world';
$enc = zlib_encode($str, ZLIB_ENCODING_DEFLATE);
echo bin2hex($enc);
?>
in c++, after having converted the hex string to binary data first: (simplified code)
z_stream d_stream;
d_stream.zalloc = (alloc_func)0 ;
d_stream.zfree = (free_func)0 ;
d_stream.opaque = (voidpf)0
d_stream.next_in = InBuffer ;
d_stream.avail_in = InBufferLen ;
d_stream.next_out = OutBuffer ;
d_stream.avail_out = OutBufferLen ;
int err = inflateInit(&d_stream) ;
while (err == Z_OK)
err = inflate(&d_stream, Z_NO_FLUSH);
err = inflateEnd(&d_stream);
OutBuffer contains "hello world" again
I was wondering if zlib inflate() also decompresses the via PHP generated zlib_encode($str, ZLIB_ENCODING_RAW); raw data ?
From the zlib documentation I think not:
The deflate compression method (the only one supported in this
version).
#define Z_DEFLATED 8
But PHP's function name zlib_encode() and define ZLIB_ENCODING_RAW seem to suggest zlib does support it ? If so what function and/or parameters do I use ?
The PHP designations are (as usual) confusing. I will assume that ZLIB_ENCODING_RAW means raw deflate data (per RFC 1951), and it appears that ZLIB_ENCODING_DEFLATE actually means zlib-wrapped deflate data (per RFC 1950).
If that's correct, they should have called them ZLIB_ENCODING_DEFLATE and ZLIB_ENCODING_ZLIB, respectively. But I digress.
You can decode raw deflate data with the zlib library by using inflateInit2() instead of inflateInit(), and giving -15 as the second argument.
How do I correctly compress a string, so PHP would be able to decompress?
I tried this:
public static byte[] compress(String string) throws IOException {
ByteArrayOutputStream os = new ByteArrayOutputStream(string.length());
DeflaterOutputStream gos = new DeflaterOutputStream(os);
// ALSO TRIED GZOutputStream, same results!
gos.write(string.getBytes());
gos.close();
byte[] compressed = os.toByteArray();
os.close();
return compressed;
}
But PHP does not recognize output as valid GZip compressed string...
The problem seems to be in some headers / footers being added by Android...
For example when I compress something word via PHP with gzcompress I got similar results as with Android, but not similar enough, so PHP could read it:
something (HEX DUMP):
Android: 1f8b08000000000000002bcecf4d2dc9c8cc4b0700fb31da0909000000
PHP: 789c2bcecf4d2dc9c8cc4b0700134703cf
The weirdest thing is that by changing GZOutputStream to DeflaterOutputStream it fixed the problem with something word, but the problem still appears with longer strings...
PS. Removing heading 10 characters from Android generated data does not help at all.
EDIT: I tried to decompress it in PHP with:
gzdecode() - this function does not exist in standard Debian PHP5
version.
gzdecompress() - does not work
And some functions to emulate gzdecode() from PHP site comments that don't really do much.
All above, with removing first 10 bytes and leaving them.
PS2. I tried every single solution from Stack Overflow, and other sources, and still nothing. It is not a duplicate.
EDIT2 (BINARY DUMP): Sample data generated with Android that can't be decomprssed by gzuncompress() or pseudo-gzdecode() functions from PHP.NET: data.compressed.
It supposed to be some JSON, after decompression.
The Android data that starts with 1f8b is a gzip stream. In php you use gzdecode() for that. gzencode() on php makes gzip streams.
The php data that starts with 789c is a zlib stream. You used gzcompress() to make that, and you would use gzuncompress() to decode it.
The compressed data contained within both of those streams, starting with 2bce is raw deflate data. You can use gzinflate() to decode that if you happened to make it somewhere, and you can use gzdeflate() to generate raw deflate.
Just to rant, gzencode(), gzcompress(), and gzdeflate() are some of the most misleading function names ever concocted, since only one of them is related to gzip yet all start with gz, and nothing in the name gzcompress() indicates zlib.
Update:
The "EDIT2" data is, for some reason, doubly compressed. It was compressed first to the zlib format, and then that zlib stream was compressed to the gzip format. (Though gzip couldn't compress the already compressed data, so it's a little bigger.)
You should repair the problem that made it doubly compressed. Or if you have no control over that, you can doubly decompress it, first stripping the gzip header using the RFC 1952 specification and then gzinflate() on the raw deflate data, and then using gzdecompress() on the result.
I try to decompress blocks of data which were compressed with zlib and author made remarks that for decompress i must use inflate_init and inflate with Z_SYNC_FLUSH. I sure that this must work because that works on php in this way :
$temp = substr($temp, 2, -4);
$temp{0} = chr(ord($temp{0}) | 1);
$temp = gzinflate($temp);
but i ckecked many method for decompress this on C++ and every time fail.
Here is one of them :
char compressedblockbuffer[3371];
char uncompressedblockbuffer[8192];
is.read(compressedblockbuffer, 3371);
z_stream strm;
strm.zalloc = Z_NULL;
strm.zfree = Z_NULL;
strm.opaque = Z_NULL;
strm.avail_in = 3371;
strm.next_in = (Bytef *)compressedblockbuffer;
strm.avail_out = 8192;
strm.next_out = (Bytef *)uncompressedblockbuffer;
inflateInit(&strm);
inflate(&strm, Z_SYNC_FLUSH);
inflateEnd(&strm);
It's not full code, just example to show problem and thats why i specified already known sizes.
I use last zlib realize so may be something change in the zlib inflate since 2003-2004 years?
So the result is :
So seems that uncompressedblockbuffer contains '\0' at the 2,3,4 indexes and many other and if i print this to console i just see two first elements.
UPD:
If gzinflate() in PHP works on the data, then your code won't. gzinflate() expects raw deflate data. Your code is looking for zlib-wrapped deflate data. If you want to decode raw deflate data, you need to use inflateInit2(&strm, -15) instead.
Your call to inflate() is likely returning an error that you are not checking for. You need to always check the return codes of the zlib routines, or for that matter any function that has the potential to return an error.
What kind of data are you decompressing? Many binary formats are perfectly accepting of NUL bytes in their data, since it just reads as a value of 0. For example, inside of image data in many formats, it'd just represent a value of 0 in either that channel or pixel (depending on data size). Not to mention, binary formats don't necessarily read as bytes. A NUL byte may actually be a part of a 2- or 4-byte value.
This is the problem with trying to read binary data as a character string. Binary data needn't follow the rules of text. This is why usually the data boundary is a separate size value, because it can't terminate on NUL values like text.
If you have the original uncompressed data for comparison, either load that data into memory and compare the data, or save the decompressed data to a file and use a diff tool to do a binary comparison of the files.
i faced PHP: Call to undefined function gzdecode() error
few days back so i have been using this in php
function gzdecode($data)
{
return gzinflate(substr($data,10,-8));
}
ref: PHP: Call to undefined function gzdecode()
i found this old code snippet while looking in old php files , so can i use above code instead of this one ?
function gzdecode($data) {
$g=tempnam('/tmp', 'php-gz');
#file_put_contents($g, $data);
ob_start();
readgzfile($g);
$d=ob_get_clean();
unlink($g);
return $d;
}
If you have access to your php.ini, simply enable the zlib extension to use gzencode. Each compression method has some slight differences.
Please read an abstract of this answer https://stackoverflow.com/a/621987/382536 by #thomasrutter
All of these are usable. I would use gzencode() because it is self-contained; its output is the same as if you had used the gzip command line tool.
There is not really much difference between the three.
gzencode() uses the fully self-contained gzip format, same as the
gzip command line tool
gzcompress() uses the raw ZLIB format. It is
similar to gzencode but with less header data, etc. I think it was
intended for streaming.
gzdeflate() uses the raw DEFLATE algorithm on
its own, which is the basis for both the other formats.
I'm trying since several hours to implement the behaviour of PHP gzinflate() in C. In PHP it's just:
gzinflate($str);
In Python it's:
import zlib
...
return zlib.decompress( decoded_data , -15)
...
But I just don't manage to implement it in C. Can anybody help me with that? I'm really stuck.. I tried to do something with Zlib but it didn't work.. Anybody got a point?
Thanks in advance,
nemo
This zlib usage example is very thorough.
Note that you are closer to the bare metal here than in Python or PHP, so the usage isn't as simple.
Addendum:
The PHP gzinflate and gzdeflate functions perform input and output raw DEFLATE format. The zlib functions, on the other hand, work by default with zlib streams, which are the same with the addition of a 2 byte header and a 4 byte trailer.
You can either switch to using the PHP gzcompress and gzuncompress functions, which produce ZLIB format, or (if you have a recent version of zlib) use the deflateInit2 function instead of deflateInit and specify a negative value for windowBits, which requests raw DEFLATE format.