I have a incoming stream that is compressed using the zlib functions, but I cannot tell the ending of the compressed data, so am having a lot of trouble getting the data out.
I also have a snippet of the source code where it is being uncompressed in AS3 flash, which should have been enough for me to figure it out, but I am at a loss.
Two files included:
http://falazar.com/projects/irnfll/version4/test//stackoverflow/as3_code.as.txt
http://falazar.com/projects/irnfll/version4/test//stackoverflow/bin_data_file
Snippet of the binary data, and what I know:
00 00 02 34 2c 02 31 78 5e ed dc cd 6e da 40 a5
21 19 40 f5 f2 c4 b7 e9 18 85 e1 5b 89 66 3d 42
31 95 90 cd 15 74 99 55 37 51 14 59 c9 a8 8c 54
0234 appears to be a size marker - 564
2c = 44, the code to match as3 COMMAND_WORLD_DATA, that is ok
0231 another size marker, always 3 smaller than above
785e - is the header marker for the zlib compress weakest compression, ZLIB_ENCODING_DEFLATE level 4
Later there is also a 789c which is another larger compressed block
I need to uncompress the two of these to move forward in project, thank you for your help.
There is also mention in the script of bigendian conversion, and am I not sure if I need to handle that.
I have written a couple scripts to try and solve this, including a php snippet that chops off the end until and loops trying to uncompress with no luck.
falazar.com/projects/irnfll/version4/test//stackoverflow/php_test.php.txt
Ideal solution in php or c#, but anything I can see that works will translate into another language easy enough.
(Using Free hex editor nero to view the binary)
You mean zlib.
Use PHP's gzuncompress() starting at each zlib header (e.g. 789c).
Related
I have inherited a zlib compressed file and long story short, I need to UN-zlib-compress this puppy back to its original content.
I have been racking my brain trying to figure out what in the world is happening, but I am hitting a wall and I am hoping you good people will help me out to figure out what's going on.
I have done alot of things so far, I won't bore you with every single thing, but this is what I landed on last, and all I get garbled output, don't know what in the heck is wrong, especially that the last step of decode complains about the data saying:
Warning: gzuncompress(): data error in
C:\xampp\htdocs\test-box\index.php on line 6
Warning: zlib_decode():
data error in C:\xampp\htdocs\test-box\index.php on line 8
and this is the code - nothing fancy, I am trying to get it to work before going too crazy with it yet and so the simplicity should allow us to better analyze it.
<?php
$filename = 'c5ytvbg4y.x'; // this is the zlib compressed file
$file = filesize($filename); // using this for the length
$zd = gzopen($filename, "r"); // create valid pointer
$contents = gzread($zd, $file); // binary safe read the content
$decoded = gzuncompress($contents); // using gzdecode produces the same issue
gzclose($zd); // close the pointer
zlib_decode($decoded); // decode it but I get nothing but garble
?>
Any assistance would be appreciated. Ideally I want to be able to open it uncompress it back to normal and save it to a new file. But at the moment I would be happy just to find out why in the heck I get nothing but garbled text back. Also keep in mind that I know the $file above is not ideal, I will put a while !feof($zd) or something to that effect later, I wanted to keep it simple for now while trying get the larger issue figured out.
Any thoughts, recommendations, suggestions, code assistance, or whatnot would be greatly appreciated, TIA.
Additions
#Mark's Request:
0A 12 0F 04 04 D8 44 DA BF 63 C4 93 93 3B 49 51 17 A2 6F E3 0C 12 4D E4 24 F6 C8 BA D0 60 76 81
It is definitely not a "zlib compressed file", at least not the first 32 bytes, nor is it any format that uses the deflate compression method (e.g. gzip, zip, png, etc.), because there is no valid deflate compressed data in the provided bytes.
The zlib header typically starts with hexadecimal 78. Your data starts with 0A, which isn't valid as part of a zlib header. (Technically it is sort of valid, but it implies a compression format that isn't supported by any version of zlib.)
The gzip header starts with hexadecimal 1F 8B. That isn't present in your data either.
So, I'm not sure what this data is, but it's neither gzip nor zlib data. You'll need to do some more research to figure out what it is.
I read via PHP function snmp2_real_walk OID value '.1.3.6.1.2.1.17.4.3.1.1' for get MAC address from Cisco switch device. Problem is, that some results (randomly around the 50 results of the 200) return bad characters (yet I have found a mistake if I read only MAC addresses). Ie.:
correct output examples:
[Dot1dTpFdbAddres] => 30 05 5C 38 A7 8C
[Dot1dTpFdbAddres] => C0 7B BC 0E 56 18
wrong output examples:
[Dot1dTpFdbAddres] => ,v�?.b (HEX DUMP: 0 : 20 2c 76 8a 3f 2e 62 20 [ ,v.?.b ])
[Dot1dTpFdbAddres] => ,A8��7 (HEX DUMP: 0 : 20 2c 41 38 82 d9 37 20 [ ,A8..7 ])
[Dot1dTpFdbAddres] => xE�\ � (HEX DUMP: 0 : 20 78 45 c4 2a 5c 20 d9 20 [ xE.\ . ])
If I try read OID '.1.3.6.1.2.1.17.4.3.1.1' from program Getif, I have got correct results.
I can't find solution for this problem - can you please help me?
The tested solution that fails
PHP - Chnaged snmp2_real_walk by snmprealwalk
Changed encoding of files (UTF8, ANSI)
SNMP longer timeout and try add PHP sleep() function
PHP directive: snmp_set_oid_numeric_print(1)
PHP directive: snmp_set_quick_print(true)
PHP directive: snmp_set_enum_print(true)
PHP directive: snmp_set_valueretrieval(SNMP_VALUE_LIBRARY and SNMP_VALUE_PLAIN)
About server:
PHP Version 5.5.3
Apache/2.4.4 (Win32) OpenSSL/1.0.1e PHP/5.5.3
LAMP are equivalent to WAMP (error too)
New important informations:
Device return randomly none-hex value for hex values, ie.:
[iso.3.6.1.2.1.17.4.3.1.1.92.38.10.129.123.27] => Hex-STRING: 5C 26 0A 81 7B 1B
[iso.3.6.1.2.1.17.4.3.1.1.120.69.196.42.25.241] => Hex-STRING: 78 45 C4 2A 19 F1
[iso.3.6.1.2.1.17.4.3.1.1.120.69.196.42.27.169] => Hex-STRING: 78 45 C4 2A 1B A9
[iso.3.6.1.2.1.17.4.3.1.1.120.69.196.42.34.45] => STRING: "xE�*\"-"
[iso.3.6.1.2.1.17.4.3.1.1.120.172.192.142.199.214] => STRING: "x�����"
[iso.3.6.1.2.1.17.4.3.1.1.124.30.179.254.9.201] => Hex-STRING: 7C 1E B3 FE 09 C9
Mibs are imported correctly to Apache server, devices look fine.
Is there a way in PHP how to write for snmp2_real_walk() all returned values as Hex-STRING?
*Similar problems (without results):
http://forums.cacti.net/viewtopic.php?f=15&t=50806
http://comments.gmane.org/gmane.network.net-snmp.user/31569
http://forums.cacti.net/viewtopic.php?f=21&t=46068*
Thanks and best regards,
Petr
Use the following config:
snmp_set_valueretrieval(SNMP_VALUE_LIBRARY);
snmp_set_quick_print(1);
snmp_set_enum_print(0);
I am parsing a GIF 89a (yes, I need to) file and I am stuck on Application Extension blocks.
They have 13 byte header (including the beginning 21 FF 0B bytes) and then there is some data. How much data is there? How do I know know much to read?
You can skip the section below if you know the answer and just tell me :)
This page says:
ApplicationData contains the information that is used by the software application. This field is structured in a series of sub-blocks identical to the data found in a Plain Text Extension block."
Each sub-block begins with a byte that indicates the number of data bytes that follow. From 1 to 255 data bytes may follow this byte. There may be any number of sub-blocks in this field.
This way I can parse NETSCAPE 2.0 blocks which are:
03 01 00 00 00
so I have a loop in PHP:
for (;;)
{
$size = ord(fread($handle, 1));
if ($size == 0) break;
fseek($handle, $size);
}
or the same in Delphi, if you prefer:
while F.Position < F.Size do begin
F.Read(Size, 1); // F is TFileStream
if Size = 0 then break;
F.Position := F.Position + Size;
end;
The iteration goes:
size = read 1 byte; //size = 3;
read 3 byte;
size = read 1 byte;
size = 0 so break
So far, so good, here comes the problem: the XMP Data
So the bytes in this block go like this (ASCII below):
21 FF 0B 58 4D 50 20 44 61 74 61 58 4D 50
!`.XMP DataXMP
and then goes ASCII XML dump:
<?xpacket begin="" id="W5M0MpCehiHzreSzNTczkc9d"?>
for about 500 bytes.
I obviously can't read it the same way I read NETSCAPE 2.0 blocks.
It seems to be terminated with 00 byte.
Should it just always read until 00 byte? Then if would fail on NETSCAPE 2.0 blocks!
How should a GIF decoder behave on Application Extension blocks? How much data is in them?
Problematic XMP Data image
Ok- the NETSCAPE 2.0 block approach might be fine and it was failing on the XML because my file could be corruptly read.
I have a PHP script which fetches data from Salesforce via the Salesforce API and writes the output to a file using file_put_contents. The data is a mixture of Korean characters and English characters.
When I run the script on a box (1) running Red Hat Enterprise Linux ES release 4 (Nahant Update 8) with PHP 5.2.8 and a similar box (2) running PHP 5.3.6 the spaces in between the Korean characters disappear.
e.g. (Using K to represent a Korean character and E to represent an English character)
EEEEEEEEK KKK KKKK EEE KKKK is appearing as EEEEEEEEKKKKKKKK EEE KKKK
However when I run the script on a box (3) running CentOs with PHP 5.3.5 or on (4) my local windows machine with PHP 5.3.6 the text in the file is correct.
Can anyone suggest what the problem might be?
EDIT - Originally I was accessing the php script via a browser however to (hopefully) simplify the problem I am currently storing the output in a text file and downloading it to my windows machine.
EDIT - Hex version
Original text - CFD란 무엇입니까?
Hex from (1) - 43 46 44 eb 9e 80 eb ac b4 ec 97 87 ec 9e 85 eb 8b 88 ea b9 8c 3f
Hex from (3) - 43 46 44 eb 9e 80 20 eb ac b4 ec 97 87 ec 9e 85 eb 8b 88 ea b9 8c 3f
EDIT - Code used to select text (with user, pass, table, id and path omitted)
<?php
ini_set("soap.wsdl_cache_enabled", "0");
require_once ("../soapclient/SforcePartnerClient.php");
require_once ("../soapclient/SforceHeaderOptions.php");
$partner_wsdl = "../soapclient/new-partner.wsdl.xml";
$client = new SforcePartnerClient();
$client->createConnection($partner_wsdl);
$loginResult = $client->login('--user--', '--pass--');
$query = "Select Name FROM --table-- WHERE Id = '--id--'";
$response = $client->query($query);
echo'<pre>';print_r($response);echo'</pre>';
$queryResult = new QueryResult($response);
foreach ($queryResult->records as $qr) {
$content = $qr->fields->Name;
file_put_contents('--path--',$content);
}
?>
After more research I discovered a function in the SforcePartnerClient.php
$QueryResult = $this->sforce->query(array ('queryString' => $query))->result;
is returning different values depending on the box used.
Box 1 and 2:
<sf:Name>CFD란 무엇입니까?</sf:Name>
Box 3 and 4:
<sf:Name>CFD란 무엇입니까?</sf:Name>
When this is combined with / parsed / converted using the XML parser (later in the file) and WSDL file the XML parser strips any white space that occurs between consecutive &#xxxxx; s - I believe this is related to a bug https://bugs.php.net/bug.php?id=33240 to avoid this I suggest commenting out line 364 of SforcePartnerClient.php
xml_parser_set_option( $parser, XML_OPTION_SKIP_WHITE, 1 );
Unfortunately I do not know if this will have any adverse effects on other code using SforcePartnerClient.php.
I have a PHP script to which I upload JPEG images (through an HTML form). You can see the code here, but I will attempt to present the relevant parts in this post. The form is declared like so:
<form action="adm_addphoto.php" method="POST" enctype="multipart/form-data" name="myform">
The MAX_FILE_SIZE form field is set to 5MB:
<input type="hidden" name="MAX_FILE_SIZE" value=5242880>
The images I want to upload are about 3MB in size.
Once it's uploaded, I turn the image file into a GD jpeg:
$filename = $_FILES['file']['tmp_name'];
$myImage = imagecreatefromjpeg($filename);
Sometimes the upload works fine, and sometimes imagecreatefromjpeg emits warnings about the JPEG being corrupt. For example (line breaks added for readability):
Warning: imagecreatefromjpeg() [function.imagecreatefromjpeg]:
gd-jpeg, libjpeg: recoverable error: Corrupt JPEG data:
47 extraneous bytes before marker 0xd9 in
/path/adm_addphoto.php on line 97
Warning: imagecreatefromjpeg() [function.imagecreatefromjpeg]:
'/tmp/phpwlSS9x' is not a valid JPEG file in
/path/adm_addphoto.php on line 97
The thing is, this doesn't happen reliably. That is, if I try the same image several times in a row, sometimes it will upload successfully and sometimes there will be errors. And within the attempts that result in errors, the specifics of the error message will vary too. (With the particular photo that produced the above messages, the number of "extraneous bytes" is sometimes 47, sometimes 20, occasionally 68.)
What might be causing the files to arrive corrupt on some attempts but not others?
PS. I know that there is an ini setting that tells GD to try hard to work with corrupt JPEGs. But that's not the point, I want to know why the result of the upload is inconsistent.
PPS. Here are the values of some possibly-relevant PHP ini settings:
memory_limit .......... 128M
post_max_size ......... 8M
file_uploads .......... On
max_file_uploads ...... 20
upload_max_filesize ... 128M
upload_tmp_dir ........ no value
UPDATE: I added code to echo out the size and MD5 checksum of the file once it was uploaded. The size is always the correct filesize (the size of my local copy of the file). The MD5, however, varies between attempts. When the MD5 is correct (matches the MD5 of my local copy), there is no error message. When there is an error message, the MD5 is always different from expected. Also, this is just a perceived difference, but errors seemed to be occurring more frequently last night than this morning - only a minority of transfers this morning resulted in an error.
Here are some "bad" uploads (curiously, there were a couple of uploads that gave an MD5 checksum different to that expected but did not produce errors):
Correct MD5: f7b9587f39c7332e62a08adf34cefbd0
-----------------------------------------------------------------
38 extraneous bytes: 2a28c46079071d9d2e2fd49865b35d59
ce1c69f798953b201dcc35f85f3b29b4
b013a0428a71adff674a46e92372d46b
71 extraneous bytes: a271928f3559b6deaa19804704b5bcb3
92 extraneous bytes: cab2a10ad8535addaca3b19bcb607a30
premature end of data segment: 4514d39db1d94ab691d6da26c0832cdb
No error (!): 83b2e3624ddcfdeb3efc10be81631916
07d0a97b21d423fdeb4c6f88d76f8cd3
UPDATE: In response to Andre and Pekka: Here are links to two versions of the same picture. This one is the original copy of the image, as stored on my machine; I uploaded it the way I usually upload files to my webhosting, i.e. I used an FTP client. This one is the same image, uploaded using my script (but with a line added to move it to that location using move_uploaded_file()). The error message on that upload was "Corrupt JPEG data: 30 extraneous bytes...". You will notice that the bottom part of the image appears incorrect. I apologise for the large size of the images, but I can only use examples of the type of images that are giving me problems.
I should make it clear that this is a different image to the one that I used for the above MD5 checksums. I didn't think to use the same image.
UPDATE: Using the two images I linked above, I found the differences between the two files. There are two sequences of 32 bytes that are different in one image from the other. The files are otherwise identical.
n.b. the first character is numbered as character 1, not character 0
n.b. 2116624 and 2493456 are both 16 modulo 32.
The difference between them is 368 * 1024.
character 2116624 through line 2116655 (32 characters):
original image:
3c bc 20 19 eb 93 cd 34 db 93 68 7c 8d 5e 37 d4
d3 84 91 70 7e 7c 82 3e e9 e8 3d eb 2e ff 00 ce
bad image:
89 c9 54 a6 e5 3d f4 fc 8e 7f 68 d5 47 14 41 f6
55 11 7d a6 55 24 a8 e0 f7 a8 a4 43 06 18 c2 c1
character 2493456 through character 2493487 (32 characters):
original image:
3d a4 61 37 19 75 63 2e 51 62 ca 07 e0 9e 49 35
5d 65 8d 22 01 7f 5e b4 a7 19 3a 7a ef 72 1c e3
bad image:
4f 99 26 39 6a db d9 cd 3e 5b ec 63 46 3c b4 ef
2d b6 30 35 0f 12 a1 b6 9a 11 68 1a 69 07 0c 49
Are you sure there is no problem with your internet connection? Have you tried it from a different connection or even ISP?
Its a stretch, but one possible issue I can think of is that you're working on the temporary file, try using move_uploaded_file to move the file to somewhere you've specified, and see if that makes a difference?
Rather than attempt a conversion at this point, dump the temp file info and compare the file size etc.
Because of the random nature of this, it is probably some form of transmission problem. You can probably rule out your own hardware by trying to move a decent size zip file (for example) from the computer you are running to something else on the local network (multiple times) and then seeing if the zip file passes the crc check or becomes corrupted.
Also: are you running a firewall appliance which sniffs http traffic? (e.g. untangle)