I seem to be stuck at sending the compressed messages from PHP to NodeJS over Amazon SQS.
Over on the PHP side I have:
$SQS->sendMessage(Array(
'QueueUrl' => $queueUrl,
'MessageBody' => 'article',
'MessageAttributes' => Array(
'json' => Array(
'BinaryValue' => bzcompress(json_encode(Array('type'=>'article','data'=>$vijest))),
'DataType' => 'Binary'
)
)
));
NOTE 1: I also tried putting compressed data directly in the message, but the library gave me an error with some invalid byte data
On the Node side, I have:
body = decodeBzip(message.MessageAttributes.json.BinaryValue);
Where message is from sqs.receiveMessage() call and that part works since it worked for raw (uncompressed messages)
What I am getting is TypeError: improper format
I also tried using:
PHP - NODE
gzcompress() - zlib.inflateraw()
gzdeflate() - zlib.inflate()
gzencode() - zlib.gunzip()
And each of those pairs gave me their version of the same error (essentially, input data is wrong)
Given all that I started to suspect that an error is somewhere in message transmission
What am I doing wrong?
EDIT 1: It seems that the error is somewhere in transmission, since bin2hex() in php and .toString('hex') in Node return totally different values. It seems that Amazon SQS API in PHP transfers BinaryAttribute using base64 but Node fails to decode it. I managed to partially decode it by turning off automatic conversion in amazon aws config file and then manually decoding base64 in node but it still was not able to decode it.
EDIT 2: I managed to accomplish the same thing by using base64_encode() on the php side, and sending the base64 as a messageBody (not using MessageAttributes). On the node side I used new Buffer(messageBody,'base64') and then decodeBzip on that. It all works but I would still like to know why MessageAttribute is not working as it should. Current base64 adds overhead and I like to use the services as they are intended, not by work arounds.
This is what all the SQS libraries do under the hood. You can get the php source code of the SQS library and see for yourself. Binary data will always be base64 encoded (when using MessageAttributes or not, does not matter) as a way to satisfy the API requirement of having form-url-encoded messages.
I do not know how long the data in your $vijest is, but I am willing to bet that after zipping and then base64 encoding it will be bigger than before.
So my answer to you would be two parts (plus a third if you are really stubborn):
When looking at the underlying raw API it is absolutely clear that not using MessageAttributes does NOT add additional overhead from base64. Instead, using MessageAttributes adds some slight additional overhead because of the structure of the data enforced by the SQS php library. So not using MessageAttributes is clearly NOT a workaround and you should do it if you want to zip the data yourself and you got it to work that way.
Because of the nature of a http POST request it is a very bad idea to compress your data inside your application. Base64 overhead will likely nullify the compression advantage and you are probably better off sending plain text.
If you absolutely do not believe me or the API spec or the HTTP spec and want to proceed, then I would advise to send a simple short string 'teststring' in the BinaryValue parameter and compare what you sent with what you got. That will make it very easy to understand the transformations the SQS library is doing on the BinaryValue parameter.
gzcompress() would be decoded by zlib.Inflate(). gzdeflate() would be decoded by zlib.InflateRaw(). gzencode() would be decoded by zlib.Gunzip(). So out of the three you listed, two are wrong, but one should work.
Related
When using PHP with get_file_contents() to read in a file that has been encrypted, it doesn't change any of the encrypted contents.
When replicating it in NodeJS though through fs.readFile(file, "utf8") or even fs.readFileSync(file, "utf8") the encryption is somehow manipulated as when being checked the same way as the PHP version it fails.
Reading in that same file without encryption works with fs.readFile(file, "utf8").
I have tried ignoring encoding all together, swapping to binary encoding and nothing seems to work.
Is there a way to keep fs.readFile() or fs.readFileSync() from change encryption characters?
You should not use the "utf8" parameter in fs.readFile(file, "utf8").
It seems like the right parameter for you would be "latin1" or its legacy alias "binary" which communicates your intention more clearly.
You can check the available encodings here.
the goal is to make a http request (empty) from Angular 7 to PHP to receive binary data in Angular for the use with protobuf3.
More specifically, the binary data (encoded like described here: https://developers.google.com/protocol-buffers/docs/encoding) in PHP (source) is encapsulated in a string, while the goal in Angular is a Uint8Array.
Therefore, I currently have the following working code:
PHP Code (a simple ProcessWire root template):
header('Content-Type: application/b64-protobuf');
…
echo base64_encode($response->serializeToString());
Angular:
let res = this.httpClient.get(`${this.API_URL}`, { responseType: 'text' });
res.subscribe((data) => {
let binary_string = atob(data);
let len = binary_string.length;
let bytes = new Uint8Array(len);
for (let i = 0; i < len; i++) {
bytes[i] = binary_string.charCodeAt(i);
}
let parsedResponse = pb.Response.deserializeBinary(bytes)
})
As you can see I encode the data as base64 before sending it. So, it is not as efficient as it could be, because base64 reduces the amount of information per character. I tried already quite a lot to get binary transmission working, but in the end the data always gets corrupted, i.e. the variable bytes is not identical to the argument of base64_encode.
But still, according to some sources (e.g. PHP write binary response, Binary data corrupted from php to AS3 via http (nobody says it would not be possible)) it should be possible.
So my question is: What must change to directly transfer binary data? Is it even possible?
What have I tried?
using different headers, such as
header('Content-Type:binary/octet-stream;'); or using Blob in Angular.
I also tried to remove base64_encode from the PHP Code and atob
from the Angular Code. The result: the content of the data is modified between serializeToString and deserializeBinary(bytes), which is not desired.
I checked for possible characters before <?php
Specifications:
PHP 7.2.11
Apache 2.4.35
Angular 7.0.2
If further information is needed, just let me know in the comments. I am eager to provide it. Thanks.
I'm trying to mimick an application that sends octet streams to and from a server. The data contained in the body looks like raw bytes, and I'm fairly certain the data being sent for each command is static, so I'm hoping to map the bytes to something more readable in my application. For example, I'll have an array that does: "test" => "&^D^^&#*#dgkel" So I can call "test" and get the real bytes that need to be sent. Trouble is, PHP seems to convert these bytes. I'm not sure if it is an encoding problem or what, but what has been happing is I'll give it some bytes (for example, �ھ����#�qs��������������������X����������������������������) which has a length of 67 I believe, but PHP will say (when I do a var_dump of the HTTP request) that the headers sent contained "Content-Length: 174" or something close to that and the bytes will look like �ھ����#�qs��������������������X����������������������������
So I'm not really sure how to fix this.. Anyone have any ideas? Cheers!
Edit, a little PHP:
$request = new HttpRequest($this->GetMessageURL(), HTTP_METH_POST);
$request->addHeaders($headers);
$request->addRawPostData($buttonMapping[$button]);
$request->send();
I have a PHP webservice which currently returns a zip archive as its only output. I'm reading the zip archive from disk using file_get_contents and sending it back as the body of the response.
I'd like it to return some additional metadata, in a JSON format:
{
"generatedDate": "2012-11-28 12:00:00",
"status": "unchanged",
"rawData": <zip file in raw form>
}
The iOS app which talks to this service will receive this response, parse the JSON, and then store the zip file locally for its own use.
However, if I try to stuff the result of file_get_contents into json_encode, it rightfully complains that the string is not in UTF-8 format. If I UTF-8-encode it using mb_convert_encoding($rawData, 'UTF-8',
mb_detect_encoding($rawData, 'UTF-8, ISO-8859-1', true));, it will encode it happily, but I can't find a way to reverse the operation on the client (calling [dataString dataUsingEncoding:NSUTF8StringEncoding] and then treating the result as a zip file fails with BOM could not extract archive: Couldn't read pkzip local header.
Can anyone suggest a good way to insert a blob of raw data as one field in a JSON response?
Surely if you successfully included the raw data in the JSON then you'd have the opposite problem at the other end, when you try to decode the JSON and whatever you use to decode can't handle the raw data?
Instead, I would suggest that you send the raw data only in the response body, and use headers to send the metadata.
Strike this question.
It turns out that UTF-8 encoding raw data like this is nonstandard at best, and the standard solution is base-64 encoding it and then using a base-64 decoder to recover it on the client:
$this->response(200, array('rawData' => base64_encode($rawData)));
...
NSString *rawDataString = [[response responseJSON] objectForKey:#"rawData"];
NSData *rawData = [Base64 decode:rawDataString];
ZIP archives are not text—they are binary files! Trying to convert your archive from ISO-8859-1 to UTF-8 makes as much sense as trying to rotate it.
There're several algorithms to serialize binary streams as text but they'll all increase the file size. If that's not an issue, have a look at:
base64_encode()
bin2hex()
unpack()
We have an iOS app that sends data in an encoded format. In PHP the following code will decode it properly.
bson_decode(pack("H*", $hex_string));
In Python, the following code will create a valid encoded object that the PHP code can then decode (data is a dict in this).
from bson import BSON
def encode(data):
return str(BSON.encode(data)).encode('hex')
The following Python code will decode a string that was encoded by the above Python code:
from bson import BSON
def parse(str):
hexed = str.decode('hex')
return BSON.decode(BSON(hexed))
In theory that should decoded data sent from the app as well. But it throws the following exceptions:
bson.errors.InvalidBSON: bad eoo
It looks like the Objective C code that encodes the data in the app adds some extra padding. If I remove the last characters from the app encoded string it works. Is there anything I can do to account for this? Changing the app code is NOT possible. Even if it were there are millions of device running the old code which I need to support so I still need to have a fix for this.
According the BSON specification, BSON documents must be terminated with a NULL byte (\x00). Have you checked if the byte string you are trying to decode is NULL terminated? If not, you may need to append a NULL byte at the end.