I have an array that i need to pack with python to 16bit depth with
I have been doing this with php without any issues.
Array is just just large set of numbers like this - [1, 2, 3, 4, 5, 700, 540...]
With php I do this process in one line:
$encoded_string = pack("s*", ...$array); // Done
I can not for the love of god figure out how same can be done in python
I have read the documentation, I looked at examples and I can not get this done
Best I have is below and it does not work in any variation i have tried.
encoded_string = struct.pack('h', *array)
You have to call struct.pack on each member of your array:
import struct
nums = [1, 2, 3, 4, 5, 700, 540]
as_bytes = [struct.pack('h', i) for i in nums]
# Produces
[b'\x01\x00', b'\x02\x00', b'\x03\x00', b'\x04\x00', b'\x05\x00', b'\xbc\x02', b'\x1c\x02', b'\x08\x00']
and then that you can join into a single byte string if you want:
>>> b''.join(as_bytes)
b'\x01\x00\x02\x00\x03\x00\x04\x00\x05\x00\xbc\x02\x1c\x02'
Note: you can also use the endianness modifiers to specify the alignment for the output bytes.
Edit: #Proper reminded me that struct.pack's formatting also supports specifying the number of target packed types, so this can be done more easily by including the data length in an f-string with the format specifier:
>>> struct.pack(f'{len(data)}h', *data)
b'\x01\x00\x02\x00\x03\x00\x04\x00\x05\x00\xbc\x02\x1c\x02'
Thank you for the reply b_c to be honest I hate python with a passion at this point, there was another problem that i had to fix, the array was created as str and not int after it was "exploded". So it had to be remapped to int.
Your code does work, thank you.
There is a way to do it with my initial code, however you have to define number of values you want to process. It is possible to simply count number of values in the array and add than in to make it automated
data_array = map(int,data) # converts all values in the array to int
encoded_string = struct.pack('240s',*data_array) # 240 is number of values in the array
Related
I am writing a class for the decoding of fax data encoded with modified Huffman code.
Data is coded line by line: data describes each pixel row. Lines are coded as records of variable length. The pixel bits are stored in the bits of code words, least significant first.
Recently the code word list (182 elements) is defined as an array:
/**
* [0] code word
* [1] length of code word
* [2] run length of color bits
* [3] 0 = white / 1 = black
* [4] 1 = termination codes / 0 = make up codes
*/
const CODEWORDS = [
[0b00110101, 8, 0, 0, 1], // termination codes white
[0b000111, 6, 1, 0, 1],
[0b0111, 4, 2, 0, 1],
[0b1000, 4, 3, 0, 1],
[0b1011, 4, 4, 0, 1],
[0b1100, 4, 5, 0, 1],
[0b1110, 4, 6, 0, 1],
[0b1111, 4, 7, 0, 1],
[0b10011, 5, 8, 0, 1],
...
];
Before usage the array is sorted in descending order according to the length of the code words.
In a first approach I´m able to find the correct code words with repeating foreach-iterations over this array - but it's (not surprising!) terribly slow.
It is clear to me, that an increase in performance can only be achieved using a binary tree.
But even after looking at several explanations here or solutions (libraries) in GitHub, I can't find access to
how to transfer the data from the array into a binary tree
how to browse the tree to get the right leaf
If someone could help me there, I would be very grateful.
Once you have the correct codes (see my comments on your question), then you start by building one set of codes for white and one for black. For each, you start the tree with a branch for the first bit being zero, and another branch for one. Break up your set of codes into two sets, one set where all the codes start with zero, and the other where they all start with one. For each of those, make two branches. Break up each set based on the second bit. Once you get to a branch with one code and you just used the last bit of that code, you now have a leaf. In that leaf you store the symbol for the code, e.g. 63 for white code 00110100. If you get to a branch and there are no codes, then you again have a leaf, but this time it will result in a decoding error if it is reached.
To decode, take the first bit and go down that branch. Choose the second branch depending on the second bit. And so on until you get to a leaf. Then emit that symbol and start back again at the root with the subsequent bit. Or terminate if you end up at an error leaf.
I want to convert byte array to UINT64 using PHP.
I can do this easily in C# but I want to do this in PHP.
Here is C# code.
bytes = Encoding.UTF8.GetBytes(hashed);
BitConverter.ToUInt64(bytes, 0);
I want to convert this to PHP.
I tried to use pack() function but this does not works.
Let's say this is a byte array.
$bytes = [101,102,54,55,99,55,56,49];
pack("J*","101","102","54","55","99","55","56","49");
This shows a warning.
epack(): 7 arguments unused on line
How can I fix this?
The major issue here (if I understand it correctly) is you're using PHP numbers to represent a byte array however unpack requires an input string. If you keep the array as is then PHP seems to just convert the numbers to strings meaning a 101 will be '101' which in turn is 3 bytes, which breaks the whole thing down.
You need to first convert the numbers to bytes. A byte is essentially as an unsigned char so you could first pack your array into unsigned chars and then unpack them:
$bytes = [101,102,54,55,99,55,56,49];
$unpacked = unpack("J", pack('C*', ...$bytes));
echo current($unpacked);
Explanation:
C is the pack code for unsigned char and * indicates that you need to use all array entries. This will generate a string of characters based on the array. You can then unpack this string using J (if you know for a fact that the bytes were generated in a big endian byte order) or P if you know the bytes were generated in little endian order or Q if you want to use the machine order. If the bytes were generated in the same machine then Q would probably be a better choice than J otherwise you need to know the endianess.
Example: http://sandbox.onlinephpfunctions.com/code/5cba2c29522f7b9f9a0748b99fac768012e759ce
Note: This is my personal understanding of what is happening so anyone with better pack/unpack knowledge can let me know if I got things wrong.
I posted this (php pack: problems with data types and verification of my results) and found that I had two problems.
So here again only one issue (I solved the other one) Hopefully this is easy to understand:
I want to use the PHP pack() function.
1) My aim is to convert any integer number info a hex one of length 2-Bytes.
Example: 0d37 --> 0x0025
2) Second aim is to toggle high / low byte of each value: 0x0025 --> 0x2500
3) There are many input values which will form 12-Bytes of binary data.
Can anyone help me?
You just have to lookup the format table in the pack() manual page and it is quite easy.
2 bytes means 16 bits, or also called a "short". I assume you want that unsigned ... so we get n for big endian (high) and v for little endian (low) byte order.
The only potentially tricky part is figuring out how to combine the format and parameters, as each format character is tied to a value argument:
bin2hex(pack('nv', 34, 34)) // returns 00222200
If you need a variable number of values, you'll need agument unpacking (a PHP language feature, not to be confused with unpack()):
$format = 'nv';
$values = [34, 34];
pack($format, ... $values); // does the same thing
And alternatively, if all of your values should be packed with the same format, you could do this:
pack('v*', $values); // will "pack" as many short integers as you want
Assume that I have a file named data.txt with the contents "Blah Blah !".
So when I use the code below
$hnd=fopen('data.txt','r');
echo fgets($hnd,2);
it displays just one character "B" instead of "Bl". Later I read the manual stating:
length
Reading ends when length - 1 bytes have been read, or a newline (which is included in the return value), or an EOF (whichever comes first). If no length is specified, it will keep reading from the stream until it reaches the end of the line.
Can anyone explain to me why it is this way? I mean why is it length-1 and not length.
The C fgets() function reads length - 1 bytes, because it has to add a terminating zero to turn the data into a proper string.
My best guess is that PHP's fgets() exhibits the same behaviour because it is either:
a legacy from the bad old days when PHP functions were little more that wrappers around the corresponding C functions, and string functions were binary unsafe (eg. strings could not contain embedded NUL characters). Changing the behaviour of the fgets() function would introduce new bugs in existing programs. Or,
a deliberate decision to make the PHP function compatible with the C function to avoid unnecessary surprises.
or both.
Interestingly, it looks like PHP internally adds a terminating zero when storing string values, for example in _php_stream_get_line() (called from fgets()) and zend_string_init().
Since _zend_string objects store the string length anyway, it shouldn't be necessary to store the terminating zero, unless there are still binary unsafe functions in PHP.
Because PHP, like many C-derivatives count from 0, and not from 1. They have Zero-based numbering
Eg for arrays: An array of length/size n has 0 to n - 1, elements.
i.e. 0, 1, 2 , 3, 4 .... n-1
So an array of length 5 has elements 0, 1, 2, 3, 4
So you will find that whether reading byte, strings, arrays... they always reference the to the (n-1)th element or marker, for an n-size structure
Please use following code for your raised questionarries
$hnd=fopen('E:\\data.txt','r');
echo fgets($hnd,2);
I'm working in a LAMP environment, so PHP is the language; at least i can use python.
As the title said i have two unordered integer arrays.
$array_A = array(13, 4, 59, 38, 9, 69, 72, 93, 1, 3, 5)
$array_B = array(29, 72, 21, 3, 6)
I want to know how many integers these array have in common; in the example as you see the result is 2. I'm not interested in what integers are in common, like (72, 3).
I need a faster method than take every element of array B and check if it's in array A ( O(nxm) )
Arrays can be sorted through asort or with sql ordering (they came from a sql result).
An idea that came to me is to create a 'vector' for every array where the integer is a position who gets value 1 and integers not present get 0.
So, for array A (starting at pos 1)
(1, 0, 1, 1, 1, 0, 0, 0, 1, 0, ...)
Same for array B
(0, 0, 1, 0, 0, 1, ...)
And then compare this two vectors with one cycle. The problem is that in this way the vector length is about 400k.
Depending on your data (size) you might want to use array_intersect_key() instead of array_intersect(). Apparently the implementation of array_intersect (testing php 5.3) does not use any optimization/caching/whatsoever but loops through the array and compares the values one by one for each element in array A. The hashtable lookup is incredibly faster than that.
<?php
function timefn($fn) {
static $timer = array();
if ( is_null($fn) ) {
return $timer;
}
$x = range(1, 120000);
$y = range(2, 100000);
foreach($y as $k=>$v) { if (0===$k%3) unset($y[$k]); }
$s = microtime(true);
$fn($x, $y);
$e = microtime(true);
#$timer[ $fn ] += $e - $s;
}
function fnIntersect($x, $y) {
$z = count(array_intersect($x,$y));
}
function fnFlip($x, $y) {
$x = array_flip($x);
$y = array_flip($y);
$z = count(array_intersect_key($x, $y));
}
for ($i=0; $i<3; $i++) {
timefn( 'fnIntersect' );
timefn( 'fnFlip' );
}
print_r(timefn(null));
printsArray
(
[fnIntersect] => 11.271192073822
[fnFlip] => 0.54442691802979
)which means the array_flip/intersect_key method is ~20 times faster on my notebook.
(as usual: this is an ad hoc test. If you spot an error, tell me ...I'm expecting that ;-) )
I don't know a great deal about PHP so you may get a more specific answer from others, but I'd like to present a more language-agnostic approach.
By checking every element in A against every element in B, it is indeed O(n2) [I'll assume the arrays are of identical length here to simplify the equations but the same reasoning will hold for arrays of differing lengths].
If you were to sort the data in both arrays, you could reduce the time complexity to O(n log n) or similar, depending on the algorithm chosen.
But you need to keep in mind that the complexity only really becomes important for larger data sets. If those two arrays you gave were typical of the size, I would say don't sort it, just use the "compare everything with everything" method - sorting won't give you enough of an advantage over that. Arrays of 50 elements would still only give you 2,500 iterations (whether that's acceptable to PHP, I don't know, it would certainly be water off a duck's back for C and other compiled languages).
And before anyone jumps in and states that you should plan for larger data sets just in case, that's YAGNI, as unnecessary as premature optimization. You may never need it in which case you've wasted time that would have been better spent elsewhere. The time to implement that would be when it became a problem (that's my opinion of course, others may disagree).
If the data sets really are large enough to make the O(n2) unworkable, I think sorting then walking through the arrays in parallel is probably your best bet.
One other possibility is if the range of numbers is not too big - then your proposed solution of a vector of booleans is quite workable since that would be O(n), walking both arrays to populate the vector followed by comparisons of fixed locations within the two vectors. But I'm assuming your range is too large or you wouldn't have already mentioned the 400K requirement. But again, the size of the data sets will dictate whether or not that's worth doing.
The simplest way would be:
count(array_intersect($array_A, $array_B));
if I understand what you're after.
Should be fast.
If both arrays came from SQL, could you not write an SQL query with an inner join on the 2 sets of data to get your result?
You want the array_intersect() function. From there you can count the result. Don't worry about speed until you know you have a problem. The built-in function execute much faster than anything you'll be able to write in PHP.
I have written a PHP extension that provides functions for efficient set operations like union, intersection, binary search, etc. Internal data layout is an ordinary int32_t array stored in a PHP string. Operations are based on merge algorithms.
Example:
// Create two intarrays
$a = intarray_create_from_array(array(1, 2, 3));
$b = intarray_create_from_array(array(3, 4, 5));
// Get a union of them
$u = intarray_union($a, $b);
// Dump to screen
intarray_dump($u);
It's available here: https://github.com/tuner/intarray