I'm trying to convert this piece of PHP code to Python. I've figured out how to do the crc32b hash but I'm stuck on proc_open(). Could you help me out?
What I want is to get the value of $pipeOutput in Python code.
PHP
$hashedData = strrev(hash("crc32b", $data, true)) . $data;
$xzBinary = trim(`which xz`);
$xzProcess = proc_open("$xzBinary '--format=raw' '--lzma1=lc=3,lp=0,pb=2,dict=128KiB' '-c' '-'", [
0 => [
"pipe",
"r"
],
1 => [
"pipe",
"w"
]
], $xzProcessPipes);
fwrite($xzProcessPipes[0], $hashedData);
fclose($xzProcessPipes[0]);
$pipeOutput = stream_get_contents($xzProcessPipes[1]);
fclose($xzProcessPipes[1]);
proc_close($xzProcess);
Python
import zlib
from subprocess import Popen
def crc32b(x):
h = zlib.crc32(x)
x='%08X' % (h & 0xffffffff,)
return x.lower()
data = 'some_data'
hashed_data = crc32b(data)[::-1]+data #strrev(hash("crc32b", $data, true)) . $data;
xz_binary = '//usr/bin/xz'
r = Popen([xz_binary, "--format=raw",'--lzma1=lc=3,lp=0,pb=2,dict=128KiB','-c','-'])
EDIT
hashed_data = crc32b(data)[::-1]+data
xz_binary = '//usr/bin/xz'
p = Popen([xz_binary, "--format=raw",'--lzma1=lc=3,lp=0,pb=2,dict=128KiB','-c','-'],stdin=PIPE, stdout=PIPE,stderr=PIPE)
p.communicate()
stdout_file = p.stdout
stdin_file = p.stdin
stdin_file.write(hashed_data)
says that stdin_file is closed but I'm not sure if this is what I should do
ValueError: I/O operation on closed file
Related
I need to obtain the uncompressed filesize of a zip, without storing the entire zip into memory (files range up to 30gb+)
From research I understand this information can be obtained from the central-directory that all Zips have, and that it's stored at the end of the file.
In most scenarios, I can find the central-directory within the last 64kb of a file.
function findCentralDirectory($data) {
$offset = 0;
while ($offset < 65536) {
$pos = strpos($data, "\x50\x4b\x01\x02", $offset);
if ($pos !== false) {
return $pos;
}
$offset++;
}
return false;
}
function findEndCentralDirectory($data) {
$offset = 0;
while ($offset < 65536) {
$pos = strpos($data, "\x50\x4b\x05\x06", $offset);
if ($pos !== false) {
return $pos;
}
$offset++;
}
return false;
}
$data = $this->extractLast64Kb($path);
$startOffset = $this->findCentralDirectory($data);
$endOffset = $this->findEndCentralDirectory(substr($data, $startOffset));
$centralDirectory = substr($data, $startOffset, $endOffset);
This appears to work correctly as I'll get this in response:
PK?4>U
~��test.txtPK6*
(this zip in particular has one file, called test.txt, tested with multiple zips, all showing the correct list of files in this output)
I'm lead to believe that in the encoded part of this, is the filesize, however being new to this kind of programming, i'm struggling to work out how I can "decode" this into an array I can sum.
Just looking to get this working for 32-bit zips before I worry about 64-bit.
Any help appreciated, thanks
I was surprised that there isn't a package out there that does this already.
The snippet below is the extract of the "central directory file header" and before it was copied/pasted here; it contained an entry for each file (and directory) within the zip which included information like uncompressed filesize etc (everything you should need to know about a file).
What we've got
PK?4>U
~��test.txtPK6*
As shown in the question, this is the binary data extract between 0x504b0102 and 0x504b0506.
One thing to note is that for ZIP64 archives, the "end of central directory" signature is 0x504b0606 (which you can translate to \x50\x4b\x06\x06) and you should check for this signature if the 32-bit end offset was not found.
What we do with it
So what we do now is we feed that binary data into a buffer, for my implementation I used the nelexa/buffer package and as we've extracted the exact binary data that we need to parse we can start the buffer at position 0 to make things a bit easier.
$binaryData = '..';
$buffer = new \Nelexa\Buffer\StringBuffer($binaryData);
$bufferPosition = 0;
$entries = [];
// ensure our position does not exceed the size of our data
while ($bufferPosition < $buffer->size()) {
$nameLength = $buffer
->setPosition($bufferPosition + 28)
->getArrayBytes(1)[0];
$extraLength = $buffer
->setPosition($bufferPosition + 30)
->getArrayBytes(1)[0];
$commentLength = $buffer
->setPosition($bufferPosition + 32)
->getArrayBytes(1)[0];
$size = $buffer
->setPosition($bufferPosition + 24)
->getUnsignedInt();
$crc32 = $buffer
->setPosition($bufferPosition + 16)
->getUnsignedInt();
$entries[] = [
'isDirectory' => $size === 0 && $crc32 === 0,
'size' => $size,
'crc32' => $crc32,
'filename' => $buffer
->setPosition($bufferPosition + 46)
->getString($nameLength);
'comment' => $buffer
->setPosition($bufferPosition + 46 + $nameLength + $extraLength)
->getString($commentLength);
];
// set the position of the next file entry
$bufferPosition += 46 + $nameLength + $extraLength + $commentLength;
}
var_dump($entries);
The code above should give you a working example and the general gist of how it works, you can find more information about the offsets/positions used here: Wikipedia - ZIP (file format).
Output for the binary data above should be similar to:
array:1 [
0 => array:5 [
"isDirectory" => false
"filename" => "test.txt"
"size" => 307
"crc32" => 1071205881
"comment" => ""
]
]
Gotchas
It is important to note that you may encounter endianness issues depending on the system in use (which is a fancy way of saying your bytes will be reversed). In such a scenario you would note that the example above will output incorrect values for file size and the CRC-32.
A good way to check if your system is little-endian or big-endian is:
function isLittleEndian(): bool
{
$testInteger = 0x00FF;
$packed = pack('S', $testInteger);
$unpacked = (array)unpack('v', $packed);
return $testInteger === current($unpacked);
}
If your system is little-endian, then you'll want to convert integers to big-endian instead, using this byte-swap function:
function convertToBigEndian(int $value): int
{
return (($value & 0x000000FF) << 24) | (($value & 0x0000FF00) << 8) | (($value & 0x00FF0000) >> 8) | (($value & 0xFF000000) >> 24);
}
and then we can make sure we have correct values for $size and $crc32 from our example above:
$size = ...
$crc32 = ...
// convert endianness
if (isLittleEndian()) {
$size = convertToBigEndian($size);
$crc32 = convertToBigEndian($crc32);
}
WARNING: It is still possible to parse the central-directory from corrupt zips (e.g zips that the native ZipArchive cannot open due an error). Therefore the existence of the central-directory should not be used as an indicator of whether the ZIP is compliant/valid/working.
A few bonus examples of fetching the last 100kib of files using a few different methods
$range = "bytes=-".(100*1024*1024);
Using AWS S3 client
$result = $s3client->getObject([
'Bucket' => 'your-bucket',
'Region' => 'your-region',
'Key' => 'path/to/file.zip',
'Range' => $range,
]);
$data = (string) $result['Body'];
Using cURL
$url = 'https://somewhere.com/path/to/file.zip';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RANGE, '-'.$bytes);
$data = (string) curl_exec($ch);
curl_close($ch);
Using file_get_contents()
$url = 'https://somewhere.com/path/to/file.zip';
$opts = [
'http' => [
'method' => 'GET',
'header' => [
'Range: '.$range
],
]
];
$context = stream_context_create($opts);
$data = (string) file_get_contents($url, false, $context);
Using Guzzle
$url = 'https://somewhere.com/path/to/file.zip';
$client = new \GuzzleHttp\Client();
$res = $client->request('GET', $url, [
'headers' => [
'Range' => $range,
]
]);
$data = (string) $res->getBody();
Hope this helps someone, there was very little information out there on this topic and even less for PHP.
Disclaimer: I am an amateur at this kind of thing, in fact this is my first time having any success at it. Please comment if I have made any mistakes
I am trying to develop a web console using php and xterm.js,
I managed to get the pseudo tty allocated and attach it to xterm.js via websocket but I am not able to tell the process what is the size of terminal to make it work correctly with the size, and I couldn't find any documentation for this.
// using react/child-process
$process = new Process('/usr/bin/env bash -l', null, null, [
0 => ['pty', 'r'],
1 => ['pty', 'w'],
2 => ['pty', 'w'],
]);
$process->start($this->loop);
I found out run stty on the allocated devpts will do the tricks
if (isset($this->processes[spl_object_id($conn)])) {
$process = $this->processes[spl_object_id($conn)];
$data = unpack('i3', $raw);
$col = $data[2];
$row = $data[3];
$getstream = (function () {
return $this->stream;
});
$stty = new Process("stty cols $col rows $row", null, null, [
$getstream->call($process->stdin), // ex: /dev/pts/0
$getstream->call($process->stdout),
$getstream->call($process->stderr),
]);
$stty->start($this->loop);
}
alternatively can do this
use React\ChildProcess\Process;
$gettty = escapeshellcmd(PHP_BINARY).' -r "echo posix_ttyname(STDIN).PHP_EOL;"';
$bash = "setsid bash -l"; // setsid is important here
$process = new Process(
"$gettty && $bash",
null, null,
[
['pty', 'r'],
['pty', 'w'],
['pty', 'w'],
]
);
// TODO: read the first line and get the tty path so can run stty on it later.
Here is a simple example:
https://gist.github.com/eslym/d3bd7809681aa9c1eb34913043df9bb6
I have this code to send a packet in PHP, and I wanted to know how to do it in perl, I have tried sending via hex data, but it is not working. Here is the PHP code:
$sIPAddr = "37.221.175.211";
$iPort = 7777;
$sPacket = "";
$aIPAddr = explode('.', $sIPAddr);
$sPacket .= "SAMP";
$sPacket .= chr($aIPAddr[0]);
$sPacket .= chr($aIPAddr[1]);
$sPacket .= chr($aIPAddr[2]);
$sPacket .= chr($aIPAddr[3]);
$sPacket .= chr($iPort & 0xFF);
$sPacket .= chr($iPort >> 8 & 0xFF);
$sPacket .= 'c';
$rSocket = fsockopen('udp://'.$sIPAddr, $iPort, $iError, $sError, 2);
fwrite($rSocket, $sPacket);
fclose($rSocket);
How would I go about doing this in Perl? I want to use a raw socket as well to send it.
This is what I tried, but the server is not replying to it, which makes me think that the data is corrupted somewhere:
$packet = Net::RawIP->new({
ip => {
saddr => $saddr,
daddr => $dest,
},
udp => {
source => $rsport,
dest => $port,
data => "\x53\x41\x4d\x50\x25\xdd\xaf\xd3\x61\x1e\x63", # this is the data from the PHP file in HEX
},
});
$packet->send;
Don't know about Net::RawIP, but here's the Perl variant that sends the exact same packet as your PHP code, using IO::Socket::INET module. For docs for it, see https://metacpan.org/pod/IO::Socket::INET
use strict;
use warnings;
use IO::Socket;
my $sIPAddr = '37.221.175.211';
my $iPort = 7777;
my $sPacket = 'SAMP' . join( '', map chr,
split(/\./, $sIPAddr),
$iPort & 0xFF,
$iPort >> 8 & 0xFF,
) . 'c';
my $sock = IO::Socket::INET->new(
Proto => 'udp',
PeerPort => $iPort,
PeerAddr => $sIPAddr,
) or die "Could not create socket: $!\n";
$sock->send( $sPacket );
I have a few console command in Symfony2 and I need to execute one command from another command with some parameters.
After successfull execution of the second command I need to get the result (as an array for example), not the display output.
How can I do that?
Here you can have a basic command inside a command. The output from the second command can be a json, then you just have to decode the output json to retrieve your array.
$command = $this->getApplication()->find('doctrine:fixtures:load');
$arguments = array(
//'--force' => true
''
);
$input = new ArrayInput($arguments);
$returnCode = $command->run($input, $output);
if($returnCode != 0) {
$text .= 'fixtures successfully loaded ...';
$output = json_decode(rtrim($output));
}
you have to pass the command in the arguments array, and to avoid the confirmation dialog in doctrine:fixtures:load you have to pass --append and not --force
$arguments = array(
'command' => 'doctrine:fixtures:load',
//'--append' => true
''
);
or it will fail with error message “Not enough arguments.”
There is an new Output class (as of v2.4.0) called BufferedOutput.
This is a very simple class that will return and clear the buffered output when the method fetch is called:
$output = new BufferedOutput();
$input = new ArrayInput($arguments);
$code = $command->run($input, $output);
if($code == 0) {
$outputText = $output->fetch();
echo $outputText;
}
I did the following
use Symfony\Component\Console\Input\ArrayInput;
use Symfony\Component\Console\Output\StreamOutput;
$tmpFile = tmpfile();
$output = new StreamOutput($tmpFile);
$input = new ArrayInput(array(
'parameter' => 'value',
));
$command = . . .
$command->run($input, $output);
fseek($tmpFile, 0);
$output = fread($tmpFile, 1024);
fclose($tmpFile);
echo $output;
¡it works!
I understand it's old post and above answers solves the problem with a bit of digging. In Symfony2.7, I had a bit issue making it work, so with above suggestions, I dug a little and have compiled the full answer here. Hope it will be useful for someone.
Using Console command under console command
As an update for Onema's answer, in Symfony 3.4.x (used by Drupal 8),
you also need to set setAutoExit(false),
and the command returns an int(0) if successful.
Here's the updated example that I'm using to script composer commands in php for a Drupal 8.8 project. This gets a list of all composer packages as json, then decodes that into a php object.
<?php
require __DIR__.'/vendor/autoload.php';
use Symfony\Component\Console\Output\BufferedOutput;
use Symfony\Component\Console\Input\ArrayInput;
use Composer\Console\Application;
$input = new ArrayInput([
'command' => 'show',
'--format'=>'json',
]);
$output = new BufferedOutput();
$application = new Application();
// required to use BufferedOutput()
$application->setAutoExit(false);
// composer package list, formatted as json, will be barfed into $output
$status = $application->run($input, $output);
if($status === 0) {
// grab the output from the $output buffer
$json = $output->fetch();
// decode the json string into an object
$list = json_decode($json);
// Profit!
print_r($list);
}
?>
The output will be something like this:
stdClass Object
(
[installed] => Array
(
... omitted ...
[91] => stdClass Object
(
[name] => drupal/core
[version] => 8.9.12
[description] => Drupal is an open source content management platform powering millions of websites and applications.
)
... omitted ...
)
)
With the help of Onema's hint Google found the rest of the solution for me here.
i run a python script whitch cache some data
self.cache.set('test', 'my sample data', 300)
data = self.cache.get('test')
self.p(data)
this program will result in print of 'my sample data' ... everything its good, but when i try to access this key from php
$data = $this->cache->get('test');
print_r($test);
i only get empty result
so i check the server stats
$list = array();
$allSlabs = $this->cache->getExtendedStats('slabs');
$items = $this->cache->getExtendedStats('items');
foreach($allSlabs as $server => $slabs) {
foreach($slabs AS $slabId => $slabMeta) {
$cdump = $this->cache->getExtendedStats('cachedump',(int)$slabId);
foreach($cdump AS $server => $entries) {
if($entries) {
foreach($entries AS $eName => $eData) {
$list[$eName] = array(
'key' => $eName,
'server' => $server,
'slabId' => $slabId,
'detail' => $eData,
'age' => $items[$server]['items'][$slabId]['age'],
);
}
}
}
}
}
ksort($list);
print_r($list);
and this key 'test' is there ... but i cannot access it
if i cache something in php i get the result everytime, but somehow this python + php cache wont work
if someone has an idea where could be a problem plese advice ... i try everything
Could it be that the hashes don't match between PHP and Python? A solution is here: http://www.ruturaj.net/python-php-memcache-hash
Add the following to your Python script to change how hashes are calculated...
import memcache
import binascii
m = memcache.Client(['192.168.28.7:11211', '192.168.28.8:11211
', '192.168.28.9:11211'])
def php_hash(key):
return (binascii.crc32(key) >> 16) & 0x7fff
for i in range(30):
key = 'key' + str(i)
a = m.get((php_hash(key), key))
print i, a