Reading the ZIP file central directory in PHP - php

Question

I need to obtain the uncompressed filesize of a zip, without storing the entire zip into memory (files range up to 30gb+)
From research I understand this information can be obtained from the central-directory that all Zips have, and that it's stored at the end of the file.
In most scenarios, I can find the central-directory within the last 64kb of a file.
function findCentralDirectory($data) {
$offset = 0;
while ($offset < 65536) {
$pos = strpos($data, "\x50\x4b\x01\x02", $offset);
if ($pos !== false) {
return $pos;
}
$offset++;
}
return false;
}
function findEndCentralDirectory($data) {
$offset = 0;
while ($offset < 65536) {
$pos = strpos($data, "\x50\x4b\x05\x06", $offset);
if ($pos !== false) {
return $pos;
}
$offset++;
}
return false;
}
$data = $this->extractLast64Kb($path);
$startOffset = $this->findCentralDirectory($data);
$endOffset = $this->findEndCentralDirectory(substr($data, $startOffset));
$centralDirectory = substr($data, $startOffset, $endOffset);
This appears to work correctly as I'll get this in response:
PK?4>U
~��test.txtPK6*
(this zip in particular has one file, called test.txt, tested with multiple zips, all showing the correct list of files in this output)
I'm lead to believe that in the encoded part of this, is the filesize, however being new to this kind of programming, i'm struggling to work out how I can "decode" this into an array I can sum.
Just looking to get this working for 32-bit zips before I worry about 64-bit.
Any help appreciated, thanks

Answer 1

I was surprised that there isn't a package out there that does this already.
The snippet below is the extract of the "central directory file header" and before it was copied/pasted here; it contained an entry for each file (and directory) within the zip which included information like uncompressed filesize etc (everything you should need to know about a file).
What we've got
PK?4>U
~��test.txtPK6*
As shown in the question, this is the binary data extract between 0x504b0102 and 0x504b0506.
One thing to note is that for ZIP64 archives, the "end of central directory" signature is 0x504b0606 (which you can translate to \x50\x4b\x06\x06) and you should check for this signature if the 32-bit end offset was not found.
What we do with it
So what we do now is we feed that binary data into a buffer, for my implementation I used the nelexa/buffer package and as we've extracted the exact binary data that we need to parse we can start the buffer at position 0 to make things a bit easier.
$binaryData = '..';
$buffer = new \Nelexa\Buffer\StringBuffer($binaryData);
$bufferPosition = 0;
$entries = [];
// ensure our position does not exceed the size of our data
while ($bufferPosition < $buffer->size()) {
$nameLength = $buffer
->setPosition($bufferPosition + 28)
->getArrayBytes(1)[0];
$extraLength = $buffer
->setPosition($bufferPosition + 30)
->getArrayBytes(1)[0];
$commentLength = $buffer
->setPosition($bufferPosition + 32)
->getArrayBytes(1)[0];
$size = $buffer
->setPosition($bufferPosition + 24)
->getUnsignedInt();
$crc32 = $buffer
->setPosition($bufferPosition + 16)
->getUnsignedInt();
$entries[] = [
'isDirectory' => $size === 0 && $crc32 === 0,
'size' => $size,
'crc32' => $crc32,
'filename' => $buffer
->setPosition($bufferPosition + 46)
->getString($nameLength);
'comment' => $buffer
->setPosition($bufferPosition + 46 + $nameLength + $extraLength)
->getString($commentLength);
];
// set the position of the next file entry
$bufferPosition += 46 + $nameLength + $extraLength + $commentLength;
}
var_dump($entries);
The code above should give you a working example and the general gist of how it works, you can find more information about the offsets/positions used here: Wikipedia - ZIP (file format).
Output for the binary data above should be similar to:
array:1 [
0 => array:5 [
"isDirectory" => false
"filename" => "test.txt"
"size" => 307
"crc32" => 1071205881
"comment" => ""
]
]
Gotchas
It is important to note that you may encounter endianness issues depending on the system in use (which is a fancy way of saying your bytes will be reversed). In such a scenario you would note that the example above will output incorrect values for file size and the CRC-32.
A good way to check if your system is little-endian or big-endian is:
function isLittleEndian(): bool
{
$testInteger = 0x00FF;
$packed = pack('S', $testInteger);
$unpacked = (array)unpack('v', $packed);
return $testInteger === current($unpacked);
}
If your system is little-endian, then you'll want to convert integers to big-endian instead, using this byte-swap function:
function convertToBigEndian(int $value): int
{
return (($value & 0x000000FF) << 24) | (($value & 0x0000FF00) << 8) | (($value & 0x00FF0000) >> 8) | (($value & 0xFF000000) >> 24);
}
and then we can make sure we have correct values for $size and $crc32 from our example above:
$size = ...
$crc32 = ...
// convert endianness
if (isLittleEndian()) {
$size = convertToBigEndian($size);
$crc32 = convertToBigEndian($crc32);
}
WARNING: It is still possible to parse the central-directory from corrupt zips (e.g zips that the native ZipArchive cannot open due an error). Therefore the existence of the central-directory should not be used as an indicator of whether the ZIP is compliant/valid/working.
A few bonus examples of fetching the last 100kib of files using a few different methods
$range = "bytes=-".(100*1024*1024);
Using AWS S3 client
$result = $s3client->getObject([
'Bucket' => 'your-bucket',
'Region' => 'your-region',
'Key' => 'path/to/file.zip',
'Range' => $range,
]);
$data = (string) $result['Body'];
Using cURL
$url = 'https://somewhere.com/path/to/file.zip';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RANGE, '-'.$bytes);
$data = (string) curl_exec($ch);
curl_close($ch);
Using file_get_contents()
$url = 'https://somewhere.com/path/to/file.zip';
$opts = [
'http' => [
'method' => 'GET',
'header' => [
'Range: '.$range
],
]
];
$context = stream_context_create($opts);
$data = (string) file_get_contents($url, false, $context);
Using Guzzle
$url = 'https://somewhere.com/path/to/file.zip';
$client = new \GuzzleHttp\Client();
$res = $client->request('GET', $url, [
'headers' => [
'Range' => $range,
]
]);
$data = (string) $res->getBody();
Hope this helps someone, there was very little information out there on this topic and even less for PHP.
Disclaimer: I am an amateur at this kind of thing, in fact this is my first time having any success at it. Please comment if I have made any mistakes