Cannot create large zip file on GAE / Cloud Storage / PHP - php

Question

I've read similar posts on ZipArchive and GAE but they do not solve my problem. Creating the .zip to the GAE tmp folder exhausts memory as the zip may be 1G or more. Therefore, I'm trying to write directly to a storage bucket.
The output below shows the results of running my zip function. I'm sort of new to GAE/Storage so hoping some more eyes could help me out!
One thing I notice is that $zip->filename = /workspace/gs:/X... Why is "workspace" there and why only one "/" after gs:?
If there is some other method to achieve this, I'm open to suggestions!
I have this working in LAMP, but need to move to GAE. Here's my code of the function that does the zipping:
<?php
function create_zip($files = array(),$fileDestinations = array(),$destination='',$overwrite = false) {
$valid_files = array();
if(is_array($files)) {
//cycle through each file
foreach($files as $file) {
//make sure the file exists OR THE ZIP WON'T BE CREATED
if(file_exists($_SESSION['devStorage'].$file)) {
$valid_files[] = $_SESSION['devStorage'].$file;
}else{
$valid_files[] = $_SERVER['DOCUMENT_ROOT']."/images/default.jpg";
}
}
}
//if we have good files...
if(count($valid_files)) {
if (is_writable(str_replace("https://storage.googleapis.com/","gs://",$_SESSION['devStorage']))) {
echo '<br>'.str_replace("https://storage.googleapis.com/","gs://",$_SESSION['devStorage']).' Folder is writable<br>';
} else {
echo '<br>'.str_replace("https://storage.googleapis.com/","gs://",$_SESSION['devStorage']).' Folder not writable<br>';
}
//create the archive
$zip = new ZipArchive();
if($zip->open($destination, ZIPARCHIVE::CREATE | ZIPARCHIVE::OVERWRITE)!==true) {
return "<br>can't open ".$destination;
}
//add the files
foreach($valid_files as $file) {
$zip->addFile($file,current($fileDestinations));
next($fileDestinations);
}
//debug
echo '<br>The zip archive contains ',$zip->numFiles,' files with a status of ',$zip->status;
echo '<br>zip->filename = '.$zip->filename;
//close the zip -- done!
if($zip->close()!==true) {
echo '<br>zip->close error';
}
//check to make sure the file exists
if (!file_exists($destination)) {
return "<br>File does not exist at ".$destination;
}else{
return true;
}
} else {
return "no valid files";
}
}
?>
The resulting output:
gs://X.appspot.com/uploads/ Folder is writable
The zip archive contains 139 files with a status of 0
zip->filename = /workspace/gs:/X.appspot.com/uploads/XSpring2021asof122920_06-03-2021_11-18-45
zip->close error
File does not exist at gs://X.appspot.com/uploads/XSpring2021asof122920_06-03-2021_11-18-45
Thanks for any help offered!

Answer 1

Ended up using ZipStream. My function for zipping is below:
/* creates a compressed zip file */
function create_zip($files = array(),$fileDestinations = array(),$destination='') {
// Autoload the dependencies
require 'vendor/autoload.php';
// enable output of HTTP headers
$options = new ZipStream\Option\Archive();
$options->setSendHttpHeaders(true);
$options->setFlushOutput(true);
$options->setZeroHeader(true);
$options->setStatFiles(true);
// create a new zipstream object
$zip = new ZipStream\ZipStream($destination, $options);
$valid_files = array();
if(is_array($files)) {
//cycle through each file
foreach($files as $file) {
if (($_SERVER["REMOTE_ADDR"]=="127.0.0.1") || ($_SESSION["LAMP"]==1)) {
// //make sure the file exists OR THE ZIP WON'T BE CREATED
//if(file_exists($_SERVER["REQUEST_SCHEME"]."://".$_SERVER['HTTP_HOST'].$file)) {
$valid_files[] = $_SERVER["REQUEST_SCHEME"]."://".$_SERVER['HTTP_HOST'].$file;
//}else{
// $valid_files[] = $_SERVER["REQUEST_SCHEME"]."://".$_SERVER['HTTP_HOST']."/images/default.jpg";
//}
}else{
// //make sure the file exists OR THE ZIP WON'T BE CREATED
//if(file_exists($file)) {
$valid_files[] = $file;
//}else{
// $valid_files[] = $_SERVER["REQUEST_SCHEME"]."://".$_SERVER['HTTP_HOST']."/images/default.jpg";
//}
}
}
}
//if we have good files...
if(count($valid_files)) {
foreach($valid_files as $url) {
// Create Temp File
$fp = tmpfile();
// Download File
$ch = curl_init();
curl_setopt($ch, CURLOPT_FILE, $fp);
curl_setopt($ch, CURLOPT_URL, $url);
curl_exec($ch);
// Force to write all data
fflush($fp);
// Rewind to the beginning, otherwise the stream is at the end from the start
rewind($fp);
// Add File
$zip->addFileFromStream(current($fileDestinations), $fp);
next($fileDestinations);
// Close the Temp File
curl_close($ch);
fclose($fp);
}
// Finish ZIP
$zip->finish();
return true;
}else{
return "no valid files";
}
}