How to get the progress of copying - php

I have this code that copies files and is there a possibility of getting progress of how much has been copied?
<?php
$src = '../www/';
$dst = '../../backup/';
function recurse_copy($src,$dst) {
$dir = opendir($src);
#mkdir($dst);
while(false !== ( $file = readdir($dir)) ) {
if (( $file != '.' ) && ( $file != '..' )) {
if ( is_dir($src . '/' . $file) ) {
recurse_copy($src . '/' . $file,$dst . '/' . $file);
}else {
copy($src . '/' . $file,$dst . '/' . $file);
}
}
}
closedir($dir);
}
recurse_copy($src,$dst);
}
?>
And if it is possible to get that number by ajax response. That means that response has to be sent multiple times. But I don't know how to do that.

You can track the copied files using a third argument passed by reference.
function recurse_copy($src, $dst, &$copied) {
// recurse code
$copied[] = [ 'src' => $src, 'dst' => $dst, 'file' => $file ];
copy($src.'/'.$file, $dst.'/'.$file);
}
$copied = [];
recurse_copy($src, $dst, $copied);
print_r($copied);
Other ways might be closures (function assigned to variable) and the use statement or class-properties when you put your function in a class.
Edit
For progress as in live update, see echo and flush() to see semi-live which files are beeing copied. A progress within a file copy is not possible.

There are two ways:
Knowing how many files you are going to copy, so that you can show a %
Not knowing the total number of files
In the first case, you have to get the full list of files first and then do the copying. Depending on the size of the directory tree and number of files, it can add quite some overhead, but it has its benefits. You have to decide.
In any case, you have to add a third parameter to the function:
function recurse_copy($src, $dst, callable $progress_callback = function(){}) { ... }
Bear in mind that callable is only supported from PHP 5.4 onwards.
The signature of the callback function would be:
function handle_progress($action, $data) { ... }
Then you can invoke the callback function like this:
$progress_callback('start'); // or...
$progress_callback('start', ['total' => $numfiles]);
$progress_callback('progress', ['current' => $filename]); // or...
$progress_callback('progress', ['total' => $numfiles, 'current' => $numcurrent, 'currentfile' => $filename]);
$progress_callback('finish');
You decide which data to pass to the progress callback and what to do with it once in the handler.
Now, to be able to get the progress via AJAX, you have to create a task that runs independently of the HTTP request. You return the task ID, and with AJAX you poll the progress of that ID.
In the server, you would have to use some kind of queue/task system, e.g. Laravel queues. It's only an example; you can google for more.
In the function callback, you can save to disk/redis/whatever the progress of the current task.

Related

php Scan function not returning results as expected after moving script path:

note.. all folders chmod set to 777 for testing.
Okay, so i have been trying to design a simple cloud storage file system in php.After users log in they can upload and browse files in their account.
I am having an issue with my php code that scans the user's storage area. I have a script called scan.php that is called to return all of the users files and folders that they saved.
I originally placed the scan script in the directory called files and it worked properly, when the user logged in the scan script scanned the users files using "scan(files/usernamevalue)".
However I decided that I would prefer to move the scan script inside the files area that way the php script would only have to call scan using "scan(usernamevalue)". However now my script does not return the users files and folders.
<?php
session_start();
$userfileloc = $_SESSION["activeuser"];
$dir = $userfileloc;
// Run the recursive function
$response = scan($dir);
// This function scans the files folder recursively, and builds a large array
function scan($dir)
{
$files = array();
// Is there actually such a folder/file?
$i=0;
if(file_exists($dir))
{
foreach(scandir($dir) as $f)
{
if(!$f || $f[0] === '.')
{
continue; // Ignore hidden files
}
if(!is_dir($dir . '/' . $f))
{
// It is a file
$files[] = array
(
"name" => $f,
"type" => "file",
"path" => $dir . '/' . $f,
"size" => filesize($dir . '/' . $f) // Gets the size of this file
);
//testing that code actually finding files
echo "type = file, ";
echo $f .", ";
echo $dir . '/' . $f. ", ";
echo filesize($dir . '/' . $f)." ";
echo"\n";
}
else
{
// The path is a folder
$files[] = array
(
"name" => $f,
"type" => "folder",
"path" => $dir . '/' . $f,
"items" => scan($dir . '/' . $f) // Recursively get the contents of the folder
);
//testing that code actually finding files
echo "type = folder, ";
echo $f .", ";
echo $dir . '/' . $f. ", ";
echo filesize($dir . '/' . $f)." ";
echo"\n";
}
}
}
else
{
echo "dir does not exist";
}
}
// Output the directory listing as JSON
if(!$response)
{ echo"failes to respond \n";}
header('Content-type: application/json');
echo json_encode(array(
"name" => $userfileloc,
"type" => "folder",
"path" => $dire,
"items" => $response
));
?>
As you can see i added i echoed out all of the results to see if there
was any error in the scan process, here is what i get from the output as you
can see the function returns null, but the files are being scanned, i cant
seem to figure out where i am going wrong. Your help would be greatly
appreciated. Thank you.
type = file, HotAirBalloonDash.png, test/HotAirBalloonDash.png, 658616
type = folder, New directory, test/New directory, 4096
type = file, Transparent.png, test/Transparent.png, 213
failes to respond
{"name":"test","type":"folder","path":null,"items":null}
You forgot to return files or folders in scan function, just echo values. That is the reason why you get null values in the response.
Possible solution is to return $files variable in all cases.

Is there an ImageMagick function that figures out if filetype is supported or not?

I have an array of file paths that I need ImageMagick to run through.
But some of the files are files that are not supported by ImageMagick, aborts the loop and gives me "Uncaught exception" errors. I'd like to just skip those files and move on to the next one, but I can't find any information on this.
This is my loop:
// I'd like this section to be inclosed in a function that decides
// if Imagemagick should skip the file or do the thumbnail process
$imagick = new Imagick();
$imagick->readImage($wpdm_uploads_folder . $file);
$filename = ABSPATH.'wp-content/uploads/thumbs/';
if ( ! is_wp_error( $imagick ) ) {
if($new_file['extension'] == "psd"){
$imagick->setIteratorIndex(0);
}
$imagick->thumbnailImage(200, 0);
$imagick->writeImage($filename . $new_file['filename'] . '.png');
}
$imagick->clear();
$imagick->destroy();
Imagick::pingImage is your friend for getting information about a possible image. Ping image just looks at the image's header and meta-info, and doesn't load any actual image-data. Checkout article "to Ping, or not to Ping."
<?php
$images = array(
'built-in' => 'rose:',
'valid' => '/tmp/image.png',
'invalid' => '/tmp/server.sock',
);
$wand = new Imagick();
foreach( $images as $type => $path ) {
try {
$okay = $wand->pingImage($path);
} catch (ImagickException $error) {
$okay = FALSE;
}
if($okay) {
print $path . ' is an image' . PHP_EOL;
print $path . ' has a format of ' . $wand->getImageFormat() . PHP_EOL;
} else {
print $path . ' is NOT an image' . PHP_EOL;
}
}
outputs ...
rose: is an image
rose: has a format of PPM
/tmp/out.png is an image
/tmp/out.png has a format of PNG
/tmp/server.sock is NOT an image
Edit
If you want to know the supported image formats of the system's ImageMagick library before working with files. Imagick::queryFormats can generate a list built-in, formats, and types.
For a command line approach, my first proposal would be to use
identify -ping -format "%m\n" filename
and then check return value and output on <stdout> for the magic file type. (This is probably the very same approach #emcconville used with PHP).
However, I've not tested this on a wide range of different file formats (yet).

PHP GAE readdir() doesn't work as intended

I'm trying to recursively list every file that is in my bucket. It's not too many files but I'd like to list them to test a few things. This code works on a normal file system but it's not working on Google Cloud Storage.
Anyone have any suggestions?
function recurse_look($src) {
$dir = opendir($src);
while(false !== ( $file = readdir($dir)) ) {
if (( $file != '.' ) && ( $file != '..' )) {
if ( is_dir($src . '/' . $file) ) {
recurse_look($src . '/' . $file);
}
else {
echo $src . '/' . $file;
echo "<br />";
}
}
}
closedir($dir);
}
recurse_look("gs://<BUCKET>");
Personally, I would recommend not using a filesystem-impersonation abstraction layer on top of Google Cloud Storage, for a task such as listing everything inside a bucket -- rather, just reach out for the underlying functionality.
In particular, see https://cloud.google.com/storage/docs/json_api/v1/json-api-php-samples for everything about authentication etc, and, once, that's taken care of, focus on just one line in the example:
$objects = $storageService->objects->listObjects(DEFAULT_BUCKET);
This is all you need to list all objects in a bucket (which is not the same thing as "files in a directory", and the "filesystem simulations" on top of buckets and objects, I offer as being just my personal opinion, end up hurting rather than helping despite their excellent intentions:-).
Now if the objects' names contain e.g slashes and you want to take that into account as symbolically signifying something or other, go right ahead, but at least this way you're sure you're getting all the objects actually existing in the bucket, and, nothing but those!-)
Now that glob is working, you can try something like this
function lstree($dir) {
foreach (glob($dir . '/*') as $path) {
if (is_dir($path)) {
echo $path;
lstree($path);
} else {
echo $path;
}
}
lstree('gs://{bucket}/');

Assetic is generating multiple files with same content

I have a class that uses Assetic to generate some css files to disk. I'll jump right into the code.
In my layout header, I'm doing something like this:
$assetify = new Assetify();
$assetify->setDebug(true);
$assetify->setAssetDirectory(BASE_DIR . '/public/assets');
$assetify->setOutputDirectory(BASE_DIR . '/public/assets/generated');
$assetify
->addStylesheet('/assets/css/bootstrap-2.3.2.css')
->addStylesheet('/assets/css/select2-3.4.3.css')
->addStylesheet('/assets/css/main.css');
echo $assetify->dump();
My "Assetify" class runs this through Assetic. I'll paste what I hope are only the relevant portions from the dump() function:
// The Asset Factory allows us to not have to do all the hard work ourselves.
$factory = new AssetFactory($this->assetDirectory, $this->debug);
$factory->setDefaultOutput('/generated/*.css');
// The Filter Manager allows us to organize filters for the asset handling.
// For other filters, see: https://github.com/kriswallsmith/assetic
$fm = new FilterManager();
$fm->set('yui_css', new Yui\CssCompressorFilter('/usr/local/bin/yuicompressor-2.4.7.jar'));
$fm->set('yui_js', new Yui\JsCompressorFilter('/usr/local/bin/yuicompressor-2.4.7.jar'));
$factory->setFilterManager($fm);
// The Asset Manager allows us to keep our assets organized.
$am = new AssetManager();
$factory->setAssetManager($am);
// The cache-busting worker prefixes every css with what amounts to a version number.
$factory->addWorker(new CacheBustingWorker());
$assetCollection = array();
foreach ($assetGroups as $assetGroup) {
foreach ($assetGroup as $media => $items) {
$fileCollection = array();
foreach ($items as $item) {
// Add this asset to the asset collection.
$fileCollection[] = new FileAsset($item);
}
$assetCollection[] = new AssetCollection($fileCollection);
}
}
$assetCollection = new AssetCollection($assetCollection);
$am->set('base_css', $assetCollection);
// Generate the required assets. Prefixing a filter name with a question mark
// will cause that filter to be omitted in debug mode.
$asset = $factory->createAsset(
array('#base_css'),
array('?yui_css')
);
// Configure an internal file system cache so we don't regenerate this file on every load.
$cache = new AssetCache(
$asset,
new FilesystemCache($this->outputDirectory)
);
// And generate static versions of the files on disk.
$writer = new AssetWriter($this->assetDirectory);
$writer->writeAsset($cache);
This generates two different files, 87229eb-f47a352.css and a37c1589762f39aee5bd24e9405dbdf9. The contents of the files are exactly the same. The 87229eb-f47a352.css file seems to get generated every single time, and the other file is not regenerated unless the contents of the files change (this is what I would like). If I comment out the $writer->writeAsset($cache), no files are written to disk.
What obvious configuration am I missing? I appreciate the help, thank you.
I was able to roughly replicate your code and got the same results.
I was trying to get the same results as what I think you require but ended up writing my own code to cache and serve static files.
It's not complete by any means but it is working. It has the following features:
You can choose to cache files for different pages if you specify $filename
You can choose to create versions of your released files or delete previous versions
A cached file will be generated to your target folder only if changes have made to a source file
You just need to put the code in to a class or function and return the url to serve.
Hope it helps :)
<?php
use Assetic\Factory\AssetFactory;
use Assetic\AssetManager;
use Assetic\FilterManager;
use Assetic\Asset\AssetCollection;
use Assetic\Asset\FileAsset;
use Assetic\Filter\JSMinFilter;
// JavaScript Collection
$js_collection[] = new FileAsset(SCRIPT_PATH . 'jquery.js');
$js_collection[] = new FileAsset(SCRIPT_PATH . 'production.js');
if (file_exists(SCRIPT_PATH . $page_info['name'] . '.js')) {
$js_collection[] = new FileAsset(SCRIPT_PATH . $page_info['name'] . '.js');
}
// CSS Collection
$css_collection[] = new FileAsset(STYLE_PATH . 'theme.css');
if (file_exists(STYLE_PATH . $page_info['name'] . '.css')) {
$css_collection[] = new FileAsset(STYLE_PATH . $page_info['name'] . '.css');
}
// The Filter Manager allows us to organize filters for the asset handling.
$fm = new FilterManager();
$fm->set('js', new JSMinFilter());
$js = new AssetCollection (
$js_collection
);
$js->setTargetPath(SCRIPT_PATH . 'static');
$css = new AssetCollection (
$css_collection
);
$css->setTargetPath(STYLE_PATH . 'static');
$am = new AssetManager();
$am->set('js', $js);
$am->set('css', $css);
//** TO DO: put the below in a class and return the static file names **//
// options
$seperator = '-';
$filename = $page_info['name'];
$versions = false;
// get a list of all collection names
$collections = $am->getNames();
// get each collection
foreach ($collections as $collection_name) {
// get the collection object
$collection = $am->get($collection_name);
// ensure file types are identical
$last_ext = false;
foreach ($collection as $leaf) {
$ext = strtolower(pathinfo($leaf->getSourcePath(), PATHINFO_EXTENSION));
if (!$last_ext || $ext == $last_ext) {
$last_ext = $ext;
} else {
throw new \RuntimeException('File type mismatch.');
}
}
// get the highest last-modified value of all assets in the current collection
$modified_time = $collection->getLastModified();
// get the target path
$path = $collection->getTargetPath();
// the target path must be set
if (!$path) {
throw new \RuntimeException('Target path not specified.');
}
// build the filename to check
$file = ($filename) ? $filename . $seperator . $modified_time . '.' . $ext : $modified_time . '.' . $ext;
$cached_file = $path . '/' . $file;
// the file doesn't exist so we need to minify, dump and save as new cached file
if (!file_exists($cached_file)) {
// create the output dir if it doesnt exist
if (!is_dir($path) && false === #mkdir($path, 0777, true)) {
throw new \RuntimeException('Unable to create directory ' . $path);
}
// apply the filters
if ($fm->has($collection_name)) {
$collection->ensureFilter($fm->get($collection_name));
}
// If not versioned, delete previous version of this file
if (!$versions) {
if ($filename) {
foreach (glob($path . '/' . $filename . $seperator . '*.' . $ext) as $searchfile) {
#unlink($searchfile);
}
} else {
foreach (glob($path . '/*.' . $ext) as $searchfile) {
#unlink($searchfile);
}
}
}
// put the contents in the file
if (false === #file_put_contents($cached_file, $collection->dump())) {
throw new \RuntimeException('Unable to write file ' . $cached_file);
}
}
// return the cached file
echo 'output: ' . $cached_file . '<br>';
}
exit;
?>

Are rename() and unlink() asynchronous functions?

I have strong reason to believe that both functions rename() and unlink() are asynchronous, which, from my understanding, means that when the functions are called, the code below them are continued before it finishes its procedures on the filesystem. This is a problem for the internet app I'll explain below, because later code depends on these changes to already be set in stone. So, is there a way to make both synchronous, so that the code reader freezes when it hits these functions, until all of its tasks are fully carried out on the filesystem?
Here is the code in delete-image.php, which is called by ajax from another admin-images.php(the latter will not be shown):
`
foreach ($dirScan as $key => $value) {
$fileParts = explode('.', $dirScan[$key]);
if (isset($fileParts[1])) {
if ((!($fileParts[1] == "gif") && !($fileParts[1] == "jpg")) && (!($fileParts[1] == "png") && !($fileParts[1] == "jpg"))) {
unset($dirScan[$key]);
}
} else {
unset($dirScan[$key]);
}
}
$dirScan = array_values($dirScan);
// for thumbnail
$file = 'galleries/' . $currentGal . '/' . $currentDir . "/" . $dirScan[$imageNum - 1];
unlink($file);
for ($i = ($imageNum - 1) + 1; $i < count($dirScan); $i++) {
$thisFile = 'galleries/' . $currentGal . '/' . $currentDir . '/' . $dirScan[$i];
$thisSplitFileName = explode('.', $dirScan[$i]);
$newName = 'galleries/' . $currentGal . '/' . $currentDir . "/" . ($thisSplitFileName[0] - 1) . "." . $thisSplitFileName[1];
rename($thisFile, $newName);
}
// for large image
$fileParts = explode('.', $dirScan[$imageNum - 1]);
$file = 'galleries/' . $currentGal . '/' . $currentDir . "/large/" . $fileParts[0] . "Large." . $fileParts[1];
unlink($file);
for ($i = ($imageNum - 1) + 1; $i < count($dirScan); $i++) {
$thisSplitFileName = explode('.', $dirScan[$i]);
$thisFile = 'galleries/' . $currentGal . '/' . $currentDir . '/large/' . $thisSplitFileName[0] . "Large." . $thisSplitFileName[1];
$newName = 'galleries/' . $currentGal . '/' . $currentDir . "/large/" . ($thisSplitFileName[0] - 1) . "Large." . $thisSplitFileName[1];
rename($thisFile, $newName);
}
sleep(1);
echo 'deleted ' . $dirScan[$imageNum - 1] . " successfully!";
} else {
echo "please set the post data";
} ?>`
After this script returns its completed text, admin-images.php triggers a new function which populates an image table from these renamed and trimmed files. Sometimes it displays old names and files that were suppose to be deleted, and a simple page refresh gets rid of them. This seems to suggest that the above php script is running through all the code and spitting out echoed text to the mainfile before it completes its file-system manipulation (All of this other code is long and complicated, and hopefully unnecessary for the discussion at hand).
You'll notice, I've tried a sleep() function to halt the php script to hopefully give it time to finish. This is an ineligent, and problematic way of doing things, because I have to put a large amount of time to insure it works every-time, but I don't want the user to wait longer than she / he has to.
Mind that file-systems often use caches to reduce the load. Normally you won't notice, but sometimes you need to clear the cache if you need to have the real information. Check the configuration of your file-system if your issue is file-system related.
PHP itself uses a cache as well for some file-operations, so clear that, too.
See clearstatcache to clear the PHP stat cache.
Take note that this is a "view" issue, the file is actually deleted on disk, but PHP might still return it's there (until you clear the cache).
I suppose they are not asynchronous, because they return a result telling if the operation was successful or not.
I believe the problem happens because when you run scandir after making the modifications, it may be using "cached" data, from memory, instead of re-scanning the file system.
rename() is not, but unlink() is asynchronous on Windows.
Because there seems to be no way of waiting for a pending delete to finish, this answer suggests to rename a file before deleting it. PHP does not seem to do that, so you can assume it's asynchronous.
To use any file operation you are required to use the $_SERVER["DOCUMENT_ROOT"] to make that work. In case you wont do it.. the real operation wont work properly. Also in case you are using the Linux Server then you will be required to set the permissions for the folders in which you want to perform the file operation.
And mind it both the operations are synchronous they are not asynchronous. It also depends on the type of the server or the OS that you are using.

Categories