I have a php script that uploads a zip file of images to a folder. This script recursively seeks out only files in the zip file and places all the files in a single directory on the server.
The problem is certain files get double uploaded. This is not the script's fault, but rather, due to the ridiculousness and inferiority of apple computers, when a mac creates a zip file of images it creates a folder of the images and then another folder with the exact same images only it places "._" in front of the file names. So seeing as how we're not going to be blessed with the disappearance of apple computers anytime soon, I tried to include in my php script a simple function to search for these inferior mac abominations and delete them from the directory. However, php isn't even pulling these files when I use "ftp_nlist".
So my question is: How do I get php to pull these stupid things so I can delete them?
$contents = ftp_nlist($conn_id, '.');
foreach($contents as $key => $value){
echo $key." => ".$value."<BR>";
if(substr($value, 1, 1) == ".") {
if(ftp_delete($conn_id, $value)) {
echo "Deleting $value<BR>";
}
}
echo "<BR>";
}
exit();
EDIT:
So thanks to Stephane's suggestion I was able to come up with this which works
if($zip->open($_FILES['theFile']['tmp_name']) === TRUE){
for($i = 0; $i < $zip->numFiles; $i++) {
$filename = $zip->getNameIndex($i);
$fileinfo = pathinfo($filename);
copy("zip://".$_FILES['theFile']['tmp_name']."#".$filename, $ezPresenter['currentFolder'].'/'.$fileinfo['basename']);
}
$zip->close();
}else{
exit("Could not upload/extract file");
}
$contents = ftp_rawlist($conn_id, '-a');
foreach($contents as $key => $value){
$value = explode(" ", $value);
$value = $value[count($value)-1];
echo $key." => ".$value."<BR>";
if(strpos($value, ".") === false) {
if(ftp_delete($conn_id, $value)) {
echo "Deleting $value<BR>";
}
}
if(substr($value, 0, 2) == "._") {
if(ftp_delete($conn_id, $value)) {
echo "Deleting $value<BR>";
}
}elseif(substr($value, 0, 1) == "." && $value != "." && $value != "..") {
if(ftp_delete($conn_id, $value)) {
echo "Deleting $value<BR>";
}
}
}
Use ftp_rawlist instead.
ftp_rawlist — Returns a detailed list of files in the given directory
ftp_rawlist($connid, "-a");
Argument -a means all as on unix command-line: ls -a.
I ran into this issue before, but I wasn't using ftp_nlist. What I ended up doing was using PHP's ZipArchive to open the zip file and look for (and exclude) the __MACOSX directory. I also ignored zip files where just one directory was inside (so you don't unzip the file and then have two directories deep to get to the data - that always annoys me).
My solution may not be the best for you, as it takes some extra processing, but it worked for me :)
Anyway, without further ado... here is the code I am using. Hopefully it'll be of help to you:
//
// unzip the file
$zip = new ZipArchive;
if ($zip->open($fname) === TRUE) {
//extract zip
$zip->extractTo($dir);
$zip->close();
//detect single dir
$basedir = function($x) use (&$basedir) {
$files = glob($x.'*', GLOB_MARK);
//ignore stupid mac directory
$k = array_search($x.'__MACOSX/',$files);
if($k!==FALSE) {
unset($files[$k]);
$files = array_values($files);
}
if(sizeof($files)==1 && is_dir($files[0]))
return $basedir($files[0]);
return $x;
};
//get root directory that has files in it
$dir = substr($basedir($dir.'/'),0,-1);
//
// here I re-zipped the data from the base directory
// and uploaded this file
//
} else {
//delete the file
unlink($fname);
//
// some other error handling
//
return;
}
Related
I have a number of different hosting accounts set up for clients and need to calculate the amount of storage space being used on each account, which would update regularly.
I have a database set up to record each clients storage usage.
I attempted this first using a PHP file on each account, run by a Cron Job. If run manually by myself, it would output the correct filesize and update the correct size to the database, although when run from the Cron Job, it would output 0.
I then attempted to run this file from a Cron Job from the main account but figured this wouldn't actually work as my hosting would block files from another server and I would end up with the same result as before.
I am now playing around with FTP access to each account from a Cron Job from the main account which looks something like below, the only problem is I don't know how to calculate directory size rather than single file sizes using FTP access, and don't know how to reiterate this way? Hoping somebody might be able to help here before I end up going around in circles?
I will also add the previous first attempt too.
$ftp_conn = ftp_connect($ftp_host, 21, 420) or die("Could not connect to server");
$ftp_login = ftp_login($ftp_conn, $ftp_username, 'mypassword');
$total_size = 0;
$contents = ftp_nlist($ftp_conn, ".");
// output $contents
foreach($contents as $folder){
while($search == true){
if($folder == '..' || $folder == '.'){
} else {
$file = $folder;
$res = ftp_size($ftp_conn, $file);
if ($res != -1) {
$total_size = $total_size + $res;
} else {
$total_size = $total_size;
}
}
}
}
ftp_close($ftp_conn);
This doesn't work as it doesn't calculate folder sizes and I don't know how to open the reiterate using this method?
This second script did work but would only work if opened manually, and return 0 if run by the cron job.
class Directory_Calculator {
function calculate_whole_directory($directory)
{
if ($handle = opendir($directory))
{
$size = 0;
$folders = 0;
$files = 0;
while (false !== ($file = readdir($handle)))
{
if ($file != "." && $file != "..")
{
if(is_dir($directory.$file))
{
$array = $this->calculate_whole_directory($directory.$file.'/');
$size += $array['size'];
$files += $array['files'];
$folders += $array['folders'];
}
else
{
$size += filesize($directory.$file);
$files++;
}
}
}
closedir($handle);
}
$folders++;
return array('size' => $size, 'files' => $files, 'folders' => $folders);
}
}
/* Path to Directory - IMPORTANT: with '/' at the end */
$directory = '../public_html/';
// return an array with: size, total files & folders
$array = $directory_size->size($directory);
$size_of_site = $array['size'];
echo $size_of_site;
Please bare in mind that I am currently testing and none of the MySQLi or PHP scripts are secure yet.
If your server supports MLSD command and you have PHP 7.2 or newer, you can use ftp_mlsd function:
function calculate_whole_directory($ftp_conn, $directory)
{
$files = ftp_mlsd($ftp_conn, $directory) or die("Cannot list $directory");
$result = 0;
foreach ($files as $file)
{
if (($file["type"] == "cdir") || ($file["type"] == "pdir"))
{
$size = 0;
}
else if ($file["type"] == "dir")
{
$size = calculate_whole_directory($ftp_conn, $directory."/".$file["name"]);
}
else
{
$size = intval($file["size"]);
}
$result += $size;
}
return $result;
}
If you do not have PHP 7.2, you can try to implement the MLSD command on your own. For a start, see user comment of the ftp_rawlist command:
https://www.php.net/manual/en/function.ftp-rawlist.php#101071
If you cannot use MLSD, you will particularly have problems telling if an entry is a file or folder. While you can use the ftp_size trick, as you do, calling ftp_size for each entry can take ages.
But if you need to work against one specific FTP server only, you can use ftp_rawlist to retrieve a file listing in a platform-specific format and parse that.
The following code assumes a common *nix format.
function calculate_whole_directory($ftp_conn, $directory)
{
$lines = ftp_rawlist($ftp_conn, $directory) or die("Cannot list $directory");
$result = 0;
foreach ($lines as $line)
{
$tokens = preg_split("/\s+/", $line, 9);
$name = $tokens[8];
if ($tokens[0][0] === 'd')
{
$size = calculate_whole_directory($ftp_conn, "$directory/$name");
}
else
{
$size = intval($tokens[4]);
}
$result += $size;
}
return $result;
}
Based on PHP FTP recursive directory listing.
Regarding cron: I'd guess that the cron does not start your script with a correct working directory, so you calculate a size of a non-existing directory.
Use an absolute path here:
$directory = '../public_html/';
Though you better add some error checking so that you can see yourself what goes wrong.
I understand how to delete a specified file using the 'unlink' command in PHP, but what I need to do is to write some code that will take all (10,000+) files in a folder (possibly put them into an array), open them, and then delete ONLY files that contain specific information. (All files in the folder are .txt files containing a list, or array, of numbers. EX: I want to delete any files where 4,5,6,7th slots in the array contained 20,20,100,100 respectively).
Is this a start:
<?php
$directory = '/path/to/files';
if (! is_dir($directory)) {
exit('Invalid diretory path');
}
$files = array();
foreach (scandir($directory) as $file) {
if ('.' === $file) continue;
if ('..' === $file) continue;
$files[] = $file;
}
var_dump($files);
?>
Following on from #ShawnMehan 's and #Jigar 's comments and from my own - I have not been able to test this where I am but it might serve to point you in a direction for what you need. Not sure how many rows of arrays you have in each file, so I have used a loop but if there is only one line, you will only need the fgets().
I have assumed that your arrays in the text files are comma delimited, but if they are tab delimited you would need to explode them by "\t".
$files = array();
$each_line = array();
$file_counter = 0;
$directory = '/path/to/files';
if (! is_dir($directory)) {
exit('Invalid directory path');
}
$files = glob($directory."/*.txt"); // Scan directory for .txt files
// Check that there are .txt files in directory
if ($files !== false) {
$number_of_files = count($files); // Count number of .txt files in directory
while($file_counter < $number_of_files){
$file_handle = fopen ($files[$file_counter], "r"); // Open file
while (!feof ($file_handle)) {
// get the arrays a row at a time and put each row into array
$each_line = explode( ',', fgets($file_handle, 1000));
if($each_line[3] == 20 || $each_line[4] == 20 || $each_line [5] == 100 || $each_line[6] == 100){
unlink($file_handle);
}
}
if(file_exists($file_handle)){
fclose($file_handle);
}
$file_counter++;
}
}else{
echo "No text files found in this directory";
}
Some notes on the use of glob(); for opening files in case you may not be opening local files (it won't work on remote files): http://php.net/manual/en/function.glob.php
I have a zip file containing one folder, that contains more folders and files, like this:
myfile.zip
-firstlevel
--folder1
--folder2
--folder3
--file1
--file2
Now, I want to extract this file using PHPs ZipArchive, but without the "firstlevel" folder. At the moment, the results look like this:
destination/firstlevel/folder1
destination/firstlevel/folder2
...
The result I'd like to have would look like this:
destination/folder1
destination/folder2
...
I've tried extractTo, which produces the first mentioned result, and copy(), as suggested here, but this doesn't seem to work at all.
My current code is here:
if($zip->open('myfile.zip') === true) {
$firstlevel = $zip->getNameIndex(0);
for($i = 0; $i < $zip->numFiles; $i++) {
$entry = $zip->getNameIndex($i);
$pos = strpos($entry, $firstlevel);
if ($pos !== false) {
$file = substr($entry, strlen($firstlevel));
if(strlen($file) > 0){
$files[] = $file;
}
}
}
//attempt 1 (extractTo):
//$zip->extractTo('./test', $files);
//attempt 2 (copy):
foreach($files as $filename){
copy('zip://'.$firstlevel.'/'.$filename, 'test/'.$filename);
}
}
How can I achieve the result I'm aiming for?
Take a look at my Quick Unzipper script. I wrote this for personal use a while back when uploading large zip files to a server. It was a backup, and 1,000s of files take forever with FTP so using a zip file was faster. I use Git and everything, but there wasn't another option for me. I place this php file in the directory I want the files to go, and put the zip file in the same directory. For my script, they all have to operate in the same directory. It was an easy way to secure it for my needs, as everything I needed was in the same dir.
Quick Unzipper: https://github.com/incomepitbull/QuickUnzipper/blob/master/unzip.php
I linked the file because I am not showcasing the repo, just the code that makes the unzip tick. With modern versions of PHP, there should't be anything that isn't included on your setup. So you shouldn't need to do any server config changes to use this.
Here is the PHP Doc for the ZipArchive class it uses: http://php.net/manual/en/class.ziparchive.php
There isn't any included way to do what you want, which is a shame. So I would unzip the file to a temp directory, then use another function to copy the contents to where you want. So when using ZipArchive, you will need to return the first item to get the folder name if it is unknown. If the folder is known, ie: the same pesky folder name every time, then you could hard code the name.
I have made it return the first item from the index. So if you ALWAYS have a zip with 1 folder inside it, and everything in that folder, this would work. However, if you have a zip file without everything consolidated inside 1 folder, it would fail. The code I have added will take care of your question. You will need to add further logic to handle alternate cases.
Also, You will still be left with the old directory from when we extract it to the temp directory for "processing". So I included code to delete it too.
NOTE: The code uses a lot of if's to show the processing steps, and print a message for testing purposes. You would need to modify it to your needs.
<?php
public function copyDirectoryContents($source, $destination, $create=false)
{
if ( ! is_dir($source) ) {
return false;
}
if ( ! is_dir($destination) && $create === true ) {
#mkdir($destination);
}
if ( is_dir($destination) ) {
$files = array_diff(scandir($source), array('.','..'));
foreach ($files as $file)
{
if ( is_dir($file) ) {
copyDirectoryContents("$source/$file", "$destination/$file");
} else {
#copy("$source/$file", "$destination/$file");
}
}
return true;
}
return false;
}
public function removeDirectory($directory, $options=array())
{
if(!isset($options['traverseSymlinks']))
$options['traverseSymlinks']=false;
$files = array_diff(scandir($directory), array('.','..'));
foreach ($files as $file)
{
if (is_dir("$directory/$file"))
{
if(!$options['traverseSymlinks'] && is_link(rtrim($file,DIRECTORY_SEPARATOR))) {
unlink("$directory/$file");
} else {
removeDirectory("$directory/$file",$options);
}
} else {
unlink("$directory/$file");
}
}
return rmdir($directory);
}
$file = dirname(__FILE__) . '/file.zip'; // full path to zip file needing extracted
$temp = dirname(__FILE__) . '/zip-temp'; // full path to temp dir to process extractions
$path = dirname(__FILE__) . '/extracted'; // full path to final destination to put the files (not the folder)
$firstDir = null; // holds the name of the first directory
$zip = new ZipArchive;
$res = $zip->open($file);
if ($res === TRUE) {
$firstDir = $zip->getNameIndex(0);
$zip->extractTo($temp);
$zip->close();
$status = "<strong>Success:</strong> '$file' extracted to '$temp'.";
} else {
$status = "<strong>Error:</strong> Could not extract '$file'.";
}
echo $status . '<br />';
if ( empty($firstDir) ) {
echo 'Error: first directory was empty!';
} else {
$firstDir = realpath($temp . '/' . $firstDir);
echo "First Directory: $firstDir <br />";
if ( is_dir($firstDir) ) {
if ( copyDirectoryContents($firstDir, $path) ) {
echo 'Directory contents copied!<br />';
if ( removeDirectory($directory) ) {
echo 'Temp directory deleted!<br />';
echo 'Done!<br />';
} else {
echo 'Error deleting temp directory!<br />';
}
} else {
echo 'Error copying directory contents!<br />';
}
} else {
echo 'Error: Could not find first directory';
}
}
i have this script which i will post absolutely unmodified:
<?
chdir("data");
$files = glob("*");
shuffle($files);
var_dump($files);
$i=0;
$finfo = finfo_open(FILEINFO_MIME_TYPE); // return mime type ala mimetype extension
foreach($files as $file) {
$i++;
$k = $i;
$mime = finfo_file($finfo, $file);
if(strpos($mime,"gif") !== false) {
$ext = "gif";
} else {
$ext = "jpg";
}
if($k < 10) {
$k = "00".$k;
} else if($k < 100) {
$k = "0".$k;
}
$k = $k.".".$ext;
rename($file,$k);
echo $k."\n";
}
the folder data has some image files (jpg and gif) in it.
but when i run it, suddenly a lot of images are just gone!
2/3rd of the images just got deleted...
i don't understand how?
i have an ext3 filesystem and PHP 5.3.2
I can't see anything in the code that would definately cause this behaviour. The most likely cause I could think of is perhaps rename($file,$k); is overwriting files that already exist. You could add the following to rule this out:
if(file_exists($k.".".$ext)) {
$k .= ".0" ;
}
while(file_exists($k.".".$ext)) {
$k .= "0" ;
}
$k = $k.".".$ext;
rename($file,$k);
The other thought I had is that perhaps something is going wrong with the chdir("data") which you could check by inserting the full path before $file and $k when calling the rename. I don't think this is very likely though.
Did you run it twice?
The first time you run it it renames all the images to 0001.jpg - 00nn.jpg. The second time it starts overwriting stuff, because the source names and target names would overlap, e.g. it renames 0042.jpg to 0001.jpg, so the existing 0001.jpg disappears.
Would be good to check if $k exists before renaming $file to $k:
if(!is_file($k)) {
rename($file, $k);
}
I'm writing a photo gallery script in PHP and have a single directory where the user will store their pictures. I'm attempting to set up page caching and have the cache refresh only if the contents of the directory has changed. I thought I could do this by caching the last modified time of the directory using the filemtime() function and compare it to the current modified time of the directory. However, as I've come to realize, the directory modified time does not change as files are added or removed from that directory (at least on Windows, not sure about Linux machines yet).
So my questions is, what is the simplest way to check if the contents of a directory have been modified?
As already mentioned by others, a better way to solve this would be to trigger a function when particular events happen, that changes the folder.
However, if your server is a unix, you can use inotifywait to watch the directory, and then invoke a PHP script.
Here's a simple example:
#!/bin/sh
inotifywait --recursive --monitor --quiet --event modify,create,delete,move --format '%f' /path/to/directory/to/watch |
while read FILE ; do
php /path/to/trigger.php $FILE
done
See also: http://linux.die.net/man/1/inotifywait
What about touching the directory after a user has submitted his image?
Changelog says: Requires php 5.3 for windows to work, but I think it should work on all other environments
with inotifywait inside php
$watchedDir = 'watch';
$in = popen("inotifywait --monitor --quiet --format '%e %f' --event create,moved_to '$watchedDir'", 'r');
if ($in === false)
throw new Exception ('fail start notify');
while (($line = fgets($in)) !== false)
{
list($event, $file) = explode(' ', rtrim($line, PHP_EOL), 2);
echo "$event $file\n";
}
Uh. I'd simply store the md5 of a directory listing. If the contents change, the md5(directory-listing) will change. You might get the very occasional md5 clash, but I think that chance is tiny enough..
Alternatively, you could store a little file in that directory that contains the "last modified" date. But I'd go with md5.
PS. on second thought, seeing as how you're looking at performance (caching) requesting and hashing the directory listing might not be entirely optimal..
IMO edubem's answer is the way to go, however you can do something like this:
if (sha1(serialize(Map('/path/to/directory/', true))) != /* previous stored hash */)
{
// directory contents has changed
}
Or a more weak / faster version:
if (Size('/path/to/directory/', true) != /* previous stored size */)
{
// directory contents has changed
}
Here are the functions used:
function Map($path, $recursive = false)
{
$result = array();
if (is_dir($path) === true)
{
$path = Path($path);
$files = array_diff(scandir($path), array('.', '..'));
foreach ($files as $file)
{
if (is_dir($path . $file) === true)
{
$result[$file] = ($recursive === true) ? Map($path . $file, $recursive) : $this->Size($path . $file, true);
}
else if (is_file($path . $file) === true)
{
$result[$file] = Size($path . $file);
}
}
}
else if (is_file($path) === true)
{
$result[basename($path)] = Size($path);
}
return $result;
}
function Size($path, $recursive = true)
{
$result = 0;
if (is_dir($path) === true)
{
$path = Path($path);
$files = array_diff(scandir($path), array('.', '..'));
foreach ($files as $file)
{
if (is_dir($path . $file) === true)
{
$result += ($recursive === true) ? Size($path . $file, $recursive) : 0;
}
else if (is_file() === true)
{
$result += sprintf('%u', filesize($path . $file));
}
}
}
else if (is_file($path) === true)
{
$result += sprintf('%u', filesize($path));
}
return $result;
}
function Path($path)
{
if (file_exists($path) === true)
{
$path = rtrim(str_replace('\\', '/', realpath($path)), '/');
if (is_dir($path) === true)
{
$path .= '/';
}
return $path;
}
return false;
}
Here's what you may try. Store all pictures in a single directory (or in /username subdirectories inside it to speed things up and to lessen the stress on the FS) and set up Apache (or whaterver you're using) to serve them as static content with "expires-on" set to 100 years in the future. File names should contain some unique prefix or suffix (timestamp, SHA1 hash of file content, etc), so whenever uses changes the file its name gets changed and Apache will serve a new version, which will get cached along the way.
You're thinking the wrong way.
You should execute your directory indexer script as soon as someone's uploaded a new file and it's moved to the target location.
Try deleting the cached version when a user uploads a file to his directory.
When someone tries to view the gallery, look if there's a cached version first. If there's a cached version, load it, otherwise, generate the page, cache it, done.
I was looking for something similar and I just found this:
http://www.franzone.com/2008/06/05/php-script-to-monitor-ftp-directory-changes/
For me looks like a great solution since I'll have a lot of control (I'll be doing an AJAX call to see if anything changed).
Hope that this helps.
Here is a code sample, that would return 0 if the directory was changed.
I use it in backups.
The changed status is determined by presence of files and their filesizes.
You could easily change this, to compare file contents by replacing
$longString .= filesize($file);
with
$longString .= crc32(file_get_contents($file));
but it will affect execution speed.
#!/usr/bin/php
<?php
$dirName = $argv[1];
$basePath = '/var/www/vhosts/majestichorseporn.com/web/';
$dataFile = './backup_dir_if_changed.dat';
# startup checks
if (!is_writable($dataFile))
die($dataFile . ' is not writable!');
if (!is_dir($basePath . $dirName))
die($basePath . $dirName . ' is not a directory');
$dataFileContent = file_get_contents($dataFile);
$data = #unserialize($dataFileContent);
if ($data === false)
$data = array();
# find all files ang concatenate their sizes to calculate crc32
$files = glob($basePath . $dirName . '/*', GLOB_BRACE);
$longString = '';
foreach ($files as $file) {
$longString .= filesize($file);
}
$longStringHash = crc32($longString);
# do changed check
if (isset ($data[$dirName]) && $data[$dirName] == $longStringHash)
die('Directory did not change.');
# save hash do DB
$data[$dirName] = $longStringHash;
file_put_contents($dataFile, serialize($data));
die('0');