I have ftp server with a lot of subfolders and files in it. I need to retrieve the folder structure from ftp, which shows names of all folders and subfolders from a specified starting path. I'm not interested in files included in each folder, only the directory tree. I'm using PHP and my server does not support mlsd.
Thanks for help.
I implemented my own recursive function, which for some reason is not working.
function ftp_list_files_recursive($ftp_stream, $path) {
$lines = ftp_rawlist($ftp_stream, $path);
$result = [];
if (is_array($lines) || is_object($lines)) {
foreach ($lines as $line) {
$exp0 = explode('<', $line);
if (sizeof($exp0) > 1):
$exp1 = explode('>', $exp0[1]);
if ($exp1[0] == 'DIR') {
$file_path=$path . "/" . ltrim($exp1[1]);
$result = array_merge($result, ftp_list_files_recursive($ftp_stream, $file_path));
} else {
$result[] = $file_path;
}
endif;
}
}
return $result;
}
The ftp_rawlist returns directory info as: 01-18-20 01:00PM <DIR> DirName so first I explode on < and check whether it was successful. If yes, then it means a string had DIR in it and it can be further exploded on >. It could have been done with regular expression, but that works for me now. If I print $file_path variable I see that it changes, so I assume the recursion works. However, the $result array is always empty. Any thoughts on that?
Start here: PHP FTP recursive directory listing.
You just need to adjust the code to:
the DOS-style listing you have from your FTP server (IIS probably) and
to collect only the folders.
function ftp_list_dirs_recursive($ftp_stream, $directory)
{
$result = [];
$lines = ftp_rawlist($ftp_stream, $directory);
if ($lines === false)
{
die("Cannot list $directory");
}
foreach ($lines as $line)
{
// rather lame parsing as a quick example:
if (strpos($line, "<DIR>") !== false)
{
$dir_path = $directory . "/" . ltrim(substr($line, strpos($line, ">") + 1));
$subdirs = ftp_list_dirs_recursive($ftp_stream, $dir_path);
$result = array_merge($result, [$dir_path], $subdirs);
}
}
return $result;
}
Related
I would like to list all my folder on my web application without the files in it. At the moment I'm using this code but this will give me all files and folders:
//Connect
$conn = ftp_connect($ftp_server);
$login = ftp_login($conn, $ftp_username, $ftp_userpassword);
//
//Enable PASV ( Note: must be done after ftp_login() )
//
$mode = ftp_pasv($conn, TRUE);
$file_list = ftp_nlist($conn, "/webspace/httpdocs/brentstevens.nl/u_media/$uid/");
foreach ($file_list as $file)
{
// process folder
}
But this will output all files and folders. How can I adjust this code to only serve the folders?
If your server supports MLSD command and you have PHP 7.2 or newer, you can use ftp_mlsd function:
$files = ftp_mlsd($conn, $path);
foreach ($files as $file)
{
if ($file["type"] == "dir")
{
echo $file["name"]."\n";
}
}
If you do not have PHP 7.2, you can try to implement the MLSD command on your own. For a start, see user comment of the ftp_rawlist command:
https://www.php.net/manual/en/function.ftp-rawlist.php#101071
It is not easy to implement this without MLSD in a way that works for any FTP server. See also Check if FTP entry is file or folder with PHP.
But if you need to work against one specific FTP server only, you can use ftp_rawlist to retrieve a file listing in a platform-specific format and parse that.
The following code assumes a common *nix format.
$lines = ftp_rawlist($conn, $path);
foreach ($lines as $line)
{
$tokens = explode(" ", $line);
$name = $tokens[count($tokens) - 1];
$type = $tokens[0][0];
if ($type == 'd')
{
echo "$name\n";
}
}
If your server not supports MLSD command or you have PHP <7.2 and ftp_rawlist marks directory as <DIR>
function getList($connection, string $path): array
{
$rawlist = ftp_rawlist($connection, $path) ?: [];
$items = [];
foreach ($rawlist as $data) {
$item = array_combine(['date', 'time', 'size', 'name'], preg_split('/\s/', $data, -1, PREG_SPLIT_NO_EMPTY));
$item['type'] = $item['size'] === '<DIR>' ? 'dir' : 'file';
$item['size'] = (int)$item['size'];
$items[] = $item;
}
return $items;
}
function getFileList($connection, string $path): array
{
return array_values(array_filter(getList($connection, $path), function ($file) { return $file['type'] === 'file'; }));
}
function getDirList($connection, string $path): array
{
return array_values(array_filter(getList($connection, $path), function ($file) { return $file['type'] === 'dir'; }));
}
You have to use http://php.net/manual/en/function.ftp-rawlist.php function and with regular expression parse type which in your case is letter d
I have a number of different hosting accounts set up for clients and need to calculate the amount of storage space being used on each account, which would update regularly.
I have a database set up to record each clients storage usage.
I attempted this first using a PHP file on each account, run by a Cron Job. If run manually by myself, it would output the correct filesize and update the correct size to the database, although when run from the Cron Job, it would output 0.
I then attempted to run this file from a Cron Job from the main account but figured this wouldn't actually work as my hosting would block files from another server and I would end up with the same result as before.
I am now playing around with FTP access to each account from a Cron Job from the main account which looks something like below, the only problem is I don't know how to calculate directory size rather than single file sizes using FTP access, and don't know how to reiterate this way? Hoping somebody might be able to help here before I end up going around in circles?
I will also add the previous first attempt too.
$ftp_conn = ftp_connect($ftp_host, 21, 420) or die("Could not connect to server");
$ftp_login = ftp_login($ftp_conn, $ftp_username, 'mypassword');
$total_size = 0;
$contents = ftp_nlist($ftp_conn, ".");
// output $contents
foreach($contents as $folder){
while($search == true){
if($folder == '..' || $folder == '.'){
} else {
$file = $folder;
$res = ftp_size($ftp_conn, $file);
if ($res != -1) {
$total_size = $total_size + $res;
} else {
$total_size = $total_size;
}
}
}
}
ftp_close($ftp_conn);
This doesn't work as it doesn't calculate folder sizes and I don't know how to open the reiterate using this method?
This second script did work but would only work if opened manually, and return 0 if run by the cron job.
class Directory_Calculator {
function calculate_whole_directory($directory)
{
if ($handle = opendir($directory))
{
$size = 0;
$folders = 0;
$files = 0;
while (false !== ($file = readdir($handle)))
{
if ($file != "." && $file != "..")
{
if(is_dir($directory.$file))
{
$array = $this->calculate_whole_directory($directory.$file.'/');
$size += $array['size'];
$files += $array['files'];
$folders += $array['folders'];
}
else
{
$size += filesize($directory.$file);
$files++;
}
}
}
closedir($handle);
}
$folders++;
return array('size' => $size, 'files' => $files, 'folders' => $folders);
}
}
/* Path to Directory - IMPORTANT: with '/' at the end */
$directory = '../public_html/';
// return an array with: size, total files & folders
$array = $directory_size->size($directory);
$size_of_site = $array['size'];
echo $size_of_site;
Please bare in mind that I am currently testing and none of the MySQLi or PHP scripts are secure yet.
If your server supports MLSD command and you have PHP 7.2 or newer, you can use ftp_mlsd function:
function calculate_whole_directory($ftp_conn, $directory)
{
$files = ftp_mlsd($ftp_conn, $directory) or die("Cannot list $directory");
$result = 0;
foreach ($files as $file)
{
if (($file["type"] == "cdir") || ($file["type"] == "pdir"))
{
$size = 0;
}
else if ($file["type"] == "dir")
{
$size = calculate_whole_directory($ftp_conn, $directory."/".$file["name"]);
}
else
{
$size = intval($file["size"]);
}
$result += $size;
}
return $result;
}
If you do not have PHP 7.2, you can try to implement the MLSD command on your own. For a start, see user comment of the ftp_rawlist command:
https://www.php.net/manual/en/function.ftp-rawlist.php#101071
If you cannot use MLSD, you will particularly have problems telling if an entry is a file or folder. While you can use the ftp_size trick, as you do, calling ftp_size for each entry can take ages.
But if you need to work against one specific FTP server only, you can use ftp_rawlist to retrieve a file listing in a platform-specific format and parse that.
The following code assumes a common *nix format.
function calculate_whole_directory($ftp_conn, $directory)
{
$lines = ftp_rawlist($ftp_conn, $directory) or die("Cannot list $directory");
$result = 0;
foreach ($lines as $line)
{
$tokens = preg_split("/\s+/", $line, 9);
$name = $tokens[8];
if ($tokens[0][0] === 'd')
{
$size = calculate_whole_directory($ftp_conn, "$directory/$name");
}
else
{
$size = intval($tokens[4]);
}
$result += $size;
}
return $result;
}
Based on PHP FTP recursive directory listing.
Regarding cron: I'd guess that the cron does not start your script with a correct working directory, so you calculate a size of a non-existing directory.
Use an absolute path here:
$directory = '../public_html/';
Though you better add some error checking so that you can see yourself what goes wrong.
I have a python script I wrote that I need to port to php. It recursively searches a given directory and builds a string based on regex searches. The first function I am trying to port is below. It takes a regex and a base dir, recursively searches all files in that dir for the regex, and builds a list of the string matches.
def grep(regex, base_dir):
matches = list()
for path, dirs, files in os.walk(base_dir):
for filename in files:
fullpath = os.path.join(path, filename)
with open(fullpath, 'r') as f:
content = f.read()
matches = matches + re.findall(regex, content)
return matches
I never use PHP except for basic GET param manipulation. I grabbed some directory walking code from the web, and am struggling to make it work like the python function above due to my utter lack of the php API.
function findFiles($dir = '.', $pattern = '/./'){
$prefix = $dir . '/';
$dir = dir($dir);
while (false !== ($file = $dir->read())){
if ($file === '.' || $file === '..') continue;
$file = $prefix . $file;
if (is_dir($file)) findFiles($file, $pattern);
if (preg_match($pattern, $file)){
echo $file . "\n";
}
}
}
Here is my solution:
<?php
class FileGrep {
private $dirs; // Scanned directories list
private $files; // Found files list
private $matches; // Matches list
function __construct() {
$this->dirs = array();
$this->files = array();
$this->matches = array();
}
function findFiles($path, $recursive = TRUE) {
$this->dirs[] = realpath($path);
foreach (scandir($path) as $file) {
if (($file != '.') && ($file != '..')) {
$fullname = realpath("{$path}/{$file}");
if (is_dir($fullname) && !is_link($fullname) && $recursive) {
if (!in_array($fullname, $this->dirs)) {
$this->findFiles($fullname, $recursive);
}
} else if (is_file($fullname)){
$this->files[] = $fullname;
}
}
}
return($this->files);
}
function searchFiles($pattern) {
$this->matches = array();
foreach ($this->files as $file) {
if ($contents = file_get_contents($file)) {
if (preg_match($pattern, $contents, $matches) > 0) {
//echo $file."\n";
$this->matches = array_merge($this->matches, $matches);
}
}
}
return($this->matches);
}
}
// Usage example:
$fg = new FileGrep();
$files = $fg->findFiles('.'); // List all the files in current directory and its subdirectories
$matches = $fg->searchFiles('/open/'); // Search for the "open" string in all those files
?>
<html>
<body>
<pre><?php print_r($matches) ?></pre>
</body>
</html>
Be aware that:
It reads each file to search for the pattern, so it may require a lot of memory (check the "memory_limit" configuration in your PHP.INI file).
It does'nt work with unicode files. If you are working with unicode files you should use the "mb_ereg_match" function rather than the "preg_match" function.
It does'nt follow symbolic links
In conclusion, even if it's not the most efficient solution at all, it should work.
i wonder if theres a tutorial out there or you could give me a quick and simple approach how i could do the following task.
i have a folder on my server. i want to build a kind of cms where i can easily delete files from a folder. i know how to upload them, i already found a tutorial.
i think of simply running through all files, creating a list of them with a checkbox in front, selecting the checkbox and pressing a DELETE button.
is this a rather difficult task to get done? do you maybe kno any tutorial or something.
thank you very much!
Start with
List all files
<?php
// file_array() by Jamon Holmgren. Exclude files by putting them in the $exclude
// string separated by pipes. Returns an array with filenames as strings.
function file_array($path, $exclude = ".|..", $recursive = false) {
$path = rtrim($path, "/") . "/";
$folder_handle = opendir($path);
$exclude_array = explode("|", $exclude);
$result = array();
while(false !== ($filename = readdir($folder_handle))) {
if(!in_array(strtolower($filename), $exclude_array)) {
if(is_dir($path . $filename . "/")) {
if($recursive) $result[] = file_array($path, $exclude, true);
} else {
$result[] = $filename;
}
}
}
return $result;
}
?>
then create hyperlinks to the above files to a delete function like;
unlink('filename.jpg');
I'm writing a photo gallery script in PHP and have a single directory where the user will store their pictures. I'm attempting to set up page caching and have the cache refresh only if the contents of the directory has changed. I thought I could do this by caching the last modified time of the directory using the filemtime() function and compare it to the current modified time of the directory. However, as I've come to realize, the directory modified time does not change as files are added or removed from that directory (at least on Windows, not sure about Linux machines yet).
So my questions is, what is the simplest way to check if the contents of a directory have been modified?
As already mentioned by others, a better way to solve this would be to trigger a function when particular events happen, that changes the folder.
However, if your server is a unix, you can use inotifywait to watch the directory, and then invoke a PHP script.
Here's a simple example:
#!/bin/sh
inotifywait --recursive --monitor --quiet --event modify,create,delete,move --format '%f' /path/to/directory/to/watch |
while read FILE ; do
php /path/to/trigger.php $FILE
done
See also: http://linux.die.net/man/1/inotifywait
What about touching the directory after a user has submitted his image?
Changelog says: Requires php 5.3 for windows to work, but I think it should work on all other environments
with inotifywait inside php
$watchedDir = 'watch';
$in = popen("inotifywait --monitor --quiet --format '%e %f' --event create,moved_to '$watchedDir'", 'r');
if ($in === false)
throw new Exception ('fail start notify');
while (($line = fgets($in)) !== false)
{
list($event, $file) = explode(' ', rtrim($line, PHP_EOL), 2);
echo "$event $file\n";
}
Uh. I'd simply store the md5 of a directory listing. If the contents change, the md5(directory-listing) will change. You might get the very occasional md5 clash, but I think that chance is tiny enough..
Alternatively, you could store a little file in that directory that contains the "last modified" date. But I'd go with md5.
PS. on second thought, seeing as how you're looking at performance (caching) requesting and hashing the directory listing might not be entirely optimal..
IMO edubem's answer is the way to go, however you can do something like this:
if (sha1(serialize(Map('/path/to/directory/', true))) != /* previous stored hash */)
{
// directory contents has changed
}
Or a more weak / faster version:
if (Size('/path/to/directory/', true) != /* previous stored size */)
{
// directory contents has changed
}
Here are the functions used:
function Map($path, $recursive = false)
{
$result = array();
if (is_dir($path) === true)
{
$path = Path($path);
$files = array_diff(scandir($path), array('.', '..'));
foreach ($files as $file)
{
if (is_dir($path . $file) === true)
{
$result[$file] = ($recursive === true) ? Map($path . $file, $recursive) : $this->Size($path . $file, true);
}
else if (is_file($path . $file) === true)
{
$result[$file] = Size($path . $file);
}
}
}
else if (is_file($path) === true)
{
$result[basename($path)] = Size($path);
}
return $result;
}
function Size($path, $recursive = true)
{
$result = 0;
if (is_dir($path) === true)
{
$path = Path($path);
$files = array_diff(scandir($path), array('.', '..'));
foreach ($files as $file)
{
if (is_dir($path . $file) === true)
{
$result += ($recursive === true) ? Size($path . $file, $recursive) : 0;
}
else if (is_file() === true)
{
$result += sprintf('%u', filesize($path . $file));
}
}
}
else if (is_file($path) === true)
{
$result += sprintf('%u', filesize($path));
}
return $result;
}
function Path($path)
{
if (file_exists($path) === true)
{
$path = rtrim(str_replace('\\', '/', realpath($path)), '/');
if (is_dir($path) === true)
{
$path .= '/';
}
return $path;
}
return false;
}
Here's what you may try. Store all pictures in a single directory (or in /username subdirectories inside it to speed things up and to lessen the stress on the FS) and set up Apache (or whaterver you're using) to serve them as static content with "expires-on" set to 100 years in the future. File names should contain some unique prefix or suffix (timestamp, SHA1 hash of file content, etc), so whenever uses changes the file its name gets changed and Apache will serve a new version, which will get cached along the way.
You're thinking the wrong way.
You should execute your directory indexer script as soon as someone's uploaded a new file and it's moved to the target location.
Try deleting the cached version when a user uploads a file to his directory.
When someone tries to view the gallery, look if there's a cached version first. If there's a cached version, load it, otherwise, generate the page, cache it, done.
I was looking for something similar and I just found this:
http://www.franzone.com/2008/06/05/php-script-to-monitor-ftp-directory-changes/
For me looks like a great solution since I'll have a lot of control (I'll be doing an AJAX call to see if anything changed).
Hope that this helps.
Here is a code sample, that would return 0 if the directory was changed.
I use it in backups.
The changed status is determined by presence of files and their filesizes.
You could easily change this, to compare file contents by replacing
$longString .= filesize($file);
with
$longString .= crc32(file_get_contents($file));
but it will affect execution speed.
#!/usr/bin/php
<?php
$dirName = $argv[1];
$basePath = '/var/www/vhosts/majestichorseporn.com/web/';
$dataFile = './backup_dir_if_changed.dat';
# startup checks
if (!is_writable($dataFile))
die($dataFile . ' is not writable!');
if (!is_dir($basePath . $dirName))
die($basePath . $dirName . ' is not a directory');
$dataFileContent = file_get_contents($dataFile);
$data = #unserialize($dataFileContent);
if ($data === false)
$data = array();
# find all files ang concatenate their sizes to calculate crc32
$files = glob($basePath . $dirName . '/*', GLOB_BRACE);
$longString = '';
foreach ($files as $file) {
$longString .= filesize($file);
}
$longStringHash = crc32($longString);
# do changed check
if (isset ($data[$dirName]) && $data[$dirName] == $longStringHash)
die('Directory did not change.');
# save hash do DB
$data[$dirName] = $longStringHash;
file_put_contents($dataFile, serialize($data));
die('0');