List of all files with PHP in a huge site - php

Question

I need to do a list of all files on a server from an other server.
I don't have access to PHP config like maximum timeout of the remote server.
The maximum timeout could be very short like 30s. In some case, the following code gives a Timeout issue, because the iterator don't have enough time to get all the files.
public function getStructure($path)
{
$structure = new \stdClass();
$structure->dirs = array();
$structure->files = array();
$iterator = new \RecursiveIteratorIterator(
new \RecursiveDirectoryIterator($path), \RecursiveIteratorIterator::SELF_FIRST);
foreach ($iterator as $file)
{
if ($file->isDir())
{
$structure->dirs[] = $file->getRealpath();
}
else
{
$structure->files[] = $file->getRealpath();
}
}
return $structure;
}
I'm looking for a way to get the structure in multiple calls. Something like : myremotesite.com/api/v1/structrue?start=xxxx where start is the point where the last call stop.
Thanks for your help

Answer 1

sounds like you need a dir with save/resume functionality... i would probably implement it in SQLite, due to its synchronous-by-default nature.
since i was bored.. UNTESTED, but should work in theory. DO NOT try to implement beginTransaction() / commit() optimizations to this code, that would defeat the whole "synchronous and tolerates crashing at any moment" part of the code;
<?php
//will return bool(true) when it's finished creating the database.
//should be timeout/unstable system resistant,
//relying on SQLite's syncronous-by-default nature.
function dirToSQLite($dir,$sqldb){
if(!is_readable($dir)){
throw new InvalidArgumentException('argument 1 must be a readable dir, but is not readable.');
}
if(!is_dir($dir)){
throw new InvalidArgumentException('argument 1 is not a valid dir');
}
$db=new PDO('sqlite:'.$sqldb,'','',array(PDO::ATTR_EMULATE_PREPARES => false,PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION));
$db->exec('CREATE TABLE IF NOT EXISTS `dir` (`id` INTEGER PRIMARY KEY,`path` TEXT UNIQUE,`type` TEXT);');
$db->query('INSERT OR IGNORE INTO `dir` (`path`,`type`) VALUES('.$db->quote($dir).',\'dirUnexplored\');');
$stm=$db->prepare('INSERT INTO `dir` (`path`,`type`) VALUES(:path,:type);');
$stmExplored=$db->prepare('UPDATE `dir` SET `type` = \'dir\' WHERE id = ? ');
$path='';
$type='';
$stm->bindParam(':path',$path,PDO::PARAM_STR);
$stm->bindParam(':type',$type,PDO::PARAM_STR);
while(true){
$found=false;
foreach($db->query('SELECT `id`,`path` FROM `dir` WHERE `type` = \'dirUnexplored\'') as $res)
{
$found=true;
$di=new DirectoryIterator($res['path']);
foreach($di as $file){
if($file->isDot()){
continue;
} else
if($file->isLink()){
$type='link';
} else
if($file->isDir()){
$type='dirUnexplored';
} else
if($file->isFile()){
$type='file';
} else
{
$type='unknown';
}
$path=$file->getPathname();
$stm->execute();
}
$stmExplored->execute(array($res['id']));
}
if(!$found){
break;
}
}
return true;
}
if(true===dirToSqlite('/home/foo','homefoo.db3')){
echo "finished!";
}else {
throw new Exception();
}
then just keep calling that url until it returns the string "finished!", then you can download the SQLite database directly, no php involved in the download.