We have an application where in user can create his own webpages and host them.We are using S3 to store the pages as they are static.Here,as we have a limitation of 100 buckets per user,we decided to go with folders for each user inside a bucket.
Now,if a user wants to host his website on his domain,we ask him for the domain name(when he starts we publish it on our subdomain) and I have to rename the folder.
S3 being a flat file system I know there are actually no folders but just delimeter / separated values so I cannot go into the folder and check how many pages it contains.The API allows it one by one but for that we have to know the object names in the bucket.
I went through the docs and came across iterators,which I have not implemented yet.This uses guzzle of which I have no experience and facing challenges in implementing
Is there any other path I can take or I need to go this way.
You can create an iterator for the contents of a "folder" by doing the following:
$objects = $s3->getIterator('ListObjects', array(
'Bucket' => 'bucket-name',
'Prefix' => 'subfolder-name/',
'Delimiter' => '/',
));
foreach ($objects as $object) {
// Do things with each object
}
If you just need a count, you could this:
echo iterator_count($s3->getIterator('ListObjects', array(
'Bucket' => 'bucket-name',
'Prefix' => 'subfolder-name/',
'Delimiter' => '/',
)));
Bit of a learning curve with s3, eh? I spent about 2 hours and ended up with this codeigniter solution. I wrote a controller to loop over my known sub-folders.
function s3GetObjects($bucket) {
$CI =& get_instance();
$CI->load->library('aws_s3');
$prefix = $bucket.'/';
$objects = $CI->aws_s3->getIterator('ListObjects', array(
'Bucket' => $CI->config->item('s3_bucket'),
'Prefix' => $prefix,
'Delimiter' => '/',
));
foreach ($objects as $object) {
if ($object['Key'] == $prefix) continue;
echo $object['Key'].PHP_EOL;
if (!file_exists(FCPATH.$object['Key'])) {
try {
$r = $CI->aws_s3->getObject(array(
'Bucket' => $CI->config->item('s3_bucket'),
'Key' => $object['Key'],
'SaveAs' => FCPATH.$object['Key']
));
} catch (Exception $e) {
echo $e->getMessage().PHP_EOL;
//return FALSE;
}
echo PHP_EOL;
} else {
echo ' -- file exists'.PHP_EOL;
}
}
return TRUE;
}
Related
I'm using aws-sdk-php and I must copy a folder with all file to a new path
This is my solution:
$objects = $s3->getIterator('ListObjects', array('Bucket' => $bucket, 'Prefix' => $folderToCopy));
foreach ($objects as $object) {
$newPath = str_replace($folderToCopy, $whereToPaste, $object['Key']);
$s3->copyObject(array(
'Bucket' => $bucket,
'Key' => $newPath,
'CopySource' => "{$bucket}/{$object['Key']}"
));
}
Works fine, but with thousands of file require a lot of time..
Is there another solution?
The scenario:
I use this function to upload a entire directory to AWS bucket at once and some folders are really big (like 30GBs) of photos.
$client->uploadDirectory(
MY_SOURCE,
SPACES_NAME,
DESTINATION,
array(
'concurrency' => 1,
'debug' => TRUE,
'force' => FALSE,
'options' => array(
'ServerSideEncryption' => 'AES256',
),
)
);
The error:
Even with concurrency = 1, after a while my script end's up with the following error:
503 Slowdown Please reduce your request rate.
My question is
Is there some parameter that limit requests? Readding docs I can't find a way to make this function slow down requests. I know there's a limit of 100 files/second and I want to obey this limit, but don't know where to put this.
You can try to use Middlewares to slow down the requests. Something like this:
use Aws\Middleware;
use Psr\Http\Message\RequestInterface;
...
$s3Client->getHandlerList()->appendInit(Middleware::mapRequest(function (RequestInterface $request) {
sleep(1);
return $request;
}));
$s3Client->uploadDirectory(...);
See the docs.
Ok, I'v found a solution:
First, I have a specific server to do this, scripts run without time limit and can use a really good ammount of memory.
$counter = 0;
$files = scanDirAndSubdir($folder);
foreach($files as $file){
if(is_file($file)){
$ACL = 'private';
$insert[] = $client->getCommand('putObject',array(
'Bucket' => SPACES_NAME,
'Key' => $file,
'SourceFile' => $file,
'ACL' => $ACL,
));
if($counter==100){
// Executes all commands at once
$pool = new Aws\CommandPool($client, $insert);
$promisse = $pool->promise();
$promisse->wait();
$counter = 0;
sleep(1);
}
$counter ++;
}
}
I need to grab the contents on one of my buckets. I try to do this using the AWS PHP SDK, but nothing is returned. Here's my code:
use Aws\S3\S3Client;
$s3client = S3Client::factory(array('credentials' => array(
'key' => '???????',
'secret' => '???????' ), 'region' => '?????', 'version' => 'latest', ));
try {
$data = $s3client->getIterator('ListObjects', array("Bucket" => "?????"));
print_r($data);
} catch (S3Exception $e) {
echo $e->getMessage() . "\n";
}
Here's the ouput:
Generator Object ( )
The output I get from that code is showing there's nothing wrong. However, there should be some content. The credentials I use are the same ones I use for uploading objects to the bucket, so I don't think those are bad. Am I missing something? How do I retrieve my buckets keys?
you are getting the iterator and not the objects.
To get to the objects you need to use the iterator. Something like:
foreach ($data as $object) {
echo $object['Key'] . "\n";
}
I'm reading: http://docs.aws.amazon.com/aws-sdk-php/latest/class-Aws.S3.S3Client.html#_getBucketCors
I have a partial key ex: "/myfolder/myinnerfolder/"
However there are actually many objects (files) inside of myinnerfolder.
I believe that I can call something like this:
$result = $client->getObject(array(
'Bucket' => $bucket,
'Key' => $key
));
return $result;
If I have the full key. How can I call something like the above but have it return all of the objects and or their names to me? In Python you can simply request by the front of a key but I don't see an option to do this. Any ideas?
You need to use the listObjects() method with the 'Prefix' parameter.
$result = $client->listObjects(array(
'Bucket' => $bucket,
'Prefix' => 'myfolder/myinnerfolder/',
));
$objects = $result['Contents'];
To make this even easier, especially if you have more than 1000 objects with that prefix (which would normally require multiple requests), you can use the Iterators feature of the SDK.
$objects = $client->getIterator('ListObjects', array(
'Bucket' => $bucket,
'Prefix' => 'myfolder/myinnerfolder/',
));
foreach ($objects as $object) {
echo $object['Name'];
}
I need to use jstree to display the directory structure of a ftp account's homedir.
I somehow got stuck at retrieving the full directory structure (and create a json object, xml file, html code, whatever). It is most likely something easy that is just slipping my mind, but anyway, here's what i tried so far:
function draw_tree($path)
{
global $con;
$list = ftp_nlist($con,$path);
$dirs = array(); $files = array();
foreach($list as $file)
{
if(ftp_is_dir($file))
{
$dir[] = array(
'attr' => array('data-path' => $path . '/' . $file,
'data' => $file,
'state' => 'closed',
'children' => // ??? some recursive function should
// probably go here
);
}
else {
$files[] = array(
'attr' => array('data-path' => $file)
);
}
}
return array_merge($dirs,$files);
}