The scenario:
I use this function to upload a entire directory to AWS bucket at once and some folders are really big (like 30GBs) of photos.
$client->uploadDirectory(
MY_SOURCE,
SPACES_NAME,
DESTINATION,
array(
'concurrency' => 1,
'debug' => TRUE,
'force' => FALSE,
'options' => array(
'ServerSideEncryption' => 'AES256',
),
)
);
The error:
Even with concurrency = 1, after a while my script end's up with the following error:
503 Slowdown Please reduce your request rate.
My question is
Is there some parameter that limit requests? Readding docs I can't find a way to make this function slow down requests. I know there's a limit of 100 files/second and I want to obey this limit, but don't know where to put this.
You can try to use Middlewares to slow down the requests. Something like this:
use Aws\Middleware;
use Psr\Http\Message\RequestInterface;
...
$s3Client->getHandlerList()->appendInit(Middleware::mapRequest(function (RequestInterface $request) {
sleep(1);
return $request;
}));
$s3Client->uploadDirectory(...);
See the docs.
Ok, I'v found a solution:
First, I have a specific server to do this, scripts run without time limit and can use a really good ammount of memory.
$counter = 0;
$files = scanDirAndSubdir($folder);
foreach($files as $file){
if(is_file($file)){
$ACL = 'private';
$insert[] = $client->getCommand('putObject',array(
'Bucket' => SPACES_NAME,
'Key' => $file,
'SourceFile' => $file,
'ACL' => $ACL,
));
if($counter==100){
// Executes all commands at once
$pool = new Aws\CommandPool($client, $insert);
$promisse = $pool->promise();
$promisse->wait();
$counter = 0;
sleep(1);
}
$counter ++;
}
}
Related
I've obtained the following code from searching about the topic
Route::get('/test', function () {
//disable execution time limit when downloading a big file.
set_time_limit(0);
$fs = Storage::disk('local');
$path = 'uploads/user-1/1653600850867.mp3';
$stream = $fs->readStream($path);
if (ob_get_level()) ob_end_clean();
return response()->stream(function () use ($stream) {
fpassthru($stream);
},
200,
[
'Accept-Ranges' => 'bytes',
'Content-Length' => 14098560,
'Content-Type' => 'application/octet-stream',
]);
});
However when I click play on the UI, it takes a good four seconds to start playing. If I switch the disk to local though, it plays almost instantly.
Is there a way to improve the performance or, read the stream by range as per request?
Edit
My current DO config is as per below
'driver' => 's3',
'key' => env('DO_ACCESS_KEY_ID'),
'secret' => env('DO_SECRET_ACCESS_KEY'),
'region' => env('DO_DEFAULT_REGION'),
'bucket' => env('DO_BUCKET'),
'url' => env('DO_URL'),
'endpoint' => env('DO_ENDPOINT'),
'use_path_style_endpoint' => env('DO_USE_PATH_STYLE_ENDPOINT', false),
But I find two type of integration online one specifying the CDN endpoint and one doesn't. I am not sure which one is relevant, though the one that specifies CDN is for Laravel 8 and I am on Laravel 9.
I had to change my code such that:
I had to use the php SDK client for connecting to Aws for the Laravel API isn't flexible to allow passing additional arguments (at least I haven't found anything while researching)
Change to streamDownload as I can't see any description to the stream method in the docs despite that it is present in code.
So the below code allows to achieve what I was aiming for which is, download by chunk based on the range received in the request.
return response()->streamDownload(function(){
$client = new Aws\S3\S3Client([
'version' => 'latest',
'region' => config('filesystems.disks.do.region'),
'endpoint' => config('filesystems.disks.do.endpoint'),
'credentials' => [
'key' => config('filesystems.disks.do.key'),
'secret' => config('filesystems.disks.do.secret'),
],
]);
$path = 'uploads/user-1/1653600850867.mp3';
$range = request()->header('Range');
$result = $client->getObject([
'Bucket' => 'wyxos-streaming',
'Key' => $path,
'Range' => $range
]);
echo $result['Body'];
},
200,
[
'Accept-Ranges' => 'bytes',
'Content-Length' => 14098560,
'Content-Type' => 'application/octet-stream',
]);
Note:
In a live scenario, you would need to cater for if range isn't specified, the content length will need to be the actual file size
When range is present however, the content length should then be the size of the segment being echoed
I am successfully uploading folders to S3 using ->uploadDirectory(). Several hundred folders have 100's, or 1,000's of images contained within them with so using PutObject() for each file hardly seemed to make sense. The upload works, and all goes well, but the ACL, StorageClass, and metadata is not being included in the upload.
According to the docs at http://docs.aws.amazon.com/aws-sdk-php/v2/guide/service-s3.html#uploading-a-directory-to-a-bucket , the following code should accomplished this. It is further documented with the putObject() function that is also cited.
I can find no examples of this function using anything but a directory and bucket, so fail to see what might be wrong with it. Any ideas why the data in $options is being ignored?
$aws = Aws::factory('config.php');
$s3 = $aws->get('S3');
$dir = 'c:\myfolder\myfiles';
$bucket = 'mybucket;
$keyPrefix = "ABC/myfiles/";
$options = array(
'ACL' => 'public-read',
'StorageClass' => 'REDUCED_REDUNDANCY',
'Metadata'=> array(
'MyVal1'=>'Something',
'MyVal2'=>'Something else'
)
);
$result = $s3->uploadDirectory($dir, $bucket, $keyPrefix, $options);
Parameters to provide to putObject or createMultipartUpload should be in the params option, not provided as top-level values in the options array. Try declaring your options as follows:
$options = array(
'params' => array(
'ACL' => 'public-read',
'StorageClass' => 'REDUCED_REDUNDANCY',
'Metadata'=> array(
'MyVal1'=>'Something',
'MyVal2'=>'Something else',
),
),
);
I'm reading: http://docs.aws.amazon.com/aws-sdk-php/latest/class-Aws.S3.S3Client.html#_getBucketCors
I have a partial key ex: "/myfolder/myinnerfolder/"
However there are actually many objects (files) inside of myinnerfolder.
I believe that I can call something like this:
$result = $client->getObject(array(
'Bucket' => $bucket,
'Key' => $key
));
return $result;
If I have the full key. How can I call something like the above but have it return all of the objects and or their names to me? In Python you can simply request by the front of a key but I don't see an option to do this. Any ideas?
You need to use the listObjects() method with the 'Prefix' parameter.
$result = $client->listObjects(array(
'Bucket' => $bucket,
'Prefix' => 'myfolder/myinnerfolder/',
));
$objects = $result['Contents'];
To make this even easier, especially if you have more than 1000 objects with that prefix (which would normally require multiple requests), you can use the Iterators feature of the SDK.
$objects = $client->getIterator('ListObjects', array(
'Bucket' => $bucket,
'Prefix' => 'myfolder/myinnerfolder/',
));
foreach ($objects as $object) {
echo $object['Name'];
}
We have an application where in user can create his own webpages and host them.We are using S3 to store the pages as they are static.Here,as we have a limitation of 100 buckets per user,we decided to go with folders for each user inside a bucket.
Now,if a user wants to host his website on his domain,we ask him for the domain name(when he starts we publish it on our subdomain) and I have to rename the folder.
S3 being a flat file system I know there are actually no folders but just delimeter / separated values so I cannot go into the folder and check how many pages it contains.The API allows it one by one but for that we have to know the object names in the bucket.
I went through the docs and came across iterators,which I have not implemented yet.This uses guzzle of which I have no experience and facing challenges in implementing
Is there any other path I can take or I need to go this way.
You can create an iterator for the contents of a "folder" by doing the following:
$objects = $s3->getIterator('ListObjects', array(
'Bucket' => 'bucket-name',
'Prefix' => 'subfolder-name/',
'Delimiter' => '/',
));
foreach ($objects as $object) {
// Do things with each object
}
If you just need a count, you could this:
echo iterator_count($s3->getIterator('ListObjects', array(
'Bucket' => 'bucket-name',
'Prefix' => 'subfolder-name/',
'Delimiter' => '/',
)));
Bit of a learning curve with s3, eh? I spent about 2 hours and ended up with this codeigniter solution. I wrote a controller to loop over my known sub-folders.
function s3GetObjects($bucket) {
$CI =& get_instance();
$CI->load->library('aws_s3');
$prefix = $bucket.'/';
$objects = $CI->aws_s3->getIterator('ListObjects', array(
'Bucket' => $CI->config->item('s3_bucket'),
'Prefix' => $prefix,
'Delimiter' => '/',
));
foreach ($objects as $object) {
if ($object['Key'] == $prefix) continue;
echo $object['Key'].PHP_EOL;
if (!file_exists(FCPATH.$object['Key'])) {
try {
$r = $CI->aws_s3->getObject(array(
'Bucket' => $CI->config->item('s3_bucket'),
'Key' => $object['Key'],
'SaveAs' => FCPATH.$object['Key']
));
} catch (Exception $e) {
echo $e->getMessage().PHP_EOL;
//return FALSE;
}
echo PHP_EOL;
} else {
echo ' -- file exists'.PHP_EOL;
}
}
return TRUE;
}
I am to newbie to amazon s3. I downloaded zip into my project and unzipped it. Bucket is already created 'bucketToUpload'. My code for upload a file is
require 'aws/aws-autoloader.php';
use Aws\S3\S3Client;
use Aws\S3\Exception\S3Exception;
use Aws\S3\Enum\CannedAcl;
$bucket = 'bucketToUpload';
$pathToFile = getcwd().'/s3_upload_file.doc';
if(file_exists($pathToFile)) {
try {
$client = S3Client::factory(array(
'key' => 'MYKEY',
'secret' => 'MYSECRETKEY',
'region' => 'us-west-1'
));
$client->waitUntilBucketExists(array('Bucket' => $bucket));
$result = $client->putObject(array(
'Bucket' => $bucket,
'Key' => 's3_upload_file.doc',
'SourceFile' => getcwd(),
'ACL' => CannedAcl::PRIVATE_ACCESS,
'Metadata' => array(
'Foo' => 'abc',
'Baz' => '123'
)
));
} catch (S3Exception $e) {
echo "The file was not uploaded: " . $e->getMessage();
}
var_dump($result);
}
I am getting Fatel error: Maximum execution time of 15 seconds exceeded.
Really don't know what am I doing wrong. Any help could be really appreciable.
Thanks
I see a few things you may need to do.
First, you are specifying
'SourceFile' => getcwd(),
When I think you probably meant to do
'SourceFile' => $pathToFile,
Second, you are doing var_dump($result); which will probably not show you what you are expecting to see. Try var_dump($result->toArray()); instead, but make sure to checkout the user guide page about response models for more information about working with results.
Third, the error you are seeing Fatal error: Maximum execution time of 15 seconds exceeded is related to PHP's max_execution_time INI setting. You should increase that limit if needed. You can also use the set_time_limit function within a single process.
I hope that helps!