I need to download thousands of images from an S3 bucket. I'm using the PHP SDK for AWS S3, but I can't find a way to remove the memory it's using when downloading the images.
$keyname = 'XXXXXXXXXX';
$bucket = 'XXXXXXXXXX';
$region = 'eu-west-1';
$credentials = new Aws\Credentials\Credentials('XXXXXXXXXX', 'XXXXXXXXXX');
$s3 = new Aws\S3\S3Client([
'version' => 'latest',
'region' => $region,
'credentials' => $credentials,
]);
$idDownloadedImages = array();
do {
$files = $this->check_new_images();
if ( !empty( $files ) ) {
foreach ( $files as $file ) {
$UrlMedia = basename( $file->UrlMedia );
$pathFile = $pathToPlugin . 'images/' . $UrlMedia;
$object = $keyname . $UrlMedia;
try {
// Get the object.
$result = $s3->getObject([
'Bucket' => $bucket,
'Key' => $object,
'SaveAs' => $pathFile,
]);
//gc_collect_cycles();
//xdebug_debug_zval('result');
$idDownloadedImages[] = $file->idMedia;
} catch ( S3Exception $e ) {
$errorCode = '(Error Code: ' . $e->getStatusCode() . ') ' . $e->getAwsErrorMessage();
}
//unset($result);
$result['Body']->__destruct();
}
$idDownloadedImages = implode( ',', $idDownloadedImages );
$wpdb->query( 'UPDATE wp_icat_Media
SET error=0
WHERE idMedia IN (' . $idDownloadedImages .')' );
$idDownloadedImages = array();
}
} while ( $this->check_new_images() );
I have tried to force the PHP garbage collector every time I download an image, to do a set() of $result and $idDownloadedImages and even to do $result['Body']->__destruct(), but I still don't know why it stores the images in memory.
Memory is only freed by forcing the gc_collect_cycles() at the end of the script.
From the documentation, Amazon recommends calling gc_collect_cycles on the before_upload event.
Here is the sample code:
$uploader = new MultipartUploader($client, $source, [
'bucket' => 'your-bucket',
'key' => 'your-key',
'before_upload' => function(\Aws\Command $command) {
gc_collect_cycles();
}
]);
Related
I am trying to use AWS to store all files from my app there. I was successfully able to upload them to the bucket, but I am not able to retrieve the object to show it to the user. I am getting an error "You cannot call GetObject on the service resource." I am not sure what is wrong? I thought it was permission issues? but if so how come I can upload the file?
Here is what I have
function aws_file_upload($key,$file)
{
$aws = aws();
// Get a resource representing the S3 service.
$s3 = $aws->s3;
$bucket = $s3->bucket('my-bucket-name');
try{
$result = $object = $bucket->putObject([
'Key' => $key,
'Body' => fopen($file, 'r'),
]);
$status = 'OK';
}catch(Exception $e){
$status = 'error';
}
return $status;
}
function aws_file_get($key)
{
$aws = aws();
$s3 = $aws->s3;
$result = $s3->getObject([
'Bucket' => 'my-bucket-name',
'Key' => $key
]);
// Display the object in the browser.
header("Content-Type: {$result['ContentType']}");
echo $result['Body'];
}
$key = 'Cases/my-file.pdf';
$file_name = 'my-file.pdf';
$res = aws_file_upload($key,$file_name);/// this will put the file into the AWS bucket
$result = aws_file_get($key);/// this will through the error
You need to create an s3 client.
Here is some sample code from Creating and Using Amazon S3 Buckets with the AWS SDK for PHP Version 3 - AWS SDK for PHP:
require 'vendor/autoload.php';
use Aws\S3\S3Client;
use Aws\Exception\AwsException;
$s3Client = new S3Client([
'profile' => 'default',
'region' => 'us-west-2',
'version' => '2006-03-01'
]);
$result = $s3Client->putObject([
'Bucket' => $bucket,
'Key' => $key,
'SourceFile' => $file_Path,
]);
For anyone wanting an answer here was my solution.
$aws = aws();
$s3 = $aws->s3;
$bucket = $s3->bucket('my-bucket-name');
$result = $bucket->object($key)->get();
header("Content-Type: {$result['ContentType']}");
echo $result['Body'];
I have used vlaim\fileupload\FileUpload; and yii\web\UploadedFile;
$image = UploadedFile::getInstance($model, 'flag');
$model->flag = new FileUpload(FileUpload::S_S3, [
'version' => 'latest',
'region' => 'us-west-2',
'credentials' => [
'key' => 'KEY',
'secret' => 'SECRET'
],
'bucket' => 'mybucket/uploads/flags/'.$model->code
]);
$uploader = $model->flag;
$model->flag = $uploader->uploadFromFile($image)->path;
In db i'm saving the path. How to customize the url?
Now my url looks like https://s3-us-west-2.amazonaws.com/mybucket%2Fuploads%2Fflags%2Fus/uploads%5C9f%5C7e%5Cc093ad5a.png
I need the url like https://mybucket.s3.amazonaws.com/uploads/flags/us.png
S3 does not have the concept of folders, It is an object store, with key/value pairs. They key for your file would be uploads/flags/us.png
with the PHP SDK it's easy to set the key of the object.
$USAGE = "\n" .
"To run this example, supply the name of an S3 bucket and a file to\n" .
"upload to it.\n" .
"\n" .
"Ex: php PutObject.php <bucketname> <filename>\n";
if (count($argv) <= 2){
echo $USAGE;
exit();
}
$bucket = $argv[1];
$file_Path = $argv[2];
$key = basename($argv[2]);
try{
//Create a S3Client
$s3Client = new S3Client([
'region' => 'us-west-2',
'version' => '2006-03-01'
]);
$result = $s3Client->putObject([
'Bucket' => $bucket,
'Key' => $key,
'SourceFile' => $file_Path,
]);
} catch (S3Exception $e) {
echo $e->getMessage() . "\n";
}
yii2 i think you need to set setFsUrl()
http://www.yiiframework.com/extension/yii2-file-upload/#hh8
setFsUrl(string $url)
(Only for Local mode)
Sets url. For example, if you set path to 'http://static.example.com' file after uploading will have URL http://static.example.com/path/to/your/file
Default to /
php $uploader->setFsPath('http://pathtoyoursite.com');
I am working on a project, now i need to rename a key using its prefix in s3 php sdk api. i couldn't find it, if any can help. Thanks
function moveFile($oldPath,$newPath){
$oKey = $this->getKey($oldPath);
$nKey = $this->getKey($newPath);
try{
// Copy an object.
$this->o->copyObject(array(
'Bucket' => $this->bucket,
'ACL' => 'public-read',
'Key' => $nKey,
'CopySource' => "{$this->bucket}/{$oKey}"
));
$this->deleteFile($oldPath);
} catch (S3Exception $e) {
echo $e->getMessage() . "\n";
return false;
}
}
You can rename s3 files using below code :
$s3sdk = new Sdk($awsConfig);
$s3 = $s3sdk->createS3();
$s3->registerStreamWrapper();
rename($oldName, $newName);
both names need to contain the full s3 path e.g:
"s3://yourBucketName/path/to/file"
Basically registerStreamWrapper() enables PHP filesystem commands for s3 files.
I did this, you guys answered late. i did it myself but LuFFy answer is also correct.
function renameFolder($oldPath,$newPath){
$oKey = $this->getKey($oldPath);
if(strpos($oKey,'/')==false){$oKey.='/';}
//echo '<br>oKey: '.$oKey.'<br>';
try{
// Copy an object.
/*$this->o->copyObject(array(
'Bucket' => $this->bucket,
'ACL' => 'public-read',
'Key' => $nKey,
'CopySource' => "{$this->bucket}/{$oKey}"
));*/
$result = $this->o->listObjects([
'Bucket' => $this->bucket, // REQUIRED
'Prefix' => $oKey,
]);
foreach($result['Contents'] as $file){
//echo '<br>objectKey: '.$file['Key'].'<br>';
$nKey = str_replace($this->getLastKey($oldPath),$this->getLastKey($newPath),$file['Key']);
//echo '<br>nKey: '.$nKey.'<br>';
$this->o->copyObject(array(
'Bucket' => $this->bucket,
'ACL' => 'public-read',
'Key' => $nKey,
'CopySource' => "{$this->bucket}/".$file['Key'].""
));
}
$this->deleteDir($oldPath);
}catch(S3Exception $e) {
echo $e->getMessage() . "\n";
return false;
}
}
I have managed to rename existing files on the batch using below steps:
Lets say your config/filesystems.php looks like this:
'disks' => [
's3_test_bucket' => [
'driver' => 's3',
'key' => env('AWS_KEY', 'your_aws_key_here'),
'secret' => env('AWS_SECRET','your_aws_secret_here'),
'region' => env('AWS_REGION', 'your_aws_region_here'),
'version' => 'latest',
'bucket' => 'my-test-bucket',
],
];
Let's say, you have my-test-bucket on your AWS S3.
Lets say you have following files inside the my-test-bucket/test-directory directory.
i.e.
test-files-1.csv
test-files-2.csv
test-files-3.csv
Call below function to rename existing files on a selected directory on S3 Bucket.
$directoryPath = 'test-directory';
$storage = new MyStorageRepository();
$storage->renameAnyExistingFilesOnImportDirectory('my-test-bucket', 'test-directory');
Output: files should be rename as below on my-test-bucket/test-directory directory:
test-files-1--1548870936.csv
test-files-2--1548870936.csv
test-files-3--1548870936.csv
Include the below library class or methods on your class and you should be good.
use Illuminate\Support\Facades\App;
use Illuminate\Support\Facades\Storage;
class MyStorageRepository
{
public function renameAnyExistingFilesOnImportDirectory($bucket, $directoryPath)
{
$directoryPath = App::environment() . '/' . $directoryPath;
$storage = Storage::disk('s3_test_bucket');
$suffix = '--' . time(); // File suffix to rename.
if ($storage->exists($directoryPath)) {
$this->renameStorageDirectoryFiles($directoryPath, $storage, $suffix);
}
}
private function getNewFilename($filename, $suffix = null)
{
$file = (object) pathinfo($filename);
if (!$suffix) {
$suffix = '--' . time();
}
return $file->dirname . '/' . $file->filename . $suffix . '.' . $file->extension;
}
private function renameStorageDirectoryFiles($directoryPath, $storage = null, $suffix = null, $filesystemDriver = null)
{
if (!$storage) {
$storage = Storage::disk($filesystemDriver);
}
// List all the existing files from the directory
$files = $storage->files($directoryPath);
if (count($files) < 1 ) return false;
foreach($files as $file) {
// Get new filename
$newFilename = Helpers::getNewFilename($file, $suffix);
// Renamed the files
$storage->move($file, $newFilename);
}
}
}
I'm having some issues when trying to upload an image to AWS S3. It seems to upload the file correctly but, whenever I try to download or preview, it can't be opened. Currently, this is the upload code I'm using:
<?php
require_once 'classes/amazon.php';
require_once 'includes/aws/aws-autoloader.php';
use Aws\S3\S3Client;
$putdata = file_get_contents("php://input");
$request = json_decode($putdata);
$image_parts = explode(";base64,", $request->image);
$image_type_aux = explode("image/", $image_parts[0]);
$image_type = $image_type_aux[1];
$image_base64 = $image_parts[1];
$dateTime = new DateTime();
$fileName = $dateTime->getTimestamp() . "." . $image_type;
$s3Client = S3Client::factory(array(
'region' => 'eu-west-1',
'version' => '2006-03-01',
'credentials' => array(
'key' => Amazon::getAccessKey(),
'secret' => Amazon::getSecretKey(),
)
));
try {
$result = $s3Client->putObject(array(
'Bucket' => Amazon::getBucket(),
'Key' => 'banners/' . $fileName,
'Body' => $image_base64,
'ContentType' => 'image/' . $image_type,
'ACL' => 'public-read'
));
echo $result['ObjectURL'] . "\n";
} catch(S3Exception $e) {
echo $e->getMessage() . "\n";
}
?>
So, when I check the console after uploading the image file, it has the expected size, permissions and headers but, as I said, whenever I try to open the file, it fails.
What could be the problem here? Thanks in advance.
The issue here is you appear to be uploading the base64 encoded version of the image and not the raw bytes of the image. Take $image_base64 and decode into raw bytes first http://php.net/manual/en/function.base64-decode.php . I am sure if you tried to open those "images" in a text editor you would see base64 hex data.
you can upload "on the fly" by using the function $s3Client->upload like the following example:
<?php
$bucket = 'bucket-name';
$filename = 'image-path.extension';
$imageData = base64_decode(end(explode(",", $base64)));
$upload = $s3Client->upload($bucket, $filename, $imageData, 'public-read');
$upload->get('ObjectURL');
I use silex php 2.0 and the following code fount
$s3 = $app['aws']->createS3();
$putdata = file_get_contents("php://input");
$data = json_decode($request->getContent(), true);
$data = (object) $data;
$image_parts = explode(";base64,", $data->image);
$image_type_aux = explode("image/", $image_parts[0]);
$image_type = $image_type_aux[1];
$image_base64 = base64_decode($image_parts[1]);
$result = $s3->putObject([
'ACL' => 'public-read',
'Body' => $image_base64,
'Bucket' => 'name-bucket',
'Key' => 'test_img.jpeg',
'ContentType' => 'image/' . $image_type,
]);
var_dump($result['ObjectURL']);
I'm trying to upload a picture on my amazon S3 via their PHP SDK. So I made a little script to do so. However, my script doesn't work and my exception doesn't send me back any error message.
I'm new with AWS thank you for your help.
Here is the code :
Config.php
<?php
return array(
'includes' => array('_aws'),
'services' => array(
'default_settings' => array(
'params' => array(
'key' => 'PUBLICKEY',
'secret' => 'PRIVATEKEY',
'region' => 'eu-west-1'
)
)
)
);
?>
Index.php
<?php
//Installing AWS SDK via phar
require 'aws.phar';
use Aws\S3\S3Client;
use Aws\S3\Exception\S3Exception;
$bucket = 'infact';
$keyname = 'myImage';
// $filepath should be absolute path to a file on disk
$filepath = 'image.jpg';
// Instantiate the client.
$s3 = S3Client::factory('config.php');
// Upload a file.
try {
$result = $s3->putObject(array(
'Bucket' => $bucket,
'Key' => $keyname,
'SourceFile' => $filePath,
'ContentType' => 'text/plain',
'ACL' => 'public-read',
'StorageClass' => 'REDUCED_REDUNDANCY'
));
// Print the URL to the object.
echo $result['ObjectURL'] . "\n";
} catch (S3Exception $e) {
echo $e->getMessage() . "\n";
}
?>
EDIT : I'm now using this code but its still not working. I don't even have error or exception message.
<?php
require 'aws.phar';
use Aws\S3\S3Client;
use Aws\S3\Exception\S3Exception;
$bucket = 'infactr';
$keyname = 'sample';
// $filepath should be absolute path to a file on disk
$filepath = 'image.jpg';
// Instantiate the client.
$s3 = S3Client::factory(array(
'key' => 'key',
'secret' => 'privatekey',
'region' => 'eu-west-1'
));
try {
// Upload data.
$result = $s3->putObject(array(
'Bucket' => $bucket,
'Key' => $keyname,
'SourceFile' => $filePath,
'ACL' => 'public-read',
'ContentType' => 'image/jpeg'
));
// Print the URL to the object.
echo $result['ObjectURL'] . "\n";
} catch (S3Exception $e) {
echo $e->getMessage() . "\n";
}
?>
Try something like this (from the AWS docs):
<?php
require 'aws.phar';
use Aws\S3\S3Client;
use Aws\S3\Exception\S3Exception;
$bucket = '<your bucket name>';
$keyname = 'sample';
// $filepath should be absolute path to a file on disk
$filepath = '/path/to/image.jpg';
// Instantiate the client.
$s3 = S3Client::factory(array(
'key' => 'your AWS access key',
'secret' => 'your AWS secret access key'
));
try {
// Upload data.
$result = $s3->putObject(array(
'Bucket' => $bucket,
'Key' => $keyname,
'SourceFile' => $filepath,
'ACL' => 'public-read'
));
// Print the URL to the object.
echo $result['ObjectURL'] . "\n";
} catch (S3Exception $e) {
echo $e->getMessage() . "\n";
}
?>
It works fine for me as long as you have the right credentials. Keep in mind that the key name is the name of your file in S3 so if you want to have your key have the same name of your file you have to do something like: $keyname = 'image.jpg'; . Also, a jpg is generally not a plain/text file type, you can ommit that Content-type field or you can just simply specify: image/jpeg
$s3 = S3Client::factory('config.php');
should be
$s3 = S3Client::factory(include 'config.php');
For those looking an up to date working version, this is what I am using
// Instantiate the client.
$s3 = S3Client::factory(array(
'credentials' => [
'key' => $s3Key,
'secret' => $s3Secret,
],
'region' => 'us-west-2',
'version' => "2006-03-01"
));
try {
// Upload data.
$result = $s3->putObject(array(
'Bucket' => $s3Bucket,
'Key' => $fileId,
'SourceFile' => $filepath."/".$fileName
));
return $result['ObjectURL'];
} catch (S3Exception $e) {
return false;
}
An alternative way to explain is by showing the curl, and how to build it in php - the pragmatic approach.
Please don't stone me for ugly code, just thought that this example is easy to follow for uploading to Azure from PHP, or other language.
$azure1 ='https://viperprodstorage1.blob.core.windows.net/bucketnameAtAzure/';
$azure3 ='?sr=c&si=bucketnameAtAzure-policy&sig=GJ_verySecretHashFromAzure_aw%3D';
$shellCmd='ls -la '.$outFileName;
$lsOutput=shell_exec($shellCmd);
#print_r($lsOutput);
$exploded=explode(' ', $lsOutput);
#print_r($exploded);
$fileLength=$exploded[7];
$curlAzure1="curl -v -X PUT -T '" . $outFileName . "' -H 'Content-Length: " . $fileLength . "' ";
$buildedCurlForUploading=$curlAzure1."'".$azure1.$outFileName.$azure3."'";
var_dump($buildedCurlForUploading);
shell_exec($buildedCurlForUploading);
This is the actual curl
shell_exec("curl -v -X PUT -T 'fileName' -H 'Content-Length: fileSize' 'https://viperprodstorage1.blob.core.windows.net/bucketnameAtAzure/fileName?sr=c&si=bucketNameAtAzure-policy&sig=GJ_verySecretHashFromAzure_aw%3D'")
Below are the code for upload image/file in amazon s3 bucket.
function upload_agreement_data($target_path, $source_path, $file_name, $content_type)
{
$fileup_flag = false;
/*------------- call global settings helper function starts ----------------*/
$bucketName = "pilateslogic";
//$global_setting_option = '__cloud_front_bucket__';
//$bucketName = get_global_settings($global_setting_option);
/*------------- call global settings helper function ends ----------------*/
if(!$bucketName)
{
die("ERROR: Template bucket name not found!");
}
// Amazon profile_template template js upload URL
$target_profile_template_js_url = "/".$bucketName."/".$target_path;
// Chatching profile_template template js upload URL
//$source_profile_template_js_url = dirname(dirname(dirname(__FILE__))).$source_path."/".$file_name;
// file name
$template_js_file = $file_name;
$this->s3->setEndpoint("s3-ap-southeast-2.amazonaws.com");
if($this->s3->putObjectFile($source_path, $target_profile_template_js_url, $template_js_file, S3::ACL_PUBLIC_READ, array(), array("Content-Type" => $content_type)))
{
$fileup_flag = true;
}
return $fileup_flag;
}