I need Help on AWS multi-part Upload using PHP 5.3
I am using
1. WAMP Server.
2. PHP 5.3
3. OS: Windows 8
I am trying to upload a Big file to My Amazon S3 bucket.
And tried in many ways. I followed the code procedure from
http://docs.aws.amazon.com/AmazonS3/latest/dev/LLuploadFilePHP.html
I wrote PHP code but I don't know what mistake I did.
My Code:
<?php
require_once './sdk/sdk.class.php';
use \AmazonS3;
$ufile = $_FILES['ufile'];
$filepath = $ufile['tmp_name'];
$bucket = '**My_bkt**';
var_dump($ufile);
print $keyname = \date("Y") . "/" . \date("F") . "/" . \date("d") . "/" . $ufile['name'] . "<BR />";
$api_key = "***my_api_key***";
$secret_key = "***my_secret_key***";
// Define a megabyte.
define('MB', 1048576);
// Instantiate the class
//$s3 = new AmazonS3();
$s3 = new AmazonS3(array(
'key' => $api_key,
'secret' => $secret_key,
));
// 1. Initiate a new multipart upload.
/* #var $response type */
$response = $s3->initiate_multipart_upload($bucket, $keyname);
// Get the Upload ID.
$upload_id = (string) $response->body->UploadId;
// 2. Upload parts.
// Get part list for a given input file and given part size.
// Returns an associative array.
$parts = $s3->get_multipart_counts(filesize($filepath), 3 * MB);
foreach ($parts as $i => $part) {
// Upload part and save response in an array.
$responses[] = $s3->upload_part($bucket, $keyname, $upload_id, array(
'fileUpload' => $filepath,
'partNumber' => ($i + 1),
'seekTo' => (integer) $part['seekTo'],
'length' => (integer) $part['length'],
)
);
}
// 3. Complete multipart upload. We need all part numbers and ETag values.
$parts = $s3->list_parts($bucket, $keyname, $upload_id);
$response = $s3->complete_multipart_upload($bucket, $keyname, $upload_id, $parts);
var_dump($response);
Please help me.
I know you're using the older SDK 1 here, so my answer doesn't apply directly to what you've posted. That said, SDK 1.x is no longer being updated and SDK 2.x is what all new work should be using (as per the AWS SDK for PHP team).
If you do update your project to use SDK 2, then take a look at S3Client::upload(). It should greatly simplify what you're trying to do here.
I have Updated My SDK and Changed My Code like bellow,
////////////////// AWS Code Begin ////////////////////
/////////////////////////// Step 1 /////////////////////////////
$ufile = $_FILES['Filedata'];
$filename = $ufile['tmp_name'];
$filesize = $ufile['size'];
/* * ************ Calculating Number of Parts ******************* */
$number_of_parts = 0;
$r = $filesize % PART; // Remainder
$q = floor($filesize / PART); // Quotient
if ($r != 0) {
$number_of_parts = $q + 1;
} else {
$number_of_parts = $q;
}
$bucket = 'isource123';
$keyname = date("Y") . "/" . date("F") . "/" . date("d") . "/" . $ufile['name'];
///////////////////////////// Step 2 /////////////////////////////
// Create a service builder using a configuration file
$aws = Aws::factory('./aws/Aws/Common/Resources/aws-config.php');
// Get the client from the builder by namespace
$client = $aws->get('S3');
$uploader = \Aws\S3\Model\MultipartUpload\UploadBuilder::newInstance()
->setClient($client)
->setSource($filename)
->setBucket($bucket)
->setKey($keyname)
->setOption('Metadata', array('Foo' => 'Bar'))
->setOption('CacheControl', 'max-age=3600')
->setConcurrency($number_of_parts)
->build();
try {
$uploader->upload();
echo "Upload complete.\n";
} catch (MultipartUploadException $e) {
$uploader->abort();
echo "Upload failed.\n";
}
I updated My SDK to version 2 and it's working fine.
Related
Is there any documentation to get a SAS URL to download a file from a Snapshot of a Azure Share File?
Using this is easy to download a direct Azure File with SAS, but not any snapshot:
GenerateFileDownloadLinkWithSAS (https://github.com/Azure/azure-storage-php/blob/master/samples/FileSamples.php)
Here my code:
use MicrosoftAzure\Storage\Common\Exceptions\ServiceException;
use MicrosoftAzure\Storage\Common\Internal\Resources;
use MicrosoftAzure\Storage\Common\Internal\StorageServiceSettings;
use MicrosoftAzure\Storage\Common\Models\Range;
use MicrosoftAzure\Storage\Common\Models\Metrics;
use MicrosoftAzure\Storage\Common\Models\RetentionPolicy;
use MicrosoftAzure\Storage\Common\Models\ServiceProperties;
use MicrosoftAzure\Storage\File\FileRestProxy;
use MicrosoftAzure\Storage\File\FileSharedAccessSignatureHelper;
use MicrosoftAzure\Storage\File\Models\CreateShareOptions;
use MicrosoftAzure\Storage\File\Models\ListSharesOptions;
use MicrosoftAzure\Storage\File\Models\ListDirectoriesAndFilesOptions;
function MapFileURL($shareName,$filePath)
{
global $fileRestProxy;
global $mapConString;
$prepareFilePath = implode('/', array_map(function ($v)
{
return rawurlencode($v);
}, explode('/', $filePath))
);
// Create a SharedAccessSignatureHelper
$settings = StorageServiceSettings::createFromConnectionString($mapConString);
$accountName = $settings->getName();
$accountKey = $settings->getKey();
$helper = new FileSharedAccessSignatureHelper(
$accountName,
$accountKey
);
$endDate=MapIsoDate(time() + 13300);
// Generate a file readonly SAS token
// Refer to following link for full candidate values to construct a service level SAS
// https://learn.microsoft.com/en-us/rest/api/storageservices/constructing-a-service-sas
$sas = $helper->generateFileServiceSharedAccessSignatureToken(
Resources::RESOURCE_TYPE_FILE,
$shareName . "/" . $prepareFilePath,
'r', // Read
$endDate
);
$connectionStringWithSAS = Resources::FILE_ENDPOINT_NAME.'='.'https://'.$accountName.'.'.Resources::FILE_BASE_DNS_NAME.';'.Resources::SAS_TOKEN_NAME.'='.$sas;
$fileClientWithSAS = FileRestProxy::createFileService($connectionStringWithSAS);
// Get a downloadable file URL
$fileUrlWithSAS = sprintf(
'%s%s?%s',
(string)$fileClientWithSAS->getPsrPrimaryUri(),
$shareName . "/" . $prepareFilePath,
$sas
);
return $fileUrlWithSAS;
}
What would be missing to be able to download the file from a Azure File snapshot?
What would be missing to be able to download the file from a Azure
File snapshot?
What you need to do is append the share's snapshot date/time to your SAS URL. Something like:
https://account.file.core.windows.net/share/file.png?sastoken&snapshot=2021-05-01T13:49:56.0000000Z
Here is the code that works:
function MapSnapshotFileURL($shareName, $filePath, $snapshotTime)
{
global $fileRestProxy;
global $mapConString;
// Preparar path para enviar a la funciĆ³n azure.
$prepareFilePath = implode('/', array_map(function ($v)
{
return rawurlencode($v);
}, explode('/', $filePath))
);
// Create a SharedAccessSignatureHelper
$settings = StorageServiceSettings::createFromConnectionString($mapConString);
$accountName = $settings->getName();
$accountKey = $settings->getKey();
$helper = new FileSharedAccessSignatureHelper(
$accountName,
$accountKey
);
$endDate=MapIsoDate(time() + 13300);
//$endDate='2019-07-16T08:30:00Z';
// Generate a file readonly SAS token
// Refer to following link for full candidate values to construct a service level SAS
// https://learn.microsoft.com/en-us/rest/api/storageservices/constructing-a-service-sas
$sas = $helper->generateFileServiceSharedAccessSignatureToken(
Resources::RESOURCE_TYPE_FILE,
$shareName . "/" . $prepareFilePath,
'r', // Read
$endDate // '2020-01-01T08:30:00Z' // A valid ISO 8601 format expiry time
);
$connectionStringWithSAS = Resources::FILE_ENDPOINT_NAME.'='.'https://'.$accountName.'.'.Resources::FILE_BASE_DNS_NAME.';'.Resources::SAS_TOKEN_NAME.'='.$sas;
$fileClientWithSAS = FileRestProxy::createFileService($connectionStringWithSAS);
// Get a downloadable file URL
$fileUrlWithSAS = sprintf(
'%s%s?%s&%s',
(string)$fileClientWithSAS->getPsrPrimaryUri(),
$shareName . "/" . $prepareFilePath,
$sas,
"snapshot=".$snapshotTime
);
return $fileUrlWithSAS;
}
I'm looking for a good solution to get the signed url from amazon s3.
I have a version working with it, but not using laravel:
private function getUrl ()
{
$distribution = $_SERVER["AWS_CDN_URL"];
$cf = Amazon::getCFClient();
$url = $cf->getSignedUrl(array(
'url' => $distribution . self::AWS_PATH.rawurlencode($this->fileName),
'expires' => time() + (session_cache_expire() * 60)));
return $url;
}
I don't know if this is the best way to do with laravel, considering it has a entire file system to work...
But if don't have another way, how do I get the client? Debugging I've found an instance of it inside the Filesystem object, but it is protected...
For Laravel 5.5 and up,
you can now use temporary URLs/s3 presigned URL.
use \Storage;
// Make sure you have s3 as your disk driver
$url = Storage::disk('s3')->temporaryUrl(
'file1.jpg', Carbon::now()->addMinutes(5)
);
This only works for s3 storage driver AFAIK.
https://laravel.com/docs/5.5/filesystem#retrieving-files
In Laravel,
$s3 = \Storage::disk('s3');
$client = $s3->getDriver()->getAdapter()->getClient();
$expiry = "+10 minutes";
$command = $client->getCommand('GetObject', [
'Bucket' => \Config::get('filesystems.disks.s3.bucket'),
'Key' => "file/in/s3/bucket"
]);
$request = $client->createPresignedRequest($command, $expiry);
return (string) $request->getUri();
Make sure you have the AWS for flysystem composer package too (version will vary):
"league/flysystem-aws-s3-v3": "1.0.9"
After lot of bugs, at last, I found the solution of accessing private content of s3 bucket using below code:-
use Storage;
use Config;
$client = Storage::disk('s3')->getDriver()->getAdapter()->getClient();
$bucket = Config::get('filesystems.disks.s3.bucket');
$command = $client->getCommand('GetObject', [
'Bucket' => $bucket,
'Key' => '344772707_360.mp4' // file name in s3 bucket which you want to access
]);
$request = $client->createPresignedRequest($command, '+20 minutes');
// Get the actual presigned-url
echo $presignedUrl = (string)$request->getUri();
the above explain answer (#brian_d) is ok, but it takes too much time to generate presigned url. i wasted almost 4-5 days to overcome that. finally following worked for me. Thanks to #Kenth.
use Carbon\Carbon;
use Illuminate\Support\Facades\Storage;
$disk = Storage::disk('s3');
$url = $disk->getAwsTemporaryUrl($disk->getDriver()->getAdapter(), $value, Carbon::now()->addMinutes(5), []);
Here is a full code to upload image to S3 and create signed URL;
First, get a file object and create object name
$filename = $_FILES["image"]["name"];
$array = explode('.', $filename);
$fileExt = $array[1]; // to get file extension
$objectName = 'images/' . time() .$fileExt; // create object name
$document = fopen($_FILES["image"]["tmp_name"],'r');
// Code to upload document on s3
$s3 = \Storage::disk('s3');
$s3->put($objectName,$document,'public');
// If you want to get uploaded document Url you can use below code:
$image_name = \Storage::disk('s3')->url($name); // it will return stored document url
Code to create/get signed URL, you can put it as a saperate function also:
$s3 = \Storage::disk('s3');
$client = $s3->getDriver()->getAdapter()->getClient();
$expiry = "+10 minutes";
$command = $client->getCommand('GetObject', [
'Bucket' => config('params.YOUR_AWS_BUCKET_NAME'), // bucket name
'Key' => $objectName
]);
$request = $client->createPresignedRequest($command, $expiry);
$url = (string) $request->getUri(); // it will return signed URL
$file_path = "profile/image.png";
$client = Storage::disk('s3')->getDriver()->getAdapter()->getClient();
$bucket = config('filesystems.disks.s3.bucket');
$command = $client->getCommand('GetObject', [
'Bucket' => $bucket,
'Key' => $file_path // file name in s3 bucket which you want to access
]);
$request = $client->createPresignedRequest($command, '+5 minutes');
return $presignedUrl = (string) $request->getUri(); // Get the actual presigned-url
For Laravel 9 example to show all images from a protected folder:
$function = function ($image){
return Storage::disk('s3')->temporaryUrl(
$image, Carbon::now()->addMinutes(60)
);
};
$images = Storage::disk('s3')->allFiles('images/client');
return array_map($function,$images);
This is the simple code I`m using to upload an image in my rackspace account with PHP :
<?php
$api_username = 'lolcat';
$api_key = 'sexyapi';
require 'cloudfiles.php';
$auth = new CF_Authentication($api_username, $api_key);
$auth->authenticate();
$connection = new CF_Connection($auth);
$images = $connection->create_container("pitch");
$url = $images->make_public();
echo 'The url is '.$url;
$file = 'sherlock.jpg';
$img = $images->create_object($file);
$img->load_from_filename($file)
?>
Also everytime it gives diffrent errors. Like:
"Strict standards: Only variables should be passed by reference in C:\wamp\www\cloudfiles.php on line 1969 "
" Fatal error: Maximum execution time of 30 seconds exceeded in C:\wamp\www\cloudfiles_http.php on line 1249"
A sample of errors (imgur image)
Please help I`ve been trying this simple thing for 2 hours now.
Php-cloudfiles bindings are deprecated. I suggest you to give php-opencloud a try. I shouldn't be hard to port your code. Here's a quick example:
<?php
require 'vendor/autoload.php';
use OpenCloud\Rackspace;
$client = new Rackspace(Rackspace::US_IDENTITY_ENDPOINT, array(
'username' => 'foo',
'apiKey' => 'bar'
));
$service = $client->objectStoreService('cloudFiles', 'DFW');
$container = $service->createContainer('pitch');
$container->enableCdn();
$cdn = $container->getCdn();
//Print "Container SSL URL: " . serialize($cdn);
$files = array(
array(
'name' => 'file_1.txt',
'body' => fopen('files/file_1.txt', 'r+')
)
);
$container->uploadObjects($files);
You can find php-opencloud docs at: https://github.com/rackspace/php-opencloud/blob/master/docs/getting-started.md
pretty new with the aws sdk, looking to start. i've installed the sdk and everything but how do I start the ec2 instances using the php sdk? Some code samples would really be useful.
Here is a basic example of starting a machine from a defined AMI:
$image_id = 'ami-3d4ff254'; //Ubuntu 12.04
$min = 1; //the minimum number of instances to start
$max = 1; //the maximum number of instances to start
$options = array(
'SecurityGroupId' => 'default', //replace with your security group id
'InstanceType' => 't1.micro',
'KeyName' => 'keypair', //the name of your keypair for auth
'InstanceInitiatedShutdownBehavior' => 'terminate' //terminate on shutdown
);
require_once('AWSSDKforPHP/sdk.class.php');
$ec2 = new AmazonEC2();
$response = $ec2->run_instances($image_id, $min, $max, $options);
if(!$response->isOK()){
echo "Start failed\n";
}
This is assuming you have your AWS credentials setup properly ... Hopefully this gets you pointed in the right direction ...
Here is a more detailed script if you are interested:
// Sleep time to allow EC2 instance to start up
$sleeptime = 15;
$username = "ec2-user";
// For AWS PHP SDK
putenv('HOME=/home/ec2-user/');
require_once 'AWSSDKforPHP/sdk.class.php';
// Get data from HTTP POST
$ami = $_POST['amis'];
$instancetype = $_POST['instancetype'];
$keyname = $_POST['key'];
$securitygroup = $_POST['securitygroups'];
// Instantiate the AmazonEC2 class
$ec2 = new AmazonEC2();
// Boot an instance of the image
$response = $ec2->run_instances($ami, 1, 1, array(
'KeyName' => $keyname,
'InstanceType' => $instancetype,
'SecurityGroupId' => $securitygroup,
));
if (!($response->isOK())) {
echo "<p class='error'>ERROR! Could not create new instance!</p>";
return;
}
$instance = $response->body->instancesSet->item->instanceId;
$message = "<p>Your instance has been successfully created.</p>";
$message .= ("<p>Instance ID is: <b>$instance</b></p>");
// Give instance some time to start up
sleep ($sleeptime);
// Get the hostname from a call to the DescribeImages operation.
$response = $ec2->describe_instances(array(
'Filter' => array(
array('Name' => 'instance-id', 'Value' => "$instance"),
)
));
if (!($response->isOK())) {
echo "<p class='error'>ERROR! Could not retrieve hostname for instance!</p>";
return;
}
$hostname = $response->body->reservationSet->item->instancesSet->item->dnsName;
// Output the message
$message .= "<p>Your instance hostname is: <b>$hostname</b></p>";
$message .= "<p>You can connect to your instance using this command:<br>" .
"<b>ssh -i $keyname.pem $username#" . $hostname . "</b></p>";
echo $message;
Pretty much the same as #dleiftah's, except that it outputs the hostname of the new instance upon completion.
My code was working just fine a couple days ago. Then, all of a sudden, BAM: it stopped working. My PUTs stopped going through, with a SignatureDoesNotMatch error. Help?
require_once 'Crypt/HMAC.php';
require_once 'HTTP/Request.php';
function uploadFile($path_to_file, $store_file_as, $bucket, $debugmode = false) {
$S3_URL = "http://s3.amazonaws.com/";
$filePath = $path_to_file;
$contentType = 'audio/mpeg';
$keyId = 'THISISMYKEY, YES I DOUBLE CHECKED IT';
$secretKey = 'THIS IS MYSECRET, YES I DOUBLED CHECKED IT';
$key = $store_file_as;
$resource = $bucket . "/" . $key;
$acl = "public-read";
$verb = "PUT";
$httpDate = gmdate("D, d M Y H:i:s T");
$stringToSign = "PUT\n\naudio/mpeg\n$httpDate\nx-amz-acl:$acl\n/$resource";
$hasher =& new Crypt_HMAC($secretKey, "sha1");
$str = $hasher->hash($stringToSign);
$raw = '';
for ($i=0; $i < strlen($str); $i+=2) {
$raw .= chr(hexdec(substr($str, $i, 2)));
}
$signature = base64_encode($raw);
$req =& new HTTP_Request($S3_URL . $resource);
$req->setMethod('PUT');
$req->addHeader("content-type", $contentType);
$req->addHeader("Date", $httpDate);
$req->addHeader("x-amz-acl", $acl);
$req->addHeader("Authorization", "AWS " . $keyId . ":" . $signature);
$req->setBody(file_get_contents($filePath));
$req->sendRequest();
echo $req->getResponseBody();
}
Run your signature against the Amazon S3 JavaScript signature tester.
If the two signatures do not match, you know something is wrong with the keys or signing procedure.
If the two match, your keys and signing process are correct and the problem is somewhere else.
The JS tester is invaluable for troubleshooting signature generation problems.