I am trying to generate a URL that other system can use to upload a file to my S3 bucket. I looked over the documentation and similar issues, however, I cannot find the right solution.
I have tried creating the PreSigned URL multiple ways ($this->s3 is a reference so S3Client):
1:
$signedUrl = $this->s3->getCommand(
'PutObject',
array(
'Bucket' => $this->bucket,
'Key' => 'recording_test.mp3',
'Body' => ''
)
)->createPresignedUrl('+2 hours');
2:
$command = $this->s3->getCommand('PutObject', array('Bucket' => $this->bucket, 'Key' => 'recording_test3.mp3', 'Body' => ''));
$request = $command->prepare();
$signedUrl = $this->s3->createPresignedUrl($request, '+2 hours');
When trying to simply access the URL I get following error:
The request signature we calculated does not match the signature you provided. Check your key and signing method.
I have also tried it this way:
$key = 'recording_test2.mp3';
$url = $this->bucket.'/'.$key;
$request = $this->s3->put($url);
$signedUrl = $this->s3->createPresignedUrl($request, '+2 hours');
However, this generates a URL in format of s3.amazonaws.com/bucket/key..., which yields an error about invalid endpoint, and that endpoint in format bucket.s3.amazonaws.com/key... should be used. Manually changing the URL gives the same invalid signature error as per above.
I cannot see what am I doing wrong or any other way to generate the PreSigned URL?
Thanks for help
When creating your signed URL, you should not use the Body key. The signed URL generated includes a signature of the body to be uploaded. Then, when you try upload a file using the URL, it can't match the empty string you are using to the content of the file.
Also, if you are using Curl to put the file, I recommend using \CURLFile, and remember to use curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "PUT"); for your PUT request.
Related
I am trying to upload a file using a signed url but I keep getting key too long error. running this via curl on cli works fine.
$result = "hl7/1/2df8c3ef-8eca-4e19-aae0-41b074f37c8e.hl7?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=CREDENTIALS%2F20211206%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-Date=20211206T144541Z&X-Amz-Expires=3600&X-Amz-Security-Token=IQoJb3JpZ2luX2VjELX%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCWV1LXdlc3QtMSJGMEQCIH%2BBvsVggdM0wLpK0Kd7gGVopCVYrDNxrHoJWdJZs%2Bl7AiBwFw1i3LNWsoDsK6SZzz8%2BPdHxYh%2Bn6T4DbHt5Ya5w8Sr6AwiO%2F%2F%2F%2F%2F%2F%2F%2F%2F%2F8BEAIaDDc4MTcyNzU3MTE3NyIMK4bkPUK4yBIM3U65Ks4DQAkLuCkrZ%2F7tcoN7ItKGxmSBM0TMYbGq7dhsS1SWSxqffm%2BOM9QzEROrQRpt%2Bm%2BUQ3R5o7vz7Y4x5lxhkvIOyzXxVUqNznCBnWEYwEdK8aLPlZLiFkh2wznL2SeTLZsYO4LD3WXoY3AiVZZYZroNkJcUicNlectdsuoFWov5DIJ7k6PyAoEaRFk00Afa9W8NX5pN4NevvgclgqpceqeLD7L%2FBHBje13F68aJ6sv%2F%2FLeWZ4pvSR5ySgWaUaQ1aHHIsNVhuuZCgqC3R8wlN2AGRQ4h5M%2Byb6Tq6pFJwgI2CeP%2BgAFmMUh0feyJDROLq88i%2BRTSIEsUlH8GQGDg%2FSwIPBlL3Fg8TxDSzL8wa8Quk2h%2BgpNalvdRmIUR6iYem4DoNUxkihU%2F77g0TEOVLIl%2Fk20freTIWSeR8ZNCmIZOhUvm9Ea7TQl88B3jWq3oVWWG3xsKkEvCZ%2FwdnQAI20fUp8EtrnODtjjNNanmgWSafHURLTAm2HjtNbnKri%2B2mQmGQoHo2s2xX%2Fi6CYuEZI5yJBYuUSFZHhSum0h%2Bja643YDOp14TmYO38xFcIThxKDqaEMBjdRo15WvVDFcXSMsUdXLElXvOPMfHBoMdT4meMMiTuI0GOqYBT%2FMubfy97zprO%2FNP8xyb2u8qfDYeO0P7xBXdNFrbfVgX%2Fdr6%2Fhh8WSEz2K9PojxLN3NLGnfiMaUaUpev2kHcThP%2BuMkb%2FurH9Pl9QCFKsgKnhylrIKC0bQ%2B4TaEbJoRwh%2BRQUZGqaJfaPrMR5tAyYZ5kN4w1PXLRjpqi2U27B3qpGfu3jruEtOsuHtxM57tFQgyUHy3J%2FOdV7YVzYX%2B8cHSHkx2ndA%3D%3D&X-Amz-Signature=e653b4bbc96cba74979a087b3bc0f14ea6ff9fcd8eff0bd28234d897833240d1&X-Amz-SignedHeaders=host&x-id=PutObject";
$client = new S3Client( [
'version' => '2006-03-01',
'region' => 'eu-west-1',
'validate' => false
] );
$params = [
'Bucket' => env('RESULTS_UPLOAD_SERVICE_BUCKET'),
'Key' => $result,
'Body' => file_get_contents($fileData)
];
$response = $client->putObject($params);
Here is my curl command which works
curl --request PUT --upload-file 20211008192601_60461040005062.HL7 "https://domain.s3.eu-west-1.amazonaws.com/hl7/1/2df8c3ef-8eca-4e19-aae0-41b074f37c8e.hl7?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=ASIA3MAUYDTUUN444IX6%2F20211206%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-Date=20211206T144541Z&X-Amz-Expires=3600&X-Amz-Security-Token=IQoJb3JpZ2luX2VjELX%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCWV1LXdlc3QtMSJGMEQCIH%2BBvsVggdM0wLpK0Kd7gGVopCVYrDNxrHoJWdJZs%2Bl7AiBwFw1i3LNWsoDsK6SZzz8%2BPdHxYh%2Bn6T4DbHt5Ya5w8Sr6AwiO%2F%2F%2F%2F%2F%2F%2F%2F%2F%2F8BEAIaDDc4MTcyNzU3MTE3NyIMK4bkPUK4yBIM3U65Ks4DQAkLuCkrZ%2F7tcoN7ItKGxmSBM0TMYbGq7dhsS1SWSxqffm%2BOM9QzEROrQRpt%2Bm%2BUQ3R5o7vz7Y4x5lxhkvIOyzXxVUqNznCBnWEYwEdK8aLPlZLiFkh2wznL2SeTLZsYO4LD3WXoY3AiVZZYZroNkJcUicNlectdsuoFWov5DIJ7k6PyAoEaRFk00Afa9W8NX5pN4NevvgclgqpceqeLD7L%2FBHBje13F68aJ6sv%2F%2FLeWZ4pvSR5ySgWaUaQ1aHHIsNVhuuZCgqC3R8wlN2AGRQ4h5M%2Byb6Tq6pFJwgI2CeP%2BgAFmMUh0feyJDROLq88i%2BRTSIEsUlH8GQGDg%2FSwIPBlL3Fg8TxDSzL8wa8Quk2h%2BgpNalvdRmIUR6iYem4DoNUxkihU%2F77g0TEOVLIl%2Fk20freTIWSeR8ZNCmIZOhUvm9Ea7TQl88B3jWq3oVWWG3xsKkEvCZ%2FwdnQAI20fUp8EtrnODtjjNNanmgWSafHURLTAm2HjtNbnKri%2B2mQmGQoHo2s2xX%2Fi6CYuEZI5yJBYuUSFZHhSum0h%2Bja643YDOp14TmYO38xFcIThxKDqaEMBjdRo15WvVDFcXSMsUdXLElXvOPMfHBoMdT4meMMiTuI0GOqYBT%2FMubfy97zprO%2FNP8xyb2u8qfDYeO0P7xBXdNFrbfVgX%2Fdr6%2Fhh8WSEz2K9PojxLN3NLGnfiMaUaUpev2kHcThP%2BuMkb%2FurH9Pl9QCFKsgKnhylrIKC0bQ%2B4TaEbJoRwh%2BRQUZGqaJfaPrMR5tAyYZ5kN4w1PXLRjpqi2U27B3qpGfu3jruEtOsuHtxM57tFQgyUHy3J%2FOdV7YVzYX%2B8cHSHkx2ndA%3D%3D&X-Amz-Signature=e653b4bbc96cba74979a087b3bc0f14ea6ff9fcd8eff0bd28234d897833240d1&X-Amz-SignedHeaders=host&x-id=PutObject"
it errors with
<?xml version=\"1.0\" encoding=\"UTF-8\"?>
<Error><Code>KeyTooLongError</Code><Message>Your key is too long</Message><Size>1 (truncated...)
at /var/www/snapshot/vendor/guzzlehttp/guzzle/src/Exception/RequestException.php:113)
[stacktrace]
You are receiving that error because you are passing pre-signed url result in to put request as key, That put request method should actually be used to upload directly with php sdk.
It seems like you want to upload file to a pre-signed url, Once you generate pre-signed url you will need to do a HTTP PUT request, you can do it directly from browser that is the usual way why we generate pre-signed url that expires after certain time so you don't expose your keys to public.
If you are using PHP directly you could simply use your keys to upload using above code or if you still want to upload to a pre-signed url with php see this link
How do I upload a file to an AWS Presigned URL with PHP Curl?
I am going to answer my own question here. It took me hours to figure this out as there is no info on this anywhere so I thought I should post this somewhere I would look first.
I was using the AWS PHP SDK to send a PUT request to add a lifecycle policy to my Digital Ocean space and it would not take since it requires a ContentMD5 header. There are two problems here, the first problem is the SDK URL encodes the path/key, which is a problem with /?lifecycle, /?location, and /?acl since they become "/%3Flifecycle" -- skip this paragraph if this isn't part of your request path. To temporarily stop this to add or update a bucket policy you have to find the file RestSerializer.php in the SDK files, if you added the API with composer it will be in a path like /vendor/aws/aws-sdk-php/src/Api/Serializer/RestSerializer.php in your composer/websites root, which will likely be in /var/www. In RestSerializer.php find the two rawurlencode function calls and remove them but leave the value/argument "rawurlencode($varspecs[$k])" becomes "$varspecs[$k]".
Now the request is going to the correct URL, to generate the ContentMD5 you need to construct a little PHP code depending on what you're doing. If you have put the XML text for your policy in a file use md5_file(PATH_TO_FILE_HERE, true) if you are using a string use md5(STRING_HERE, true). Then wrap that in base64_encode() so it looks something like base64_encode(md5_file('/path/file.xml', true)). Finally, add that to your putObject array with 'ContentMD5' => base64_encode(md5_file('/path/file.xml', true)) .
PHP Example with File:
// $spaceS3Client is a new S3Client object.
// since its a file, I need to get the file first
$xmlfile = fopen('/spaces.xml', 'r+');
$request = $spaceS3Client->putObject([
'Bucket' => 'myspacename',
'Key' => '?lifecycle',
'Body' => $xmlfile,
'ContentType' => 'application/xml',
'ContentMD5' => base64_encode(md5_file('/spaces.xml'', true))
]);
// close file
fclose($xmlfile);
// if you are having trouble connecting to your space in the first place with an S3Client object, since its set up for AWS and not DO you need to add an 'endpoint' to the array in new S3Client like 'endpoint' => 'https://'.$myspace.'.'.$myspaceregion.'.digitaloceanspaces.com'. You also need to add 'bucket_endpoint' => true.
There are two problems here, the first problem is the SDK URL encodes the path/key, which is a problem with /?lifecycle, /?location, and /?acl since they become "/%3Flifecycle" -- skip this paragraph if this isn't part of your request path. To temporarily stop this to add or update a bucket policy you have to find the file RestSerializer.php in the SDK files, if you added the API with composer it will be in a path like /vendor/aws/aws-sdk-php/src/Api/Serializer/RestSerializer.php in your composer/websites root, which will likely be in /var/www. In RestSerializer.php find the two rawurlencode function calls and remove them but leave the value/argument "rawurlencode($varspecs[$k])" becomes "$varspecs[$k]".
Now the request is going to the correct URL, to generate the ContentMD5 you need to construct a little PHP code depending on what you're doing. If you have put the XML text for your policy in a file use md5_file(PATH_TO_FILE_HERE, true) if you are using a string use md5(STRING_HERE, true). Then wrap that in base64_encode() so it looks something like base64_encode(md5_file('/path/file.xml', true)). Finally, add that to your putObject array with 'ContentMD5' => base64_encode(md5_file('/path/file.xml', true)) .
PHP Example with File:
// $spaceS3Client is a new S3Client object.
// since its a file, I need to get the file first
$xmlfile = file_get_contents('/spaces.xml', 'r+');
$request = $spaceS3Client->putObject([
'Bucket' => 'myspacename',
'Key' => '?lifecycle',
'Body' => $xmlfile,
'ContentType' => 'application/xml',
'ContentMD5' => base64_encode(md5_file('/spaces.xml', true))
]);
// if you are having trouble connecting to your space in the first place with an S3Client object, since its set up for AWS and not DO you need to add an 'endpoint' to the array in new S3Client like 'endpoint' => 'https://'.$myspace.'.'.$myspaceregion.'.digitaloceanspaces.com'. You also need to add 'bucket_endpoint' => true.
// to check the rules have been set use a getObject request and then use the code below to parse the response.
header('Content-type: text/xml');
$request = $request->toArray()["Body"];
echo $request;
I am trying to make a PutObject presigned request using the AWS S3 PHP SDK.
I have gotten the request to work but now I only want to allow my users to be able to only upload video files.I have tried a lot of combinations and searched a lot but I could not get it to work.
Here is the sample code I use:
$cmd = $this->s3client->getCommand('PutObject', [
'Bucket' => 'myBucket',
'Key' => 'inputs/' . $movie->getId(),
'ACL' => 'private',
'Conditions' => ['Starts-With', '$Content-Type', 'video/'], // I have tried other combinations but it seems to not work
]);
$request = $this->s3client->createPresignedRequest($cmd, '+30 minutes');
$movie->setSignedUrl((string)$request->getUri());
The signed url generated does never include the Content-Type in the X-Amz-SignedHeaders query parameter, only the host is included.
The putObject() request has no documented Conditions key.
You appear to be confusing S3's PUT upload interface with the pre-signed POST capability, which supports policy document conditions like ['Starts-With', '$Content-Type', 'video/'],
PUT does not support "starts with". It requires the exact Content-Type and the key for this (which should result in the header appearing in the X-Amz-SignedHeaders query string parameter) is simply ContentType. It goes in the outer parameters array, just like Bucket and Key.
But if you want to support multiple content types without knowing the specific type in advance, you need to use POST uploads.
I am using s3 direct uploads along with a database to store the URLS of the files (along with other data like who uploaded etc).
To allow direct upload to s3, I'm creating a presigned URL like :
$s3 = App::make('aws')->createClient('s3', [
'credentials' => [
'key' => 'AAAAAAAAAAAAAAA',
'secret' => 'YYYYYYYYYYYYYYYYYYYY',
]
]);
$command = $s3->getCommand('PutObject', [
'#use_accelerate_endpoint'=>true,
'Bucket' => 'remdev-experimental',
'Key' => "newest newest.txt",
'Metadata' => array(
'foo' => "test",
)
]);
return response()->json(((string)$s3->createPresignedRequest($command, '+1 minutes')->getUri()));
Now, after the file from the client has finished uploading , I want my server to know about it. So I will require the client to send me a request , notifying about the fact that he has finished uploading. For this, I think the simplest(and also secure) way is to just allow the client to send back the signed URL that he just sent back.
Is there a way to parse the URL ?
I am interested in getting the object key , and more importantly , I want to verify that the URL has not been tampered with (meaning, the signature in the URL should match the rest of the contents). How can I do this in php sdk ?
The signed URL is the file's URL with the signature information in the query data. So a signed request for bucket: remdev-experimental file: abc.txt looks like https://s3.amazonaws.com/remdev-experimental/abc.txt?X-Amz-Date=date&X-Amz-Expires=60&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Signature=signature&X-Amz-Credential=SOMEID/20160703/us-east-1/s3/aws4_request&X-Amz-SignedHeaders=Host&x-amz-security-token=some-long-token so all you need to do is get the URL's Path (/remdev-experimental/abc.txt and take everything after the 2nd slash.
Also you should be aware that you can have S3 redirect the browser to a URL using success_action_redirect in an HTTP post policy
Lastly you can have S3 trigger a notification to your server (via SQS, SNS, or Lambda) whenever a file is uploaded.
I googled for a solution to share a private S3 object using AWS SDK for PHP 2.
I can only find the solution for .Net, Java, and Visual Studio.
http://docs.aws.amazon.com/AmazonS3/latest/dev/ShareObjectPreSignedURL.html
I also want to generate this presigned url for just 15 minutes of expiry time.
Currently, there is a way to use the latest AWS SDK for PHP 2 to accomplish the above.
Read this.
This link will show you 2 ways to do this.
Most common way is:
$signedUrl = $client->getObjectUrl($bucket, 'data.txt', '+15 minutes');
The second way is to use the command object method which is reproduced below.
// Get a command object from the client and pass in any options
// available in the GetObject command (e.g. ResponseContentDisposition)
$command = $client->getCommand('GetObject', array(
'Bucket' => $bucket,
'Key' => 'data.txt',
'ResponseContentDisposition' => 'attachment; filename="data.txt"'
));
// Create a signed URL from the command object that will last for
// 15 minutes from the current time
$signedUrl = $command->createPresignedUrl('+15 minutes');
$signedUrl will give you a string that looks something like this:
https://bucketname.s3.amazonaws.com/keytothefile.ext?AWSAccessKeyId=AASDASDFDFGTSSYCQ&Expires=1380861181&Signature=eD6qtV81278nmashdkp0huURXc%3D