I am trying to upload a file using a signed url but I keep getting key too long error. running this via curl on cli works fine.
$result = "hl7/1/2df8c3ef-8eca-4e19-aae0-41b074f37c8e.hl7?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=CREDENTIALS%2F20211206%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-Date=20211206T144541Z&X-Amz-Expires=3600&X-Amz-Security-Token=IQoJb3JpZ2luX2VjELX%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCWV1LXdlc3QtMSJGMEQCIH%2BBvsVggdM0wLpK0Kd7gGVopCVYrDNxrHoJWdJZs%2Bl7AiBwFw1i3LNWsoDsK6SZzz8%2BPdHxYh%2Bn6T4DbHt5Ya5w8Sr6AwiO%2F%2F%2F%2F%2F%2F%2F%2F%2F%2F8BEAIaDDc4MTcyNzU3MTE3NyIMK4bkPUK4yBIM3U65Ks4DQAkLuCkrZ%2F7tcoN7ItKGxmSBM0TMYbGq7dhsS1SWSxqffm%2BOM9QzEROrQRpt%2Bm%2BUQ3R5o7vz7Y4x5lxhkvIOyzXxVUqNznCBnWEYwEdK8aLPlZLiFkh2wznL2SeTLZsYO4LD3WXoY3AiVZZYZroNkJcUicNlectdsuoFWov5DIJ7k6PyAoEaRFk00Afa9W8NX5pN4NevvgclgqpceqeLD7L%2FBHBje13F68aJ6sv%2F%2FLeWZ4pvSR5ySgWaUaQ1aHHIsNVhuuZCgqC3R8wlN2AGRQ4h5M%2Byb6Tq6pFJwgI2CeP%2BgAFmMUh0feyJDROLq88i%2BRTSIEsUlH8GQGDg%2FSwIPBlL3Fg8TxDSzL8wa8Quk2h%2BgpNalvdRmIUR6iYem4DoNUxkihU%2F77g0TEOVLIl%2Fk20freTIWSeR8ZNCmIZOhUvm9Ea7TQl88B3jWq3oVWWG3xsKkEvCZ%2FwdnQAI20fUp8EtrnODtjjNNanmgWSafHURLTAm2HjtNbnKri%2B2mQmGQoHo2s2xX%2Fi6CYuEZI5yJBYuUSFZHhSum0h%2Bja643YDOp14TmYO38xFcIThxKDqaEMBjdRo15WvVDFcXSMsUdXLElXvOPMfHBoMdT4meMMiTuI0GOqYBT%2FMubfy97zprO%2FNP8xyb2u8qfDYeO0P7xBXdNFrbfVgX%2Fdr6%2Fhh8WSEz2K9PojxLN3NLGnfiMaUaUpev2kHcThP%2BuMkb%2FurH9Pl9QCFKsgKnhylrIKC0bQ%2B4TaEbJoRwh%2BRQUZGqaJfaPrMR5tAyYZ5kN4w1PXLRjpqi2U27B3qpGfu3jruEtOsuHtxM57tFQgyUHy3J%2FOdV7YVzYX%2B8cHSHkx2ndA%3D%3D&X-Amz-Signature=e653b4bbc96cba74979a087b3bc0f14ea6ff9fcd8eff0bd28234d897833240d1&X-Amz-SignedHeaders=host&x-id=PutObject";
$client = new S3Client( [
'version' => '2006-03-01',
'region' => 'eu-west-1',
'validate' => false
] );
$params = [
'Bucket' => env('RESULTS_UPLOAD_SERVICE_BUCKET'),
'Key' => $result,
'Body' => file_get_contents($fileData)
];
$response = $client->putObject($params);
Here is my curl command which works
curl --request PUT --upload-file 20211008192601_60461040005062.HL7 "https://domain.s3.eu-west-1.amazonaws.com/hl7/1/2df8c3ef-8eca-4e19-aae0-41b074f37c8e.hl7?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=ASIA3MAUYDTUUN444IX6%2F20211206%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-Date=20211206T144541Z&X-Amz-Expires=3600&X-Amz-Security-Token=IQoJb3JpZ2luX2VjELX%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCWV1LXdlc3QtMSJGMEQCIH%2BBvsVggdM0wLpK0Kd7gGVopCVYrDNxrHoJWdJZs%2Bl7AiBwFw1i3LNWsoDsK6SZzz8%2BPdHxYh%2Bn6T4DbHt5Ya5w8Sr6AwiO%2F%2F%2F%2F%2F%2F%2F%2F%2F%2F8BEAIaDDc4MTcyNzU3MTE3NyIMK4bkPUK4yBIM3U65Ks4DQAkLuCkrZ%2F7tcoN7ItKGxmSBM0TMYbGq7dhsS1SWSxqffm%2BOM9QzEROrQRpt%2Bm%2BUQ3R5o7vz7Y4x5lxhkvIOyzXxVUqNznCBnWEYwEdK8aLPlZLiFkh2wznL2SeTLZsYO4LD3WXoY3AiVZZYZroNkJcUicNlectdsuoFWov5DIJ7k6PyAoEaRFk00Afa9W8NX5pN4NevvgclgqpceqeLD7L%2FBHBje13F68aJ6sv%2F%2FLeWZ4pvSR5ySgWaUaQ1aHHIsNVhuuZCgqC3R8wlN2AGRQ4h5M%2Byb6Tq6pFJwgI2CeP%2BgAFmMUh0feyJDROLq88i%2BRTSIEsUlH8GQGDg%2FSwIPBlL3Fg8TxDSzL8wa8Quk2h%2BgpNalvdRmIUR6iYem4DoNUxkihU%2F77g0TEOVLIl%2Fk20freTIWSeR8ZNCmIZOhUvm9Ea7TQl88B3jWq3oVWWG3xsKkEvCZ%2FwdnQAI20fUp8EtrnODtjjNNanmgWSafHURLTAm2HjtNbnKri%2B2mQmGQoHo2s2xX%2Fi6CYuEZI5yJBYuUSFZHhSum0h%2Bja643YDOp14TmYO38xFcIThxKDqaEMBjdRo15WvVDFcXSMsUdXLElXvOPMfHBoMdT4meMMiTuI0GOqYBT%2FMubfy97zprO%2FNP8xyb2u8qfDYeO0P7xBXdNFrbfVgX%2Fdr6%2Fhh8WSEz2K9PojxLN3NLGnfiMaUaUpev2kHcThP%2BuMkb%2FurH9Pl9QCFKsgKnhylrIKC0bQ%2B4TaEbJoRwh%2BRQUZGqaJfaPrMR5tAyYZ5kN4w1PXLRjpqi2U27B3qpGfu3jruEtOsuHtxM57tFQgyUHy3J%2FOdV7YVzYX%2B8cHSHkx2ndA%3D%3D&X-Amz-Signature=e653b4bbc96cba74979a087b3bc0f14ea6ff9fcd8eff0bd28234d897833240d1&X-Amz-SignedHeaders=host&x-id=PutObject"
it errors with
<?xml version=\"1.0\" encoding=\"UTF-8\"?>
<Error><Code>KeyTooLongError</Code><Message>Your key is too long</Message><Size>1 (truncated...)
at /var/www/snapshot/vendor/guzzlehttp/guzzle/src/Exception/RequestException.php:113)
[stacktrace]
You are receiving that error because you are passing pre-signed url result in to put request as key, That put request method should actually be used to upload directly with php sdk.
It seems like you want to upload file to a pre-signed url, Once you generate pre-signed url you will need to do a HTTP PUT request, you can do it directly from browser that is the usual way why we generate pre-signed url that expires after certain time so you don't expose your keys to public.
If you are using PHP directly you could simply use your keys to upload using above code or if you still want to upload to a pre-signed url with php see this link
How do I upload a file to an AWS Presigned URL with PHP Curl?
I'm upload multiple images inside Picture(like folder) under Google Cloud Storage bucket.
Eg. Picture/a.jpg, Picture/b.jpg, Pictuer/c.jpg, .....
Now, I want to directly show those multiple images from Cloud Storage into my cakephp 2.x web application as a list.
According to Google Cloud Storage documentation, to access each object inside bucket, Signed URLs must be generated.
So, I created signed URLs for each object based on my selecting data from database. Following is my sample code.
<?php
# Imports the Google Cloud client library
use Google\Cloud\Storage\StorageClient;
use Google\Cloud\Core\Exception\GoogleException;
function index() {
//$rsl = 'select data from database';
for($i=0; $i<$count; $i++) {
# create url to access image from google cloud storage
$file_path = $rsl[$i]['pic']['file_path'];
if(!empty($file_path)) {
$rsl[$i]['pic']['real_path'] = $this->get_object_v4_signed_url($file_path);
} else {
$rsl[$i]['pic']['real_path'] = '';
}
}
}
/**
* Generate a v4 signed URL for downloading an object.
*
* #param string $bucketName the name of your Google Cloud bucket.
* #param string $objectName the name of your Google Cloud object.
*
* #return void
*/
function get_object_v4_signed_url($objectName) {
$cloud = parent::connect_to_google_cloud_storage();
$storage = $cloud[0];
$bucketName = $cloud[1];
$bucket = $storage->bucket($bucketName);
$object = $bucket->object($objectName);
if($object->exists()) {
$url = $object->signedUrl(
# This URL is valid for 1 minutes
new \DateTime('1 min'),
[
'version' => 'v4'
]
);
} else {
$url = '';
}
return $url;
}
?>
The problem is, it takes too long to generate because each file is generated to v4 signed URL. When I read Google Cloud Storage documentation, I didn't see to generate v4 signed URL of multiple objects at once(may be I'm wrong). So, is there anyway to speed it up this generating process?
As you mention there is no explanation in the Google Cloud Storage documentation on how to generate signed URLs for multiple objects using PHP as you do. However, I have found that using the gsutil signurl command, you can specify multiple paths or even use a wildcard:
gsutil signurl -d 1m ~/sandbox/key.json gs://bucket_name/object_1.jpg gs://bucket_name/object_2.jpg ...
gsutil signurl -d 1m ~/sandbox/key.json gs://bucket_name/*
This is specified in the gsutil help signurl page: "Multiple gs:// urls may be provided and may contain wildcards. A signed url will be produced for each provided url, authorized for the specified HTTP method and valid for the given duration."
Another option could be using multi-threading in your APP, here you'll find how to perform multi-threading using the pthreads API. If you choose to use this API you should bear in mind (from the documentation):
Warning
The pthreads extension cannot be used in a web server environment. Threading in PHP is therefore restricted to CLI-based applications only.
Warning
pthreads (v3) can only be used with PHP 7.2+: This is due to ZTS mode being unsafe in 7.0 and 7.1.
But since you have a web APP, I think that neither of those solutions would be useful for you. Besides that you could try using Cloud Functions to generate the signed URLs.
I am trying to make a PutObject presigned request using the AWS S3 PHP SDK.
I have gotten the request to work but now I only want to allow my users to be able to only upload video files.I have tried a lot of combinations and searched a lot but I could not get it to work.
Here is the sample code I use:
$cmd = $this->s3client->getCommand('PutObject', [
'Bucket' => 'myBucket',
'Key' => 'inputs/' . $movie->getId(),
'ACL' => 'private',
'Conditions' => ['Starts-With', '$Content-Type', 'video/'], // I have tried other combinations but it seems to not work
]);
$request = $this->s3client->createPresignedRequest($cmd, '+30 minutes');
$movie->setSignedUrl((string)$request->getUri());
The signed url generated does never include the Content-Type in the X-Amz-SignedHeaders query parameter, only the host is included.
The putObject() request has no documented Conditions key.
You appear to be confusing S3's PUT upload interface with the pre-signed POST capability, which supports policy document conditions like ['Starts-With', '$Content-Type', 'video/'],
PUT does not support "starts with". It requires the exact Content-Type and the key for this (which should result in the header appearing in the X-Amz-SignedHeaders query string parameter) is simply ContentType. It goes in the outer parameters array, just like Bucket and Key.
But if you want to support multiple content types without knowing the specific type in advance, you need to use POST uploads.
I am trying to generate a URL that other system can use to upload a file to my S3 bucket. I looked over the documentation and similar issues, however, I cannot find the right solution.
I have tried creating the PreSigned URL multiple ways ($this->s3 is a reference so S3Client):
1:
$signedUrl = $this->s3->getCommand(
'PutObject',
array(
'Bucket' => $this->bucket,
'Key' => 'recording_test.mp3',
'Body' => ''
)
)->createPresignedUrl('+2 hours');
2:
$command = $this->s3->getCommand('PutObject', array('Bucket' => $this->bucket, 'Key' => 'recording_test3.mp3', 'Body' => ''));
$request = $command->prepare();
$signedUrl = $this->s3->createPresignedUrl($request, '+2 hours');
When trying to simply access the URL I get following error:
The request signature we calculated does not match the signature you provided. Check your key and signing method.
I have also tried it this way:
$key = 'recording_test2.mp3';
$url = $this->bucket.'/'.$key;
$request = $this->s3->put($url);
$signedUrl = $this->s3->createPresignedUrl($request, '+2 hours');
However, this generates a URL in format of s3.amazonaws.com/bucket/key..., which yields an error about invalid endpoint, and that endpoint in format bucket.s3.amazonaws.com/key... should be used. Manually changing the URL gives the same invalid signature error as per above.
I cannot see what am I doing wrong or any other way to generate the PreSigned URL?
Thanks for help
When creating your signed URL, you should not use the Body key. The signed URL generated includes a signature of the body to be uploaded. Then, when you try upload a file using the URL, it can't match the empty string you are using to the content of the file.
Also, if you are using Curl to put the file, I recommend using \CURLFile, and remember to use curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "PUT"); for your PUT request.
I have a bucket in S3 that I linked up to a CNAME alias. Let's assume for now that the domain is media.mycompany.com. In the bucket are image files that are all set to private. Yet they are publicly used on my website using URL signing. A signed URL may look like this:
http://media.mycompany.com/images/651/38935_small.JPG?AWSAccessKeyId=05GMT0V3GWVNE7GGM1R2&Expires=1466035210&Signature=Uk3K7qVNRFHuIUnMaDadCOPjV%2BM%3D
This works fine as it is. I'm using a S3 helper library in PHP to generate such URLs. Here's the identifier of that library:
$Id: S3.php 44 2008-12-23 15:38:38Z don.schonknecht $
I know that it is old, but I'm relying on a lot of methods in this library, so it's not trivial to upgrade, and as said, it works well for me. Here's the relevant method in this library:
public static function getAuthenticatedURL($bucket, $uri, $lifetime, $hostBucket = false, $https = false) {
$expires = time() + $lifetime;
$uri = str_replace('%2F', '/', rawurlencode($uri)); // URI should be encoded (thanks Sean O'Dea)
return sprintf(($https ? 'https' : 'http').'://%s/%s?AWSAccessKeyId=%s&Expires=%u&Signature=%s',
$hostBucket ? $bucket : $bucket.'.s3.amazonaws.com', $uri, self::$__accessKey, $expires,
urlencode(self::__getHash("GET\n\n\n{$expires}\n/{$bucket}/{$uri}")));
}
In my normal, working setup, I'd call this method like this:
$return = $this->s3->getAuthenticatedURL('media.mycompany.com', $dir . '/' . $filename,
$timestamp, true, false);
This returns the correctly signed URL as shared earlier in this post, and all is good.
However, I'd now like to generate HTTPS URLs, and this is where I'm running into issues. Simply adding HTTPs to the current URL (by setting the last param of the method to true) will not work, it will generate a URL like this:
https://media.mycompany.com/images/651/38935_small.JPG?AWSAccessKeyId=05GMT0V3GWVNE7GGM1R2&Expires=1466035210&Signature=Uk3K7qVNRFHuIUnMaDadCOPjV%2BM%3D
This will obviously not work, since my SSL certificate (which is created from letsencrypt) is not installed on Amazon's domain, and as far as I know, there's no way to do so.
I've learned of an alternative URL format to access the bucket over SSL:
https://media.mycompany.com.s3.amazonaws.com/images/651/38935_small.JPG?AWSAccessKeyId=05GMT0V3GWVNE7GGM1R2&Expires=1466035210&Signature=Uk3K7qVNRFHuIUnMaDadCOPjV%2BM%3D
This apparently works for some people, but not for me, from what I know, it's due to having a dot (.) character in my bucket name. I cannot change the bucket name, it would have large consequences in my setup.
Finally, there's this format:
https://s3.amazonaws.com/media.mycompany.com/images/2428/39000_small.jpg?AWSAccessKeyId=05GMT0V3GWVNE7GGM1R2&Expires=1466035210&Signature=6p3W6GHQtddJNnCoUXaNl970x9s%3D
And here I am getting very close. If I take a working non-secure URL, and edit the URL to take on this format, it works. The image is shown.
Now I'd like to have it working in the automated way, from the signing method I showed earlier. I'm calling it like this:
$return = $this->s3->getAuthenticatedURL("s3.amazonaws.com/media.mycompany.com", $dir . '/' . $filename,
$timestamp, true, true);
The change here is the alternative bucket name format, and the last parameter being set to true, indicating HTTPs. This leads to an output like this:
https://s3.amazonaws.com/media.mycompany.com/images/2784/38965_small.jpg?AWSAccessKeyId=05GMT0V3GWVNE7GGM1R2&Expires=1466035210&Signature=Db2ynwWOV852Mn4rpcWA0Q1DrH0%3D
As you can see, it has the same format as the URL I manually crafted to work. But unfortunately, I'm getting signature errors:
<Code>SignatureDoesNotMatch</Code>
<Message>
The request signature we calculated does not match the signature you provided. Check your key and signing method.
</Message>
I'm stuck figuring out why these signatures are incorrect. I tried setting the 4th parameter of the signing method to true and false, but it makes no difference.
What am I missing?
Edit
Based on Michael's answer below I tried to do the simple string replace after the call to the S3 library, which works. Quick and dirty code:
$return = $this->s3->getAuthenticatedURL("media.mycompany.com", $dir . '/' . $filename, $timestamp, true, true);
$return = substr_replace($return, "s3.amazonaws.com/", strpos($return, "media.mycompany.com"), 0);
The change here is the alternative bucket name format
Almost. This library doesn't quite appear to have what you need in order to do what you are trying to do.
For Signature Version 2 (which is what you're using), your easiest workaround will be to take the signed URL with https://bucket.s3.amazonaws.com/path and just doing a string replace to https://s3.amazonaws.com/bucket/path.¹ This works because the signatures are equivalent in V2. It wouldn't work for Signature V4, but you aren't using that.
That, or you need to rewrite the code in the supporting library to handle this case with another option for path-style URLs.
The "hostbucket" option seems to assume a CNAME or Alias named after the bucket is pointing to the S3 endpoint, which won't work with HTTPS. Setting this option to true is actually causing the library to sign a URL for the bucket named s3.amazonaws.com/media.example.com, which is why the signature doesn't match.
If you wanted to hide the "S3" from the URL and use your own SSL certificate, this can be done by using CloudFront in front of S3. With CloudFront, you can use your own cert, and point it to any bucket, regardless of whether the bucket name matches the original hostname. However, CloudFront uses a very differential algorithm for signed URLs, so you'd need code to support that. One advantage of CloudFront signed URLs -- which may or may not be useful to you -- is that you can generate a signed URL that only works from the specific IP address you include in the signing policy.
It's also possible to pass-through signed S3 URLs with special configuration of CloudFront (configure the bucket as a custom origin, not an S3 origin, and forward the query string to the origin) but this defeats all caching in CloudFront, so it's a little bit counterproductive... but it would work.
¹ Note that you have to use the regional endpoint when you rewrite like this, unless your bucket is in us-east-1 (a.k.a. US Standard) so the hostname would be s3-us-west-2.amazonaws.com for buckets in us-west-2, for example. For US Standard, either s3.amazonaws.com or s3-external-1.amazonaws.com can be used with https URLs.
Spent days going round in circles trying to setup custom CNAME/host for presigned URLs and it seemed impossible.
All forums said it cannot be done, or you have to recode your whole app to use cloudfront instead.
Changing my DNS to point from MYBUCKET.s3-WEBSITE-eu-west-1.amazonaws.com to MYBUCKET.s3-eu-west-1.amazonaws.com fixed it instantly.
Hope this helps others.
Working code:
function get_objectURL($key) {
// Instantiate the client.
$this->s3 = S3Client::factory(array(
'credentials' => array(
'key' => s3_key,
'secret' => s3_secret,
),
'region' => 'eu-west-1',
'version' => 'latest',
'endpoint' => 'https://example.com',
'bucket_endpoint' => true,
'signature_version' => 'v4'
));
$cmd = $this->s3->getCommand('GetObject', [
'Bucket' => s3_bucket,
'Key' => $key
]);
try {
$request = $this->s3->createPresignedRequest($cmd, '+5 minutes');
// Get the actual presigned-url
$presignedUrl = (string)$request->getUri();
return $presignedUrl;
} catch (S3Exception $e) {
return $e->getMessage() . "\n";
}
}