I'm trying to upload the base64 data of an image directly through javascript to Google Storage using signed URLs as authentication, which is apparently possible to do.
According to developers.google.com/storage/docs/reference-methods#putobject there are only six headers that need to be set for this to work. Also for the header 'Authorization' I'm attempting to use the last option here:
developers.google.com/storage/docs/reference-headers#authorization
Which is 'A signature' developers.google.com/storage/docs/authentication#service_accounts
The only thing I want to use PHP for is to get the signature. Here is what I have been trying to get working with no success.
PHP & JS page/code
<?php
$theDate = Date(DATE_RFC822);
function signedURL( $filename, $bucket, $method = 'PUT' ) {
$signature = "";
$duration = 30;
$emailID = "980000000000-ytyertyr#developer.gserviceaccount.com";
$certs = array();
$priv_key = file_get_contents("9999999999999999999999999999-privatekey.p12");
if (!openssl_pkcs12_read($priv_key, $certs, 'notasecret')) { echo "Unable to parse the p12 file. OpenSSL error: " . openssl_error_string(); exit(); }
$expires = time() + $duration;
$to_sign = ( $method . "\n\n\n" . $expires . "\n" . "/" . $bucket . "/" . $filename );
$RSAPrivateKey = openssl_pkey_get_private($certs["pkey"]);
if (!openssl_sign( $to_sign, $signature, $RSAPrivateKey, 'sha256' ))
{
error_log( 'openssl_sign failed!' );
$signature = 'failed';
} else {
$signature = urlencode( base64_encode( $signature ) );
}
return (
'http://storage.googleapis.com/' . $bucket . '/' . $filename . '?GoogleAccessId=' . $emailID . '&Expires=' . $expires . '&Signature=' . $signature
);
openssl_free_key($RSAPrivateKey);
}
?>
<script>
var base64img = 'data:image/png;base64,AAABAAIAICA....snip...A';
var xhr = new XMLHttpRequest();
//PUT test - PUT status "(Canceled)" - OPTION status 200 (OK)
xhr.open("PUT", "<?php echo signedURL('test.png', 'mybucket'); ?>");
//xhr.setRequestHeader("Content-type", "image/png");
xhr.setRequestHeader("x-goog-acl", "public-read"); //try to set public read on file
xhr.setRequestHeader("Content-Length", base64img.length); // Chrome throws error (Refused to set unsafe header "Content-Length" )
xhr.send( base64img );
//GET test.txt temp file - working and returning 200 status (signing must be working ?)
/*
xhr.open("GET", "<?php echo signedURL('test.txt', 'mybucket', 'GET'); ?>");
xhr.send();
*/
//
</script>
Cors xml (seems to be fine) - I've set a wildcard only while testing and a low cache/maxage time
<?xml version="1.0" ?>
<CorsConfig>
<Cors>
<Origins>
<Origin>*</Origin>
</Origins>
<Methods>
<Method>GET</Method>
<Method>HEAD</Method>
<Method>OPTIONS</Method>
<Method>PUT</Method>
</Methods>
<ResponseHeaders>
<ResponseHeader>accept-encoding</ResponseHeader>
<ResponseHeader>cache-control</ResponseHeader>
<ResponseHeader>content-length</ResponseHeader>
<ResponseHeader>content-type</ResponseHeader>
<ResponseHeader>expect</ResponseHeader>
<ResponseHeader>if-modified-since</ResponseHeader>
<ResponseHeader>origin</ResponseHeader>
<ResponseHeader>range</ResponseHeader>
<ResponseHeader>referer</ResponseHeader>
<ResponseHeader>x-goog-acl</ResponseHeader>
<ResponseHeader>x-goog-api-version</ResponseHeader>
</ResponseHeaders>
<MaxAgeSec>900</MaxAgeSec>
</Cors>
</CorsConfig>
I've tested the GET method on a file and get a 200 status back now (\n\n - fix)
Update:
Looking in Firefox it does return a 403, unlike Chrome.
So the following lines are weird, as the conflate signed URLs with OAuth and PUT with POST:
# This looks like a PUT to signed URL
xhr.open("PUT", '<?php echo signedURL('imgfile.png','PUT',30,'mybucketname'); ?>', true);
# But multipart requires POST
xhr.setRequestHeader("Content-type", "multipart/form-data; boundary="+boundary);
# And here's a second form of authorization
xhr.setRequestHeader("Authorization", "OAuth <?php echo $signature; ?>");
multipart/form-data uploads require POST verb and are intended for html forms: Google Cloud Storage : PUT Object vs POST Object to upload file.?.
As long as you are sending a custom headers in an XMLHttpRequest I would recommend using PUT with either OAuth credentials:
xhr.open("PUT", "https://storage.googleapis.com/mybucketname/imgfile.png");
xhr.setRequestHeader("Authorization", "OAuth Bearer 1234567abcdefg");
xhr.setRequestHeader("Content-Length", raw_img_bytes.length);
xhr.send(raw_img_bytes);
or a signed url:
xhr.open("PUT", "https://storage.googleapis.com/mybucketname/imgfile.png?" +
"GoogleAccessId=1234567890123#developer.gserviceaccount.com&" +
"Expires=136891473&" +
"Signature=BClz9e...WvPcwN%2BmWBPqwg...sQI8IQi1493mw%3D");
xhr.setRequestHeader("Content-Length", raw_img_bytes.length);
xhr.send(raw_img_bytres);
I gess your Content-Type is something known (like Content-Type:video/mp4 for instance)? Try to upload a file with not known extention. For me, PUT is working in this case, not when Content-Type is not empty...
I don't understand why...
Related
I'm building a PHP based webpage. I want to use REST APIs to store, read and update data in an AZURE Cosmos DB.
First challenge is to generate the authentication token with the masterkey. I used the Microsoft documentation: https://learn.microsoft.com/de-de/rest/api/cosmos-db/access-control-on-cosmosdb-resources
and a postman collection* as a reference to build the code below.
*see
https://www.youtube.com/watch?v=2ndj_-zp82Y
https://github.com/MicrosoftCSA/documentdb-postman-collection
I used the console log from postman to compare each step and figured out that there are different results** but I have no clue how to get the same results from my PHP like from Postman
**
key64 PHP:
WUtDMFF5VWZkZ1NsTFp5UmU5ZVBMbW9jV3ZSN.....==
key64 Postman:
60a0b443251f7604a52d9c917bd78f2e6a1c5af4790d3f67dc7dbd513d173418... NO == at the end
authstring PHP:
type%253Dmaster%2526ver%253D1.0%2526sig%.....
authstring POSTMAN:
type%3Dmaster%26ver%3D1.0%26sig%3.....
POSTMAN (JS)
// store our master key for documentdb
var mastKey = postman.getEnvironmentVariable("DocumentDBMasterKey");
console.log("mastKey = " + mastKey);
// store our date as RFC1123 format for the request
var today = new Date();
var UTCstring = today.toUTCString();
postman.setEnvironmentVariable("RFC1123time", UTCstring);
// Grab the request url
var url = request.url.trim();
console.log("request url = " + url);
// strip the url of the hostname up and leading slash
var strippedurl = url.replace(new RegExp('^https?://[^/]+/'),'/');
console.log ("stripped Url = " + strippedurl);
// push the parts down into an array so we can determine if the call is on a specific item
// or if it is on a resource (odd would mean a resource, even would mean an item)
var strippedparts = strippedurl.split("/");
var truestrippedcount = (strippedparts.length - 1);
console.log(truestrippedcount);
// define resourceId/Type now so we can assign based on the amount of levels
var resourceId = "";
var resType = "";
// its odd (resource request)
if (truestrippedcount % 2)
{
console.log("odd");
// assign resource type to the last part we found.
resType = strippedparts[truestrippedcount];
console.log("resType");
console.log(resType);
if (truestrippedcount > 1)
{
// now pull out the resource id by searching for the last slash and substringing to it.
var lastPart = strippedurl.lastIndexOf("/");
resourceId = strippedurl.substring(1,lastPart);
console.log(resourceId);
}
}
else // its even (item request on resource)
{
console.log("even");
// assign resource type to the part before the last we found (last is resource id)
resType = strippedparts[truestrippedcount - 1];
console.log("resType");
// finally remove the leading slash which we used to find the resource if it was
// only one level deep.
strippedurl = strippedurl.substring(1);
console.log("strippedurl");
// assign our resourceId
resourceId = strippedurl;
console.log("resourceId");
console.log(resourceId);
}
// assign our verb
var verb = request.method.toLowerCase();
// assign our RFC 1123 date
var date = UTCstring.toLowerCase();
// parse our master key out as base64 encoding
var key = CryptoJS.enc.Base64.parse(mastKey);
console.log("key = " + key);
// build up the request text for the signature so can sign it along with the key
var text = (verb || "").toLowerCase() + "\n" +
(resType || "").toLowerCase() + "\n" +
(resourceId || "") + "\n" +
(date || "").toLowerCase() + "\n" +
"" + "\n";
console.log("text = " + text);
// create the signature from build up request text
var signature = CryptoJS.HmacSHA256(text, key);
console.log("sig = " + signature);
// back to base 64 bits
var base64Bits = CryptoJS.enc.Base64.stringify(signature);
console.log("base64bits = " + base64Bits);
// format our authentication token and URI encode it.
var MasterToken = "master";
var TokenVersion = "1.0";
auth = encodeURIComponent("type=" + MasterToken + "&ver=" + TokenVersion + "&sig=" + base64Bits);
console.log("auth = " + auth);
// set our auth token enviornmental variable.
postman.setEnvironmentVariable("authToken", auth);
PHP CODE
#PHP Script
function generateAuthKey($url, $method){
$key = "****************";
$date = new DateTime('');
$date = $date->format('D, d M Y H:i:s O');
$ressourcetype = "";
$strippedurl = parse_url($url, PHP_URL_PATH);
$strippedparts = explode("/", $strippedurl);
$strippedurlcount = sizeof($strippedparts)-1;
#GET RESSOURCE TYPE
if ($strippedurlcount % 2){
$resType = $strippedparts[$strippedurlcount];
if ($strippedurlcount > 1){
$ressourcetype = $strippedparts[$strippedurlcount];
}
}
else{
$ressourcetype = $strippedparts[$strippedurlcount-1];
}
$sig = nl2br(strtolower($method)."\n".strtolower($ressourcetype)."\n".$strippedurl."\n".strtolower($date)."\n".""."\n");
$sig = utf8_encode($sig);
$key64 = base64_encode($key);
echo $key64."\n";
$hmac = hash_hmac('sha256',$sig,$key64);
$token = "type=master&ver=1.0&sig=".$hmac;
return urlencode($token);
}
How can I change the PHP script to provide the same output as Postman (JS)?
I believe the issue is with the following line of code:
$key64 = base64_encode($key);
As per the REST API documentation, you should be doing a base64_decode of your key as the key is already base64 encoded.
Please try by changing your code to:
$key64 = base64_decode($key);
While this is an old question, I'll note a couple issues with the author's code:
It was not necessary to utf8_encode() -- trying to utf8 encode a string that is already valid ISO-8859-1 can produce unexpected results. Also note that this method is deprecated as of PHP 8.2.0.
The author was returning the string output of hash_hmac(), rather than binary.
Here is how you correctly generate a signature in PHP. Bear in mind that all requests to the Cosmos REST API, needs to include an x-ms-date header, which matches the same date used to generate the token. It's up to you how to want to handle that, but in my case, I chose to return the date and token as an array from the function. You could also consider making the function return the entire header array all at once.
I am using Carbon and Guzzle in this example.
private function cosmosAuth(string $method, string $resourceType, string $resourceLink)
{
$date = Carbon::now()->toRfc7231String();
// alternatively: gmdate('D, d M Y H:i:s T')
$key = base64_decode(MY_COSMOS_KEY);
$body = $method . "\n" .
$resourceType . "\n" .
$resourceLink . "\n" .
$date . "\n" .
"\n";
$hash = hash_hmac('sha256', strtolower($body), $key, true);
$signature = base64_encode($hash);
$tokenType = "master";
$tokenVersion = "1.0";
$token = urlencode("type={$tokenType}&ver={$tokenVersion}&sig={$signature}");
return [
'date' => $date,
'token' => $token
];
}
Here is an example of a request to update a document. Note if you created your collection with a partition key, you also must pass the partition key value in the x-ms-documentdb-partitionkey header. The format here is a little janky, with Microsoft expecting a string representation of an array containing the value.
$resource = "dbs/{$database}/colls/{$collection}/docs/{$documentId}";
$auth = $this->cosmosAuth("PUT", "docs", $resource);
try {
$client = new \GuzzleHttp\Client();
$client->request("PUT", "https://{$account}.documents.azure.com/{$resource}", [
'headers' => [
'authorization' => $auth['token'],
'x-ms-date' => $auth['date'],
'x-ms-documentdb-partitionkey' => '["'.$documentId.'"]',
],
'json' => $documentData
]);
}
catch (Exception $e) {
$this->log("error: {$e->getMessage()}");
}
Microsoft resources:
Constructing a hashed token
Replace a document
I'm trying to use the Form Post middleware (enabled) with OpenStack Swift. I've the following code in php:
$expires = intval(time() + 60*1000);
$path = '/v1/AUTH_xxxxxxxxx/mycontainer';
$max_file_count = 1;
$max_file_size = 104857600;
$redirect = '';
$key = 'testkey';
$hmac_body = sprintf("%s\n%s\n%d\n%d\n%d", $path, $redirect, $max_file_size, $max_file_count, $expires);
$sig = hash_hmac('sha1', $hmac_body, $key);
And my Ajax call is the following:
Upload.http({
url: uploadLink.url, //includes path
method: 'POST',
data: {
redirect: uploadLink.redirect,
max_file_size: uploadLink.max_file_size,
max_file_count: uploadLink.max_file_count,
expires: uploadLink.expires,
signature: uploadLink.sig,
file: file
}
})
But the result is 401 status code. When I try with the tempUrl middleware, it works fine (so the Temp-URL-Key works and CORS too). I've tried with Postman, it did not work.
Do you have any idea?
I have a question here, I try to transfer a variable to my PHP script in order to retrieve data from Bing Search API.
I use the following AJAX code:
var bingquery = 'bingquery=' + $('#query').val();
console.log(bingquery);
$.ajax({
method: "POST",
url: "hw8.php",
dataType: "json",
data: bingquery,
success: function(jsondata){
console.log('***Test for News Feeds***');
console.log(jsondata);
}
});
And my PHP is:
if (isset($_POST["bingquery"])){
// Replace this value with your account key
$accountKey = '***myaccountkey***';
$WebSearchURL = 'https://api.datamarket.azure.com/Bing/Search/v1/' + 'News?$format=json&Query=';
$cred = sprintf('Authorization: Basic %s', base64_encode($accountKey . ":" . $accountKey) );
$context = stream_context_create(array(
'http' => array(
'header' => $cred
)
));
$request = $WebSearchURL . urlencode( '\'' . $_POST["bingquery"] . '\'');
//if I hard code the request URL here, it does work.
$response = file_get_contents($request, 0, $context);
echo $response;
}
I wonder if there is something wrong with my URL encoding? Because the console says file_get_contents(0%27MYSYMBOL%27) fails, MYSYMBOL is the string I want in the search.
Thank you so much for your help!
There is nothing wrong about the encoding at all, urlencode is supposed to make the input string url safe, and that is exactly what it is doing, \ has special meaning in a url and hence it is being encoded by the function.
UPDATE
You are adding up two strings, in PHP . is used to concatenate two strings, make the following changes,
$WebSearchURL = 'https://api.datamarket.azure.com/Bing/Search/v1/News';
$request = $WebSearchURL .'?Query='.urlencode($_POST["bingquery"]).'&$format=json;
I get my file via:
require_once 'google/appengine/api/cloud_storage/CloudStorageTools.php';
use google\appengine\api\cloud_storage\CloudStorageTools;
$public_link = CloudStorageTools::getPublicUrl("gs://bucket/file.pdf", false);
If I go to $public_link in the browser, it shows the PDF inside the browser. I am trying to figure out how I can force the download of this file.
Google App Engine only has a 60 second timeout so I'm afraid the serve function wont work via GAE. Does anyone have any suggestions?
--
EDIT
Andrei Volga's previous answer in this post suggests I use a Signed URL with a response-content-distribution header.
So far, I am able to create a signed URL that successfully shows the file but I am not able to generate a signed url that has any sort of header at all aka create a signed URL that will force the download instead of just showing it.
This is what I have so far, most of which is courtesy of mloureiro.
function googleBuildConfigurationString($method, $expiration, $file, array $options = [])
{
$allowedMethods = ['GET', 'HEAD', 'PUT', 'DELETE'];
// initialize
$method = strtoupper($method);
$contentType = $options['Content_Type'];
$contentMd5 = $options['Content_MD5'] ? base64_encode($options['Content_MD5']) : '';
$headers = $options['Canonicalized_Extension_Headers'] ? $options['Canonicalized_Extension_Headers'] . PHP_EOL : '';
$file = $file ? $file : $options['Canonicalized_Resource'];
// validate
if(array_search($method, $allowedMethods) === false)
{
throw new RuntimeException("Method '{$method}' is not allowed");
}
if(!$expiration)
{
throw new RuntimeException("An expiration date should be provided.");
}
return <<<TXT
{$method}
{$contentMd5}
{$contentType}
{$expiration}
{$headers}{$file}
TXT;
}
function googleSignString($p12FilePath, $string)
{
$certs = [];
if (!openssl_pkcs12_read(file_get_contents($p12FilePath), $certs, 'notasecret'))
{
echo "Unable to parse the p12 file. OpenSSL error: " . openssl_error_string(); exit();
}
$RSAPrivateKey = openssl_pkey_get_private($certs["pkey"]);
$signed = '';
if(!openssl_sign( $string, $signed, $RSAPrivateKey, 'sha256' ))
{
error_log( 'openssl_sign failed!' );
$signed = 'failed';
}
else $signed = base64_encode($signed);
return $signed;
}
function googleBuildSignedUrl($serviceEmail, $file, $expiration, $signature)
{
return "http://storage.googleapis.com{$file}" . "?GoogleAccessId={$serviceEmail}" . "&Expires={$expiration}" . "&Signature=" . urlencode($signature);
}
$serviceEmail = '<EMAIL>';
$p12FilePath = '../../path/to/cert.p12';
$expiration = (new DateTime())->modify('+3hours')->getTimestamp();
$bucket = 'bucket';
$fileToGet = 'picture.jpg';
$file = "/{$bucket}/{$fileToGet}";
$string = googleBuildConfigurationString('GET', $expiration, $file, array("Canonicalized_Extension_Headers" => ''));
$signedString = googleSignString($p12FilePath, $string);
$signedUrl = googleBuildSignedUrl($serviceEmail, $file, $expiration, $signedString);
echo $signedUrl;
For small files you can use serve option instead of public URL with save-as option set to true. See documentation.
For large files you can use a Signed URL with response-content-disposition parameter.
You can add and additional query string only.
https://cloud.google.com/storage/docs/xml-api/reference-headers#responsecontentdisposition
response-content-disposition
A query string parameter that allows content-disposition to be overridden for authenticated GET requests.
Valid Values URL-encoded header to return instead of the content-disposition of the underlying object.
Example
?response-content-disposition=attachment%3B%20filename%3D%22foo%22
I've written a PHP script that generates a signed CloudFront URL for RTMP with use in Flowplayer that's working just fine, but when I use the same signature generation method to create a download URL I get an AccessDenied XML file from Amazon. I've tried just about everything and I'm at my wits end. Anyone know why the signature would work for RTMP streaming, but that same signature generation method would fail for a download?
$keyPairId = 'APK...';
$privateKey = '/var/www/certs/pk-APK....pem';
$rtmp = false;
$distribution = 'd2m...';
// Get extension.
$extension = substr($this->getFilename(), strrpos($this->getFilename(), '.') + 1);
$fileName = substr($this->getFilename(), 0, strrpos($this->getFilename(), '.'));
$expires = strtotime(gmdate('Y-m-d H:i:s', strtotime('+3 hours')));
$json = '{"Statement":[{"Resource":"' . $fileName . '","Condition"{"DateLessThan":{"AWS:EpochTime":' . $expires . '}}}]}';
// read cloudfront private key pair
$fp = fopen($privateKey, 'r');
$priv_key = fread($fp, 8192);
fclose($fp);
// create the private key
$key = openssl_get_privatekey($priv_key);
// sign the policy with the private key
// depending on your php version you might have to use
// openssl_sign($json, $signed_policy, $key, OPENSSL_ALGO_SHA1)
openssl_sign($json, $signed_policy, $key);
openssl_free_key($key);
// create url safe signed policy
$base64_signed_policy = base64_encode($signed_policy);
$signature = str_replace(array('+', '=', '/'), array('-', '_', '~'), $base64_signed_policy);
// construct the url
$urlParams = urlencode($this->getFilename()) . '?Expires=' . $expires .'&Signature=' . $signature . '&Key-Pair-Id=' . $keyPairId;
$keyPairId;
if ($rtmp) {
$url = ( ($this->getExtension() != 'flv') ? $this->getExtension() . ':' : '' ) . $urlParams;
} else {
$url = 'https://' . $distribution . '.cloudfront.net/' . $urlParams;
}
First of all, signed RTMP urls are made differently than regular urls
RTMP distributions: Include only the stream name. For example, if the
full URL for a streaming video is:
rtmp://s5c39gqb8ow64r.cloudfront.net/videos/mp3_name.mp3
then use the following value for Resource:
videos/mp3_name
Regular signed urls contain the entire path.
Secondly, cloudfront RTMP distributions only distribute streaming media over RTMP. You said that you wanted a download url, so using an RTMP distribution will not enable you to download the file.
You probably want to create a cloudfront web distribution and link it to the same bucket, then generate a signed url using the web distribution, and access it that way.