Performance of Laravel storage streaming file from Digital ocean spaces - php

I've obtained the following code from searching about the topic
Route::get('/test', function () {
//disable execution time limit when downloading a big file.
set_time_limit(0);
$fs = Storage::disk('local');
$path = 'uploads/user-1/1653600850867.mp3';
$stream = $fs->readStream($path);
if (ob_get_level()) ob_end_clean();
return response()->stream(function () use ($stream) {
fpassthru($stream);
},
200,
[
'Accept-Ranges' => 'bytes',
'Content-Length' => 14098560,
'Content-Type' => 'application/octet-stream',
]);
});
However when I click play on the UI, it takes a good four seconds to start playing. If I switch the disk to local though, it plays almost instantly.
Is there a way to improve the performance or, read the stream by range as per request?
Edit
My current DO config is as per below
'driver' => 's3',
'key' => env('DO_ACCESS_KEY_ID'),
'secret' => env('DO_SECRET_ACCESS_KEY'),
'region' => env('DO_DEFAULT_REGION'),
'bucket' => env('DO_BUCKET'),
'url' => env('DO_URL'),
'endpoint' => env('DO_ENDPOINT'),
'use_path_style_endpoint' => env('DO_USE_PATH_STYLE_ENDPOINT', false),
But I find two type of integration online one specifying the CDN endpoint and one doesn't. I am not sure which one is relevant, though the one that specifies CDN is for Laravel 8 and I am on Laravel 9.

I had to change my code such that:
I had to use the php SDK client for connecting to Aws for the Laravel API isn't flexible to allow passing additional arguments (at least I haven't found anything while researching)
Change to streamDownload as I can't see any description to the stream method in the docs despite that it is present in code.
So the below code allows to achieve what I was aiming for which is, download by chunk based on the range received in the request.
return response()->streamDownload(function(){
$client = new Aws\S3\S3Client([
'version' => 'latest',
'region' => config('filesystems.disks.do.region'),
'endpoint' => config('filesystems.disks.do.endpoint'),
'credentials' => [
'key' => config('filesystems.disks.do.key'),
'secret' => config('filesystems.disks.do.secret'),
],
]);
$path = 'uploads/user-1/1653600850867.mp3';
$range = request()->header('Range');
$result = $client->getObject([
'Bucket' => 'wyxos-streaming',
'Key' => $path,
'Range' => $range
]);
echo $result['Body'];
},
200,
[
'Accept-Ranges' => 'bytes',
'Content-Length' => 14098560,
'Content-Type' => 'application/octet-stream',
]);
Note:
In a live scenario, you would need to cater for if range isn't specified, the content length will need to be the actual file size
When range is present however, the content length should then be the size of the segment being echoed

Related

How to setup amazon timestream in php?

I have found the documentation for it here. I have PHP SDK installed. Now when I go through the documents there is not so much in detail about the PHP one. I have the following questions:
Here how can I specify the $client
$result = $client->createDatabase([
'DatabaseName' => '<string>', // REQUIRED
'KmsKeyId' => '<string>',
'Tags' => [
[
'Key' => '<string>', // REQUIRED
'Value' => '<string>', // REQUIRED
],
// ...
],
]);
Is there any good documents or videos regarding the timestream in PHP from where I can get some help.
There are two client classes. One for writing and one for reading.
TimestreamWriteClient
https://docs.aws.amazon.com/aws-sdk-php/v3/api/class-Aws.TimestreamWrite.TimestreamWriteClient.html
and
TimestreamQueryClient
https://docs.aws.amazon.com/aws-sdk-php/v3/api/class-Aws.TimestreamQuery.TimestreamQueryClient.html
You can use the function createTimestreamQuery and createTimestreamWrite on the $sdk object to instantiate those classes.
A sample Timestream client and query below.
// Create client
$client = new \Aws\TimestreamQuery\TimestreamQueryClient([
'version' => 'latest',
'region' => AWS_REGION, /* eg: eu-west-1 */
'endpoint' => AWS_TIMESTREAM_ENDPOINT, /* eg: https://query-cell3.timestream.eu-west-1.amazonaws.com */
'credentials' => new \Aws\Credentials\Credentials(AWS_KEY, AWS_SECRET)
]);
// Perform a basic query with the client
$client->query([
'QueryString' => 'select * from "db_timestream"."tbl_usage_logs"',
'ValidateOnly' => true,
]);
If you receive endpoint warning, such as "The endpoint required for this service is currently unable to be retrieved"
You can find endpoint using AWS CLI command,
aws timestream-query describe-endpoints --region eu-west-1
Sample response:
{
"Endpoints": [
{
"Address": "query-cell3.timestream.eu-west-1.amazonaws.com",
"CachePeriodInMinutes": 1440
}
]
}
One can create TimestreamWriteClient and write records in a similar way.
The documentation seems sparse and a bit misleading, to me anyhow.
This is how I got it going for a write client (assuming SDK is installed).
//Create the client
$client = new \Aws\TimestreamWrite\TimestreamWriteClient([
'version' => 'latest',
'region' => 'eu-west-1',
'credentials' => new \Aws\Credentials\Credentials('***KEY***', '***SECRET***')
]);
Note that the 'endpoint' is not specified, as I've seen in some examples. There seems to be some misleading documentation of what the endpoint should be for any given region. The SDK does some magic and creates a suitable endpoint; providing a specific endpoint didn't work for me.
$result = $client->writeRecords(
[
'DatabaseName' => 'testDB',
'TableName' => 'history',
'Records' =>
[
[
'Dimensions' => [
[
'DimensionValueType' => 'VARCHAR',
'Name' => 'Server',
'Value' => 'VM01',
],
],
'MeasureName' => 'CPU_utilization',
'MeasureValue' => '1.21',
'MeasureValueType' => 'DOUBLE',
'Time' => strval(time()),
'TimeUnit' => 'SECONDS',
]
]
]
);
This seems to be the minimum set of things needed to write a record to Timestream successfully. The code above writes one record, with one dimension, in this case, a 'Name' of a 'Server', recording its CPU utilization at time().
Note:
Time is required, although the documentation suggested it is optional.
Time has to be a String.

Laravel Dropbox API v2 - Empty File on Upload

What I have till now
Right now I have a working oauth2 authentication between a laravel user and the dropbox API. Every user can upload files to their personal folder.
The Problem
After Uploading a file with laravel with the Dropbox API v2 I can see that there is a empty (0 Bytes) file uploaded.
Used to accomplish this task:
Laravel
Guzzle
Dropbox API Library
What am I missing?
The Code
My function for processing a form looks like this:
$formFile = $request->file('fileToUpload');
$path = $formFile->getClientOriginalName();
$file = $formFile->getPathName();
$result = Dropbox::files()->upload($path, $file);
return redirect('dropboxfiles');
And my files->upload function in my Dropbox Library looks like this:
$client = new Client;
$response = $client->post("https://content.dropboxapi.com/2/files/upload", [
'headers' => [
'Authorization' => 'Bearer '.$this->getAccessToken(),
'Content-Type' => 'application/octet-stream',
'Dropbox-API-Arg' => json_encode([
'path' => $path,
'mode' => 'add',
'autorename' => true,
'mute' => true,
'strict_conflict' => false
])
],
'data-binary' => '#'.$file
]);
The file, as I said, gets uploaded successfully. Correct name, but 0 Bytes. So empty file.
Thank you so much in advance for your help!
Update
With the following code I made it work. My question is though if there is a better "Laravel-Like" Solution instead of using fopen?
$response = $client->post("https://content.dropboxapi.com/2/files/upload", [
'headers' => [
'Authorization' => 'Bearer '.$this->getAccessToken(),
'Dropbox-API-Arg' => json_encode([
'path' => $path,
'mode' => 'add',
'autorename' => true,
'mute' => true,
'strict_conflict' => false
]),
'Content-Type' => 'application/octet-stream',
],
'body' => fopen($file, "r"),
]);
How #Greg mentioned (see cross-linking reference) I was able to solve this issue by using
'body' => fopen($file, "r"),
instead of
'data-binary' => '#'.$file
This is, how Greg mentioned, because data-binary is used in Curl requests. Other HTTP Clients, like Guzzle in my case use different names.

Have a problem with cloudfront signed urls (No account found for the given parameters)

I'm trying to create signed urls from cloudfront with aws-sdk-php
I have created both Distributions WEB and RTMP
and this is the code i used to do that
this is start.php
<?php
require 'vendor/autoload.php';
use Aws\S3\S3Client;
use Aws\CloudFront\CloudFrontClient;
$config = require('config.php');
// S3
$client = new Aws\S3\S3Client([
'version' => 'latest',
'region' => 'us-east-2',
]);
// CloudFront
$cloudfront = CloudFrontClient::factory([
'version' => 'latest',
'region' => 'us-east-2',
]);
and this is config.php
<?php
return [
's3'=>[
'key' => 'XXXXXXXXXXXXXXXXXXXXXXXXXX',
'secret' => 'XXXXXXXXXXXXXXXXXXXXXXXXXX',
'bucket' => 'hdamovies',
'region' => 'us-east-2',
],
'cloudFront' => [
'url' => 'https://d2t7o0s69hxjwd.cloudfront.net',
],
];
and this is index.php
<?php
require 'config/start.php';
$video = 'XXXXXXXXXXX.mp4';
$expiry = new DateTime( '+1 hour' );
$url = $cloudfront->getSignedUrl([
'private_key' => 'pk-XXXXXXXXXXXXXXXXXXXXX.pem',
'key_pair_id' => 'XXXXXXXXXXXXXXXXXXXXX',
'url' => "{$config['cloudFront']['url']}/{$video}",
'expires' => strtotime('+10 minutes'),
]);
echo "Downlod";
When i click on the link i get that error
<Error>
<Code>KMS.UnrecognizedClientException</Code>
<Message>No account found for the given parameters</Message>
<RequestId>0F0A772FE67F0503</RequestId>
<HostId>juuIQZKHb1pbmiVkP7NVaKSODFYmBtj3T9AfDNZuXslhb++LcBsw9GNjpT0FG8MxgeQGqbVo+bo=</HostId></Error>
What is the problem here and how can i solve that?
CloudFront does not support downloading objects that were stored, encrypted, in S3 using KMS Keys, apparently because the CloudFront Origin Access Identity is not an IAM user, so it's not possible to authorize it to have the necessary access to KMS.
https://forums.aws.amazon.com/thread.jspa?threadID=268390
I had this issue and had it resolved after setting up the correctly Identities. However, I had a lot of issues with the error even after setting things up correctly. This was because I was attempting to download a file that was originally uploaded when the bucket was KMS encrypted, then later when I changed it to SSE-S3, it still was throwing a KMS error.
After reuploading the file, it seemed to work without any issues. Hope this helps someone else

Uploading file to S3 using presigned URL in PHP

I am developing a Web Application using PHP. In my application, I need to upload the file to the AWS S3 bucket using Presigned URL. Now, I can read the private file from the S3 bucket using pre-signed like this.
$s3Client = new S3Client([
'version' => 'latest',
'region' => env('AWS_REGION', ''),
'credentials' => [
'key' => env('AWS_IAM_KEY', ''),
'secret' => env('AWS_IAM_SECRET', '')
]
]);
//GetObject
$cmd = $s3Client->getCommand('GetObject', [
'Bucket' => env('AWS_BUCKET',''),
'Key' => 'this-is-uploaded-using-presigned-url.png'
]);
$request = $s3Client->createPresignedRequest($cmd, '+20 minutes');
//This is for reading the image. It is working.
$presignedUrl = (string) $request->getUri();
When I access the $presignedUrl from the browser, I can get the file from the s3. It is working. But now, I am uploading a file to S3. Not reading the file from s3. Normally, I can upload the file to the S3 like this.
$client->putObject(array(
'Bucket' => $bucket,
'Key' => 'data.txt',
'Body' => 'Hello!'
));
The above code is not using the pre-signed URL. But I need to upload the file using a pre-signed URL. How, can I upload the file using a pre-signed URL. For example, what I am thinking is something like this.
$client->putObject(array(
'presigned-url' => 'url'
'Bucket' => $bucket,
'Key' => 'data.txt',
'Body' => 'Hello!'
));
How can I upload?
It seems reasonable that you can create a pre-signed PutPobject command by running:
$cmd = $s3Client->getCommand('PutObject', [
'Bucket' => $bucket,
'Key' => $key
]);
$request = $s3Client->createPresignedRequest($cmd, '+20 minutes')->withMethod('PUT');
Then you might want to perform the PUT call from PHP using:
file_put_contents(
$request->getUri(),
'Hello!',
stream_context_create(['http' => [ 'method' => 'PUT' ]])
);
If you want to create a URL that a browser can submit, then you need to have the browser send the file as a form POST. This AWS documentation explains how to create a pre-signed POST request with the fields that you then need to put into an HTML form and display to the user: https://docs.aws.amazon.com/sdk-for-php/v3/developer-guide/s3-presigned-post.html
Also, this answer might be useful: https://stackoverflow.com/a/59644117/53538

AWS S3 authorization mechanism - Signature 4

I'm currently struggling with getting a S3 download link working. I've been using this code as reference but when I try to open the file, I get the error:
The authorization mechanism you have provided is not supported. Please
use AWS4-HMAC-SHA256.
I tried a few other scripts floating around, but all ended with some other error message.
Is there an easy way to migrate the script I'm using to make it work with Signature v4?
UPDATE: as suggested by hjpotter92, I used the AWS-SDK and came up with this working code:
$client = S3Client::factory([
'version' => 'latest',
'region' => 'eu-central-1',
'signature' => 'v4',
'credentials' => [
'key' => '12345',
'secret' => 'ABCDE'
]
]);
$cmd = $client->getCommand('GetObject', [
'Bucket' => '###name###',
'Key' => $fileName
]);
$request = $client->createPresignedRequest($cmd, '+2 minutes');
$presignedUrl = (string) $request->getUri();
return $presignedUrl;

Categories