S3: putObject() a stream receiving via POST - php

Question

I'm receiving files (up to 4 GB): the file-content is streamed to me in the body of a POST request.
I want to upload this stream directly to a s3 bucket, without saving it locally first.
Already tried different approaches which failed for different reasons.
My current approach:
use GuzzleHttp\Psr7\Stream;
use Aws\S3\S3Client;
$s3 = new \Aws\S3\S3Client([
'version' => 'latest',
'region' => 'eu-west-1',
'credentials' => [
'key' => 'abc',
'secret' => '123'
]
]);
$stream = new \GuzzleHttp\Psr7\Stream(fopen('php://input', 'r'));
$result = $s3->putObject(array(
'Bucket' => $bucket,
'Key' => $keyname,
'ContentLength' => (int)$_SERVER['CONTENT_LENGTH'],
'Body' => $stream->getContents(),
'ACL' => 'private',
'StorageClass' => 'STANDARD_IA',
));
The following error occurs while trying to stream a 80 MB file:
PHP message: PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 78847383 bytes) in /var/www/slimapi/vendor/slim/slim/Slim/Http/Stream.php on line 403
Line 403 of Stream.php is:
if (!$this->isReadable() || ($contents = stream_get_contents($this->stream)) === false) {
So the error is probably caused by triying to load the whole content of the stream into a string, which exceeds the memory limit.
(It's irritating why the error is occuring within Slim/Stream, as I'm trying to use guzzle\Stream.)
So my questions is:
How can I stream the incoming POST data directly to a s3 bucket without buffering issues leading to memory problems?
I already tried:
$stream = Psr7\stream_for(fopen('php://input', 'r'));
$stream
= fopen('php://input', 'r');
within putObject(): 'Body' => Stream::factory(fopen('php://input', 'r')),

Answer 1

I know this is old topic but it's not marked as solved, so...
PHP SDK does support stream source as you can see in the SDK specs (https://docs.aws.amazon.com/aws-sdk-php/v3/api/api-s3-2006-03-01.html#putobject) - see Parameter syntax:
$result = $client->putObject([
// ...
'Body' => <string || resource || Psr\Http\Message\StreamInterface>,
// ...
]);
It means your code is almost OK, the only thing is that you should pass $stream instead of $stream->getContents():
$stream = new \GuzzleHttp\Psr7\Stream(fopen('php://input', 'r'));
$result = $s3->putObject(array(
'Bucket' => $bucket,
'Key' => $keyname,
'ContentLength' => (int)$_SERVER['CONTENT_LENGTH'],
'Body' => $stream,
'ACL' => 'private',
'StorageClass' => 'STANDARD_IA',
));
As simple as that.

Answer 2

That PHP SDK call does not support directly reading a stream. So what appears to be happening to me is that PHP is exhausting memory as it loads the entire object from that stream into a variable, before it actually calls the SDK to PUT that string of data to the object.
You'll want to consider using the S3 Stream Wrapper.
This example seems most appropriate, but you'll need to pass the data between both streams. While the S3 Stream Wrapper appears to support creating a stream say from a local file, I didn't see a direct example of passing an existing stream to it.
In this example, we read 4096 bytes from the source if available (or less if 4096 isn't available, and if the returned value is non-empty then we write that to the S3 object. We continue this until the source reached EOF (in this example source must support and EOF).
$client = new Aws\S3\S3Client([/** options **/]);
// Register the stream wrapper from an S3Client object
$client->registerStreamWrapper();
$stream = fopen('s3://bucket/key', 'w');
while (!$stream_source->stream_eof()) {
$string = $stream_source->stream_read (4096)
if (!empty($string)) {
fwrite($stream, $string);
}
}
fclose($stream);