PHP - file encoding issue with files downloaded from AWS bucket - php

Using following code in PHP, I am trying to download file from AWS bucket.
I am able to download file successfully, but the downloaded file is not readable. File encoding is set to ANSI.
In AWS bucket the meta data for this file is as follows
Content-Type: text/csv;%20charset=utf-16le
Content-Encoding: gzip
require '/aws/aws-autoloader.php';
use Aws\S3\S3Client;
// Instantiate the S3 client with your AWS credentials
$key = [access_key];
$secret = [secret_key];
$client = S3Client::factory(array(
'credentials' => array(
'key' => $key,
'secret' => $secret
)
));
$bucket = [bucket_name];
$file=[bucket_file_location];
$fileSaveAs = [download_file_location];
// Get an object using the getObject operation & download file
$result = $client->getObject(array(
'Bucket' => $bucket,
'Key' => $file,
'SaveAs' => $fileSaveAs
));
Can anyone explain, what is wrong here ?
Edit 1:
This file downloads nicely when I download it from AWS bucket directly.
Edit 2 :
I just noticed that downloaded CSV File is always of 1KB size.
Downloaded file damage pattern:
`‹½”ÍJÃ#F¿µOÑPi+h]V[Tð§hâFÚ4HÐjI¬Ð—W¿¤
Edit 3 :
All these files are transferred from Google play bucket using gsutil

Files received from AWS bucket are gzip. (Content-Encoding: gzip)
So need to decode gzip compressed string using gzdecode function
The following code solves problem
$content = gzdecode(file_get_contents($fileSaveAs));
file_put_contents($fileSaveAs,$content);

Related

AWS PHP SDK: Limit S3 file upload size in presigned URL

I'm working on a project involving generating S3 URLs that someone else can use to upload files to my S3 bucket. Here's a minimal working example:
<?php
require('aws.phar');
use Aws\S3\S3Client;
$s3client = new S3Client(...); // credentials go here
$id = uniqid(); // generate some kind of key
$command = $s3client->getCommand('PutObject', [
'ACL' => 'private',
'Body' => '',
'Bucket' => 'mybucket',
'Key' => 'tmp/' . $id]);
echo (string) $s3client->createPresignedRequest($command, '+5 minutes')->getURI();
?>
Now, if I put that file at a location accessible by the internet, my web server can be used to fetch new signed upload URLs:
$ curl http://my.domain.com/some/page.php
https://s3.amazonaws.com/mybucket/tmp/someID?x-amz-acl=private&lots-of-aws-params...
$ curl -X PUT -d "#someFile" https://s3.amazonaws.com/mybucket/tmp/someID?x-amz-acl=private&lots-of-aws-params...
$
This successfully uploads a local file to my bucket, so I can play with it in S3.
Let's suppose that I'm not too worried about people generating many URLs and uploading many files to my bucket in a short period of time, but I would like to limit the size of uploaded files. Many resources suggest attaching a policy to the signed URL:
<?php
require('aws.phar');
use Aws\S3\S3Client;
$s3client = new S3Client(...); // credentials go here
$id = uniqid(); // generate some kind of key
$policy = [
'conditions' => [
['acl' => 'private'],
['bucket' => 'mybucket'],
['content-length-range', 0, 8*1024], // 8 KiB
['starts-with', '$key', 'tmp/']
], 'expiration' =>
(new DateTime())->modify('+5 minutes')->format(DateTime::ATOM)];
$command = $s3client->getCommand('PutObject', [
'ACL' => 'private',
'Body' => '',
'Bucket' => 'mybucket',
'Key' => 'tmp/' . $id,
'Policy' => $policy]);
echo (string) $s3client->createPresignedRequest($command, '+5 minutes')->getURI();
?>
This version generates URLS (without any indication of errors) that can be used in the same way. I'm not sure if I need some of those conditions in the policy (acl, bucket, starts-with), but I don't think that including them would break the policy.
In theory, attempting to use this signed URL to upload a file larger than 8 KiB should cause S3 to abort the upload. However, testing this with a larger file shows that curl still happily uploads the file:
$ ls -lh file.txt
-rw-rw-r-- 1 millinon millinon 210K Jan 2 00:41 file.txt
$ curl http://my.domain.com/some/page.php
https://s3.amazonaws.com/mybucket/tmp/someOtherID?x-amz-acl=private&lots-of-aws-params...
$ curl -X PUT -d "#file.txt" https://s3.amazonaws.com/mybucket/tmp/someOtherID?x-amz-acl=private&lots-of-aws-params...
$
Checking the bucket shows that, indeed, the large file was uploaded, and the file's size is larger than the policy supposedly indicates.
Since various pages show different ways of attaching the policy, I have also tried the following versions:
'Policy' => json_encode($policy)
'Policy' => base64_encode(json_encode($policy))
However, URLs generated with any of these versions allow files larger than the specified size to be uploaded.
Am I attaching the policy incorrectly, or is there a fundamental limitation to restricting uploads to S3 in this manner?
For my web server, I'm using HHVM 3.11.1 with version 3.14.1 of the AWS SDK for PHP.
An S3 upload policy cannot be used with pre-signed URLs.
A policy document can be used with browser uploads to S3 using HTML POST Forms.
Pre-signed URLs and HTML POST forms are two different methods of uploading to S3. The former is arguably simpler, but less flexible, than the latter.
UPDATE
If you must upload the files without the use of a browser, the HTML POST Form's request can be reproduced using PHP's curl functions, a library such as Guzzle, or using the command line as follows:
curl 'https://s3-bucket.s3.amazonaws.com/' \
-F 'key=uploads/${filename}' \
-F 'AWSAccessKeyId=YOUR_AWS_ACCESS_KEY' \
-F 'acl=private' \
-F 'success_action_redirect=http://localhost/' \
-F 'policy=YOUR_POLICY_DOCUMENT_BASE64_ENCODED' \
-F 'signature=YOUR_CALCULATED_SIGNATURE' \
-F 'Content-Type=image/jpeg' \
-F 'file=#file.jpg'
can you try to add the specific policy to your bucket
$s3client->putBucketPolicy(array(
'Bucket' => $bucket,
'Policy' => json_encode(array(
'conditions' => array(
array(
'content-length-range', 0, 8*1024,
'starts-with', '$key', 'tmp/'
...
and after this, just use the normal putObject command
$command = $s3client->getCommand('PutObject', [
'ACL' => 'private',
'Body' => '',
'Bucket' => 'mybucket',
'Key' => 'tmp/' . $id]);

Downloading files from google cloud storage and search good documantation by apies for php

I began to work with Google Cloud Storage
I am currently using the library Google API PHP Client.
The documentation is here however I couldn't find any php documentation related to the APIs of the library.
And I've problem with downloading files from storage. I know how upload, but I don't know how download.
$res = $storage->objects->insert(
"bucket",
$obj,
['name' => $file_name, 'data' => file_get_contents('path'), 'uploadType' => 'multipart', 'mimeType'=>'text/xml']
);
I can get data of file, but I can't download.
$res = $storage->objects->get(
"bucket",
$file_name,
[]
);
Thanks
$storage->objects->get
This api returns mediaLink parameter and by this link we can download from storage

How to get file size from Amazon AWS S3 Version 2?

I am forced to use Version 2 of AWS S3, because i cannot update PHP to 5.5 on this server in order to use Version 3.
I made this PHP script to download files from AWS, which works good:
//http://docs.aws.amazon.com/aws-sdk-php/v2/api/class-Aws.S3.S3Client.html#_createPresignedUrl
// Get a command object from the client and pass in any options
// available in the GetObject command (e.g. ResponseContentDisposition)
$command = $s3Client->getCommand('GetObject', array(
'Bucket' => $bucket,
'Key' => $objectKey,
'ResponseContentDisposition' => 'attachment; filename="' . $originFilename . '"'
));
// Create a signed URL from the command object that will last for
// 10 minutes from the current time
$signedUrl = $command->createPresignedUrl('+1000 minutes');
$file = file_get_contents($signedUrl);
The problem is that i want to be sure that the file_get_contents() downloads the entire file and to detect and fix any error (like server going offline during a download, etc...), so i thought the following flow:
I ask AWS the file size
I download the file
I check the size. If it's not equal i re-download the file
So, how to get file size from AWS? I found this, but it doesn't work for my version.
You can use the HEAD Object REST API to determine the size of the object stored on S3.
HEAD Object will return the meta-data associated with the stored S3 Object, including the size on disk of the object, within the Content-Length header.
http://docs.aws.amazon.com/aws-sdk-php/v2/api/class-Aws.S3.S3Client.html#_headObject

How to upload a folder into AWS

Iam using AWS to upload images, css and some zip files for my site and they are fine to upload them.But now I want like I first upload a zip on localhost and I will extract them into one folder and I want to upload that entire folder into aws.Can anyone help me to do it.Thanks in advance.
Iam using function to upload files like
require_once 'aws-sdk-for-php/sdk.class.php';
$s3 = new AmazonS3();
$response = $s3->create_object($bucket, $filename, array(
'fileUpload' => $filepath,
'acl' => AmazonS3::ACL_PUBLIC,
'storage' => AmazonS3::STORAGE_REDUCED,
'headers' => array(
'Cache-Control' => 'max-age=2592000'
);
It is working fine for single images.But I dont know how to do it for entore folder.
There is no API call for Amazon S3 that can upload an entire folder. You can loop through your list of local files and then upload each individually to S3. If you're capable, doing it in parallel can greatly speed the upload, too.
You could also cheat by calling out to the AWS Command Line Interface (CLI). The CLI can upload/download a recursive list of files and can also do multi-part upload for large files. There is also an aws s3 sync command that can intelligently upload only new/modified files.
Using PHP you can upload entire directory:
$client->uploadDirectory(
SOURCE_FOLDER,
YOUR_BUCKET_NAME,
DESTINATION,
array(
'concurrency' => 5,
'debug' => TRUE,
'force' => FALSE,
'params' => array(
'ServerSideEncryption' => 'AES256',
),
)
);

Create a zip file using PHP class ZipArchive without writing the file to disk?

I would like to create a zip file in memory using a ZipArchive (or a native PHP class) and read the content of the file back to the client. Is this possible? If so, how?
The files that I want to zip in this application are a maximum of 15 MB total. I think we should be in good shape memory-wise.
Try ZipStream (link to GitHub repo, also supports install via Composer).
From original author's website (now dead):
ZipStream is a library for dynamically streaming dynamic zip files
from PHP without writing to the disk at all on the server.
Thanks to Frosty Z for the great library ZipStream-PHP. We have a use case to upload some large zip files in quantity and size to S3. The official documentation does not mention how to upload to S3.
So we've had an idea to stream the zip created by ZipStream output directly to S3 without creating the zip file on the server.
Here is a working sample code that we came up with:
<?php
# Autoload the dependencies
require 'vendor/autoload.php';
use Aws\S3\S3Client;
use ZipStream\Option\Archive as ArchiveOptions;
//s3client service
$s3Client = new S3Client([
'region' => 'ap-southeast-2',
'version' => 'latest',
'credentials' => [
'key' => '<AWS_KEY>',
'secret' => '<AWS_SECRET_KEY>',
]
]);
$s3Client->registerStreamWrapper(); //required
$opt = new ArchiveOptions();
$opt->setContentType('application/octet-stream');
$opt->setEnableZip64(false); //optional - for MacOs to open archives
$bucket = 'your_bucket_path';
$zipName = 'target.zip';
$zip = new ZipStream\ZipStream($zipName, $opt);
$path = "s3://{$bucket}/{$zipName}";
$s3Stream = fopen($path, 'w');
$zip->opt->setOutputStream($s3Stream); // set ZipStream's output stream to the open S3 stream
$filePath1 = './local_files/document1.zip';
$filePath2 = './local_files/document2.zip';
$filePath3 = './local_files/document3.zip';
$zip->addFileFromPath(basename($filePath1), $filePath1);
$zip->addFileFromPath(basename($filePath2), $filePath2);
$zip->addFileFromPath(basename($filePath3), $filePath3);
$zip->finish(); // sends the stream to S3
?>
Take a look at the following library, it allows creating zip files and returning them as a stream: PHPClasses.org.
There is another thread talking about this:
Manipulate an Archive in memory with PHP (without creating a temporary file on disk)
nettle has suggested to use zip.lib.php from phpmyadmin. I think this is a rather solid solution.
FYI zip.lib.php does not exist anymore, it has been replace by ZipFile.php in the same libraries/ folder.

Categories