Access Google Cloud Storage from Local Env PHP - php

I am just trying to do a very simple file_get_contents('gs://[bucket][file]');
I am getting this error when I run locally.
"\google\appengine\ext\cloud_storage_streams\CloudStorageStreamWrapper::stream_open" call failed.
However, when I run it on my production env it works fine.
I know I am missing something with the setup, and I am just not sure what.
I have set the project gcloud config set project PROJECT_ID, set the account gcloud config set account ACCOUNT, and run gcloud auth login. Any ideas on what I am missing in my setup here?
I ran composer install , composer require google/cloud, I don't get any errors when including use Google\Cloud\Storage\StorageClient; I created the service-account-keyfile as an owner of the bucket. Set owners to full permissions. Ran this code...
require __DIR__ . '/vendor/autoload.php';
use Google\Cloud\Storage\StorageClient;
$storage = new StorageClient([
'keyFilePath' => __DIR__ .'/../keys/appdocs-com.json',
]);
echo file_get_contents('gs://[myBucket][myFile]');
and I am still getting ...streamWrapper::stream_open" call failed
I even tried the service builder approach.
require __DIR__ . '/vendor/autoload.php';
use Google\Cloud\ServiceBuilder;
use Google\Cloud\Storage\StorageClient;
$gcloud = new ServiceBuilder([
'keyFilePath' => __DIR__ .'/../keys/appdocs-com.json',
'projectId' => 'appdocs-com',
]);
$storage = $gcloud->storage();
$bucket = $storage->bucket('test-appdocs-sendgrid-inbound');
$object = $bucket->object('4/envelope.json');
$stream = $object->downloadAsStream();
echo $stream->getContents();
this returns
Fatal error: Uncaught exception 'Google\Cloud\Core\Exception\ServiceException' with message 'No system CA bundle could be found in any of the the common system locations. PHP versions earlier than 5.6 are not properly configured to use the system's CA bundle by default. In order to verify peer certificates, you will need to supply the path on disk to a certificate bundle to the 'verify' request option: http://docs.guzzlephp.org/en/latest/clients.html#verify. If you do not need a specific certificate bundle, then Mozilla provides a commonly used CA bundle which can be downloaded here (provided by the maintainer of cURL): https://raw.githubusercontent.com/bagder/ca-bundle/master/ca-bundle.crt. Once you have a CA bundle available on disk, you can set the 'openssl.cafile' PHP ini setting to point to the path to the file, allowing you to omit the 'verify' request option. See http://curl.haxx.se/docs/sslcerts.html for more information.' in /Users/thom/Engine/appdocs-com/vendor/google/cloud-core/RequestWrapper.php:241 Stack trac in /Users/thom/Engine/appdocs-com/vendor/google/cloud-core/RequestWrapper.php on line 241
I have my runtime at php55. Could that be preventing me from using StorageClient or serviceBuilder

When outside the appengine environment, you will need to manually register the cloud storage stream wrapper to access files using the gs:// protocol:
use Google\Cloud\Storage\StorageClient;
$storage = new StorageClient([
'keyFilePath' => '/path/to/service-account-keyfile.json'
]);
$storage->registerStreamWrapper();
This example assumes you have Google Cloud PHP installed.
More information is available in the Google Cloud PHP docs.
To resolve the guzzle ssh error, refer to this answer. NOTE that setting verify to false is not safe, and should only be used to test, never in production.

As #jdp mentioned in his answer, the error I was getting was related to my installation of curl.
So I looked at Google's php_ini docs to see if I could set the openssl.cafile. https://cloud.google.com/appengine/docs/standard/php/config/php_ini
If you want to use Google Storage Client you need to change from standard to flexible env.
https://cloud.google.com/appengine/docs/flexible/php/quickstart
Then read through cloud storage docs to see how to download dependencies for the API usage: https://cloud.google.com/storage/docs/reference/libraries
Once you have set up your flex env and gathered dependencies, you can then follow these directions to set up the connection.
Set up a service account key In Google Cloud Console:
Select "your-project"->IAM & ADMIN-> Service Accounts.
Click the "Create Service Account" button.
Assign role (in my case I assigned the role "owner" since this service account was specific to my local development), mark the checkbox "Furnish New Private Key".
This just downloaded your "service-account-keyfile.json"
Then in the code initiate Cloud Storage like so:
use Google\Cloud\Storage\StorageClient;
$storage = new StorageClient([
'keyFilePath' => '/path/to/service-account-keyfile.json'
]);
$storage->registerStreamWrapper();
You can now retrieve bucket data like you would from the production env.
echo file_get_contents('gs://[Bucket][File]');
Make sure you have set the bucket permissions to allow the previously selected role to access the bucket.
However, I would still love to find a way to set up a local environment with with php standard that can interact with gs://[bucket]/[file].

Related

Error retrieving credentials from the instance profile metadata server with credentials defined in .aws folder

I have a web page on an AWS instance located at /var/www/html/
Until now this website used the keys AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in the code itself to access files hosted on S3.
For security reasons,
I have removed these keys from my code and used the aws configure command to ssh to store them on the server as recommended by AWS.
I see that in my directory ~.aws/ folder has been created with 2 files: credentials and config.
Both seem to be correct but in the web logs now I get the following error when trying to access files from S3:
PHP Fatal error: Uncaught Aws\Exception\CredentialsException: Error retrieving credentials from the instance profile metadata server. (Client error: 'GET http://169.254.169.254/latest/meta-data/iam/security-credentials /' resulted in a '404 Not Found resulted in a '404 Not Found' response:
<! DOCTYPE html PUBLIC "- // W3C // DTD XHTML 1.0 Transitional // EN"
"http: // www. (truncated ...)
) in /var/www/html/aws/Aws/Credentials/InstanceProfileProvider.php:88
I don't know what that URL is but I can't access it through the browser.
I have tried it with environment variables:
export AWS_ACCESS_KEY_ID = xxxxx...
I have copied the .aws folder to / var / www
I have given more permissions to .aws, I have changed the owner and group from root to ec2-user ...
How should I do the configuration so that my code correctly calls S3 and gets the files?
Call example that fails:
$s3 = new Aws\S3\S3Client ([
'version' => 'latest',
'region' => 'eu-central-1'
]);
if ($s3) {
$result = $ s3-> getObject (array (
'Bucket' => AWS_S3_BUCKET,
'Key' => $s3_key,
'Range' => 'bytes ='. $Startpos .'- '. ($Startpos + 7)
));
You probably need to move the .aws folder to the home folder of the service (apache) and not your home folder. The aws sdk can't find it and you receive this error. However, it isn't a good idea to use aws configure inside an EC2 instance.
The http://169.254.169.254/latest/meta-data/ is the meta-data url only available from inside an EC2 instance. For services running in EC2 (or other AWS compute service) you SHOULD NOT use AWS credentials to access services. Instead, you should create an IAM role and add assign it to the instance. From the console, you can do that with the Actions button:
Only assign required permissions to the role (S3 read/write).
Your code ($s3 = new Aws\S3\S3Client) will try to load the default credentials. It will first try to call the meta-data service and get temporary credentials that correspond to the IAM role permissions.

Getting started with Google Storage

I am having really difficult time trying to connect Google Storage. All I need is to be able to upload PDF file to a bucket that I've created on Google Storage Console. The documentation seems to be all over the place and lacking simple examples of PHP code. So here's what I've done so far:
Installed cloud storage
$ composer require google/cloud-storage
Added billing as per Google's requirement. Enabled Cloud Storage API. Created project and added a bucket.
Attempted to use the following example:
require '../vendor/autoload.php';
define("PROJECT_ID", "my-project");
define("BUCKET_NAME", "my-bucket");
$client = new Google_Client();
$client->setApplicationName("API_Cloud_Storage");
$client->useApplicationDefaultCredentials();
$client->setScopes(["https://www.googleapis.com/auth/cloud-platform"]);
$service = new Google_Service_Storage($client);
$request = $service->buckets->listBuckets(PROJECT_ID);
foreach ($request["items"] as $bucket)
printf("%s\n", $bucket->getName());
Keep on getting error on
Fatal error: Uncaught Error: Class 'Google_Client' not found in /home/domain/public_html/test.php:11 ...
I know the vendor/autoload.php file is loading because I have no issues with AWS in a different script.
I didn't even pass the first line. What am I missing?
You have the documentation page for Cloud Storage Client Libraries with a PHP sample and detailed instructions. There's also a link to the GitHub repo for that quickstart, in addition to the one shared by ceejayoz in the comments.

Class not found in Google Cloud app

I am getting the following error:
Fatal error: Class 'Google\Cloud\Datastore\DatastoreClient' not found in /base/data/home/apps/e~brookes-room-usage/1.3998538263957262‌​38/record-usage.php on line 84
so presumably I have failed to point my app correctly to that class.
I have enabled the Datastore API on the cloud console.
I have tried to find documentation relating to how to set up a PHP connection to the Datastore API.
I tried to create a settings.yml file using the instructions here but I didn't know what my client ID or my client secret were.
# Google credentials and configuration
google_client_id: YOUR_CLIENT_ID
google_client_secret: YOUR_CLIENT_SECRET
google_project_id: YOUR_PROJECT_ID
# options are "cloudsql", "mongodb", or "datastore"
bookshelf_backend: datastore
(source on Github)
What do I need to get my PHP app on Google Cloud to recognise Google\Cloud\Datastore\DatastoreClient and connect to the Datastore API so that I can retrieve and post data?
I think you forgot to include google/cloud-datastore library. Install via composer as below.
$ composer require google/cloud-datastore
and you can use as below.
<?php
require 'vendor/autoload.php';
use Google\Cloud\Datastore\DatastoreClient;
$datastore = new DatastoreClient([
'projectId' => 'my_project'
]);
// Create an entity
$bob = $datastore->entity('Person');
$bob['firstName'] = 'Bob';
$bob['email'] = 'bob#example.com';
$datastore->insert($bob);
// Update the entity
$bob['email'] = 'bobV2#example.com';
$datastore->update($bob);
// If you know the ID of the entity, you can look it up
$key = $datastore->key('Person', '12345328897844');
$entity = $datastore->lookup($key);
More details: https://github.com/GoogleCloudPlatform/google-cloud-php#google-cloud-datastore-ga
I am already using this and working fine for me.
So it turns out that you can only run PHP with Datastore on a Google Cloud app on your local device, not on Google Appengine, unless you use a third-party library.
Which is why I am now writing my app in Python instead.
(Update: this may no longer be the case. This answer was written in 2017.)

Using local google Datastore with dev_appserver.pyp

At the moment I am able to write to the datastore once I deploy my code, but I can't write to the datastore emulator with code running locally since it throws a ca-bundle error. The local datastore is visible at localhost:8000
use google\appengine\api\users\User;
use google\appengine\api\users\UserService;
use google\appengine\api\app_identity\AppIdentityService;
echo AppIdentityService::getApplicationId()."<br>";
echo AppIdentityService::getDefaultVersionHostname()."<br>";
# Includes the autoloader for libraries installed with composer
require __DIR__ . '/vendor/autoload.php';
use Google\Cloud\ServiceBuilder;
$cloud = new ServiceBuilder([
'projectId' => AppIdentityService::getApplicationId(),
'keyFilePath'=>'review-9504000716d8.json'
]);
$datastore = $cloud->datastore();
# The kind for the new entity
$kind = 'Task';
# The name/ID for the new entity
$name = 'sampletask1';
# The Cloud Datastore key for the new entity
$taskKey = $datastore->key($kind, $name);
# Prepares the new entity
$task = $datastore->entity($taskKey, ['description' => 'Buy milk']);
# Saves the entity
$datastore->upsert($task);
This code runs without any issues when deployed. But locally throws:
Fatal error: Uncaught exception 'Google\Cloud\Exception\ServiceException' with message 'No system CA bundle could be found in any of the the common system locations. PHP versions earlier than 5.6 are not properly configured to use the system's CA bundle by default. In order to verify peer certificates, you will need to supply the path on disk to a certificate bundle to the 'verify' request option: http://docs.guzzlephp.org/en/latest/clients.html#verify. If you do not need a specific certificate bundle, then Mozilla provides a commonly used CA bundle which can be downloaded here (provided by the maintainer of cURL): https://raw.githubusercontent.com/bagder/ca-bundle/master/ca-bundle.crt. Once you have a CA bundle available on disk, you can set the 'openssl.cafile' PHP ini setting to point to the path to the file, allowing you to omit the 'verify' request option. See http://curl.haxx.se/docs/sslcerts.html for more information.' in D:\Google\php\appengine-php-guestbook-phase0-helloworld\appengine-php-guestbook-phase0-hellowo in D:\Google\php\appengine-php-guestbook-phase0-helloworld\appengine-php-guestbook-phase0-helloworld\vendor\google\cloud\src\RequestWrapper.php on line 219
I didn't manage to make the local server even consider the php.ini file nor did I manage to upgrade the bundled php55 to at least php56.
Thus I actually have 2 questions:
how to properly connect from the local instance (dev_appserver.py) on windows to Google's remote datastore?
how to properly connect from the local instant to the local emulated datastore so I can view the data on localhost:8000?
The APIs are using CA certificate files for authentication more specifically they are using curl.cainfo.
Now you server might already have this file configured in php.ini. You can check in server file. Remember there could be different ini files for different environments like apache, cli.
Now you can either copy that file or Create your own authority file
Option 1:
Set absolute path in php.ini
Option 2:
Use ini_set to set this config.
Option 3:
Try with some other mode of authentication, i am sure google will have that.
Option 4:
As given in your question itself.
If you do not need a specific certificate bundle, then Mozilla provides a commonly used CA bundle which can be downloaded here
https://raw.githubusercontent.com/bagder/ca-bundle/master/ca-bundle.crt. Once you have a CA bundle available on disk, you can set the 'openssl.cafile' PHP ini setting to point to the path to the file, allowing you to omit the 'verify' request option

Google Analytics service API (ERR_CONNECTION_RESET)

IMPORTANT EDIT: Found out that issue is caused by my web server (I don't know how to fix that) - tags changed. I use latest XAMPP.
I am trying to set up server to server connection with Google Analytics API with OAuth following this guide:
Service Applications and Google Analytics API V3: Server-to-server OAuth2 authentication?
I have following code:
require_once 'google-api-php-client/src/Google_Client.php';
require_once 'google-api-php-client/src/contrib/Google_AnalyticsService.php';
// create client object and set app name
$client = new Google_Client();
$client -> setApplicationName("------------"); // name of your app
// set assertion credentials
$client->setAssertionCredentials(
new Google_AssertionCredentials(
"------------#developer.gserviceaccount.com", // email you added to GA
array('https://www.googleapis.com/auth/analytics.readonly'),
file_get_contents("-:/-----------/-----/------/----/---------------------------------------------------.p12") // keyfile you downloaded
));
// other settings
$client->setClientId("------------.apps.googleusercontent.com"); // from API console
$client->setAccessType('offline_access'); // this may be unnecessary?
// create service and get data
$service = new Google_AnalyticsService($client);
$ids = "ga:--------";
$startDate = "2013-05-01";
$endDate = "2013-05-26";
$metrics = "ga:visitors";
var_dump($service->data_ga->get($ids, $startDate, $endDate, $metrics));
Which causes HTTP error #101 (ERR_CONNECTION_RESET). My issue is similar to: Service Applications and Google Analytics API V3: Error 101 (net::ERR_CONNECTION_RESET) but I have no other calls to Google_AnalyticsService().
I have PHP version 5.4.7 (XAMPP version) with enabled OpenSSL (enabled via uncommeting line in php.ini).
Thank you in advance for any answers. How to deal with ones like this (producing some HTTP error and stopping? Are there logs that produce any useful informations?)?
EDIT: Copying code from post: "Not sufficient permissions" google analytics API service account to me causes same error (no other messages printed).
I found way to deal with this issue. It's caused by Apache and PHP together - they use different version of OpenSSL.
First workaround (one that does not work for me as I write it 05-28-3013) on XAMPP 1.8.1 (release date 30.9.2012) is to replace files ssleay32.dll and libeay32.dll within Apache /bin/ folder with these from php folder. It does not work for me because mod_ssl can't work with PHP's DLLs.
Second workaround is to install other package. Personally, I switched to ZendServer which has synced versions of OpenSSL and Apache.
Of course you can always build your own Apache and PHP making sure you have OpenSSL and mod_ssl cooperating and in latest version.

Categories