Authenticate and use Google's BigQuery in my php code - php

I'm setting up my client app using my BigQuery Data, I need to build it with php (which I'm not expert).
I want to authenticate and be able to build queries using php in my server.
I'm having trouble authenticating, as I see different solutions which seem outdated or incomplete.
I've got the JSON key and the project I'd but don't know the proper wait to authenticate to BigQuery.
Can someone tell me how to properly do that? If I need another library (like google cloud auth?)

Here is the code snippet:
public function getServiceBuilder() {
putenv('GOOGLE_APPLICATION_CREDENTIALS=path_to_service_account.json');
$builder = new ServiceBuilder(array(
'projectId' => PROJECT_ID
));
return $builder;
}
then you can use like this
$builder = $this->getServiceBuilder();
$bigQuery = $builder->bigQuery();
$job = $bigQuery->runQueryAsJob('SELECT .......');
$backoff = new ExponentialBackoff(8);
$backoff->execute(function () use ($job) {
$job->reload();
$this->e('reloading job');
if (!$job->isComplete()) {
throw new \Exception();
}
});
if (!$job->isComplete()) {
$this->e('Job failed to complete within the allotted time.');
return false;
}
$queryResults = $job->queryResults();
if ($queryResults->isComplete()) {
$i = 0;
$rows = $queryResults->rows();
foreach ($rows as $row) {
(edited)
you may need to add:
use Google\Cloud\BigQuery\BigQueryClient;
use Google\Cloud\ServiceBuilder;
use Google\Cloud\ExponentialBackoff;
and composer.json (this is just an example it may differ the actual version)
{
"require": {
"google/cloud": "^0.99.0"
}
}

Related

Elasticsearch doesn't index something in a php application

I've a problem using elasticsearch in a php application. The application is built with zend and uses a .env to hold the following configuration:
ELASTICSEARCH_MAX_DOCUMENTS=250
ELASTICSEARCH_MAX_BULK_SIZE=3M
ELASTICSEARCH_HOST=my-elasticsearch.intern.rz
ELASTICSEARCH_PORT=80
ELASTICSEARCH_USER=user
ELASTICSEARCH_PASSWORD=pw
The call to index the new files is part of a import service class and looks like this: 
public function flush(FlushInterface $flushInterface = null) {
$bulk = $this->getSearchDocumentBulk();
if (!empty($bulk->getActions())) {
$response = $bulk->send();
$this->resetSearchDocumentBulk();
if (0 === $response->count()) {
$data = $response->getData();
throw new BulkException(isset($data['message']) ? strip_tags($data['message']) : '');
}
}
$this->documentCache = [];
if ($flushInterface instanceof FlushInterface) {
$flushInterface->flush();
}
return $this;
}
protected function getSearchDocumentBulk() {
if (!($this->searchDocumentBulk instanceof Bulk)) {
$this->searchDocumentBulk = new Bulk($this->getSearchClient()->getClient());
$this->searchDocumentBulk->setIndex(self::INDEX);
}
return $this->searchDocumentBulk;
}
I know it's only a short snippet but it's quite difficult to pull out the relevant code. So please let me know if I have to post some more methods. 
The application is started by a symfony command and I'm able to curl to elasticsearch (version 5.1) without any errors.
My problem is that no document is indexed. If I check elasticsearch-head I see that there was no data transfer anymore. 
But there's also no error, no exception or something like that. The process is completed with 100% (100,000 of 100,000 files imported). So I've no idea what happens or how to find out the bug.

voryx thruway multiple publish

I need to publish messages from php script, I can publish a single message fine. But now I need to publish different messages in loop, can't find proper way how to do it, here is what I tried:
$counter = 0;
$closure = function (\Thruway\ClientSession $session) use ($connection, &$counter) {
//$counter will be always 5
$session->publish('com.example.hello', ['Hello, world from PHP!!! '.$counter], [], ["acknowledge" => true])->then(
function () use ($connection) {
$connection->close(); //You must close the connection or this will hang
echo "Publish Acknowledged!\n";
},
function ($error) {
// publish failed
echo "Publish Error {$error}\n";
}
);
};
while($counter<5){
$connection->on('open', $closure);
$counter++;
}
$connection->open();
Here I want to publish $counter value to subscribers but the value is always 5, 1.Is there a way that I open connection before loop and then in loop I publish messages
2.How to access to $session->publish() from loop ?
Thanks!
There are a couple different ways to accomplish this. Most simply:
$client = new \Thruway\Peer\Client('realm1');
$client->setAttemptRetry(false);
$client->addTransportProvider(new \Thruway\Transport\PawlTransportProvider('ws://127.0.0.1:9090'));
$client->on('open', function (\Thruway\ClientSession $clientSession) {
for ($i = 0; $i < 5; $i++) {
$clientSession->publish('com.example.hello', ['Hello #' . $i]);
}
$clientSession->close();
});
$client->start();
There is nothing wrong with making many short connections to the router. If you are running in a daemon process though, it would probably make more sense to setup something that just uses the same client connection and then use the react loop to manage the loop instead of while(1):
$loop = \React\EventLoop\Factory::create();
$client = new \Thruway\Peer\Client('realm1', $loop);
$client->addTransportProvider(new \Thruway\Transport\PawlTransportProvider('ws://127.0.0.1:9090'));
$loop->addPeriodicTimer(0.5, function () use ($client) {
// The other stuff you want to do every half second goes here
$session = $client->getSession();
if ($session && ($session->getState() == \Thruway\ClientSession::STATE_UP)) {
$session->publish('com.example.hello', ['Hello again']);
}
});
$client->start();
Notice that the $loop is now being passed into the client constructor and also that I got rid of the line disabling automatic reconnect (so if there are network issues, your script will reconnect).

zend gdata and google spreadsheet not connecting

ive been using Zend Gdata for a while now, and today im getting an error of
Notice: Undefined offset: ClientLogin.php on line 150
via php, this has been working for a while now, and today without changing anything it stopped working, im guessing some deprecated service on behalf of google with the zend gdata maybe the Zend_Gdata_ClientLogin::getHttpClient( ) method or something, can any one confirm or help me with this issue. the code im using to connect is as follows:
require_once('Zend/Loader.php');
Zend_Loader::loadClass('Zend_Gdata');
Zend_Loader::loadClass('Zend_Gdata_ClientLogin');
Zend_Loader::loadClass('Zend_Gdata_Docs');
Zend_Loader::loadClass('Zend_Gdata_Spreadsheets');
require_once 'Zend/Gdata.php';
require_once 'Zend/Gdata/AuthSub.php';
require_once 'Zend/Gdata/Spreadsheets.php';
require_once 'Zend/Gdata/Spreadsheets/DocumentQuery.php';
require_once 'Zend/Gdata/Spreadsheets/ListQuery.php';
require_once 'Zend/Loader.php';
$sourceUser = "myemail";
$sourcePass = "mysuperawesomepassword";
$service = Zend_Gdata_Spreadsheets::AUTH_SERVICE_NAME;
$sourceClient = Zend_Gdata_ClientLogin::getHttpClient($sourceUser, $sourcePass, $service);
$connection = new Zend_Gdata_Spreadsheets($sourceClient);
i am using the zend gdata with the google spreadsheets
also the error points specifically to this line
$sourceClient = Zend_Gdata_ClientLogin::getHttpClient($sourceUser, $sourcePass, $service);
as i said, i was using this for a while now, and nothing has changed on my end
I uses ClientLogin for a server to server application.
needed to switch to oAuth2 in a hurry today. I merged some examples I found to get the authorisation token using a 'Service account'
function get_token() {
$client_email = '0-1.apps.googleusercontent.com';
$client_email = '0-1#developer.gserviceaccount.com';
$private_key = file_get_contents('abc.p12');
$scopes = array('https://spreadsheets.google.com/feeds');
$credentials = new Google_Auth_AssertionCredentials(
$client_email,
$scopes,
$private_key,
'notasecret', // Default P12 password
'http://oauth.net/grant_type/jwt/1.0/bearer' // Default grant type
);
$client = new Google_Client();
$client->setAssertionCredentials($credentials);
if ($client->getAuth()->isAccessTokenExpired()) {
$client->getAuth()->refreshTokenWithAssertion();
}
$tokenData = json_decode($client->getAccessToken());
return $tokenData->access_token;
}
I needed to use the developer email not the app email, same email must be added as a user to the spreadsheet
the generated token can then be used with php-google-spreadsheet-client
$accessToken = get_token();
$serviceRequest = new DefaultServiceRequest($accessToken);
ServiceRequestFactory::setInstance($serviceRequest);
$spreadsheetService = new Google\Spreadsheet\SpreadsheetService();
$spreadsheetFeed = $spreadsheetService->getSpreadsheets();
$spreadsheet = $spreadsheetFeed->getByTitle('Hello World');
In the end I'm ending with something like this(very raw code so far, but enought for those who's seaching for solution. You need php google spreadsheet client of https://github.com/asimlqt/php-google-spreadsheet-client). Thats tiny example of inserting one row to my spreadsheet (sorry about my code, but showing just working example)
Thanks for bram brambring showing better way to authorization - answer by bram brambring
<?php
/*
* Google Spreadsheet class to work with google spreadsheets obviously ;D [using OAuth 2.0, as Zend Gdata is not anymore working]
*/
require_once('/Google/Spreadsheet/ServiceRequestInterface.php');
require_once('/Google/Spreadsheet/DefaultServiceRequest.php');
require_once('/Google/Spreadsheet/ServiceRequestFactory.php');
require_once('/Google/Spreadsheet/Spreadsheet.php');
require_once('/Google/Spreadsheet/SpreadsheetFeed.php');
require_once('/Google/Spreadsheet/SpreadsheetService.php');
require_once('/Google/Spreadsheet/Exception.php');
require_once('/Google/Spreadsheet/UnauthorizedException.php');
require_once('/Google/Spreadsheet/Spreadsheet.php');
require_once('/Google/Spreadsheet/Util.php');
require_once('/Google/Spreadsheet/Worksheet.php');
require_once('/Google/Spreadsheet/WorksheetFeed.php');
require_once('/Google/Spreadsheet/ListFeed.php');
require_once('/Google/Spreadsheet/ListEntry.php');
require_once('/Google/Spreadsheet/CellFeed.php');
require_once('/Google/Spreadsheet/CellEntry.php');
require_once('/Google/Config.php');
require_once('/Google/Client.php');
require_once('/Google/Auth/Abstract.php');
require_once('/Google/Auth/OAuth2.php');
require_once('/Google/Http/Request.php');
require_once('/Google/Utils.php');
require_once('/Google/IO/Abstract.php');
require_once('/Google/IO/Curl.php');
require_once('/Google/Http/CacheParser.php');
require_once('/Google/Logger/Abstract.php');
require_once('/Google/Logger/Null.php');
require_once('/Google/Exception.php');
require_once('/Google/Auth/Exception.php');
require_once('/Google/Auth/AssertionCredentials.php');
require_once('/Google/Cache/Abstract.php');
require_once('/Google/Cache/File.php');
require_once('/Google/Signer/Abstract.php');
require_once('/Google/Signer/P12.php');
use Google\Spreadsheet\DefaultServiceRequest;
use Google\Spreadsheet\ServiceRequestFactory;
class Google_Spreadsheet
{
private $default = array(
'worksheetCols' => 12,
'worksheetRows' => 25
);
private $spreadsheetKey;
private $spreadsheetName;
private $worksheetName;
private $spreadsheetFeed;
public $initialized = true;
public function __construct($spreadsheetKey, $worksheetName, $spreadsheetName = '')
{
$this->spreadsheetKey = $spreadsheetKey;
$this->worksheetName = $worksheetName;
$this->spreadsheetName = $spreadsheetName;
$this->initialized = $this->initialize();
return true;
}
private function getToken() {
$client_email = '318977712937456456454656563tcfjblgoi#developer.gserviceaccount.com';
$private_key = file_get_contents('API Project-f10e456456b60.p12');
$scopes = array('https://spreadsheets.google.com/feeds');
$credentials = new Google_Auth_AssertionCredentials(
$client_email,
$scopes,
$private_key,
'notasecret', // Default P12 password
'http://oauth.net/grant_type/jwt/1.0/bearer' // Default grant type
);
$client = new Google_Client();
$client->setAssertionCredentials($credentials);
if ($client->getAuth()->isAccessTokenExpired()) {
$client->getAuth()->refreshTokenWithAssertion();
}
$tokenData = json_decode($client->getAccessToken());
return $tokenData->access_token;
}
public function initialize(/*$reInitialized = false*/)
{
// load OAuth2 token data - exit if false
$tokenData = $this->getToken();
$serviceRequest = new DefaultServiceRequest($tokenData);
ServiceRequestFactory::setInstance($serviceRequest);
$spreadsheetService = new Google\Spreadsheet\SpreadsheetService();
try {
$spreadsheetFeed = $spreadsheetService->getSpreadsheets();
} catch (\Google\Spreadsheet\UnauthorizedException $e) {
Google_Spreadsheet::warnAdmin($e->getMessage());
return false;
}
$this->spreadsheetFeed = $spreadsheetFeed;
return true;
}
public function insertRow($rowData, $default_fields = array()) {
$spreadsheetFeed = $this->spreadsheetFeed;
$spreadsheet = $this->spreadsheetKey ? $spreadsheetFeed->getByKey($this->spreadsheetKey) : $spreadsheetFeed->getByTitle($this->spreadsheetName);
if(!$spreadsheet && !empty($this->spreadsheetName)) {
$spreadsheet = $spreadsheetFeed->getByTitle($this->spreadsheetName);
}
if(!$spreadsheet) {
Google_Spreadsheet::warnAdmin('No spreadsheet', serialize($rowData));
return false;
}
$worksheetFeed = $spreadsheet->getWorksheets();
$worksheet = $worksheetFeed->getByTitle($this->worksheetName);
if(!$worksheet) {
//create worksheet if not exist
$worksheet = $spreadsheet->addWorksheet($this->worksheetName, $this->default['worksheetRows'], $this->default['worksheetCols']);
$cellFeed = $worksheet->getCellFeed();
for( $i= 1 ; $i <= $this->default['worksheetCols']; $i++ ) {
if(isset($default_fields[$i])) {
$cellFeed->editCell(1, $i, $default_fields[$i]);
}
else {
$cellFeed->editCell(1, $i, "head");
}
$cellFeed->editCell(2,$i,"content");
}
}
if(!$worksheet) {
Google_Spreadsheet::warnAdmin('No worksheet', serialize($rowData));
return false;
}
$listFeed = $worksheet->getListFeed();
$data = array();
foreach ($listFeed->getEntries() as $entry) {
$values = $entry->getValues();
$data[] = $values;
break; //only first row needed, as we need keys
}
$keys = array();
if(!count($data)) {
Google_Spreadsheet::warnAdmin('No data', serialize($rowData));
return false;
}
foreach ($data[0] as $key => $value) {
$keys[] = $key;
}
$newRow = array();
$count = 0;
foreach($keys as $key) {
if(isset($rowData[$count])) {
$newRow["$key"] = $rowData[$count];
}
else {
$newRow["$key"] = '';
}
$count++;
}
$listFeed->insert($newRow);
return true;
}
static function warnAdmin($reason = '', $content = '') {
//temporal function to warn myself about all the stuff happening wrong :)
}
}
And in main model I'm using:
$spreadsheet = new Google_Spreadsheet("spreadsheet name or ID", $worksheetname, "My spreadsheet name");
if(!$spreadsheet->initialized) {
Google_Spreadsheet::warnAdmin('cannot initialize spreadsheet', serialize($rowValues));
}
if(!$spreadsheet->initialized || !$spreadsheet->insertRow($rowValues, $this->default_fields)) {
Google_Spreadsheet::warnAdmin('failed to insert row ');
}
#Edit, thanks to bram brambring for his token solution, seems more easier than my. Working like a charm now, I hope his way it gonna refresh token normally. THANK YOU DUDE!
Been stuck with this issue aswell, seems like google's ClientLogin was finally removed
Important: Do not use ClientLogin for new applications. Instead, use
the more secure OAuth authentication protocol. ClientLogin is a
deprecated authentication protocol and is being turned down on April
20, 2015. At that time, ClientLogin requests will no longer be
answered. If you have existing applications that use ClientLogin, we
encourage you to migrate to OAuth. The ClientLogin support in this
library will be removed in the next major release.
Going to try remake my spreadsheets with OAuth now, this may be the solution: removed cuz no more reputation, will post link into commets
edit; having some troubles with magnetikonline, so gonna try this solution: removed cuz no more reputation, will post link into comments
edit2; /'\ this solution is much better, I must go now, but I've got some success with this library, it works much better as far as I see and provide more functionality, try it.
For your problem with token, try https://github.com/asimlqt/php-google-oauth . THis way also worked for me and its much easier, I reveived array with my token info(if u gonna try that token with second library, ur token gonna be array, while u need only $accessToken['access_token'] part in order to make $serviceRequest = new DefaultServiceRequest($accessToken);
Another big tutorial I found: http://konstantinshkut.com/blog/2014/11/01/how_to_get_data_from_google_spreadsheet_in_yii_php_application (thats for second solution, asimlqt/php-google-spreadsheet-client), step 6 helps a lot to understand how my file must be looking.
I also have had php scripts using Zend for Google Spreadsheets. They have been running great for years, but stopped working today at noon for no reason. What has changed on Google's side and how to easily fix it?
That deprecated ClientLogin, it works again!
It was such a surprise on the eve of the conference Google I / O?
I had to write a self-made solution. Third-party libraries were in basically unusable,
I wrote about 10 methods, I think it is better than the alternatives, which are too complex and cumbersome. You can buy from me))
Not yet ready to put on github)

How to generate entity id for GAE datastore in php?

i am trying to insert new entity using PHP client library into datastore, i am using datastore_connect.php file from this example, https://github.com/amygdala/appengine_php_datastore_example
I want to insert entity with auto id, not the name. I see that there is function setId(), but i dont know how to generate proper id. Whats the best practice in doing so?
Thanks
function createKeyForTestItem () {
$path = new Google_Service_Datastore_KeyPathElement();
$path->setKind("testkind");
$path->setName("testkeyname");
//$path->setId(??)
$key = new Google_Service_Datastore_Key();
$key->setPath([$path]);
return $key;
}
You can have Cloud Datastore generate the ID for you by populating the insertAutoId field on the mutation instead of the upsert field.
Here's a code snippet (adapted from the datastore_connect.php file you posted):
function create_key() {
$path = new Google_Service_Datastore_KeyPathElement();
$path->setKind("testkind");
// Neither name nor ID is set.
$key = new Google_Service_Datastore_Key();
$key->setPath([$path]);
return $key;
}
function create_entity() {
$entity = new Google_Service_Datastore_Entity();
$entity->setKey(create_key());
// Add properties...
return $entity;
}
function create_commit_request() {
$entity = create_entity();
$mutation = new Google_Service_Datastore_Mutation();
$mutation->setInsertAutoId([$entity]); // Causes ID to be allocated.
$req = new Google_Service_Datastore_CommitRequest();
$req->setMode('NON_TRANSACTIONAL');
$req->setMutation($mutation);
return $req;
}
If you're looking for a PHP library to take away most of the headache of Cloud Datastore, you could try my new library, which sits on top of the official google-api-php-client:
https://github.com/tomwalder/php-gds
And here's a sample code snippet to create an Entity with an auto-generated ID
$obj_book = new GDS\Entity();
$obj_book->title = 'Romeo and Juliet';
$obj_book->author = 'William Shakespeare';
$obj_book->isbn = '1840224339';
// Write it to Datastore
$obj_book_store->upsert($obj_book);
More code snippets and documentation on GitHub.

Amazon AWS PHP SDK with Guzzle's MultiCurl?

I need to perform some fairly heavy queries with Amazon's AWS SDK for PHP.
The most efficient way would be to use PHP's MultiCurl. It seems that Guzzle already has functionality for MultiCurl built in.
Does using the standard methods provided by the AWS SDK automatically use MultiCurl or do I have to specify it's usage directly? E.g. calling $sns->Publish() 30 times.
Thanks!
Parallel requests work exactly the same in the SDK as in plain Guzzle and do take advantage of MultiCurl. For example, you could do something like this:
$message = 'Hello, world!';
$publishCommands = array();
foreach ($topicArns as $topicArn) {
$publishCommands[] = $sns->getCommand('Publish', array(
'TopicArn' => $topicArn,
'Message' => $message,
));
}
try {
$successfulCommands = $sns->execute($publishCommands);
$failedCommands = array();
} catch (\Guzzle\Service\Exception\CommandTransferException $e) {
$successfulCommands = $e->getSuccessfulCommands();
$failedCommands = $e->getFailedCommands();
}
foreach ($failedCommands as $failedCommand) { /* Handle any errors */ }
$messageIds = array();
foreach ($successfulCommands as $successfulCommand) {
$messageIds[] = $successfulCommand->getResult()->get('MessageId');
}
// Also Licensed under version 2.0 of the Apache License.
The AWS SDK for PHP User Guide has more information about working with command objects in this way.

Categories