PHP BigQuery Client Error Code: 401 running long job - php

I got a PHP BigQueryClient that I use to export big tables to csv from BQ, but after running for a while they throw and error with code 401:
Google\Cloud\Core\Exception\ServiceException
{
"error": {
"code": 401,
"message": "Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.",
"errors": [
{
"message": "Invalid Credentials",
"domain": "global",
"reason": "authError",
"location": "Authorization",
"locationType": "header"
}
],
"status": "UNAUTHENTICATED"
}
}
it occurs when running a long while that goes over the rows of the table like this
$bigQuery = new BigQueryClient([
'keyFile' => $array[$key],
'projectId' => $this->projectId
]);
//This is to get total row count so we can loop over it
$info = $bigQuery->dataset($this->datasetId)->table($tableName)->info();
//Output row count, and status message
$this->info($this->messagePrepend . "Rows: " . number_format($info['numRows']) . ", Chunk size: " . number_format($this->chunkSize) . " – Processing.." . PHP_EOL);
$startTime = Carbon::now();
//Open our output file
$file = fopen(storage_path("tmp/{$tableName}.csv"), 'w');
fputcsv($file, $this->tableHeaders);
$orderBy = match ($keyword) {
default => "id"
};
while ($info['numRows'] > $offset) {
$config = $bigQuery->query("SELECT * FROM {$this->datasetId}.{$tableName} ORDER BY {$orderBy} ASC LIMIT {$this->chunkSize} OFFSET {$offset}");
$job = $bigQuery->startQuery($config);
$queryResults = collect($job->queryResults());
$queryResults->map(function ($row) {
$line = [FORM LINE]
fputcsv($file, $line);
});
$offset += $this->chunkSize;
}
I've tried looking for the causes for this, and it seems like the issue is that the client token expires after an hour.
Despite that I've not found a way to refresh it, cause it doesn't seem to do so automatically, could anyone help me figure out how to do that?
I've read cloud bigquery docs but they didn't provide me with much answers, and neither did google

Resolved it by updating the instance of BigQueryClient when an exception is thrown:
try {
$job = $bigQuery->startQuery($config);
} catch (\Throwable $th) {
$this->warn("Code: " . $th->getCode() . ": " . $th->getMessage(), 'vvv');
$bigQuery = new BigQueryClient([
'keyFile' => $array[$key],
'projectId' => $this->projectId
]);
$job = $bigQuery->startQuery($config);
}

Related

Google Sheets API - 0 cells updated

I have a sheet that needs to be updated with some data.
Code that should add «hello» and «bye» to my sheet:
require __DIR__ . '/vendor/autoload.php';
function getClient()
{
...
}
$client = getClient();
$service = new Google_Service_Sheets($client);
$spreadsheetId = 'ID';
$range = 'A1';
$value = [
["hello", "bye"]
];
$body = new Google_Service_Sheets_ValueRange(
["majorDimension" => "ROWS"],
["values" => $value]
);
$params = array('valueInputOption' => 'USER_ENTERED');
$requestBody = new Google_Service_Sheets_ValueRange();
$result = $service->spreadsheets_values->update($spreadsheetId, $range,
$body, $params);
printf("%d cells updated.", $result->getUpdatedCells());
result:
0 cells updated.
I also tried to change $range = 'A1’; to something like Sheet1!A1 or A1-B2 and variety of another ranges; but in that case I receive error:
{ "error":
{ "code": 400, "message": "Unable to parse range: A1-B2",
"errors":
[
{ "message": "Unable to parse range: A1-B2",
"domain": "global",
"reason": "badRequest" } ],
"status": "INVALID_ARGUMENT" } }
How about this modification?
Answer 1:
0 cells updated.
About the reason of above issue, if you want to use "majorDimension" => "ROWS", please modify as follows.
From:
$body = new Google_Service_Sheets_ValueRange(
["majorDimension" => "ROWS"],
["values" => $value]
);
To:
$body = new Google_Service_Sheets_ValueRange([
"majorDimension" => "ROWS",
"values" => $value
]);
In your case, I think that the following modification can be also used.
$body = new Google_Service_Sheets_ValueRange([
"values" => $value
]);
Answer 2:
also tried to change $range = 'A1’; to something like Sheet1!A1 or A1-B2 and variety of another ranges; but in that case I receive error:
About the reason of above issue, if you want to use the range, how about the following modification?
From:
$range = 'A1';
To:
$range = 'Sheet1!A1';
and
$range = 'Sheet1!A1:B1';
References:
Method: spreadsheets.values.update
A1 notation

Unable to set up dialogFlow API

I am trying to use this quick guide to set up dialogFlow API on my laravel project - https://cloud.google.com/dialogflow/docs/quick/api.
This is the method I have created to detect intents, etc.
function detect_intent_texts($projectId, $texts, $sessionId, $languageCode = 'en-US'){
// new session
$filePath = base_path('dialogflow.json');
putenv("GOOGLE_APPLICATION_CREDENTIALS=".$filePath);
$sessionsClient = new SessionsClient();
$session = $sessionsClient->sessionName($projectId, $sessionId ?: uniqid());
printf('Session path: %s' . PHP_EOL, $session);
// query for each string in array
foreach ($texts as $text) {
// create text input
$textInput = new TextInput();
$textInput->setText($text);
$textInput->setLanguageCode($languageCode);
// create query input
$queryInput = new QueryInput();
$queryInput->setText($textInput);
// get response and relevant info
//dd($queryInput);
$response = $sessionsClient->detectIntent($session, $queryInput);
$queryResult = $response->getQueryResult();
$queryText = $queryResult->getQueryText();
$intent = $queryResult->getIntent();
$displayName = $intent->getDisplayName();
$confidence = $queryResult->getIntentDetectionConfidence();
$fulfilmentText = $queryResult->getFulfillmentText();
// output relevant info
print(str_repeat("=", 20) . PHP_EOL);
printf('Query text: %s' . PHP_EOL, $queryText);
printf('Detected intent: %s (confidence: %f)' . PHP_EOL, $displayName,
$confidence);
print(PHP_EOL);
printf('Fulfilment text: %s' . PHP_EOL, $fulfilmentText);
}
$sessionsClient->close();
}
When I try to run the function using this -
$array = ['hello'];
detect_intent_texts(env('DIALOGFLOW_PROJECT_ID'), $array, '134253474848');
I get this error-
{ "message": "IAM permission 'dialogflow.sessions.detectIntent' on 'projects\/********\/agent' denied.", "code": 7, "status": "PERMISSION_DENIED", "details": [] }
Please what might be the problem here?
It sounds like you have not properly setup roles when configuring your service account. See the auth setup steps.

How can I fix an error on update Google spreadsheet

I'm using Google Sheet Api on PHP framework and facing error while updating sheet.
Fatal error: Uncaught Google_Service_Exception: {
"error": {
"code": 403,
"message": "Request had insufficient authentication scopes.",
"errors": [
{
"message": "Request had insufficient authentication scopes.",
"domain": "global",
"reason": "forbidden"
}
],
"status": "PERMISSION_DENIED"
}
How can I fix it?
function add($spreadsheetId, $range, $value, $service){
$result = $service->spreadsheets_values->get($spreadsheetId, $range);
$numRows = $result->getValues() != null ? count($result->getValues()) : 0;
printf("%d rows retrieved.", $numRows);
$body = new Google_Service_Sheets_ValueRange([
'values' => [$value]
]);
$end = chr(65 + count($value) - 1);
$range = $range."!A".($numRows + 1).":".$end;//ex:Sheet!A6:H
printf("%s\n", $range);
$result = $service->spreadsheets_values->update($spreadsheetId, $range, $body, ['valueInputOption' => 'USER_ENTERED']);
printf("%d cells added.\n", $result->getUpdatedCells());
}
I also had error like that.
And I solved it by downloading credential.json file again.
Additionally, Please check if spreadsheetId is correct or wrote $client->setScopes(Google_Service_Sheets::SPREADSHEETS); before creating service.

php Google spreadsheet API "Request had insufficient authentication scopes"

i have a Problem with the Spreadsheed api and the "scopes".
With these script i want to update Cells on a Sheet.
I do not work with composer ich have just download the package in intereating it. The Token is already there and the error is from these row:
"$response = $service->spreadsheets_values->get($spreadsheetId, $range);"
<?php
session_start();
require_once __DIR__.'/vendor/autoload.php';
$client = new Google_Client();
$client->setAuthConfig('oauth-credentials.json');
$client->addScope(Google_Service_Drive::DRIVE_METADATA_READONLY);
if (isset($_SESSION['access_token']) && $_SESSION['access_token'])
{
$client->setAccessToken($_SESSION['access_token']);
echo "<pre>";
$service = new Google_Service_Sheets($client);
$spreadsheetId = 'xxx';
$range = 'Tabellenblatt1!A2:E';
$response = $service->spreadsheets_values->get($spreadsheetId, $range);
$values = $response->getValues();
if (count($values) == 0) {
print "No data found.\n";
} else {
print "Name, Major:\n";
foreach ($values as $row) {
// Print columns A and E, which correspond to indices 0 and 4.
printf("%s, %s <br>", $row[0], $row[4]);
}
}
} else {
$redirect_uri = 'http://' . $_SERVER['HTTP_HOST'] . '/api/oauth2callback.php';
header('Location: ' . filter_var($redirect_uri, FILTER_SANITIZE_URL));
}
?>
These Code brings the following Error
Fatal error: Uncaught exception 'Google_Service_Exception' with message '{
"error": {
"code": 403,
"message": "Request had insufficient authentication scopes.",
"errors": [
{
"message": "Request had insufficient authentication scopes.",
"domain": "global",
"reason": "forbidden"
}
],
"status": "PERMISSION_DENIED"
}
}
You have this scope defined:
$client->addScope(Google_Service_Drive::DRIVE_METADATA_READONLY);
For accessing a spreadsheet value:
$response = $service->spreadsheets_values->get($spreadsheetId, $range);
You should have:
$client->addScope(Google_Service_Sheets::SPREADSHEETS_READONLY);
or
$client->addScope(Google_Service_Sheets::SPREADSHEETS);
Source:
https://developers.google.com/identity/protocols/googlescopes#sheetsv4
You Must Update the scope as #random425 said,
but after that delete the Token.json.
that will start the process of verification again what will give you new token with new scope that you have changed to.

Estimate search result | Gmail API PHP

I am trying to get the number of emails (read & unread shown separately) for a given search. I've read that the labels.get() function does the trick but I don't know how to use it. Here's the code I have to detect if I have less or more than 100 result for a given sender.
require_once '../../vendor/autoload.php';
session_start();
$client = new Google_Client();
$client->setAuthConfigFile('../../client_secrets.json');
$client->addScope(Google_Service_Drive::DRIVE_METADATA_READONLY);
$client->addScope(Google_Service_Gmail::GMAIL_READONLY);
$client->setAccessType("offline");
$client->setApprovalPrompt('force');
$client->setAccessToken($_SESSION['token']);
$service = new Google_Service_Gmail($client);
$sender = array();
$sender[] = 'sender1#email.com';
$sender[] = 'sender2#email.com';
$sender[] = 'sender3#email.com';
function countfrom($service, $userId, $expeditor) {
try
{
unset($optParamsamz);
$optParamsamz = [];
$optParamsamz['maxResults'] = 100; // Return Only 5 Messages
$optParamsamz['q'] = "From: '".$expeditor."' ";
$messagesamz = $service->users_messages->listUsersMessages('me',$optParamsamz);
$listamz = $messagesamz->getMessages();
echo sizeof($listamz);
}
catch (Exception $e)
{
print 'An error occurred: ' . $e->getMessage();
}
}
foreach ($sender as $key => $value)
{
echo $value .': ';
countfrom($service,$_SESSION['emaile'],$value) ;
echo '<br/>';
}
------------------- EDIT ----------------------
I have tried a new solution that seems closer to what I'm looking for. The issue now comes from Google who returns some odd number for the resultestimatsize:
<?
require_once '../../vendor/autoload.php';
session_start();
$client = new Google_Client();
$client->setAuthConfigFile('../../client_secrets.json');
$client->addScope(Google_Service_Drive::DRIVE_METADATA_READONLY);
$client->addScope(Google_Service_Gmail::GMAIL_READONLY);
$client->setAccessType("offline");
$client->setApprovalPrompt('force');
$client->setAccessToken($_SESSION['token']);
$service = new Google_Service_Gmail($client);
$sender_array[] = 'sender1#sender.com';
$sender_array[] = 'sender2#sender.com';
$sender_array[] = 'sender3#sender.com';
$sender_array[] = 'sender4#sender.com';
foreach ($sender_array as $key => $expeditor)
{
$optParamsamz1['q'] = "From: '".$expeditor."' is:read ";
$optParamsamz2['q'] = "From: '".$expeditor."' ";
echo $expeditor.": ".$service->users_messages->listUsersMessages('me',$optParamsamz1)->getResultSizeEstimate() . "
".$service->users_messages->listUsersMessages('me',$optParamsamz2)->getResultSizeEstimate();
echo "<br>";
}
?>
labels.get() will be of no help in this use case, I'm afraid. It only works for labels, so you could get read/unread from e.g. INBOX or CHAT easily, but will be of no help if you want to get all read/unread from e.g. all messages sent from example#gmail.com.
An alternative solution is fairly cheap though:
List messages with the query + AND is:unread, and a second one with the same query +
AND -is:unread.
If the response contains a nextPageToken, you have 100+ read/unread. If it does not contain a nextPageToken, there are response.messages.length amount of read/unread messages.
Example
Request unread
q = from:info#berniesanders.com AND is:unread
GET https://www.googleapis.com/gmail/v1/users/me/messages?q=from%3Ainfo%40berniesanders.com+AND+is%3Aunread&access_token={YOUR_API_KEY}
Response
{
"messages": [
{
"id": "1523144d6e3feb2e",
"threadId": "1523144d6e3feb2e"
},
{
"id": "15227d879ccb601f",
"threadId": "15227d879ccb601f"
}, ...
}
// No nextPageToken => response.messages.length unread = 22 unread
Request NOT unread
q = from:info#berniesanders.com AND -is:unread
GET https://www.googleapis.com/gmail/v1/users/me/messages?q=from%3Ainfo%40berniesanders.com+AND+-is%3Aunread&access_token={YOUR_API_KEY}
Response
{
"messages": [
{
"id": "1522d4af39d7eec6",
"threadId": "1522d4af39d7eec6"
},
{
"id": "1521d6f3dbeaf886",
"threadId": "1521d6f3dbeaf886"
}, ...
"nextPageToken": "32436546446"
}
// nextPageToken in response => 100+ read
You could take it one step further and keep on listing with the nextPageToken until there is no nextPageToken in the response, and just add all the results together, but that might be to slow or inefficient for your use case.

Categories