Response logging in AWS PHP SDK v3 - php

In v2 of the AWS PHP SDK, I was able to setup logging of request and response information by simply doing this:
<?php
use Monolog\Logger;
use Guzzle\Log\MonologLogAdapter;
use Guzzle\Plugin\Log\LogPlugin;
use Aws\S3\S3Client;
$monolog = new Logger('main');
$monolog_adapter = new MonologLogAdapter($monolog);
$log_plugin = new LogPlugin($monolog_adapter);
$s3_client = S3Client::factory(['region' => 'us-east-1']);
$s3_client->addSubscriber($log_plugin);
var_dump($s3_client->doesObjectExist('my-bucket', 'object-that-doesnt-exist'));
# This is the log entry I want in the v3 version:
# [2015-10-30 14:47:20] main.ERROR: myhostname aws-sdk-php2/2.8.20 Guzzle/3.9.3 curl/7.43.0 PHP/5.5.23 - [2015-10-30T14:47:20+00:00] "HEAD /my-bucket/object-that-doesnt-exist HTTP/1.1" 404 ...
# bool(false)
In v3, I cannot seem to find the solution. Middlewares do not seem helpful as they only fire before the request is sent, and thus I cannot access the response HTTP code.
Guzzle v6 has this feature built into its Middlewares, but I do not know how to get it to work with the aws-php-sdk. https://github.com/guzzle/guzzle/blob/master/src/Middleware.php#L180
The closest I got was this:
<?php
use Monolog\Logger;
use GuzzleHttp\MessageFormatter;
use GuzzleHttp\Middleware;
use GuzzleHttp\HandlerStack;
use Aws\S3\S3Client;
$monolog = new Logger('main');
$guzzle_formatter = new MessageFormatter(MessageFormatter::CLF);
$guzzle_log_middleware = Middleware::log($monolog, $guzzle_formatter);
$guzzle_stack = HandlerStack::create();
$guzzle_stack->push($guzzle_log_middleware);
$s3_client = new S3Client([
'region' => 'us-east-1',
'version' => '2006-03-01',
'http_handler' => $guzzle_stack,
]);
var_dump($s3_client->doesObjectExist('my-bucket', 'object-that-doesnt-exist'));
# [2015-10-30 15:10:12] main.INFO: myhostname aws-sdk-php/3.9.2 - [30/Oct/2015:15:10:12 +0000] "HEAD /my-bucket/object-that-doesnt-exist HTTP/1.1" 404 [] []
# bool(true)
However, while the logging works, doesObjectExist() now returns the incorrect value because this handler does not throw an exception for 404, which the aws-php-sdk expects to happen. Some other simple requests like uploading to S3 seemed to work at first glance. Not sure where else there could be issues with this method.

The handler used in the SDK is a bit different from the one used in Guzzle. You're creating a Guzzle handler correctly, and to pass that into the SDK, you would need to create an adapter like so:
<?php
use Aws\Handler\GuzzleV6\GuzzleHandler;
use Aws\S3\S3Client;
use Monolog\Logger;
use GuzzleHttp\Client;
use GuzzleHttp\MessageFormatter;
use GuzzleHttp\Middleware;
use GuzzleHttp\HandlerStack;
$guzzle_stack = HandlerStack::create();
$guzzle_stack->push(Middleware::log(
new Logger('main'),
new MessageFormatter(MessageFormatter::CLF)
));
$handler = new GuzzleHandler(new Client(['handler' => $guzzle_stack]));
$s3_client = new S3Client([
'region' => 'us-east-1',
'version' => '2006-03-01',
'http_handler' => $handler,
]);
var_dump($s3_client->doesObjectExist('my-bucket', 'object-that-doesnt-exist'));
The GuzzleHandler object converts HTTP errors to exceptions.

With aws/aws-sdk-php 3.212.7, registering a custom logger as shown by #giaour did not work for me.
The configuration supports a debug key now:
$s3_client = new S3Client([
'region' => 'us-east-1',
'version' => '2006-03-01',
'debug' => true,
]);
This shows the data transferred, but not the URLs used.

Related

Class S3 not found if use inside function

I am new in PHP. I am trying to make S3 upload function so I can use it in my other php files. I have made s3.php file like this
<?php
use Aws\S3\S3Client;
use Aws\Exception\AwsException;
use Aws\S3\ObjectUploader;
use Aws\S3\MultipartUploader;
use Aws\Exception\MultipartUploadException;
function bucket_upload($BUCKET_ACCESS_KEY,$BUCKET_SECRET_KEY,$BUCKET_REGION,$BUCKET_URL,$BUCKET_NAME,$FILE,$FILE_NAME){
require __DIR__.'/vendor/autoload.php';
$config = [
'credentials' => new \Aws\Credentials\Credentials(
$BUCKET_ACCESS_KEY,
$BUCKET_SECRET_KEY
),
'version' => 'latest',
'region' => $BUCKET_REGION,
'endpoint' => $BUCKET_URL
];
$s3Client = new \Aws\S3\S3Client($config);
if($s3Client->putObjectFile($FILE, $BUCKET_NAME, $FILE_NAME, S3::ACL_PUBLIC_READ)) {
return true;
}else{
return false;
}
}
Its working fine if I do not use S3 inside function. For now in function its giving me error on line 23 called class S3 not found. Let me know if anyone here can help me to fix the issue.
Thanks!

Call to undefined method Goutte\Client::setClient()

I am stuck with this error...
but the client is defined.
my code like this
use Goutte\Client;
use Illuminate\Http\Request;
use GuzzleHttp\Client as GuzzleClient;
class WebScrapingController extends Controller
{
public function doWebScraping()
{
$goutteClient = new Client();
$guzzleClient = new GuzzleClient(array(
'timeout' => 60,
'verify' => false
));
$goutteClient->setClient($guzzleClient);
$crawler = $goutteClient->request('GET', 'https://duckduckgo.com/html/?q=Laravel');
$crawler->filter('.result__title .result__a')->each(function ($node) {
dump($node->text());
});
}
}
I think error from this line
$goutteClient->setClient($guzzleClient);
goutte: "^4.0" guzzle: "7.0" Laravel Framework: "6.20.4"
This answer is regarding creating instance of Goutte client, a simple PHP Web Scraper
For Version >= 4.0.0
Pass HttpClient(either guzzle httpclient , symfony httpclient) instance directly inside the instance of Goutte Client.
use Goutte\Client;
use Symfony\Component\HttpClient\HttpClient;// or use GuzzleHttp\Client as GuzzleClient;
$client = new Client(HttpClient::create(['timeout' => 60]));
// or
// $guzzleClient = new GuzzleClient(['timeout' => 60, 'verify' => false]); // pass this to Goutte Client
For Version <= 4.0.0 (i.e from 0.1.0 to 3.3.1)
use Goutte\Client;
use GuzzleHttp\Client as GuzzleClient;
$goutteClient = new Client();
$guzzleClient = new GuzzleClient(['timeout' => 60]);
$goutteClient->setClient($guzzleClient);

Is it possible to setup Guzzle + Pool over HTTP/2?

Guzzle provides a mechanism to send concurrent requests: Pool. I used the example from the docs: http://docs.guzzlephp.org/en/stable/quickstart.html#concurrent-requests. It works quite fine, sends concurrent requests and everything is awesome except one thing: it seems Guzzle ignores HTTP/2 in this case.
I've prepared a simplified script that sends two requests to https://stackoverflow.com, the first one is using Pool, the second one is just a regular Guzzle request. Only the regular request connects via HTTP/2.
<?php
include_once 'vendor/autoload.php';
use GuzzleHttp\Client;
use GuzzleHttp\Pool;
use GuzzleHttp\Psr7\Request;
$client = new Client([
'version' => 2.0,
'debug' => true
]);
/************************/
$requests = function () {
yield new Request('GET', 'https://stackoverflow.com');
};
$pool = new Pool($client, $requests());
$promise = $pool->promise();
$promise->wait();
/************************/
$client->get('https://stackoverflow.com', [
'version' => 2.0,
'debug' => true,
]);
Here is an output: https://pastebin.com/k0HaDWt6 (I highlighted important parts with "!!!!!")
Does anybody know why Guzzle does this and how to make Pool work with HTTP/2?
Found what was wrong: new Client() doesn't actually accept 'version' as an option if passed to Pool requests are created as new Request(). Either the protocol version must be provided as an option of every request or the requests must be created as $client->getAsync() (or ->postAsync or whatever).
See the corrected code:
...
$client = new Client([
'debug' => true
]);
$requests = function () {
yield new Request('GET', 'https://stackoverflow.com', [], null, '2.0');
};
/* OR
$client = new Client([
'version' => 2.0,
'debug' => true
]);
$requests = function () use ($client) {
yield function () use ($client) {
return $client->getAsync('https://stackoverflow.com');
};
};
*/
$pool = new Pool($client, $requests());
$promise = $pool->promise();
$promise->wait();
...

storageclient class can't be found?

I'm trying desperately to figure out how to create a simple audio transcription script (for longer audio files) via PHP (the only language I know). I'm getting the error Class 'Google\Cloud\Storage\StorageClient' not found
I'm using the gcloud console code editor and everything should be installed (unless there is a separate composer install just for cloud storage, although I haven't been able to find anything about it in the documentation if there is).
I also entered gcloud auth application-default print-access-token which printed out an access token, but I don't know what (if any) I'm supposed to do with that other than the "set GOOGLE_APPLICATION_CREDENTIALS" command that I copied and pasted into the console shell prompt.
Here's the php code:
<?php
namespace Google\Cloud\Samples\Speech;
require __DIR__ . '/vendor/autoload.php';
use Exception;
# [START speech_transcribe_async_gcs]
use Google\Cloud\Speech\SpeechClient;
use Google\Cloud\Storage\StorageClient;
use Google\Cloud\Core\ExponentialBackoff;
$projectId = 'xxxx';
$speech = new SpeechClient([
'projectId' => $projectId,
'languageCode' => 'en-US',
]);
$filename = "20180925_184741_L.mp3";
# The audio file's encoding and sample rate
$options = [
'encoding' => 'LINEAR16',
'sampleRateHertz' => 16000,
'languageCode' => 'en-US',
'enableWordTimeOffsets' => false,
'enableAutomaticPunctuation' => true,
'model' => 'video',
];
function transcribe_async_gcs($bucketName, $objectName, $languageCode = 'en-US', $options = [])
{
// Create the speech client
$speech = new SpeechClient([
'languageCode' => $languageCode,
]);
// Fetch the storage object
$storage = new StorageClient();
$object = $storage->bucket($bucketName)->object($objectName);
// Create the asyncronous recognize operation
$operation = $speech->beginRecognizeOperation(
$object,
$options
);
// Wait for the operation to complete
$backoff = new ExponentialBackoff(10);
$backoff->execute(function () use ($operation) {
print('Waiting for operation to complete' . PHP_EOL);
$operation->reload();
if (!$operation->isComplete()) {
throw new Exception('Job has not yet completed', 500);
}
});
// Print the results
if ($operation->isComplete()) {
$results = $operation->results();
foreach ($results as $result) {
$alternative = $result->alternatives()[0];
printf('Transcript: %s' . PHP_EOL, $alternative['transcript']);
printf('Confidence: %s' . PHP_EOL, $alternative['confidence']);
}
}
}
# [END speech_transcribe_async_gcs]
transcribe_async_gcs("session_audio", $filename, "en-US", $options);
With apologies, PHP is not a language I'm proficient with but, I suspect you haven't (and must) install the client library for Cloud Storage so that your code may access it. This would explain its report that the Class is missing.
The PHP client library page includes two alternatives. One applies if you're using Composer, the second -- possibly what you want -- a direct download which you'll need to path correctly for your code.
Some time ago, I wrote a short blog post providing a simple example (using Cloud Storage) for each of Google's supported languages. Perhaps it will help you too.

PHP AWS S3 SDK retry on network connections errors

I'm using AWS SDK V3 for PHP. Sometimes when I make calls to AWS S3 I get errors like 400 errors Bad Request RequestTimeout (client): Your socket connection to the server was not read from or written to within the timeout period due to network problem even when the object I'm trying to interact with is there. What I need to do is to implement a retry mechanism. I wonder if we can do it simply with an option in the AWS SDK to specify the number of times we want a retry after an error.
I know that I can do that with simple try catch and retry, but I'm thinking may be the SDK already provides a much cleaner way to do that.
I already found static function Middleware::retry() but I have no idea how to use it.
You can specify the number of retries when constructing a new instance of any AWS service client class:
$client = new Aws\EC2\Ec2Client([
'region' => 'eu-central-1',
'retries' => 3
]);
https://docs.aws.amazon.com/aws-sdk-php/v3/guide/guide/configuration.html#retries
You can use something like this,
public function get_s3_connection(){
$sdk = new \Aws\Sdk([
'region' => 'us-east-1',
'version' => 'latest',
'credentials' => $credentials
]);
$s3_connection = $sdk->createS3([
'retries' => 3,
'http' => [
'connect_timeout' => 15,
'timeout' => 20
],
'endpoint' => 'https://bucket.0123xyz.s3.us-east-1.amazonaws.com'
]);
return $s3_connection;
}
In the above code the retries => 3 means it will try 3 times and then stop trying.
You can also set the timeout and connect_timeout as show above.
You can also retry manually by surrounding the method with do-while loop, as shown below,
public function s3_connect_with_retry($total_retry = 3){
$retry_count = 0;
$do_retry = false;
do{
try{
$s3_connection = $this->get_s3_connection();
}catch(Exception $exception){
$retry_count++;
$do_retry = true;
}
}while($total_retry < $retry && $do_retry == true;);
}
When the connection breaks and throws exception the $do_retry value will become true and the $retry_count will increase. based on the $do_retry flag the retry happens until the $total_retry count is reached.

Categories