I need to perform some fairly heavy queries with Amazon's AWS SDK for PHP.
The most efficient way would be to use PHP's MultiCurl. It seems that Guzzle already has functionality for MultiCurl built in.
Does using the standard methods provided by the AWS SDK automatically use MultiCurl or do I have to specify it's usage directly? E.g. calling $sns->Publish() 30 times.
Thanks!
Parallel requests work exactly the same in the SDK as in plain Guzzle and do take advantage of MultiCurl. For example, you could do something like this:
$message = 'Hello, world!';
$publishCommands = array();
foreach ($topicArns as $topicArn) {
$publishCommands[] = $sns->getCommand('Publish', array(
'TopicArn' => $topicArn,
'Message' => $message,
));
}
try {
$successfulCommands = $sns->execute($publishCommands);
$failedCommands = array();
} catch (\Guzzle\Service\Exception\CommandTransferException $e) {
$successfulCommands = $e->getSuccessfulCommands();
$failedCommands = $e->getFailedCommands();
}
foreach ($failedCommands as $failedCommand) { /* Handle any errors */ }
$messageIds = array();
foreach ($successfulCommands as $successfulCommand) {
$messageIds[] = $successfulCommand->getResult()->get('MessageId');
}
// Also Licensed under version 2.0 of the Apache License.
The AWS SDK for PHP User Guide has more information about working with command objects in this way.
Related
I'm setting up my client app using my BigQuery Data, I need to build it with php (which I'm not expert).
I want to authenticate and be able to build queries using php in my server.
I'm having trouble authenticating, as I see different solutions which seem outdated or incomplete.
I've got the JSON key and the project I'd but don't know the proper wait to authenticate to BigQuery.
Can someone tell me how to properly do that? If I need another library (like google cloud auth?)
Here is the code snippet:
public function getServiceBuilder() {
putenv('GOOGLE_APPLICATION_CREDENTIALS=path_to_service_account.json');
$builder = new ServiceBuilder(array(
'projectId' => PROJECT_ID
));
return $builder;
}
then you can use like this
$builder = $this->getServiceBuilder();
$bigQuery = $builder->bigQuery();
$job = $bigQuery->runQueryAsJob('SELECT .......');
$backoff = new ExponentialBackoff(8);
$backoff->execute(function () use ($job) {
$job->reload();
$this->e('reloading job');
if (!$job->isComplete()) {
throw new \Exception();
}
});
if (!$job->isComplete()) {
$this->e('Job failed to complete within the allotted time.');
return false;
}
$queryResults = $job->queryResults();
if ($queryResults->isComplete()) {
$i = 0;
$rows = $queryResults->rows();
foreach ($rows as $row) {
(edited)
you may need to add:
use Google\Cloud\BigQuery\BigQueryClient;
use Google\Cloud\ServiceBuilder;
use Google\Cloud\ExponentialBackoff;
and composer.json (this is just an example it may differ the actual version)
{
"require": {
"google/cloud": "^0.99.0"
}
}
I need to check whether a set of keys exist in S3, for each of a large number of items. (Each set of keys relates to one of the large number of items).
I am using the PHP SDK (v2)
Currently I am calling $client->doesObjectExist(BUCKET, $key) for each of the keys, which is a bottleneck (the round-trip time to S3 for each call).
I would prefer to do something like $client->doesObjectExist(BUCKET, $batch) where $batch = array($key1, $key2 ... $keyn), and for the client to check all of those keys then come back with an array of responses (or some other similar structure).
I have come across a few references to a "batch api" which sounds promising, but nothing concrete. I'm guessing that this might have been present only in the v1 SDK.
You can do parallel requests using the AWS SDK for PHP by taking advantage of the underlying Guzzle library features. Since the doesObjectExist method actually does HeadObject operations under that hood. You can create groups of HeadObject commands by doing something like this:
use Aws\S3\S3Client;
use Guzzle\Service\Exception\CommandTransferException;
function doObjectsExist(S3Client $s3, $bucket, array $objectKeys)
{
$headObjectCommands = array();
foreach ($objectKeys as $key) {
$headObjectCommands[] = $s3->getCommand('HeadObject', array(
'Bucket' => $bucket,
'Key' => $key
));
}
try {
$s3->execute($headObjectCommands); // Executes in parallel
return true;
} catch (CommandTransferException $e) {
return false;
}
}
$s3 = S3Client::factory(array(
'key' => 'your_aws_access_key_id',
'bucket' => 'your_aws_secret_key',
));
$bucket = 'your_bucket_name';
$objectKeys = array('object_key_1', 'object_key_2','object_key_3');
// Returns true only if ALL of the objects exist
echo doObjectsExist($s3, $bucket, $objectKeys) ? 'YES' : 'NO';
If you want data from the responses, other than just whether or not the keys exist, you can change the try-catch block to do something like this instead.
try {
$executedCommands = $s3->execute($headObjectCommands);
} catch (CommandTransferException $e) {
$executedCommands = $e->getAllCommands();
}
// Do stuff with the command objects
foreach ($executedCommands as $command) {
$exists = $command->getResponse()->isSuccessful() ? "YES" : "NO";
echo "{$command['Bucket']}/{$command['Key']}: {$exists}\n";
}
Sending commands in parallel is mentioned in the AWS SDK for PHP User Guide, but I would also take a look at the Guzzle batching documentation.
The only way to do a bulk check to see if some keys exist would be to list the objects in the bucket.
For a list call AWS returns up to 1000 keys/call so it's much faster than doing a doesObjectExist call for each key. But if you have a large number of keys and you only want to check a couple of them, listing all the objects in the bucket will not be practical so in that case, your only option remains to check each object individually.
The problem is not that the PHP v2 SDK lacks the bulk functionality but that the S3 API does not implement such bulk processing.
I am building on Jeremy Lindblom's answer.
Just want to point out the OnComplete callback that you can setup on each command.
$bucket = 'my-bucket';
$keys = array('page1.txt', 'page2.txt');
$commands = array();
foreach ($keys as $key) {
$commands[] = $s3Client->getCommand('HeadObject', array('Bucket' => $bucket, 'Key' => $key))
->setOnComplete(
function($command) use ($bucket, $key)
{
echo "\nBucket: $bucket\n";
echo "\nKey: $key\n";
// see http://goo.gl/pIWoYr for more detail on command objects
var_dump($command->getResult());
}
);
}
try {
$ex_commands = $s3Client->execute($commands);
}
catch (\Guzzle\Service\Exception\CommandTransferException $e) {
$ex_commands = $e->getAllCommands();
}
// this is necesary; without this, the OnComplete handlers wouldn't get called (strange?!?)
foreach ($ex_commands as $command)
{
$command->getResult();
}
It will be wonderful if someone could shed light on why I need to call $command->getResult() to invoke the OnComplete handler.
Assume composer is installed and you need to setup an ec2 client.
Suppose SDK setup with recommended method using Composer. First call aws_setupthen create an ec2 client object with security credentials. Since composer has been invoked, it will automatically load required libraries.
Then use DescribeInstances to get all running instances.
I packaged the function countInstances so it can be reused. You can call DescribeInstances with
with an array to filter results which is posted at the end.
Setup as follows:
require('/PATH/TO/MY/COMPOSER/vendor/autoload.php');
function aws_setup()
{
$conf_aws = array();
$conf_aws['key'] = 'MYKEY';
$conf_aws['secret'] = 'MYSECRET';
$conf_aws['region'] = 'us-east-1';
return $conf_aws;
}
function countInstances($list)
{
$count = 0;
foreach($list['Reservations'] as $instances)
{
foreach($instances['Instances'] as $instance)
{
$count++;
}
}
return $count;
}
$config = aws_setup();
$ec2Client = \Aws\Ec2\Ec2Client::factory($config);
$list = $ec2Client->DescribeInstances();
echo "Number of running instances: " . countInstances($list);
If you want to filter your results try something like this as a parameter to DescribeInstances:
array('Filters' => array(array('Name' => 'tag-value', 'Values' => array('MY_INSTANCE_TAG'))));
The code executes without error, but I had to adapt it to post it here.
EDIT: Added list of instances to countInstances function. Otherwise it wouldn't be visible.
I'm trying to find some simple examples of a PHP class accessing a distributed API.
Whenever I search for a PHP API client it gives way too much information on building the API itself and little in the way of real access methods beyond the curl examples. A PHP class that includes a good curl method and a way to handle the requests would be nice. I learn by example and lots of re-arranging.
I started with a construct to catch the usual connection settings:
private $hosts = array( URI_1, URI_2, URI_3);
private $users = array( USER_1, USER_2, USER_3);
private $pass = array( PASS_1, PASS_2, PASS_3);
public function __construct($request = array())
{
if (is_array($request)) {
if(isset($request['hostname'])) {
if(in_array($request['hostname'], $hosts)) {
$this->input['hostname'] = $request['hostname'];
}
}
if (isset($request['username'])) {
if(in_array($request['username'], $users)) {
$this->input['username'] = $request['username'];
}
}
if (isset($request['password'])) {
if (in_array($request['password'], $pass)) {
$this->input['password'] = $request['password'];
}
}
if (isset($this->input['hostname']) &&
isset($this->input['username']) &&
isset($this->input['password'])) {
return true;
}
} else {
return false;
}
}
Some simple examples of best practices would make this a lot easier.
Try out Guzzle:
Guzzle takes the pain out of sending HTTP requests and the redundancy
out of creating web service clients.
Guzzle is a framework that includes the tools needed to create a
robust web service client, including: Service descriptions for
defining the inputs and outputs of an API, resource iterators for
traversing paginated resources, batching for sending a large number of
requests as efficiently as possible.
Is there any PHP class or resource for using Metaweblog api ?
I want to add this api to my own cms (like wp) so that other application could easily post (or ...) throw it
Implementation of the MetaWeblog API http://www.xmlrpc.com/metaWeblogApi in PHP.
I looked to this script I linked for inspiration to develop the implementation I'm currently using. Feel free to use the example code below as an example of implementing the metaweblog API - but please consider using a modern XMLRPC library. I've included a link to a modified version of the original "xmlrpc.php" file that the example code requires.
Here's the xmlrpc library the example code utilizes: XMLRPC library modified to work with PHP 5.4 - originally written by Keith Devens.
Doing a quick package search on packagist also provides many great options that are much more forward thinking in terms of PHP standards. ZendFramework2 even includes a component you can use in your project with minimal dependencies (10 packages - not the entire framework). I would strongly recommend that this example code, be used as such, and any new development be done with a modern XMLRPC library.
Adding the example code here in case the first link dies:
<?php
/**
* Skeleton file for MetaWeblog API http://www.xmlrpc.com/metaWeblogApi in PHP
* Requires Keith Devens' XML-RPC Library http://keithdevens.com/software/xmlrpc and store it as xmlrpc.php in the same folder
* Written by Daniel Lorch, based heavily on Keith Deven's examples on the Blogger API.
*/
require_once dirname(__FILE__) . '/xmlrpc.php';
function metaWeblog_newPost($params) {
list($blogid, $username, $password, $struct, $publish) = $params;
$title = $struct['title'];
$description = $struct['description'];
// YOUR CODE:
$post_id = 0; // id of the post you just created
XMLRPC_response(XMLRPC_prepare((string)$post_id), WEBLOG_XMLRPC_USERAGENT);
}
function metaWeblog_editPost($params) {
list($postid, $username, $password, $struct, $publish) = $params;
// YOUR CODE:
$result = false; // whether or not the action succeeded
XMLRPC_response(XMLRPC_prepare((boolean)$result), WEBLOG_XMLRPC_USERAGENT);
}
function metaWeblog_getPost($params) {
list($postid, $username, $password) = $params;
$post = array();
// YOUR CODE:
$post['userId'] = '1';
$post['dateCreated'] = XMLRPC_convert_timestamp_to_iso8601(time());
$post['title'] = 'Replace me';
$post['content'] = 'Replace me, too';
$post['postid'] = '1';
XMLRPC_response(XMLRPC_prepare($post), WEBLOG_XMLRPC_USERAGENT);
}
function XMLRPC_method_not_found($methodName) {
XMLRPC_error("2", "The method you requested, '$methodName', was not found.", WEBLOG_XMLRPC_USERAGENT);
}
$xmlrpc_methods = array(
'metaWeblog.newPost' => 'metaWeblog_newPost',
'metaWeblog.editPost' => 'metaWeblog_editPost',
'metaWeblog.getPost' => 'metaWeblog_getPost'
);
$xmlrpc_request = XMLRPC_parse($HTTP_RAW_POST_DATA);
$methodName = XMLRPC_getMethodName($xmlrpc_request);
$params = XMLRPC_getParams($xmlrpc_request);
if(!isset($xmlrpc_methods[$methodName])) {
XMLRPC_method_not_found($methodName);
} else {
$xmlrpc_methods[$methodName]($params);
}