My PHP script is working with multiple external services using API requests and theirs SDKs. It happens, that requests might fail and I would like to implement a retry mechanism.
SDKs don't have this feature built-in and they will just throw an Exception if request was not successful.
I've created multiple functions that use SDKs to retrieve / upload required data and now I would like to add retry mechanism if any of SDK requests fail. I could add a piece of code around every SDK call like this:
$retry = 0;
while ($retry < 3) {
try {
$account = $client->load('account');
} catch (\Exception $e) {
$retry++;
}
}
But this means that I will add the same piece of code to every SDK function call and it's going to be a real mess.
Is there any possibility in PHP to create a generic function or some kind of wrapper that will retry executing function if it has failed (thrown an Exception)?
P.S. SDKs use curl to send requests. Maybe, it's possible to tweak curl to implement this kind of retry mechanism?
Related
I am using the PayPal PHP SDK found here: https://github.com/paypal/Checkout-PHP-SDK
And I am somewhat puzzled in terms of how to complete the process.
On the outset this seems quite simple:
Setup your credentials
Create the Order
Check the result, and re-direct to approval link
User makes a payment and is sent to the SUCCESS link that you would have set.
i.e. http://example.com/pay/complete/paypal?token=8UK32254ES097084V&PayerID=SEQNPLB2JR9LY
And this is where things get a bit shakey.
Conveniently, a token and a PayerID is returned.
And according to the documentation, you now need to "Capturing the Order" and the following code is provided:
use PayPalCheckoutSdk\Orders\OrdersCaptureRequest;
// Here, OrdersCaptureRequest() creates a POST request to /v2/checkout/orders
// $response->result->id gives the orderId of the order created above
$request = new OrdersCaptureRequest("APPROVED-ORDER-ID");
$request->prefer('return=representation');
try {
// Call API with your client and get a response for your call
$response = $client->execute($request);
// If call returns body in response, you can get the deserialized version from the result attribute of the response
print_r($response);
}catch (HttpException $ex) {
echo $ex->statusCode;
print_r($ex->getMessage());
}
What is confusing is that the OrdersCaptureRequest requires an "APPROVED-ORDER-ID"
But all that has been returned is a "token" and a "PayerID".
So my question is, what is this APPROVED-ORDER-ID, and where do I get it?
Thank you!
what is this APPROVED-ORDER-ID, and where do I get it
At that moment, sourced from token= . It should correspond to an Order Id you received in the response to your step 2 ("Create the Order")
For step 3, it is better to use no redirects whatsoever. Instead, implement this front-end UI, which offers a far superior in-context experience that keeps your site loaded in the background: https://developer.paypal.com/demo/checkout/#/pattern/server
There is no reason for a modern website to be redirecting unnecessarily
This actually follows on from a previous question I had that, unfortunately, did not receive any answers so I'm not exactly holding my breath for a response but I understand this can be a bit of a tricky issue to solve.
I am currently trying to implement rate limiting on outgoing requests to an external API to match the limit on their end. I have tried to implement a token bucket library (https://github.com/bandwidth-throttle/token-bucket) into the class we are using to manage Guzzle requests for this particular API.
Initially, this seemed to be working as intended but we have now started seeing 429 responses from the API as it no longer seems to be correctly rate limiting the requests.
I have a feeling what is happening is that the number of tokens in the bucket is now being reset every time the API is called due to how Symfony handles services.
I am setting currently setting up the bucket location, rate and starting amount in the service's constructor:
public function __construct()
{
$storage = new FileStorage(__DIR__ . "/api.bucket");
$rate = new Rate(50, Rate::MINUTE);
$bucket = new TokenBucket(50, $rate, $storage);
$this->consumer = new BlockingConsumer($bucket);
$bucket->bootstrap(50);
}
I'm then attempting to consume a token before each request:
public function fetch(): array
{
try {
$this->consumer->consume(1);
$response = $this->client->request(
'GET', $this->buildQuery(), [
'query' => array_merge($this->params, ['api_key' => $this->apiKey]),
'headers' => [ 'Content-type' => 'application/json' ]
]
);
} catch (ServerException $e) {
// Process Server Exception
} catch (ClientException $e) {
// Process Client Exception
}
return $this->checkResponse($response);
}
I can't see anything obvious in that, that would allow it to request more than 50 times per minute unless the amount of available tokens was being reset on each request.
This is being supplied to a set of repository services that handle converting the data from each endpoint into objects used within the system. Consumers use the appropriate repository to request the data needed to complete their process.
If the amount of tokens is being reset by the bootstrap function being in service constructor, where should it be moved to within the Symfony framework that would still work with consumers?
I assume that it should work, but maybe try to move the ->bootstrap(50) call from every request? Not sure, but it can be the reason.
Anyway it's better to do that only once, as a part of your deployment (every time you deploy a new version). It doesn't have anything with Symfony, really, because the framework doesn't have any restrictions on deployment procedure. So it depends on how you do the deployment.
P.S. Have you considered to just handle 429 errors from the server? IMO you can wait (that's what BlockingConsumer does inside) when you receive 429 error. It's simpler and doesn't require an additional layer in your system.
BTW, have you considered nginx's ngx_http_limit_req_module as an alternative solution? It usually comes with nginx by default, so no additional actions to install, only a small configuration is required.
You can place an nginx proxy behind your code and the target web service and enable limits on it. Then in your code you will handle 429 as usual, but the requests will be throttled by your local nginx proxy, not by the external web service. So the final destination will get only limited amount of requests.
I have found a trick using Guzzle bundle for symfony.
I had to improve a sequential program sending GET requests to a Google API. In code example, it a pagespeed URL.
To have a rate limit, there an option to delay the requests before they are sent asynchronously.
Pagespeed rate limit is 200 requests per minute.
A quick calculation gives 200/60 = 0.3s per request.
Here is the code I tested on 300 urls, getting a fantastic result of no error, except if the url passed as a parameter in the GET request gives a 400 HTTP Error (Bad request).
I put a delay of 0.4s and the average result time is less then 0.2s, whereas it took more than a minute with a sequential program.
use GuzzleHttp;
use GuzzleHttp\Client;
use GuzzleHttp\Promise\EachPromise;
use GuzzleHttp\Exception\ClientException;
// ... Now inside class code ... //
$client = new GuzzleHttp\Client();
$promises = [];
foreach ($requetes as $i=>$google_request) {
$promises[] = $client->requestAsync('GET', $google_request ,['delay'=>0.4*$i*1000]); // delay is the trick not to exceed rate limit (in ms)
}
GuzzleHttp\Promise\each_limit($promises, function(){ // function returning the number of concurrent requests
return 100; // 1 or 100 concurrent request(s) don't really change execution time
}, // Fulfilled function
function ($response,$index)use($urls,$fp) { // $urls is used to get the url passed as a parameter in GET request and $fp a csv file pointer
$feed = json_decode($response->getBody(), true); // Get array of results
$this->write_to_csv($feed,$fp,$urls[$index]); // Write to csv
}, // Rejected function
function ($reason,$index) {
if ($reason instanceof GuzzleHttp\Exception\ClientException) {
$message = $reason->getMessage();
var_dump(array("error"=>"error","id"=>$index,"message"=>$message)); // You could write the errors to a file or database too
}
})->wait();
I'm working on trace logger of sorts that pushes log message requests onto a Queue on a Service Bus, to later be picked off by a worker role which would insert them into the table store. While running on my machine, this works just fine (since I'm the only one using it), but once I put it up on a server to test, it produced the following error:
HTTP_Request2_MessageException: Malformed response: in D:\home\site\wwwroot\vendor\pear-pear.php.net\HTTP_Request2\HTTP\Request2\Adapter\Socket.php on line 1013
0 HTTP_Request2_Response->__construct('', true, Object(Net_URL2)) D:\home\site\wwwroot\vendor\pear-pear.php.net\HTTP_Request2\HTTP\Request2\Adapter\Socket.php:1013
1 HTTP_Request2_Adapter_Socket->readResponse() D:\home\site\wwwroot\vendor\pear-pear.php.net\HTTP_Request2\HTTP\Request2\Adapter\Socket.php:139
2 HTTP_Request2_Adapter_Socket->sendRequest(Object(HTTP_Request2)) D:\home\site\wwwroot\vendor\pear-pear.php.net\HTTP_Request2\HTTP\Request2.php:939
3 HTTP_Request2->send() D:\home\site\wwwroot\vendor\microsoft\windowsazure\WindowsAzure\Common\Internal\Http\HttpClient.php:262
4 WindowsAzure\Common\Internal\Http\HttpClient->send(Array, Object(WindowsAzure\Common\Internal\Http\Url)) D:\home\site\wwwroot\vendor\microsoft\windowsazure\WindowsAzure\Common\Internal\RestProxy.php:141
5 WindowsAzure\Common\Internal\RestProxy->sendContext(Object(WindowsAzure\Common\Internal\Http\HttpCallContext)) D:\home\site\wwwroot\vendor\microsoft\windowsazure\WindowsAzure\Common\Internal\ServiceRestProxy.php:86
6 WindowsAzure\Common\Internal\ServiceRestProxy->sendContext(Object(WindowsAzure\Common\Internal\Http\HttpCallContext)) D:\home\site\wwwroot\vendor\microsoft\windowsazure\WindowsAzure\ServiceBus\ServiceBusRestProxy.php:139
7 WindowsAzure\ServiceBus\ServiceBusRestProxy->sendMessage('<queuename>/mes…', Object(WindowsAzure\ServiceBus\Models\BrokeredMessage)) D:\home\site\wwwroot\vendor\microsoft\windowsazure\WindowsAzure\ServiceBus\ServiceBusRestProxy.php:155
⋮
I've seen previous posts that describe similar issues; Namely:
Windows Azure PHP Queue REST Proxy Limit (Stack Overflow)
Operations on HTTPS do not work correctly (GitHub)
That imply that this is a known issue regarding the PHP Azure Storage libraries, where there are a limited amount of HTTPS connections allowed. Before requirements were changed, I was accessing the table store directly, and ran into this same issue, and fixed it in the way the first link describes.
The problem is that the Service Bus endpoint in the connection string, unlike Table Store (etc.) connection string endpoints, MUST be 'HTTPS'. Trying to use it with 'HTTP' will return a 400 - Bad Request error.
I was wondering if anyone had any ideas on a potential workaround. Any advice would be greatly appreciated.
Thanks!
EDIT (After Gary Liu's Comment):
Here's the code I use to add items to the queue:
private function logToAzureSB($source, $msg, $severity, $machine)
{
// Gather all relevant information
$msgInfo = array(
"Severity" => $severity,
"Message" => $msg,
"Machine" => $machine,
"Source" => $source
);
// Encode it to a JSON string, and add it to a Brokered message.
$encoded = json_encode($msgInfo);
$message = new BrokeredMessage($encoded);
$message->setContentType("application/json");
// Attempt to push the message onto the Queue
try
{
$this->sbRestProxy->sendQueueMessage($this->azureQueueName, $message);
}
catch(ServiceException $e)
{
throw new \DatabaseException($e->getMessage, $e->getCode, $e->getPrevious);
}
}
Here, $this->sbRestProxy is a Service Bus REST Proxy, set up when the logging class initializes.
On the recieving end of things, here's the code on the Worker role side of this:
public override void Run()
{
// Initiates the message pump and callback is invoked for each message that is received, calling close on the client will stop the pump.
Client.OnMessage((receivedMessage) =>
{
try
{
// Pull the Message from the recieved object.
Stream stream = receivedMessage.GetBody<Stream>();
StreamReader reader = new StreamReader(stream);
string message = reader.ReadToEnd();
LoggingMessage mMsg = JsonConvert.DeserializeObject<LoggingMessage>(message);
// Create an entry with the information given.
LogEntry entry = new LogEntry(mMsg);
// Set the Logger to the appropriate table store, and insert the entry into the table.
Logger.InsertIntoLog(entry, mMsg.Service);
}
catch
{
// Handle any message processing specific exceptions here
}
});
CompletedEvent.WaitOne();
}
Where Logging Message is a simple object that basically contains the same fields as the Message Logged in PHP (Used for JSON Deserialization), LogEntry is a TableEntity which contains these fields as well, and Logger is an instance of a Table Store Logger, set up during the worker role's OnStart method.
This was a known issue with the Windows Azure PHP, which hasn't been looked at in a long time, nor has it been fixed. In the time between when I posted this and now, We ended up writing a separate API web service for logging, and had our PHP Code send JSON strings to it over cURL, which works well enough as a temporary work around. We're moving off of PHP now, so this wont be an issue for much longer anyways.
A very open question which I need some advice on and more importantly pointers in the right direction.
I'm looking at using openstack for my private cloud (currently using VMware) as the main aim is to be able to launch a new VM instance from within our web application so this could be trigger via a php page to deploy new apache worker server for example. The next aim is to develop our code to be able to see when a server load is getting high or needs more worker servers to preform a task to auto launch an instance?
I've been looking at the openstack API to see if this is the best approach? But also looking at juju to see if you can use charms to do this and seeing if the api for juju to is best?
The aim is get this working with VMware or to replace vmware.
My current setup is running openstack on a laptop using nova as the storage so any help with the pointers would be great
I know its a open question
Well there is an SDK page listing many of the OpenStack API client SDKs that exist.
Ref:
https://wiki.openstack.org/wiki/SDKs#PHP
Listed in there are two PHP SDKs for OpenStack currently:
Ref:
https://github.com/rackspace/php-opencloud
https://github.com/zendframework/ZendService_OpenStack
I wouldn't use Juju as an interface. And frankly I am not sure OpenStack is the right tool for what you are doing. But, if you want to play with devstack and get an idea, I think rackspace's php client SDK is probably a good start. Devstack is not a bad way to get that experience either.
example of spinning up a server with php-opencloud:
$server = $compute->server();
try {
$response = $server->create(array(
'name' => 'My lovely server',
'image' => $ubuntu,
'flavor' => $twoGbFlavor
));
} catch (\Guzzle\Http\Exception\BadResponseException $e) {
// No! Something failed. Let's find out:
$responseBody = (string) $e->getResponse()->getBody();
$statusCode = $e->getResponse()->getStatusCode();
$headers = $e->getResponse()->getHeaderLines();
echo sprintf("Status: %s\nBody: %s\nHeaders: %s", $statusCode, $responseBody, implode(', ', $headers));
}
This would be a polling function:
use OpenCloud\Compute\Constants\ServerState;
$callback = function($server) {
if (!empty($server->error)) {
var_dump($server->error);
exit;
} else {
echo sprintf(
"Waiting on %s/%-12s %4s%%",
$server->name(),
$server->status(),
isset($server->progress) ? $server->progress : 0
);
}
};
$server->waitFor(ServerState::ACTIVE, 600, $callback);
I need to integrate vTiger CRM with Quickbooks 2010. So it is PHP. QB server and Linux box running vTiger are sitting in the same office. I try to figure out if there is a way to make PHP talk directly to QB server without using web connector. We need real time data access. Not like real 'real time', but report needs to be generated in like 5-30 seconds. Only reading of data, no writing. Any way to send/receive qbXML directly to QB server?
Thanks in advance.
Michael
This can be done... but be aware that QuickBooks isn't always the most reliable piece of software to integrate with for real-time data access - connections to QuickBooks tend to be both slow and fragile, which is why you usually see a queued model with the Web Connector.
What you could do is write a custom C# component (look at the examples included with the SDK, specifically this one: C:\Program Files (x86)\Intuit\IDN\QBSDK12.0\samples\qbdt\c-sharp\qbXML\CustomerAdd) which opens a connection to QuickBooks, and then does whatever.
"Whatever" might be something like:
Run a query against QuickBooks, and then HTTP POST the data back to vTiger
or
Open a socket and listen for incoming requests, in which case PHP could connect to the socket and feed the component requests which it could relay to QuickBooks, and then return the response
Be aware that generating a report in QuickBooks every 5 seconds is probably going to slow QuickBooks down considerably. Depending on how much data you're talking and what filters, it might not even be possible to get QuickBooks to generate a report that quickly.
Likewise, be aware that certain things cause connections to QuickBooks to fail, so you'll have to handle those gracefully. Specifically:
QuickBooks automatic updates will lock out integrated apps
If you have too many users logged in at once you'll get locked out
If someone else goes into single-user mode you'll get locked out
etc.
Maybe if you can provide more detail about why you need such frequent access to reports, we can provide some additional information.
Your code would look something like this:
while (true)
{
string input = "your fancy qbXML request goes here";
RequestProcessor2 rp = null;
string ticket = null;
string response = null;
try
{
rp = new RequestProcessor2 ();
rp.OpenConnection("", "Your App Name");
ticket = rp.BeginSession("", QBFileMode.qbFileOpenDoNotCare );
response = rp.ProcessRequest(ticket, input);
}
catch( System.Runtime.InteropServices.COMException ex )
{
MessageBox.Show( "COM Error Description = " + ex.Message, "COM error" );
return;
}
finally
{
if( ticket != null )
{
rp.EndSession(ticket);
}
if( rp != null )
{
rp.CloseConnection();
}
};
// Do something with the response here
...
}