Reading it at http://blog.programmableweb.com/2007/04/02/12-ways-to-limit-an-api/ I wondered how to accomplish time based limits (ie. 1 call per second). Well, if authentication required, the way is to compare the seconds tracked by PHP's time function (for instance, if ($previous_call_time == $current_call_time) { ... })? Any other suggestions?
Use a simple cache:
if(filemtime("cache.txt") < (int)$_SERVER["REQUEST_TIME"] - 3600) { // 3600 is one hour in seconds
$data = file_get_contents("http://remote.api/url/goes/here");
file_put_contents("cache.txt", $data);
} else
$data = file_get_contents("cache.txt");
Something like this will save the value of the API and let you get the information whenever you want while still limiting how often you actually pull date from the feed.
Hope this helps!
The more general limitations criteria is "number of calls per 'period'". For example - per hour, like twitter does.
You can track number of currently performed requests by adding a record to mysql for each request.
The more performant solution in this way is to use memcached and to increment the "key" (user_id + current_hour).
Something to consider: You can use an external service to do this instead of building it yourself. E.g. my company, WebServius ( http://www.webservius.com ) currently supports configurable per-API and per-API-key throttling, and we are likely going to be adding even more features such as "adaptive throttling" (automatically throttle usage when API becomes less responsive).
Related
I have a PHP app that is overloading an external API with too many calls per minute. I need to limit it to only 20 calls a minute, but I can't seem to figure it out. I have researched the issue and found this and that, but there is a lack of proper documentation and I don't know how it would work. I understand that this is called "rate limiting", but I guess I skipped that in school.
My app is just sending cURL requests in a loop. I have several loops running in the program to gather all this information together. I could just limit the one loop with a timer for 20 per minute, but I have 17 loops running and I have loops within loops. Is it possible to just limit all the cURL requests within my PHP app with a single helper or something and not edit all my code?
There is no way to rate limit PHP functions using any built-in features. You could write some simple wrapper which would call the API only a given amount of times per minute. A crude example would look like this:
function callAPI($api) {
static $lastRequest;
$maxRequestsPerMin = 20;
if (isset($lastRequest)) {
$delay = 60 / $maxRequestsPerMin; // 60 seconds / $maxRequestsPerMin
if ((microtime(true) - $lastRequest) < $delay) {
// Sleep until the delay is reached
$sleepAmount = ($delay - microtime(true) + $lastRequest) * (1000 ** 2);
usleep($sleepAmount);
}
}
$lastRequest = microtime(true);
// Call you API here
}
However, this will only rate-limit this particular script. If you execute another one, then you will start another counter. Alternatively you could store some round robin table either in a flat file or in a database, and check against it every time you want to call the API.
For advanced usages you should look into message queues or ReactPHP. You do not want to hang your server if such functionality would be exposed to the end-users.
My website has a script that will call an external API when a user visits a particular page.
An API request will be made when the page is accessed and a response is returned in xml format.
I am using the usual curl requests.
Right now, due to new implementations on the API side, if the API is getting too much requests, it will throw an exception and deny the request.
I want to limit the total calls to the API from my website to only 8 times per second.
How can I achieve this? Someone suggested me about queuing the requests but I've never done something like this before and I'm having a hard time finding a solution.
Sorry if my English has errors. Any help is appreciated.
For example: if 100 users accessed the web-page all at the same time, I need to queue those API requests 8 after 8 per second and so on until all are done.
I have give suggest you to use one api generate to create token and match token on every request and do expiry or delete token after some time. So may be resolve your multiple request issues.
$currentCount=0;
$currentSeconds;
function callAPI()
{
if($currentCount<8 || date("s") != $currentSeconds)
{
if(date("s") != $currentSeconds)
{
$currentCount=0;
}
$currentSeconds=date("s");
//call your API here
$currentCount++;
}
}
For each API call:
-Record the current time (just the seconds) in a variable.
-Make your API call.
-Increment a call counter.
-Check again if current seconds equal the previously stored value and if your call counter is under 8. If your call counter is under 8, you may make another call.
You can delay the API request for microseconds, here is sample code
usleep(1250000);//1 sec = 10,000,00 ms
function _callAPI(){
// Your code here
}
When a user visits your site, the request will fire after a few microseconds in this way you can delay the request.
You can also maintain a log when a request is fired for the API and based on the previous request, dealy the next request.
Value of 8 call per second is low, so can save in database each call attempt and calculate number of calls per last 5 second every time.
For large values usually used counters in nosql database like Cassandra or Aerospike.
I.e for each request you get current time and increase counter name "counter"+second until you got your desired limit.
Aerospike is best for this if load is really high(1000+ cps), it give very low latency.
Cassandra is simpler to use and require less memory.
Even less memory is memcashed.
I've run into a bit of a pickle. Usually I can find solutions to my problems by some extensive googling (to the right SO thread), but not this time.
I'm using an API that lets me access their data by cURL, and this API has a request limit of 500 requests per 10 minutes. If this limit is repeatedly exceeded, your API key gets suspended.
My application, written in PHP, frequently makes requests through different pages - sometimes simultaneously. I need a solution that ensures that I'm under the request limit at all times. I need to be able to check and update a variable that resets every 10 minutes, from all scripts - at the same time.
Can this even be done? How should I approach this problem?
I'm not asking for sourcecode to a perfect solution, I'd just like some pointers to how this could be solved, or if it can't be solved - what's the alternate approach?
Thank you in advance.
Sounds like memcache is what your looking for. It can hold variables like PHP objects or base types and give them as well an expiration date. Just google for the memcached extensions for PHP and look over here: http://php.net/manual/en/class.memcache.php.
Furhermore, it can be accessed via TCP and has some failover capabilities. AWS is offering a service compatible with the memcache API calling it ElastiCache
If pages are run by different users, only thing i can think of is storing in a table.
Do something like
$datestring = substr(date("Y-m-d H:i"),0, -1);
in this way you have always same string for minutes 2016-11-07 17:00:00 - 2016-11-07 17:09:59.
Then store in a table with this mechanism:
"INSERT INTO table (datestring, count) VALUES ('$datestring', '1') ON DUPLICATE KEY UPDATE count = count + 1";
of course put datestring char(15) as a unique key.
you check your counting with a select FROM.... comparing datestring you just created and you get no rows at all ora a row with a count value.
This is a solution, of course there are many.
I was playing around with the Google Complete API looking for a quick way to get hold of the top 26 most searched terms for various question prefixes - one for each letter of the alphabet.
I wouldn't count myself a programmer but it seemed like a fun task!
My script works fine locally but it takes too long on my shared server and times out after 30 seconds - and as it's shared I can't access the php.ini to lengthen the max execution time.
It made me wonder if there was a more efficient way of making the requests to the API, here is my code:
<?php
$prep = $_POST['question'];
for($i=0;$i<26;$i++){
$letters = range('a','z');
$letter = $letters[$i];
$term = $prep . $letter;
if(!$xml=simplexml_load_file('http://google.com/complete/search?output=toolbar&q=' . $term)){
trigger_error('Error reading XML file',E_USER_ERROR);
}
do{
$count = 1;
$result = ucfirst($xml->CompleteSuggestion->suggestion->attributes()->data);
$queries = number_format((int)$xml->CompleteSuggestion->num_queries->attributes()->int);
echo '<p><span>' . ucfirst($letter) . ':</span> ' . $result . '?</p>';
echo '<p class="queries">Number of queries: ' . $queries . '</p><br />';
} while ($count < 0);
}
?>
I also wrote a few lines that fed the question in to the Yahoo Answers API, which worked pretty well although it made the results take even longer and I couldn't exact match on the search term through the API so I got a few odd answers back!
Basically, is the above code the most efficient way of calling an API multiple times?
Thanks,
Rich
You should using user perspective to re-look into this issue, ask yourself,
Will you like to wait 30 seconds for a web page to load?
Obviously you dun want
How can I make the page load faster?
You are depending on an external resource (google api)
and not just calling once, but 26 times asynchronously
So, if you change the above synchronously,
the total time is reduced form 26 to 1 (with the expenses of network bandwidth)
Take a look at http://php.net/manual/en/function.curl-multi-exec.php,
here is first step of optimization
If you get the above done,
your time spent on external resource could reduce up to 95%
Will this good enough ?
Obviously not yet
Any call to external resource is not reliable, even is google
if the network down, DNS not resolvable, your page is going down too
How to prevent that ?
You need cache, basically the logic is :-
search for existing cache, if found, return from cache
if not, query google api synchronously (from a to z)
store the result into cache
return the result
However, on-demand process is still not ideal (first user issue the request have to wait longest),
if you know the permutation of user input (hopefully not that big),
you can use a scheduler (cronjob) to periodically pull result from google api,
and store the result locally
I recommend using cron jobs for this kind of work. This way you can either change the max execution time with a parameter or splitt the work into multiple operations and run the cron job more regulary to run one operation after another.
I'm attempting to make a php script that can load the current weather forecast and it uses a bit of XML pre-processing to digest the input, however it is accessed quite often and reloaded. The problem begins with my current host, which yes I do understand why, limits the amount of processing power a script takes up.
Currently takes an entire process for ever execution, which is around 3 seconds per execution. I'm limited to 12, yet I get quite a few pings.
My question to you guys is: What methods, if any, can I use to cache the output of a script so that it does not have to pre-process something it already did 5 minutes ago. Since it is weather, I can have a time difference of up to 2 hours.
I am quite familiar with php too, so don't worry xD.
~Thank you very much,
Jonny :D
You could run a cronjob that would generate the weather forecast data and then just display the whole thing from cache. You could use APC so it is always loaded in memory (plus all other added advantages).
The Zend Framework provides the Zend_Cache object with multiple backends (File, memcached, APD). Or you can roll your own with something like:
$cachFile = "/path/to/cache/file";
$ttl = 60; // 60 second time to live
if (!file_exists($cacheFile) || time()-filemtime($cacheFile) > $ttl) {
$data = getWeatherData(); // Go off and get the data
file_put_contents(serialize($cacheFile), $data);
} else {
$data = unserialize(file_get_contents($cacheFile));
}
need a code snippet to see what kind of processing you are doing. consider using xdebug to better optimize your code.
Also you may use a benchmarking tool such as AB to see how many processes your server can handle.
there are several different caching mechanisms available but without seeing what kind of process you are doing it is hard to say...
3 seconds is an extremely long execution time, as already asked, some cold would be nice to see how you process the 'input' and in what format said input is in.
A quick and dirty article about caching out of script to file is found here:
http://codestips.com/?p=153