I have 3 functions work(), timeLimits(), and search(). I'm trying to do some check in timeLimits() and if some of the returns are true the script becomes blocked for a certain time.
Everything works fine with checks and blocks. What isn't working is updating the variable which holds the time until when will be blocked and checking when the block ends.
So, here are the functions:
const TOTAL_BLOCK_TIME = 20; // sec
const TOTAL_OPERATING_TIME_BEFORE_BLOCK = 10; // sec
private $worker;
private $startTime;
private $afterBlock;
private $isBlocked = false;
public function __construct()
{
$this->worker->addServer();
$this->worker->addFunction("myfunction", array($this, "search"));
$this->startTime = time();
$this->afterBlock = time() - 1;
}
public function work()
{
while (true) {
$this->worker->work();
}
}
public function timeLimits()
{
$currentTime = time();
$timePassed = $currentTime - $this->startTime;
if ($timePassed > self::TOTAL_OPERATING_TIME_BEFORE_BLOCK) {
$this->afterBlock = $currentTime + self::TOTAL_BLOCK_TIME;
return true;
}
return false;
}
public function search($job)
{
if ( $this->timeLimits() ) {
return 'Blocked until:'.$this->afterBlock;
}
// db query, the rest of the code
// it wont be executed if $this->timeLimits() return true
}
The problem: When timeLimits() return true and the script is blocked each new request updates the $this->afterBlock again. I guess is because I have $this->afterBlock = $currentTime + self::TOTAL_WORKER_SLEEP_TIME; in the if condition and each time it is in the condition the variable gets an update.
What I don't know is how to update this variable only once e.g. when it is entered in the condition, not every time I call if ( $this->timeLimits() ) { ... }
Can anyone help here?
You can add a flag to keep track of the block status and update the $this->afterBlock only when the flag is false.
const TOTAL_BLOCK_TIME = 20; // sec
const TOTAL_OPERATING_TIME_BEFORE_BLOCK = 10; // sec
private $isBlocked = false;
public function work()
{
while (true) {
$this->work();
}
}
public function timeLimits()
{
$currentTime = time();
$timePassed = $currentTime - $this->startTime;
if ($timePassed > self::TOTAL_OPERATING_TIME_BEFORE_BLOCK && !$this->isBlocked) {
$this->afterBlock = $currentTime + self::TOTAL_BLOCK_TIME;
$this->isBlocked = true;
}
return $this->isBlocked && $currentTime < $this->afterBlock;
}
public function search($job)
{
if ( $this->timeLimits() ) {
return 'Blocked until:'.$this->afterBlock;
}
// db query, the rest of the code
// it wont be executed if $this->timeLimits() return true
}
Related
I use a variety of 3rd party web APIs, and many of them enforce rate limiting. It would be very useful to have a fairly generic PHP library that I could rate limit my calls with. I can think of a few ways to do it, perhaps by putting calls into a queue with a timestamp of when the call can be made, but I was hoping to avoid reinventing the wheel if someone else has already done this well.
You can do rate limiting with the token bucket algorithm. I implemented that for you in PHP: bandwidth-throttle/token-bucket
:
use bandwidthThrottle\tokenBucket\Rate;
use bandwidthThrottle\tokenBucket\TokenBucket;
use bandwidthThrottle\tokenBucket\storage\FileStorage;
$storage = new FileStorage(__DIR__ . "/api.bucket");
$rate = new Rate(10, Rate::SECOND);
$bucket = new TokenBucket(10, $rate, $storage);
$bucket->bootstrap(10);
if (!$bucket->consume(1, $seconds)) {
http_response_code(429);
header(sprintf("Retry-After: %d", floor($seconds)));
exit();
}
I realize this is an old thread but thought I'd post my solution since it was based on something else I found on SE. I looked for a while for an answer myself but had trouble finding something good. It's based on the Python solution discussed here, but I've added support for variable-sized requests and turned it into a function generator using PHP closures.
function ratelimiter($rate = 5, $per = 8) {
$last_check = microtime(True);
$allowance = $rate;
return function ($consumed = 1) use (
&$last_check,
&$allowance,
$rate,
$per
) {
$current = microtime(True);
$time_passed = $current - $last_check;
$last_check = $current;
$allowance += $time_passed * ($rate / $per);
if ($allowance > $rate)
$allowance = $rate;
if ($allowance < $consumed) {
$duration = ($consumed - $allowance) * ($per / $rate);
$last_check += $duration;
usleep($duration * 1000000);
$allowance = 0;
}
else
$allowance -= $consumed;
return;
};
}
It can be used to limit just about anything. Here's a stupid example that limits a simple statement at the default five "requests" per eight seconds:
$ratelimit = ratelimiter();
while (True) {
$ratelimit();
echo "foo".PHP_EOL;
}
Here's how I'm using it to limit batched requests against the Facebook Graph API at 600 requests per 600 seconds based on the size of the batch:
$ratelimit = ratelimiter(600, 600);
while (..) {
..
$ratelimit(count($requests));
$response = (new FacebookRequest(
$session, 'POST', '/', ['batch' => json_encode($requests)]
))->execute();
foreach ($response->..) {
..
}
}
Hope this helps someone!
This is essentially the same as #Jeff's answer, but I have tidied the code up a lot and added PHP7.4 type/return hinting.
I have also published this as a composer package: https://github.com/MacroMan/rate-limiter
composer require macroman/rate-limiter
/**
* Class RateLimiter
*
* #package App\Components
*/
class Limiter
{
/**
* Limit to this many requests
*
* #var int
*/
private int $frequency = 0;
/**
* Limit for this duration
*
* #var int
*/
private int $duration = 0;
/**
* Current instances
*
* #var array
*/
private array $instances = [];
/**
* RateLimiter constructor.
*
* #param int $frequency
* #param int $duration #
*/
public function __construct(int $frequency, int $duration)
{
$this->frequency = $frequency;
$this->duration = $duration;
}
/**
* Sleep if the bucket is full
*/
public function await(): void
{
$this->purge();
$this->instances[] = microtime(true);
if (!$this->is_free()) {
$wait_duration = $this->duration_until_free();
usleep($wait_duration);
}
}
/**
* Remove expired instances
*/
private function purge(): void
{
$cutoff = microtime(true) - $this->duration;
$this->instances = array_filter($this->instances, function ($a) use ($cutoff) {
return $a >= $cutoff;
});
}
/**
* Can we run now?
*
* #return bool
*/
private function is_free(): bool
{
return count($this->instances) < $this->frequency;
}
/**
* Get the number of microseconds until we can run the next instance
*
* #return float
*/
private function duration_until_free(): float
{
$oldest = $this->instances[0];
$free_at = $oldest + $this->duration * 1000000;
$now = microtime(true);
return ($free_at < $now) ? 0 : $free_at - $now;
}
}
Usage is the same
use RateLimiter\Limiter;
// Limit to 6 iterations per second
$limiter = new Limiter(6, 1);
for ($i = 0; $i < 50; $i++) {
$limiter->await();
echo "Iteration $i" . PHP_EOL;
}
As an alternate, I've (in the past) created a "cache" folder that stored the API calls so if I try to make the same call again, within a specific time range, it grabs from the cache first (more seamless) until it's okay to make a new call. May end up with archived information in the short term, but saves you from the API blocking you in the long term.
I liked mwp's answer and I wanted to convert it to OO to make me feel warm and fuzzy. I ended up drastically rewriting it to the point that it is totally unrecognizable from his version. So, here is my mwp-inspired OO version.
Basic explanation: Every time await is called, it saves the current timestamp in an array and throws out all old timestamps that arent relevant anymore (greater than the duration of the interval). If the rate limit is exceeded, then it calculates the time until it will be freed up again and sleeps until then.
Usage:
$limiter = new RateLimiter(4, 1); // can be called 4 times per 1 second
for($i = 0; $i < 10; $i++) {
$limiter->await();
echo microtime(true) . "\n";
}
I also added a little syntactic sugar for a run method.
$limiter = new RateLimiter(4, 1);
for($i = 0; $i < 10; $i++) {
$limiter->run(function() { echo microtime(true) . "\n"; });
}
<?php
class RateLimiter {
private $frequency;
private $duration;
private $instances;
public function __construct($frequency, $duration) {
$this->frequency = $frequency;
$this->duration = $duration;
$this->instances = [];
}
public function await() {
$this->purge();
$this->instances[] = microtime(true);
if($this->is_free()) {
return;
}
else {
$wait_duration = $this->duration_until_free();
usleep(floor($wait_duration));
return;
}
}
public function run($callback) {
if(!is_callable($callback)) {
return false;
}
$this->await();
$callback();
return true;
}
public function purge() {
$this->instances = RateLimiter::purge_old_instances($this->instances, $this->duration);
}
public function duration_until_free() {
return RateLimiter::get_duration_until_free($this->instances, $this->duration);
}
public function is_free() {
return count($this->instances) < $this->frequency;
}
public static function get_duration_until_free($instances, $duration) {
$oldest = $instances[0];
$free_at = $oldest + $duration * 1000000;
$now = microtime(true);
if($free_at < $now) {
return 0;
}
else {
return $free_at - $now;
}
}
public static function purge_old_instances($instances, $duration) {
$now = microtime(true);
$cutoff = $now - $duration;
return array_filter($instances, function($a) use ($duration, $cutoff) {
return $a >= $cutoff;
});
}
}
PHP source code to limit access to your API by allowing a request every 5 seconds for any user and using Redix.
Installing the Redis/Redix client :
composer require predis/predis
Download Redix (https://github.com/alash3al/redix/releases) depending on your operating system, then start the service :
./redix_linux_amd64
The following answer indicates that Redix is listening on ports 6380 for RESP protocol and 7090 for HTTP protocol.
redix resp server available at : localhost:6380
redix http server available at : localhost:7090
In your API, add the following code to the header :
<?php
require_once 'class.ratelimit.redix.php';
$rl = new RateLimit();
$waitfor = $rl->getSleepTime($_SERVER['REMOTE_ADDR']);
if ($waitfor>0) {
echo 'Rate limit exceeded, please try again in '.$waitfor.'s';
exit;
}
// Your API response
echo 'API response';
The source code for the script class.ratelimit.redix.php is :
<?php
require_once __DIR__.'/vendor/autoload.php';
Predis\Autoloader::register();
class RateLimit {
private $redis;
const RATE_LIMIT_SECS = 5; // allow 1 request every x seconds
public function __construct() {
$this->redis = new Predis\Client([
'scheme' => 'tcp',
'host' => 'localhost', // or the server IP on which Redix is running
'port' => 6380
]);
}
/**
* Returns the number of seconds to wait until the next time the IP is allowed
* #param ip {String}
*/
public function getSleepTime($ip) {
$value = $this->redis->get($ip);
if(empty($value)) {
// if the key doesn't exists, we insert it with the current datetime, and an expiration in seconds
$this->redis->set($ip, time(), self::RATE_LIMIT_SECS*1000);
return 0;
}
return self::RATE_LIMIT_SECS - (time() - intval(strval($value)));
} // getSleepTime
} // class RateLimit
I'm trying to convert a certain excel sheet to a php app.
Within the excel sheet you could do time calculations as eg:
A starting time A1 (could be negative), and for every day you add or substract a certain amount of hours and minutes.
- A1= -10:59
- A2= -7:36
- A3= -18:35 (A1 + A2)
- B2= 4:24
- B3= -14:11 (A3 + B2)
And so on ...
Here is some code i've tested with without succes ofcourse...
$startTime = new \DateTime('00:00:00');
$startVal = new \DateInterval('PT10H59M');
$startTime->sub($startVal); // -10:59:00
$timeSpan = new \DateInterval('PT7H36M'); // 7:36:00
$addTime = $startTime->add($timeSpan);
What is the best way to solve this in PHP? I've been testing around with DateTime but this won't allow me te start with a negative time value.
Hope someone can give me a hand.
Kind regards,
Jochem
I have written a simple class to handle this. It works for your given example but I was unable to test further as I cannot get negative times in my version of Excel. Let me know if it fails at some point.
class MyTime
{
private $positive = true;
private $hour=0;
private $minute=0;
/**
* MyTime constructor.
* Split the given time string into hours and minutes and whether it is positive or negative
*
* #param string $timeString In the format '-10:59', '4:35', or optionally just minutes '24'
*/
public function __construct($timeString)
{
if(!empty($timeString))
{
if(strpos($timeString,'-')===0)
{
$this->positive = false;
$timeString = substr($timeString,1);
}
$timeParts = explode(':',$timeString);
if(!empty($timeParts))
{
if(count($timeParts) == 1)
{
$this->hour = 0;
$this->minute = intval($timeParts[0]);
}
else
{
$this->hour = intval($timeParts[0]);
$this->minute = intval($timeParts[1]);
}
}
}
}
public function getHour()
{
return $this->hour;
}
public function getMinute()
{
return $this->minute;
}
/**
* Convert into minutes either negative or positive
* #return int
*/
public function inMinutes()
{
$minutes = ($this->hour*60)+$this->minute;
if(!$this->positive)
{
$minutes *= -1;
}
return $minutes;
}
/**
* Convert back to a string for output
* #return string
*/
public function __toString()
{
return ($this->positive?'':'-').$this->getHour().':'.str_pad($this->getMinute(),2,'0',STR_PAD_LEFT);
}
/**
* Add on the given time which could be negative
* #param MyTime $time
*
* #return $this
*/
public function add(MyTime $time)
{
$newMinutes = $this->inMinutes() + $time->inMinutes();
if($newMinutes<0)
{
$this->hour = (int) ceil($newMinutes/60);
}
else
{
$this->hour = (int) floor($newMinutes/60);
}
$this->minute = abs($newMinutes-($this->hour*60));
if($newMinutes<0)
{
$this->positive = false;
$this->hour *= -1;
}
else
{
$this->positive = true;
}
return $this;
}
}
$time = new MyTime('-10:59');
echo $time."\n";
echo $time->add(new MyTime('-7:36'))."\n";
echo $time->add(new MyTime('4:24'))."\n";
echo $time->add(new MyTime('18:32'))."\n";
$time = new MyTime('10:59');
echo $time."\n";
echo $time->add(new MyTime('-59'))."\n";
I'm having problems creating a cookie with laravel, the thing is that sometimes the value of this cookie changes into a null intead of its real value which makes my system give errors that I've programed to show in case this cookie dont exist.
This happens when a reaload the page, I've debugged with dd() and every time that I reload the page the value changes from'activo' to 'null' and from 'null' to 'activo', why does this happen?
My code:
public function store_inicio(Request $request)
{
$empresa_id = $request->input('empresa_id');
$empresa = Empresa::find($empresa_id);
$cierre = $empresa->inicio_caja()->orderBy('id', 'desc')->first();
if (isset($cierre)) {
if (!isset($cierre->cierre)) {
Cookie::queue('estado_caja', 'activo', 180);
Cookie::queue('numero_caja', $cierre->caja, 180);
} else {
$this->crear_inicio($request, $empresa);
}
} else {
$this->crear_inicio($request, $empresa);
}
return redirect('ventas');
}
private function crear_inicio($request, $empresa)
{
$aux_usuario = $request->input('usuario_inicio_caja');
$usuario = User::where('name', $aux_usuario)->first();
$numero_caja = $request->input('caja_inicio_caja');
$valor = $request->input('valor_inicio_caja');
$inicio_caja = new Inicio_caja();
$inicio_caja->user_id = $usuario->id;
$inicio_caja->caja = $numero_caja;
$inicio_caja->valor = $valor;
$inicio_caja->save();
$empresa->inicio_caja()->syncWithoutDetaching($inicio_caja->id);
Cookie::queue('estado_caja', 'activo', 600);
Cookie::queue('numero_caja', $numero_caja, 600);
}
The objective is to continually collect data of the current temperature. But a separate process should analyse the output of that data because I have to tweak the algorithm a lot but I want to avoid downtime so stopping the process is a no-go.
The problem is when I separate these processes, that process 2 would either continually have to make calls to the database or read from a local file to do something with the output generated by 1 process but I want to act upon it immediately and that is expensive in terms of resources.
Would it be possible to reload the class into memory somehow when the file changes by for example writing a function that keeps calculating the MD5 of the file, and if it changes than reload the class somehow? So this separate class should act as a plugin. Is there any way to make that work?
Here is a possible solution. Use Beanstalk (https://github.com/kr/beanstalkd).
PHP class to talk to Beanstalk (https://github.com/pda/pheanstalk)
Run beanstalk.
Create a process that goes into an infinite loop that reads from a Beanstalk queue. (Beanstalk queues are called "Tubes"). PHP processes are not meant to be run for a very long time. The main reason for this is memory. The easiest way to do handle this is to restart the process every once in a while or if memory gets to a certain threshold.
NOTE: What I do is to have the process exit after some fixed time or if it uses a certain amount of memory. Then, I use Supervisor to restart it.
You can put data into Beanstalk as JSON and decode it on the receiving end. The sending and receiving processes need to agree on that format. You could store your work payload in a database and just send the primary key in the queue.
Here is some code you can use:
class BeanstalkClient extends AbstractBaseQueue{
public $queue;
public $host;
public $port;
public $timeout;
function __construct($timeout=null) {
$this->loadClasses();
$this->host = '127.0.0.1';
$this->port = BEANSTALK_PORT;
$this->timeout = 30;
$this->connect();
}
public function connect(){
$this->queue = new \Pheanstalk\Pheanstalk($this->host, $this->port);
}
public function publish($tube, $data, $delay){
$payload = $this->encodeData($data);
$this->queue->useTube($tube)->put($payload,
\Pheanstalk\PheanstalkInterface::DEFAULT_PRIORITY, $delay);
}
public function waitForMessages($tube, $callback=null){
if ( $this->timeout ) {
return $this->queue->watchOnly($tube)->reserve($this->timeout);
}
return $this->queue->watchOnly($tube)->reserve();
}
public function delete($message){
$this->queue->delete($message);
}
public function encodeData($data){
$payload = json_encode($data);
return $payload;
}
public function decodeData($encodedData) {
return json_decode($encodedData, true);
}
public function getData($message){
if ( is_string($message) ) {
throw new Exception('message is a string');
}
return json_decode($message->getData(), true);
}
}
abstract class BaseQueueProcess {
protected $channelName = ''; // child class should set this
// The queue object
public $queue = null;
public $processId = null; // this is the system process id
public $name = null;
public $status = null;
public function initialize() {
$this->processId = getmypid();
$this->name = get_called_class();
$this->endTime = time() + (2 * 60 * 60); // restart every hour
// seconds to timeout when waiting for a message
// if the process isn't doing anything, timeout so they have a chance to do housekeeping.
$queueTimeout = 900;
if ( empty($this->queue) ) {
$this->queue = new BeanstalkClient($queueTimeout);
}
}
public function receiveMessage($queueMessage) {
$taskData = $this->queue->getData($queueMessage);
// debuglog(' Task Data = ' . print_r($taskData, true));
if ( $this->validateTaskData($taskData) ) {
// process the message
$good = $this->didReceiveMessage($taskData);
if ( $good !== false ) {
// debuglog("Completing task {$this->taskId}");
$this->completeTask($queueMessage);
}
else {
$this->failTask($queueMessage);
}
}
else {
// Handle bad message
$this->queue->delete($queueMessage);
}
}
public function run() {
$this->processName = $this->channelName;
// debuglog('Start ' . $this->processName);
// debuglog(print_r($this->params, true));
while(1) {
$queueMessage = $this->queue->waitForMessages($this->channelName);
if ( ! empty($queueMessage) ) {
$this->receiveMessage($queueMessage);
}
else {
// empty message
// a timeout
// // debuglog("empty message " . get_called_class());
}
$memory = memory_get_usage();
if( $memory > 20000000 ) {
// debuglog('Exit '.get_called_class().' due to memory. Memory:'. ($memory/1024/1024).' MB');
// Supervisor will restart process.
exit;
}
elseif ( time() > $this->endTime ) {
// debuglog('Exit '.get_called_class().' due to time.');
// Supervisor will restart process.
exit;
}
usleep(10);
}
}
public function completeTask($queueMessage) {
//
$this->queue->delete($queueMessage);
}
public function failTask($queueMessage) {
//
$this->queue->delete($queueMessage);
}
}
class MyProcess extends BaseQueueProcess {
public function initialize() {
$this->channelName = 'Temperature';
parent::initialize();
}
public function didReceiveMessage($taskData) {
// debuglog(print_r($taskData, true));
// process data here
// return false if something went wrong
return true;
}
}
//Sender
class WorkSender {
const TubeName = 'Temperature';
const TubeDelay = 0; // Set delay to 0, i.e. don't use a delay.
function send($data) {
$c = BeanstalkClient();
$c->publish(self::TubeName, $data, self::TubeDelay);
}
}
I'm using custom class to connect to asterisk server via php.
Here it's code:
class Asterisk_ami
{
public $ini = array();
function __construct ()
{
$this->ini["con"] = false;
$this->ini["host"] = "127.0.0.1";
$this->ini["port"] = "****";
$this->ini["lastActionID"] = 0;
$this->ini["lastRead"] = array();
$this->ini["sleep_time"]=1.5;
$this->ini["login"] = "****";
$this->ini["password"] = "****";
}
function __destruct()
{
unset ($this->ini);
}
public function connect()
{
$this->ini["con"] = fsockopen($this->ini["host"], $this->ini["port"],$a,$b,10);
if ($this->ini["con"])
{
stream_set_timeout($this->ini["con"], 0, 400000);
}
}
public function disconnect()
{
if ($this->ini["con"])
{
fclose($this->ini["con"]);
}
}
public function write($a)
{
$this->ini["lastActionID"] = rand (10000000000000000,99999999900000000);
fwrite($this->ini["con"], "ActionID: ".$this->ini["lastActionID"]."\r\n$a\r\n\r\n");
$this->sleepi();
return $this->ini["lastActionID"];
}
public function sleepi ()
{
sleep($this->ini["sleep_time"]);
}
public function read()
{
$mm = array();
$b = array();
$mmmArray=array();
$k = 0;
$s = "";
$this->sleepi();
do
{
$s.= fread($this->ini["con"],1024);
sleep(0.005);
$mmm=socket_get_status($this->ini["con"]);
array_push($mmmArray, $mmm);
} while ($mmm['unread_bytes']);
$mm = explode ("\r\n",$s);
$this->ini["lastRead"] = array();
for ($i=0;$i<count($mm);$i++)
{
if ($mm[$i]=="")
{
$k++;
}
$m = explode(":",$mm[$i]);
if (isset($m[1]))
{
$this->ini["lastRead"][$k][trim($m[0])] = trim($m[1]);
}
}
unset ($b);
unset ($k);
unset ($mm);
unset ($mm);
unset ($mmm);
unset ($i);
unset ($s);
var_dump($mmmArray);
return $this->ini["lastRead"];
//return $s;
}
public function init()
{
return $this->write("Action: Login\r\nUsername: ".$this->ini["login"]."\r\nSecret: ".$this->ini["password"]."\r\n\r\n");
}
}
And here is testAsterisk.php where I try it.
include("./lib/asteriskAmi.php");
$a = new Asterisk_ami();
$a->connect();
if ($a->ini["con"])
{
$a->init();
$a->write("Action: GetConfig\r\nFilename: extensions.conf\r\n");
print_r($a->read());
$a->disconnect();
}
I want to get extension.conf config via ami. The problem is that I don't get full config. 16 last string are alwas missing. However when I check GetConfig via asterisk console it returns full config.
As you can see while cycle is interrupted when unread bytes of socket_get_status are 0, I checked them pushing in array and dumping it, and I can see that unread_bytes are actually 0. I tried changing sleep_time and different timeout parametres, the result was the same.
What else can I check? What can cause this mistake? May be I can use some other func?
Really, only proper solution I've found is to use PAMI, instead of custom class. It's more comfortable for using and anyway it gives me full content of extensions.conf.