Rate Limiting By Session PHP Without Library's - php

I'm using PHP for my as my backend for my webservice, I need the webservice to be able to set a rate limit of maximum of 1000 requests within the last 24 hours, based off the users PHP session. Is that possible to do it without using a database and just rate limit using only PHP. I have currently already made a rate limiter for 1 request per second per session, but I am looking to set a rate limit of 1000 requests per session in the last 24 hours. PS I'm new to PHP, any help would be great.
Here the code I did for the 1 per second rate limit.
class gaqsession {
public $lastrequest;
public function __construct() {
$this->lastrequest = time();
}
public function ratelimited() {
if($this->lastrequest == time()) {
$this->lastrequest = time();
return true;
} else {
$this->lastrequest = time();
return false;
}
}
}

You can do this provided your sessions are set up to last at least 24 hours. The only real trick is saving rate counts in the session in a way that allows you to maintain them within a 24 hour window. You can do that with a simple array that uses time as keys. Here's a class that adds some features around an array like that in order to easily manage the data, along with some example code that runs it through it's paces.
/**
* Class RateLimitCounter
* Track number of events by time. Intended to be set on a session.
*/
class RateLimitCounter
{
// The time -> request count data
private $timeline = [];
/**
* Log an event in the timeline
*/
public function increment()
{
$now = time();
if (!array_key_exists($now, $this->timeline))
{
$this->timeline[$now] = 0;
}
$this->timeline[$now]++;
}
/**
* Return the total number of events logged in the counter
* #return int
*/
public function getTotal()
{
return array_sum($this->timeline);
}
/**
* Remove any timeline data older than 24 hours
*/
private function trim()
{
// Get the current time
$now = time();
// Time is in seconds, so subtract 1 day worth of seconds
$timeFloor = $now - 86400;
// Filter out all timeline entries more than 24 hours old
$this->timeline = array_filter($this->timeline, function ($key) use ($timeFloor) {
return $key > $timeFloor;
}, ARRAY_FILTER_USE_KEY);
}
public function __serialize(): array
{
return [
'timeline' => $this->timeline
];
}
/**
* Wake up! Set the timeline data and trim data older than 24 hours
* #param array $data
*/
public function __unserialize(array $data): void
{
$this->timeline = $data['timeline'];
$this->trim();
}
}
/**
* Verify that the rate limit has not been exceeded. Bail out if it has been.
* #param $counter
* #return bool
*/
function rateLimit($counter)
{
$limit = 1000;
if ($counter->getTotal() > $limit)
{
// Do whatever you need to here, throw an exception, redirect to an error page, etc.
exit('Rate limit exceeded' . PHP_EOL);
}
return true;
}
/*
* Instantiate a counter - this is what you would do if you do not already have one on the session
*/
$counter = new RateLimitCounter();
/*
* Simulate some prior activity
* Let's get close to the limit then save to the "session"
*/
for ($i = 0; $i <= 995; $i++)
{
$counter->increment();
}
// Mock session
$dummySession = ['counter' => $counter];
// Serialize the session
$serializedSession = serialize($dummySession);
// Unserialize the session
$session = unserialize($serializedSession);
$counter = $session['counter'];
// Do API calls until we hit our limit. There should be 5 remaining.
while (rateLimit($counter))
{
apiCall();
// Don't forget to increment the counter for each call
$counter->increment();
}
// Dummy function to simulate your API call
function apiCall()
{
echo 'Doing something interesting' . PHP_EOL;
}

Related

Limit the number of PHP API calls [duplicate]

I use a variety of 3rd party web APIs, and many of them enforce rate limiting. It would be very useful to have a fairly generic PHP library that I could rate limit my calls with. I can think of a few ways to do it, perhaps by putting calls into a queue with a timestamp of when the call can be made, but I was hoping to avoid reinventing the wheel if someone else has already done this well.
You can do rate limiting with the token bucket algorithm. I implemented that for you in PHP: bandwidth-throttle/token-bucket
:
use bandwidthThrottle\tokenBucket\Rate;
use bandwidthThrottle\tokenBucket\TokenBucket;
use bandwidthThrottle\tokenBucket\storage\FileStorage;
$storage = new FileStorage(__DIR__ . "/api.bucket");
$rate = new Rate(10, Rate::SECOND);
$bucket = new TokenBucket(10, $rate, $storage);
$bucket->bootstrap(10);
if (!$bucket->consume(1, $seconds)) {
http_response_code(429);
header(sprintf("Retry-After: %d", floor($seconds)));
exit();
}
I realize this is an old thread but thought I'd post my solution since it was based on something else I found on SE. I looked for a while for an answer myself but had trouble finding something good. It's based on the Python solution discussed here, but I've added support for variable-sized requests and turned it into a function generator using PHP closures.
function ratelimiter($rate = 5, $per = 8) {
$last_check = microtime(True);
$allowance = $rate;
return function ($consumed = 1) use (
&$last_check,
&$allowance,
$rate,
$per
) {
$current = microtime(True);
$time_passed = $current - $last_check;
$last_check = $current;
$allowance += $time_passed * ($rate / $per);
if ($allowance > $rate)
$allowance = $rate;
if ($allowance < $consumed) {
$duration = ($consumed - $allowance) * ($per / $rate);
$last_check += $duration;
usleep($duration * 1000000);
$allowance = 0;
}
else
$allowance -= $consumed;
return;
};
}
It can be used to limit just about anything. Here's a stupid example that limits a simple statement at the default five "requests" per eight seconds:
$ratelimit = ratelimiter();
while (True) {
$ratelimit();
echo "foo".PHP_EOL;
}
Here's how I'm using it to limit batched requests against the Facebook Graph API at 600 requests per 600 seconds based on the size of the batch:
$ratelimit = ratelimiter(600, 600);
while (..) {
..
$ratelimit(count($requests));
$response = (new FacebookRequest(
$session, 'POST', '/', ['batch' => json_encode($requests)]
))->execute();
foreach ($response->..) {
..
}
}
Hope this helps someone!
This is essentially the same as #Jeff's answer, but I have tidied the code up a lot and added PHP7.4 type/return hinting.
I have also published this as a composer package: https://github.com/MacroMan/rate-limiter
composer require macroman/rate-limiter
/**
* Class RateLimiter
*
* #package App\Components
*/
class Limiter
{
/**
* Limit to this many requests
*
* #var int
*/
private int $frequency = 0;
/**
* Limit for this duration
*
* #var int
*/
private int $duration = 0;
/**
* Current instances
*
* #var array
*/
private array $instances = [];
/**
* RateLimiter constructor.
*
* #param int $frequency
* #param int $duration #
*/
public function __construct(int $frequency, int $duration)
{
$this->frequency = $frequency;
$this->duration = $duration;
}
/**
* Sleep if the bucket is full
*/
public function await(): void
{
$this->purge();
$this->instances[] = microtime(true);
if (!$this->is_free()) {
$wait_duration = $this->duration_until_free();
usleep($wait_duration);
}
}
/**
* Remove expired instances
*/
private function purge(): void
{
$cutoff = microtime(true) - $this->duration;
$this->instances = array_filter($this->instances, function ($a) use ($cutoff) {
return $a >= $cutoff;
});
}
/**
* Can we run now?
*
* #return bool
*/
private function is_free(): bool
{
return count($this->instances) < $this->frequency;
}
/**
* Get the number of microseconds until we can run the next instance
*
* #return float
*/
private function duration_until_free(): float
{
$oldest = $this->instances[0];
$free_at = $oldest + $this->duration * 1000000;
$now = microtime(true);
return ($free_at < $now) ? 0 : $free_at - $now;
}
}
Usage is the same
use RateLimiter\Limiter;
// Limit to 6 iterations per second
$limiter = new Limiter(6, 1);
for ($i = 0; $i < 50; $i++) {
$limiter->await();
echo "Iteration $i" . PHP_EOL;
}
As an alternate, I've (in the past) created a "cache" folder that stored the API calls so if I try to make the same call again, within a specific time range, it grabs from the cache first (more seamless) until it's okay to make a new call. May end up with archived information in the short term, but saves you from the API blocking you in the long term.
I liked mwp's answer and I wanted to convert it to OO to make me feel warm and fuzzy. I ended up drastically rewriting it to the point that it is totally unrecognizable from his version. So, here is my mwp-inspired OO version.
Basic explanation: Every time await is called, it saves the current timestamp in an array and throws out all old timestamps that arent relevant anymore (greater than the duration of the interval). If the rate limit is exceeded, then it calculates the time until it will be freed up again and sleeps until then.
Usage:
$limiter = new RateLimiter(4, 1); // can be called 4 times per 1 second
for($i = 0; $i < 10; $i++) {
$limiter->await();
echo microtime(true) . "\n";
}
I also added a little syntactic sugar for a run method.
$limiter = new RateLimiter(4, 1);
for($i = 0; $i < 10; $i++) {
$limiter->run(function() { echo microtime(true) . "\n"; });
}
<?php
class RateLimiter {
private $frequency;
private $duration;
private $instances;
public function __construct($frequency, $duration) {
$this->frequency = $frequency;
$this->duration = $duration;
$this->instances = [];
}
public function await() {
$this->purge();
$this->instances[] = microtime(true);
if($this->is_free()) {
return;
}
else {
$wait_duration = $this->duration_until_free();
usleep(floor($wait_duration));
return;
}
}
public function run($callback) {
if(!is_callable($callback)) {
return false;
}
$this->await();
$callback();
return true;
}
public function purge() {
$this->instances = RateLimiter::purge_old_instances($this->instances, $this->duration);
}
public function duration_until_free() {
return RateLimiter::get_duration_until_free($this->instances, $this->duration);
}
public function is_free() {
return count($this->instances) < $this->frequency;
}
public static function get_duration_until_free($instances, $duration) {
$oldest = $instances[0];
$free_at = $oldest + $duration * 1000000;
$now = microtime(true);
if($free_at < $now) {
return 0;
}
else {
return $free_at - $now;
}
}
public static function purge_old_instances($instances, $duration) {
$now = microtime(true);
$cutoff = $now - $duration;
return array_filter($instances, function($a) use ($duration, $cutoff) {
return $a >= $cutoff;
});
}
}
PHP source code to limit access to your API by allowing a request every 5 seconds for any user and using Redix.
Installing the Redis/Redix client :
composer require predis/predis
Download Redix (https://github.com/alash3al/redix/releases) depending on your operating system, then start the service :
./redix_linux_amd64
The following answer indicates that Redix is listening on ports 6380 for RESP protocol and 7090 for HTTP protocol.
redix resp server available at : localhost:6380
redix http server available at : localhost:7090
In your API, add the following code to the header :
<?php
require_once 'class.ratelimit.redix.php';
$rl = new RateLimit();
$waitfor = $rl->getSleepTime($_SERVER['REMOTE_ADDR']);
if ($waitfor>0) {
echo 'Rate limit exceeded, please try again in '.$waitfor.'s';
exit;
}
// Your API response
echo 'API response';
The source code for the script class.ratelimit.redix.php is :
<?php
require_once __DIR__.'/vendor/autoload.php';
Predis\Autoloader::register();
class RateLimit {
private $redis;
const RATE_LIMIT_SECS = 5; // allow 1 request every x seconds
public function __construct() {
$this->redis = new Predis\Client([
'scheme' => 'tcp',
'host' => 'localhost', // or the server IP on which Redix is running
'port' => 6380
]);
}
/**
* Returns the number of seconds to wait until the next time the IP is allowed
* #param ip {String}
*/
public function getSleepTime($ip) {
$value = $this->redis->get($ip);
if(empty($value)) {
// if the key doesn't exists, we insert it with the current datetime, and an expiration in seconds
$this->redis->set($ip, time(), self::RATE_LIMIT_SECS*1000);
return 0;
}
return self::RATE_LIMIT_SECS - (time() - intval(strval($value)));
} // getSleepTime
} // class RateLimit

Redirection in multistep form CakePHP

I am creating a multistep form in CakePHP using http://bakery.cakephp.org/2012/09/09/Multistep-forms.html?
But it's not redirecting well. As I click next in step 1, it redirects me to msf_index.
Probably I'm not proceeding the right step count in the msf_setup method.
Why is this happening ?
Here is the Controller Code that I have just copy-pasted.
class UsersController extends AppController {
/**
* use beforeRender to send session parameters to the layout view
*/
public function beforeRender() {
parent::beforeRender();
$params = $this->Session->read('form.params');
$this->set('params', $params);
}
/**
* delete session values when going back to index
* you may want to keep the session alive instead
*/
public function msf_index() {
$this->Session->delete('form');
}
/**
* this method is executed before starting the form and retrieves one important parameter:
* the form steps number
* you can hardcode it, but in this example we are getting it by counting the number of files that start with msf_step_
*/
public function msf_setup() {
App::uses('Folder', 'Utility');
$usersViewFolder = new Folder(APP.'View'.DS.'Users');
$steps = count($usersViewFolder->find('msf_step_.*\.ctp'));
$this->Session->write('form.params.steps', $steps);
$this->Session->write('form.params.maxProgress', 0);
$this->redirect(array('action' => 'msf_step', 1));
}
/**
* this is the core step handling method
* it gets passed the desired step number, performs some checks to prevent smart users skipping steps
* checks fields validation, and when succeding, it saves the array in a session, merging with previous results
* if we are at last step, data is saved
* when no form data is submitted (not a POST request) it sets this->request->data to the values stored in session
*/
public function msf_step($stepNumber) {
/**
* check if a view file for this step exists, otherwise redirect to index
*/
if (!file_exists(APP.'View'.DS.'Users'.DS.'msf_step_'.$stepNumber.'.ctp')) {
$this->redirect('/users/msf_index');
}
/**
* determines the max allowed step (the last completed + 1)
* if choosen step is not allowed (URL manually changed) the user gets redirected
* otherwise we store the current step value in the session
*/
$maxAllowed = $this->Session->read('form.params.maxProgress') + 1;
if ($stepNumber > $maxAllowed) {
$this->redirect('/users/msf_step/'.$maxAllowed);
} else {
$this->Session->write('form.params.currentStep', $stepNumber);
}
/**
* check if some data has been submitted via POST
* if not, sets the current data to the session data, to automatically populate previously saved fields
*/
if ($this->request->is('post')) {
/**
* set passed data to the model, so we can validate against it without saving
*/
$this->User->set($this->request->data);
/**
* if data validates we merge previous session data with submitted data, using CakePHP powerful Hash class (previously called Set)
*/
if ($this->User->validates()) {
$prevSessionData = $this->Session->read('form.data');
$currentSessionData = Hash::merge( (array) $prevSessionData, $this->request->data);
/**
* if this is not the last step we replace session data with the new merged array
* update the max progress value and redirect to the next step
*/
if ($stepNumber < $this->Session->read('form.params.steps')) {
$this->Session->write('form.data', $currentSessionData);
$this->Session->write('form.params.maxProgress', $stepNumber);
$this->redirect(array('action' => 'msf_step', $stepNumber+1));
} else {
/**
* otherwise, this is the final step, so we have to save the data to the database
*/
$this->User->save($currentSessionData);
$this->Session->setFlash('Account created!');
$this->redirect('/users/msf_index');
}
}
} else {
$this->request->data = $this->Session->read('form.data');
}
/**
* here we load the proper view file, depending on the stepNumber variable passed via GET
*/
$this->render('msf_step_'.$stepNumber);
}
}
The problem might be here:
public function msf_setup() {
App::uses('Folder', 'Utility');
$usersViewFolder = new Folder(APP.'View'.DS.'Users');
$steps = count($usersViewFolder->find('msf_step_.*\.ctp'));
$this->Session->write('form.params.steps', $steps);
$this->Session->write('form.params.maxProgress', 0);
$this->redirect(array('action' => 'msf_step', 1));
}
Also, I have added four msf_setp files in view/Users like
msf_step_1, msf_step_2, msf_setp_3, msf_step_4.
Please help. I'm Stuck. Thankx in advance.

Store forms in session and back button

I'm trying to achieve the following scenario:
1. user display the page addBook.php
2. user starts filling the form
3. but when he wants to select the book Author from the Author combo box, the Author is not yet created in the database so the user clicks a link to add a new Author
5. user is redirected to addAuthor.php
6. the user fill the form and when he submits it, he goes back to addBook.php with all the previous data already present and the new Author selected.
The things is: I have scenarios where there is more than one level of recursion. (Example: Add Book => Add Author => Add Country)
How can I do that?
At step #3, the link submit the form so that I can save it in session.
To handle recursion, I can use a Stack and push the current from on the Stack each time I click a link. And pop the last form of the Stack when the user completes the action correctly or click a cancel button.
My problem is:
How can I handle the back button of the browser?
If instead of clicking the "cancel" button, the user click on the back button, how could I kown that I need to pop the last element?
Do you known some common pattern to achieve that?
You must use javascript on the client and hook into the window unload event, serialize the form and send the answer to the server, which saves it in the session.
$(window).unload(function() {
$.ajax({
url : 'autosave.php',
data : $('#my_form').serialize()
});
});
on server
// autosave.php
$_SESSION['autosave_data'] = $_POST['autosave_data'];
// addbook.php
if (isset($_SESSION['autosave_data'])) {
// populate the fields
}
This is the solution I developed to answer my problem.
As the problem was not a client side problem but truly a server side one. Following the php classes I used in my project:
First the main class of the stack functionality. The inclusion need to be done before the session_start as the object will be stored in the session
class Stack {
private $stack;
private $currentPosition;
private $comeFromCancelledAction = false;
public function __construct() {
$this->clear();
}
/* ----------------------------------------------------- */
/* PUBLICS METHODS */
/* ----------------------------------------------------- */
/**
* Clear the stack history
*/
public function clear() {
$this->stack = array();
$this->currentPosition = -1;
}
/**
* get the current position of the stack
*/
public function getCurrentPosition() {
return $this->currentPosition;
}
/**
* Add a new element on the stack
* Increment the current position
*
* #param $url the url to add on the stack
* #param $data optionnal, the data that could be stored with this $url
*/
public function add($url, &$data = array()) {
if (count($this->stack) != $this->currentPosition) {
// the currentPosition is not the top of the stack
// need to slice the array to discard dirty urls
$this->stack = array_slice($this->stack, 0, $this->currentPosition+1);
}
$this->currentPosition++;
$this->stack[] = array('url' => $url, 'data' => $data, 'previousData' => null, 'linked_data' => null);
}
/**
* Add the stack position parameter in the URL and do a redirect
* Exit the current script.
*/
public function redirect() {
header('location:'.$this->addStackParam($this->getUrl($this->currentPosition)), 301);
exit;
}
/**
* get the URL of a given position
* return null if the position is not valid
*/
public function getUrl($position) {
if (isset($this->stack[$position])) {
return $this->stack[$position]['url'];
} else {
return null;
}
}
/**
* get the Data of a given position
* return a reference of the data
*/
public function &getData($position) {
if (isset($this->stack[$position])) {
return $this->stack[$position]['data'];
} else {
return null;
}
}
/**
* Update the context of the current position
*/
public function storeCurrentData(&$data) {
$this->stack[$this->currentPosition]['data'] = $data;
}
/**
* store some data that need to be fixed in sub flow
* (for example the id of the parent object)
*/
public function storeLinkedData($data) {
$this->stack[$this->currentPosition]['linked_data'] = $data;
}
/**
* Update the context of the current position
*/
public function storePreviousData(&$data) {
$this->stack[$this->currentPosition]['previousData'] = $data;
}
/**
* Compute all linked data for every positions before the current one and return an array
* containing all keys / values
* Should be called in sub flow to fixed some data.
*
* Example: if you have tree pages: dad.php, mum.php and child.php
* when creating a "child" object from a "dad", the dad_id should be fixed
* but when creating a "child" object from a "mum", the mum_id should be fixed and a combo for choosing a dad should be displayed
*/
public function getLinkedData() {
$totalLinkedData = array();
for($i = 0; $i < $this->currentPosition; $i++) {
$linkedData = $this->stack[$i]['linked_data'];
if ($linkedData != null && count($linkedData) > 0) {
foreach($linkedData as $key => $value) {
$totalLinkedData[$key] = $value;
}
}
}
return $totalLinkedData;
}
/**
* Main method of the Stack class.
* Should be called on each page before any output as this method should do redirects.
*
* #param $handler StackHandler object that will be called at each step of the stack process
* Let the caller to be notified when something appens.
* #return the data
*/
public function initialise(StackHandler $handler) {
if (!isset($_GET['stack']) || !ctype_digit($_GET['stack'])) {
// no stack info, acces the page directly
$this->clear();
$this->add($this->getCurrentUrl()); //add the ?stack=<position number>
$this->storeLinkedData($handler->getLinkedData());
$this->redirect(); //do a redirect to the same page
} else {
// $_GET['stack'] is set and is a number
$position = $_GET['stack'];
if ($this->currentPosition == $position) {
// ok the user stay on the same page
// or just comme from the redirection
if (!empty($_POST['action'])) {
// user submit a form and need to do an action
if ($_POST['action'] == 'cancel') {
$currentData = array_pop($this->stack);
$this->currentPosition--;
$handler->onCancel($currentData);
// redirect to the next page with ?stack=<current position + 1>
$this->redirect();
} else {
// store the action for future use
$this->stack[$this->currentPosition]['action'] = $_POST['action'];
$currentData = $this->getData($this->currentPosition);
list($currentData, $nextUrl) = $handler->onAction($currentData, $_POST['action']);
// store current form for future use
$this->storeCurrentData($currentData);
// add the new page on the stack
$this->add($nextUrl);
// redirect to the next page with ?stack=<current position + 1>
$this->redirect();
}
} else if (isset($this->stack[$this->currentPosition]['action'])) {
// no action, and an action exists for this position
$currentData = $this->getData($this->currentPosition);
$action = $this->stack[$this->currentPosition]['action'];
if ($this->comeFromCancelledAction) {
//we return from a cancelled action
$currentData = $handler->onReturningFromCancelledAction($action, $currentData);
$this->comeFromCancelledAction = false;
} else {
$previousData = $this->getPreviousData();
if ($previousData != null) {
//we return from a sucessful action
$currentData = $handler->onReturningFromSuccesAction($action, $currentData, $previousData);
$this->resetPreviousData();
}
}
$this->storeCurrentData( $currentData );
}
$currentData = $this->getData($this->currentPosition);
if ($currentData == null) {
$currentData = $handler->getInitialData();
$this->storeCurrentData( $currentData );
}
return $currentData;
} else if ($this->getUrl($position) == $this->getCurrentUrl()) {
// seems that the user pressed the back or next button of the browser
// set the current position
$this->currentPosition = $position;
return $this->getData($position);
} else {
// the position does not exist or the url is incorrect
// redirect to the last known position
$this->redirect();
}
}
}
/**
* call this method after completing an action and need to redirect to the previous page.
* If you need to give some data to the previous action, use $dataForPreviousAction
*/
public function finishAction($dataForPreviousAction = null) {
$pop = array_pop($this->stack);
$this->currentPosition--;
$this->storePreviousData($dataForPreviousAction);
$this->redirect();
}
/* ----------------------------------------------------- */
/* PRIVATE METHODS */
/* ----------------------------------------------------- */
/**
* get the previous data for the current position
* used when a sub flow finish an action to give some data to the parent flow
*/
private function &getPreviousData() {
if (isset($this->stack[$this->currentPosition])) {
return $this->stack[$this->currentPosition]['previousData'];
} else {
return null;
}
}
/**
* get the current url without the stack parameter
*
* Attention: this method calls "basename" on PHP_SELF do strip the folder structure
* and assume that every pages are in the same directory.
*
* The "stack" parameter is removed from the query string
*
* Example: for the page "http://myserver.com/path/to/a.php?id=1&stack=2"
* PHP_SELF will be: /path/to/a.php
* QUERY_STRING wille be: id=1&stack=2
* This method will return: "a.php?id=1"
*/
private function getCurrentUrl() {
$basename = basename($_SERVER['PHP_SELF']);
if ($_SERVER['QUERY_STRING'] != '') {
return $basename.$this->removeQueryStringKey('?'.$_SERVER['QUERY_STRING'], 'stack');
} else {
return $basename;
}
}
/**
* add the "stack" parameter in an url
*/
private function addStackParam($url) {
return $url . (strpos($url, '?') === false ? '?' : '&') . 'stack=' . $this->currentPosition;
}
/**
* Usefull private method to remove a key=value from a query string.
*/
private function removeQueryStringKey($url, $key) {
$url = preg_replace('/(?:&|(\?))'.$key.'=[^&]*(?(1)&|)?/i', "$1", $url);
return $url != '?' ? $url : '';
}
/**
* reset the previous data so that the data are not used twice
*/
private function resetPreviousData() {
$this->stack[$this->currentPosition]['previousData'] = null;
}
}
Then define the abstract StackHandler class
abstract class StackHandler {
/**
* return the initial data to store for this current page
*/
public function &getInitialData() {
return null;
}
/**
* return an array containing the key/values that need to be fixed in sub flows
*/
public function getLinkedData() {
return null;
}
/**
* user ask to go to a sub page
*/
public function onAction(&$currentData, $action) {
$currentData = $_POST;
$nextUrl = $_POST['action'];
return array($currentData, $nextUrl);
}
public function onCancel(&$currentData) {
}
public function onReturningFromCancelledAction($action, &$currentData) {
}
public function onReturningFromSuccesAction($action, &$currentData, $previousData) {
}
}
Then add the following lines at the top of your pages. Adapt the handler it to fit your needs.
// be sure that a stack object exist in the session
if (!isset($_SESSION['stack'])) {
$_SESSION['stack'] = new Stack();
}
$myDad = $_SESSION['stack']->initialise(new DadStackHandler());
class DadStackHandler extends StackHandler {
/**
* return the initial data to store for this current page
*/
public function &getInitialData() {
if(! empty($_GET['id_dad']) && ctype_digit($_GET['id_dad'])){
// update
$myDad = new Dad($_GET['id_dad']);
} else {
// creation
$myDad = new Dad();
}
return $myDad;
}
/**
* return an array containing the key/values that need to be fixed in sub flows
*/
public function getLinkedData() {
$linkedData = array();
if (! empty($_GET['id_dad']) && ctype_digit($_GET['id_dad'])) {
$linkedData['id_dad'] = $_GET['id_dad'];
}
return $linkedData;
}
/**
* user ask to go to a sub page
*/
public function onAction(&$myDad, $action) {
//in order not to loose user inputs, save them in the current data
$myDad->name = $_POST['name'];
$nextUrl = null;
// find the next url based on the action name
if ($action == 'child') {
$nextUrl = 'child.php';
}
return array($myDad, $nextUrl);
}
public function onCancel(&$myDad) {
// probably nothing to do, leave the current data untouched
// or update current data
return $myDad;
}
public function onReturningFromCancelledAction($action, &$myDad) {
// probably nothing to do, leave the current data untouched
// called when returning from child.php
return $myDad;
}
public function onReturningFromSuccesAction($action, &$myDad, $newId) {
// update the id of the foreign field if needed
// or update the current data
// not a good example as in real life child should be a list and not a foreign key
// $myDad->childId = $newId;
$myDad->numberOfChildren++;
return $myDad;
}
}
...
if (user submit form and all input are correct) {
if ($myDad->save()) {
// the user finish an action, so we should redirect him to the previous one
if ($_SESSION['stack']->getCurrentPosition() > 0) {
$_SESSION['stack']->finishAction($myDad->idDad);
} else {
// default redirect, redirect to the same page in view more or redirect to a list page
}
}
}
I hope this could help others.

Multithreading or Parallel processing in PHP

I'm dealing with Godaddy auction domains, they provide some way to download domains listing. I do have a cron job developed to download & dump (insert) domains listing into my database table. This process takes few seconds from download and dumping into database. The total number of domains (records) in this case are 34000 entries.
Second, I need to update the page rank for each individual domain in database for total 34000 records. I have the PHP API for fetching the page rank live. The Godaddy downloads don't provide page rank detail so I have to fetch and update it separately.
Now, the problem is when it comes to fetching page rank live and then updating page rank into database takes too much time for total 34000 domains.
I recently did an experiment via cron job to update page rank for domains in database, it took 4 hours to update page rank just for 13383 domains from 34000 total. Since it has to first fetch and then update into database. This all was going on dedicated server.
Is there any way to speed up this process for large number of domains? The only way, I'm thinking is to accomplish this via multitasking.
Would that be possible to have 100 tasks fetching page rank and updating it into database simultaneously?
In case you need the code:
$sql = "SELECT domain from auctions";
$mozi_get=runQuery($sql);
while($results = mysql_fetch_array($mozi_get)){
/* PAGERANK API*/
if($results['domain']!='Featured Listings'){
//echo $results['domain']."<br />";
try
{
$url = new SEOstats("http://www.".trim($results['domain']));
$rank=$url->Google_Page_Rank();
if(!is_integer($rank)){
//$rank='0';
}
}
catch (SEOstatsException $e)
{
$rank='0';
}
try
{
$url = new SEOstats(trim("http://".$results['domain']));
$rank_non=$url->Google_Page_Rank();
if(!is_integer($rank_non)){
//$rank_non='0';
}
}
catch (SEOstatsException $e)
{
$rank_non='0';
}
$sql = "UPDATE auctions set rank='".$rank."', rank_non='".$rank_non."' WHERE domain='".$results['domain']."'";
runQuery($sql);
echo $sql."<br />";
}
}
Here is my updated code for pthreads:
<?php
set_time_limit(0);
require_once("database.php");
include 'src/class.seostats.php';
function get_page_rank($domain) {
try {
$url = new SEOstats("http://www." . trim($domain));
$rank = $url->Google_Page_Rank();
if(!is_integer($rank)){
$rank = '0';
}
} catch (SEOstatsException $e) {
$rank = '0';
}
return $rank;
}
class Ranking extends Worker {
public function run(){}
}
class Domain extends Stackable {
public $name;
public $ranking;
public function __construct($name) {
$this->name = $name;
}
public function run() {
$this->ranking = get_page_rank($this->name);
/* now write the Domain to database or whatever */
$sql = "UPDATE auctions set rank = '" . $this->ranking . "' WHERE domain = '" . $this->name . "'";
runQuery($sql);
}
}
/* start some workers */
$workers = array();
while (#$worker++ < 8) {
$workers[$worker] = new Ranking();
$workers[$worker]->start();
}
/* select auctions and start processing */
$domains = array();
$sql = "SELECT domain from auctions"; // RETURNS 55369 RECORDS
$domain_result = runQuery($sql);
while($results = mysql_fetch_array($domain_result)) {
$domains[$results['domain']] = new Domain($results['domain']);
$workers[array_rand($workers)]->stack($domains[$results['domain']]);
}
/* shutdown all workers (forcing all processing to finish) */
foreach ($workers as $worker)
$worker->shutdown();
/* we now have ranked domains in memory and database */
var_dump($domains);
var_dump(count($domains));
?>
Any help will be highly appreciated. Thanks
Well here's a pthreads example that will allow you to multi-thread your operations ... I have chosen the worker model and am using 8 workers, how many workers you use depends on your hardware and the service receiving the requests ... I've never used SEOstats or godaddy domain auctions, I'm not sure of the CSV fields and will leave the getting of page ranks to you ...
<?php
define ("CSV", "https://auctions.godaddy.com/trpSearchResults.aspx?t=12&action=export");
/* I have no idea how to get the actual page rank */
function get_page_rank($domain) {
return rand(1,10);
}
class Ranking extends Worker {
public function run(){}
}
class Domain extends Stackable {
public $auction;
public $name;
public $bids;
public $traffic;
public $valuation;
public $price;
public $ending;
public $type;
public $ranking;
public function __construct($csv) {
$this->auction = $csv[0];
$this->name = $csv[1];
$this->traffic = $csv[2];
$this->bids = $csv[3];
$this->price = $csv[5];
$this->valuation = $csv[4];
$this->ending = $csv[6];
$this->type = $csv[7];
}
public function run() {
/* we convert the time to a stamp here to keep the main thread moving */
$this->ending = strtotime(
$this->ending);
$this->ranking = get_page_rank($this->name);
/* now write the Domain to database or whatever */
}
}
/* start some workers */
$workers = array();
while (#$worker++ < 8) {
$workers[$worker] = new Ranking();
$workers[$worker]->start();
}
/* open the CSV and start processing */
$handle = fopen(CSV, "r");
$domains = array();
while (($line = fgetcsv($handle))) {
$domains[$line[0]] = new Domain($line);
$workers[array_rand($workers)]->stack(
$domains[$line[0]]);
}
/* cleanup handle to csv */
fclose($handle);
/* shutdown all workers (forcing all processing to finish) */
foreach ($workers as $worker)
$worker->shutdown();
/* we now have ranked domains in memory and database */
var_dump($domains);
var_dump(count($domains));
?>
Questions:
Right, 8 workers
Workers execute Stackable objects in the order they were stack()'d, this line chooses a random worker to execute the Stackable
You can traverse the list of $domains in the main process during execution, checking the status of each Stackable as you are executing
All of each workers stack will be executed before the shutdown takes place, the shutdown ensures that all work is therefore done by that point in the execution of the script.

How to remember 10 last read articles with timestamp in session for a user in Codeigniter?

I would like to make a PHP if condition code that will check if the last 10 articles or 10 minutes from the article reading by the user have already elapsed.
E.g.
A user open a page with id = 235 (this id value is in the url localhost/article/235 )
and this id value will be saved in session with a current timestamp and maybe his IP address
Then he read another article and the same will happen.
I need to remember the clicked stuff for another ten clicks and then reset that only for the first row. E.g. after the 10th click the id and timestamp will not became 11th row but will replace the 1st row in the list.
The php condition in CodeIgniter will then check these values and will update the article hit counter value in the articles table and column counter like this:
$this->db->where('id', $id);
$this->db->set('counter', 'counter+1', FALSE);
$this->db->update('articles');
But before calling this code I need to make this check from the session?
How to do that?
I think storing e.g. 10 entries in the session with timestamps per user will be enough.
Just don't save the same page in the session twice.
And the condition will check the current timestamp with the saved one and if it is more than e.g. 10 minutes or the user have read/clicked another 10 articles it will allow the update counter php code.
I don't need to have this bulletproof. Just to disable the increment using browser's refresh button.
So, if he wants to increment the counter he will need to wait ten minutes or read another 10 articles ;)
You should definitely go for Sessions. It saves you bandwidth consumption and is much easier to handle. Unless, of course, you need the data on the client-side, which, by your explanation, I assume you don't. Assuming you went for sessions, all you gotta do is store an array with the data you have. The following code should do it:
$aClicks = $this->session
->userdata('article_clicks');
// Initialize the array, if it's not already initialized
if ($aClicks == false) {
$aClicks = array();
}
// Now, we clean our array for the articles that have been clicked longer than
// 10 minutes ago.
$aClicks = array_filter(
$aClicks,
function($click) {
return (time() - $click['time']) < 600; // Less than 10 minutes elapsed
}
);
// We check if the article clicked is already in the list
$found = false;
foreach ($aClicks as $click) {
if ($click['article'] === $id) { // Assuming $id holds the article id
$found = true;
break;
}
}
// If it's not, we add it
if (!$found) {
$aClicks[] = array(
'article' => $id, // Assuming $id holds the article id
'time' => time()
);
}
// Store the clicks back to the session
$this->session
->set_userdata('article_clicks', $aClicks);
// If we meet all conditions
if (count($aClicks) < 10) {
// Do something
}
I assumne that $clicks is an array with up to ten visited articles. The id is used as key and the timestamp as value. $id is the id of the new article.
$clicks = $this->session->userdata('article_clicks');
//default value
$clicks = ($clicks)? $clicks : array();
//could be loaded from config
$maxItemCount = 10;
$timwToLive= 600;
//helpers
$time = time();
$deadline = $time - $timeToLive;
//add if not in list
if(! isset($clicks[$id]) ){
$clicks[$id] = $time;
}
//remove old values
$clicks = array_filter($clicks, function($value){ $value >= $deadline;});
//sort newest to oldest
arsort($clicks);
//limit items, oldest will be removed first because we sorted the array
$clicks = array_slice($clicks, 0, $maxItemCount);
//save to session
$this->session->>set_userdata('article_clicks',$clicks)
Usage:
//print how mch time has passed since the last visit
if(isset($clicks[$id]){
echo "visited ".($time-$clicks[$id]). "seconds ago." ;
} else {
echo "first visit";
}
EDIT: you have to use arsort not rsort or the keys will be lost, sorry
Based on Raphael_ code and your question you can try this:
<?php
$aClicks = $this->session
->userdata('article_clicks');
$nextId = $this->session->userdata('nextId');
// Initialize the array, if it's not already initialized
if ($aClicks == false) {
$aClicks = array();
$nextId = 0;
}
// Now, we clean our array for the articles that have been clicked longer than
// 10 minutes ago.
$aClicks = array_filter($aClicks, function($click) {
return (time() - $click['time']) < 600; // Less than 10 minutes elapsed
}
);
// We check if the article clicked is already in the list
$found = false;
foreach ($aClicks as $click) {
if ($click['article'] === $id) { // Assuming $id holds the article id
$found = true;
break;
}
}
// If it's not, we add it
if (!$found) {
$aClicks[$nextId] = array(
'article' => $id, // Assuming $id holds the article id
'time' => time()
);
$nextId++;
$this->session->set_userdata('nextId', $nextId);
}
$this->session->set_userdata('article_clicks', $aClicks);
if (count($aClicks) > 10 && $nextId > 9) {
$this->session->set_userdata('nextId', 0);
echo "OK!";
}
?>
I hope I understood correctly what you need.
Usage:
$this->load->library('click');
$this->click->add($id, time());
The class API is very simple and the code is commented. You can also check if an item expired(), if exists() and you can get() item saved time.
Remember that:
Each item will expire after 10 minutes (see $ttl)
Only 10 items are saved in session (see $max_entries)
class Click
{
/**
* CI instance
* #var object
*/
private $CI;
/**
* Click data holder
* #var array
*/
protected $clicks = array();
/**
* Time until an entry will expire
* #var int
*/
protected $ttl = 600;
/**
* How much entries do we store ?
* #var int
*/
protected $max_entries = 10;
// -------------------------------------------------------------------------
public function __construct()
{
$this->CI =& get_instance();
if (!class_exists('CI_Session')) {
$this->CI->load->library('session');
}
// load existing data from user's session
$this->fetch();
}
// -------------------------------------------------------------------------
/**
* Add a new page
*
* #access public
* #param int $id Page ID
* #param int $time Added time (optional)
* #return bool
*/
public function add($id, $time = null)
{
// If page ID does not exist and limit has been reached, stop here
if (!$this->exist($id) AND (count($this->clicks) == $this->max_entries)) {
return false;
}
$time = !is_null($time) ? $time : time();
if ($this->expired($id)) {
$this->clicks[$id] = $time;
return true;
}
return false;
}
/**
* Get specified page ID data
*
* #access public
* #param int $id Page ID
* #return int|bool Added time or `false` on error
*/
public function get($id)
{
return ($this->exist($id)) ? $this->clicks[$id] : false;
}
/**
* Check if specified page ID exists
*
* #access public
* #param int $id Page ID
* #return bool
*/
public function exist($id)
{
return isset($this->clicks[$id]);
}
/**
* Check if specified page ID expired
*
* #access public
* #param int $id Page ID
* #return bool
*/
public function expired($id)
{
// id does not exist, return `true` so it can added
if (!$this->exist($id)) {
return true;
}
return ((time() - $this->clicks[$id]) >= $this->ttl) ? true : false;
}
/**
* Store current clicks data in session
*
* #access public
* #return object Click
*/
public function save()
{
$this->CI->session->set_userdata('article_clicks', serialize($this->clicks));
return $this;
}
/**
* Load data from user's session
*
* #access public
* #return object Click
*/
public function fetch()
{
if ($data = $this->CI->session->userdata('article_clicks')) {
$this->clicks = unserialize($data);
}
return $this;
}
public function __destruct()
{
$this->save();
}
}
You could easily wrap that into a class of it's own that serializes the information into a string and that is able to manipulate the data, e.g. to add another value while taking care to cap at the maximum of ten elements.
A potential usage could look like, let's assume the cookie last would contain 256 at start:
echo $_COOKIE['last'] = (new StringQueue($_COOKIE['last']))->add(10), "\n";
echo $_COOKIE['last'] = (new StringQueue($_COOKIE['last']))->add(20), "\n";
echo $_COOKIE['last'] = (new StringQueue($_COOKIE['last']))->add(30), "\n";
echo $_COOKIE['last'] = (new StringQueue($_COOKIE['last']))->add(40), "\n";
echo $_COOKIE['last'] = (new StringQueue($_COOKIE['last']))->add(50), "\n";
echo $_COOKIE['last'] = (new StringQueue($_COOKIE['last']))->add(60), "\n";
echo $_COOKIE['last'] = (new StringQueue($_COOKIE['last']))->add(70), "\n";
echo $_COOKIE['last'] = (new StringQueue($_COOKIE['last']))->add(80), "\n";
echo $_COOKIE['last'] = (new StringQueue($_COOKIE['last']))->add(90), "\n";
echo $_COOKIE['last'] = (new StringQueue($_COOKIE['last']))->add(100), "\n";
And the output (Demo):
10,256
20,10,256
30,20,10,256
40,30,20,10,256
50,40,30,20,10,256
60,50,40,30,20,10,256
70,60,50,40,30,20,10,256
80,70,60,50,40,30,20,10,256
90,80,70,60,50,40,30,20,10,256
100,90,80,70,60,50,40,30,20,10
A rough implementation of that:
class StringQueue implements Countable
{
private $size = 10;
private $separator = ',';
private $values;
public function __construct($string) {
$this->values = $this->parseString($string);
}
private function parseString($string) {
$values = explode($this->separator, $string, $this->size + 1);
if (isset($values[$this->size])) {
unset($values[$this->size]);
}
return $values;
}
public function add($value) {
$this->values = $this->parseString($value . $this->separator . $this);
return $this;
}
public function __toString() {
return implode(',', $this->values);
}
public function count() {
return count($this->values);
}
}
It's just some basic string operations, here with implode and explode.

Categories