I would like to know how to implement a unique visitor counter using Redis with a timeout of 1 hour. I have a little online shop and I would like to show how many guys are viewing an article currently. What would be the best way? I'm using Predis
Thanks inadvance
You will need to do 2 things:
Generate a unique ID for each visitor (they are many ways to do it)
Use a Redis SET or ZSET to store those ids and add an expire mechanism.
Method #1: SET + SCARD + EXPIRE
Solution
The SET name could be something like [article_id]:[current_hour_timestamp], each time you SADD a visitor id inside it, do an EXPIRE. Each set will expire after a certain time.
Use SCARD [article_id]:[current_hour_timestamp] to know how many visitors are online
Notes
The key can be [article_id]:[current_hour_timestamp] or anything like articles:[article_id]:readers:[current_hour_timestamp] if you prefer...
At every hour start the SET will be empty that's not what you may want.
Method #2: ZSET + ZCARD + cron
Solution
Each time a visitor read open an article send a ZADD [article_id]:readers [expiry_time] [visitor_id], it does not matter if it reload the page, the expiry_time will just be updated.
expiry_time is the timestamp in the future when we consider the visitor to have leaved the article.
Setup a cron to run each hour:
Retrieve all article ids in databases
For each article
Send a `zremrangebyscore "article.id:readers" -inf [current_timestamp]` to remove expired visitors_id
Finally to retrieve current readers count run ZCARD [article_id]:readers.
Note
This approach is way better because it uses a sliding window so you won't have the "0 reader" effect at each new hour. However the downside is that you will need to loop over all your article id which could be an issue.
Method #3: ZSET + ZCARD + ZSET as index + cron
Solution
Same as method #2, however this one uses another zset articles:readers:zset that contains all zset keys recently updated.
You will then have to loop through this zset members instead of all article ids.
Note
If you go down that way don't forget to remove expired members from articles:readers:zset !
Related
I'm trying to replace the sql implementation of Drupal 8's flood control service with a redis based implementation.
See https://github.com/drupal/drupal/blob/8.0.x/core/lib/Drupal/Core/Flood/DatabaseBackend.php
The requirements are like this:
Each occurrence of an action/event (e.g trying to log in) is logged with an expiration, identifier and timestamp
I need to be able to prevent that a certain action can be done more than N times in a given timeframe
I want to be able to clean up expired events
In case of a threshold of 3 in 10 minutes, if the user tries once, then twice after 5 minutes, he is blocked and can try again once after 5 more minutes. Not 10. While the second would be a valid way to do this, it's not how the sql implementation works or how the tests expect it to work.
As you can see based on the API, I also don't know when registering the event what the threshold is, I only know the expiration of a single event.
My thoughts on how to implement this:
If, after N occurrences should be locked for the given time, then this would be easy with a single KEY for event:identifier that is incremented, once the max is reached, it is locked until it expires again and each INCR would also update the expiration (or not).
I found many posts that ask about expiration of list entries, which is not possible. There are workarounds using sorted sets and delete by range. Most seem to use a single global set, but then I can't easily count my event + identifier - I think.
After writing all this down, I might actually have an idea how it could work, so I guess what I'm looking for is feedback on whether that makes sense or if there's an easier way.
Each event:identifier combination is a key and contains a sorted set. That uses the expiration as score and as value a unique value, possibly creation time in microseconds. I count the non-expired records to detect if the threshold was reached. I'm updating the expiration of each event:identifier to the provided expiration window, so it will be auto-deleted assuming unless a given identifier/client doesn't give up and keeps on trying, without ever reaching the expiration. Is it worth to clean up the records inside a set e.g. when doing a new register? It seems to be fairly fast, and I could also only do it sometimes.
I would prefer to use Redis' key expiration feature, instead of reimplementing one.
A simpler alternative would be the following one:
just SET a simple value, which is the the number of attempts; use a key built on a pattern like "identifier":"event type" :
SETNX <identifier>:<event type> 1
if the response is 1, this is the first attempt, so you set a timeout on this key:
EXPIRE <identifier>:<event type> <timeout in seconds>
otherwise you increment the number of attempts
INCR <identifier>:<event type>
The response of the INCR will give you the number of attempts during the window, so you know if you can allow the action or not.
You could also use a hash instead of a simple value, if you need to store more data, like the max number of allowed attempts in the given time window. In this case you will probably use HSETNX and HINCR.
I've run into a bit of a pickle. Usually I can find solutions to my problems by some extensive googling (to the right SO thread), but not this time.
I'm using an API that lets me access their data by cURL, and this API has a request limit of 500 requests per 10 minutes. If this limit is repeatedly exceeded, your API key gets suspended.
My application, written in PHP, frequently makes requests through different pages - sometimes simultaneously. I need a solution that ensures that I'm under the request limit at all times. I need to be able to check and update a variable that resets every 10 minutes, from all scripts - at the same time.
Can this even be done? How should I approach this problem?
I'm not asking for sourcecode to a perfect solution, I'd just like some pointers to how this could be solved, or if it can't be solved - what's the alternate approach?
Thank you in advance.
Sounds like memcache is what your looking for. It can hold variables like PHP objects or base types and give them as well an expiration date. Just google for the memcached extensions for PHP and look over here: http://php.net/manual/en/class.memcache.php.
Furhermore, it can be accessed via TCP and has some failover capabilities. AWS is offering a service compatible with the memcache API calling it ElastiCache
If pages are run by different users, only thing i can think of is storing in a table.
Do something like
$datestring = substr(date("Y-m-d H:i"),0, -1);
in this way you have always same string for minutes 2016-11-07 17:00:00 - 2016-11-07 17:09:59.
Then store in a table with this mechanism:
"INSERT INTO table (datestring, count) VALUES ('$datestring', '1') ON DUPLICATE KEY UPDATE count = count + 1";
of course put datestring char(15) as a unique key.
you check your counting with a select FROM.... comparing datestring you just created and you get no rows at all ora a row with a count value.
This is a solution, of course there are many.
What would be the best way to achieve an undo function in a PHP CRUD application? The only solution I've been able to come up with is using a sort of buffer table in my database that is periodically wiped.
If the user clicks the "Undo" button after deleting a record for example, the id of the last change will be passed to a handler which will pull the buffer record and reinstate it into the main table for that data type. If the "Undo" is not done in say, 4 or 5 minutes, a reaper script will drop the entry.
Does this sound feasible? Is there a better way of accomplishing this?
You could use a flag field in your database to mark a row for delete.
And you can setup task (crontab in linux) to delete all rows with delete flag set to true and time difference > to 5 mins.
I've learned to not delete anything, but simply do as Ignacio Ocampo stated by using a flag column in your DB such as status. By default set the status column to open. If your client clicks your delete button, just update that records status column to void, or deleted..
In doing this, you'll need to update your data request to pull only those records with the status column set to open. This allows the data to not be lost, but also not seen.
all undo(s) or redo(s) if applicable can reset the open status to - or + 1 record sorted by a timestamp column.
If db space is at a premium, and you need to remove old data then crontab does work, but I prefer the simplicity phpmyadmin conjob to loop a file that will wipe all void or deleted records older than time()-'(last cron run).
Depending on what and how you're building, you might also want to consider using one of the following solutions.
1) A pure PHP CRUD solution would be something along the lines you've mentioned, with also possibly storing cookies on the client side to track which actions are being done. Every action a new cookie is created, then your application will only have to sort the cookies by date and time. You could also set the cookies to be automatically expire after x amount of time. (Although I would expire after a x amount of steps, instead of time)
2) If you are able to use HTML5 local storage (http://www.w3schools.com/html/html5_webstorage.asp) along with some Javascript would be perfect for this, since you wouldn't have to wait around for the server to respond everytime 'undo' is clicked since all the processing would be handled locally.
i just created a user login system with php session and now users can register and login to site and do another things...
now i want to create online.php which will fetch all online users.i almost search everything in google and stackoverflow for this with no success.
ok now i want to describe the system which i want to create with high performance..
when a user logged in we just update table user.lastlogin which is a timestamp and then in online.php we SELECT * every user where time interval is < 5 minutes.
for this purpose i can update this timestamp lastlogin field in database when user load each page,and this cost many mysql query to do the job... then in each page load i have to update
UPDATE user set last=now()
that will cost me many mysql query.now i am looking for some another way like using sessions or something that i found in this link
"The normal solution is to store a timestamp in the table which you update every time the user does something. Users with a recent timestamp (say, five minutes) are shown as logged in, everybody else are logged out.
It doesn't even have to be updated on every page load. Keep a timestamp in the session with the last update time as well, and only update the table when the database flag are about to expire."
but unfortunately the answer wasnt quite helpful and i need an example or more describe on this.
1.) If you need more speed in sessions there is for example memcache. A simple key - value store to save your sessions or keys. You can configure Apache that all sessions will be automaticly stored in memcache.
2.) Another solution is to make a MEMORY Table which MySQL holds in RAM. Problem is when you restart your server the data in this table is lost. But i think in your case for last logon there is no problem.
3.) PHP Shared Memory: http://php.net/manual/de/ref.shmop.php for small data.
When you have implemented a solution i would prefer the MEMORY Table or the SHM solution.
Edit: The last paragraph with the searching on HDD was related to the comment on top: Find Number of Open Sessions
I have a PHP website using a MySQL database.
We have items that users create, that are on a timer, and once this timer has counted down, without user interaction (basically next time someone sees it) the status needs to have changed.
I'm not sure how to implement this in a way to be accurate to the minute.
So we have an object X, that expires at 10:15pm tommorrow, and the next person to see object X after that time has to see it as expired.
Is the correct way to do this to be the next time object X is loaded we check if it's expired, and if so, update the database?
What happens if 10 people load object X at the same time after it's expired, what's to prevent some sort of race condition from all 10 requests attempting to update the database?
Is there a cron job that runs every minute that I can some how make use of, or any type of timer in MySQL to kick off every minute checking for these and running a script?
I have several ideas on how it -could- be done, like those listed above, but I'm not sure what the most practical is, or what the standard way to do it is as I'm positive someone has solved this problem before.
Is the correct way to do this to be the next time object X is loaded we check if it's expired, and if so, update the database?
Why do you need to update the database? It seems like you might have some redundancy in your DB table - from what you've said, it sounds like you have (for instance) an is_expired column and then an expires_at column.
Why not just get rid of the is_expired column? It's cheap to compare 2 integers, so when you want to determine if something is expired, just fetch the expires_at column and compare with the current time. This avoids any race conditions with expiry, since nothing in the DB changes.
You can do it with cron of course. Or with javascript native function setInterval( "checkFunction()", 10000 ); to alter the db. Or you could use a date field in DB and check for expiration
Make a field date_to_expire as DATE , enter the expiration date and everytime you query for it check to see if the item is expired (this can go up to seconds)