I want to use Redis basically like this, if it (hypothetically) accepted SQL:
SELECT id, data, processing_due FROM qtable WHERE processing_due < NOW()
where processing_due is an integer timestamp of some sort.
The idea is then to also remove completed "jobs" with something like:
DELETE from qtable WHERE id = $someid
Which Redis commands would I use on the producing ("insert") and consuming ("select, delete from") end?
I find that Redis can be used as a queue, but I don't want the answers in strictly the order they were inserted, but rather based on if "now" is past processing_due.
I imagine this is almost the same problem as a leaderboard?
(I try to wrap my head around how Redis works and it looks simple enough from the documentation, but I just don't get it.)
Would a decent solution be to do ZADD qtable <timestamp> <UUID> and then use the UUID as a key to store the (json) value under it?
You can use a Sorted Set, in which the score is your time (an integer as you've suggested), and then you query using ZRANGEBYSCORE. Each member would be a Json representation of your "fields". For example: {id:"1",data:"bla",processing_due:"3198382"}
Regarding delete, just use ZREM when you find the relevant member to delete. pass your Json string as a parameter and you're OK.
A possibly better variant would be to just hold generated IDs as your member, and in a separate String-type key save pairs of your IDs along with the Json representation of your data. Just remember to maintain the two structs in sync.
Related
I am currently working on a new web based project with various types of entities. This service will be accessible through an REST API, and I'm thinking about endpoints like:
api.example.com/users/{user_id}
For this, I think that an auto-incremental ID for users will be a bad approach, since anybody can hit:
api.example.com/users/1, and then api.example.com/users/2, api.example.com/users/3, and so on.
Now, I'm thinking to use UUID, but I don't know if it's a good idea, because it's a VARCHAR(36). For these reason, I do something like this when I generate the user ID on the INSERT query (I'm using MySQL):
unhex(replace(uuid(),'-',''))
With this, I'm casting the UUID to binary. And I'm storing an BINARY(16) on the database instead.
And when I want to retrieve info from database, I can use something like that:
SELECT hex(id), name FROM user;
and
SELECT hex(id), name FROM user WHERE hex(id) = '36c5f55620ef11e7b94d6c626d968e15';
So, I'm working with Hexadecimal form, but storing it in binary form.
It is this a good approach?
Almost there...
Indexes are your performance friend. Presumably id is indexed, then
WHERE id = function('...')
uses the index and goes straight to the row, but
WHERE function(id) = '...'
cannot use the index; instead, it scans all the rows, checking each one. For a large table, this is sloooow.
So...
Use this to store it:
INSERT INTO tbl (uuid, ...)
VALUES
(UNHEX(REPLACE(UUID(), '-', '')), ...);
And this to test for it:
SELECT ... WHERE
uuid = UNHEX(REPLACE('786f3c2a-21f6-11e7-9392-80fa5b3669ce', '-', ''));
If you choose to send (via REST) the 32 characters without the dashes, you can figure that minor variation.
Since that gets tedious, build a pair of Stored Functions. Oh, I have one for you; see http://mysql.rjweb.org/doc.php/uuid .
That also discusses why UUIDs are inefficient for huge tables, and a possible way around it.
I use sorted set type in Redis store.
For each user I create a own KEY and put here data:
Example of KEY:
FEED:USER:**1**, FEED:USER:**2**, FEED:USER:**3**
I want to select data from Redis for user's keys: 1, 2, 3 and sorted each by score (timestamp).
If see at problem simply, I need select from any KEY a data across time and after combine all results sorted by score.
There are a couple of ways to do this but the right one depends on what you're trying to do. For example:
You can use ZRANGEBYSCORE (or ZREVRANGEBYSCORE) in your code for each FEED:USER:n key and "merge" the replies in the client
You can do a ZUNIONSTORE on the relevant keys and then do the ZRANGEBYSCORE on the result from the client.
However, if your "feeds" are large, #2's flow should be reversed - first range and then union.
You could also do similar types of processing entirely server-side with some Lua scripting.
EDIT: further clarifications
Re. 1 - Merging could be done client-side on the results that you get from ZRANGEBYSCORE or you could use server-side Lua scripts to do that. Use the WITHSCORES to get the timestamp and merge/sort on it. Regardless the your choice of location for running this code (I'd probably use Lua for data locality), the implementation is up to you - lmk if you need help with that :)
I'm using mysql database auto-increment as an order ID. When I display the order ID to the user, I want to somehow mask/obfuscate it.
Why?
So at first glance, it is obvious to admin users what the number
refers to (orders start with 10, customers start with 20 etc)
To hide, at first glance, that this is only my 4th order.
Based on this this answer, I want the masked/obfuscated order id to:
Be only numbers
Consistent length (if possible)
Not cause collisions
Be reversible so I can decode it and get the original ID
How would I acheive this in PHP? It doesn't have to be very complex, just so at first glance it's not obvious.
I think you can use XOR operator to hide "at first glance" for example (MySQL example):
(id*121) ^ 2342323
Where 2342323 and 121 are "magic" numbers - templates for the order number.
To reverse:
(OrderNum ^ 2342323)/121
Additional advantage in this case - you can validate OrderNumber (to avoid spam or something like this in online form) if (OrderNum ^ 2342323) is divided by 121 with no remainder.
SQLFiddle demo
A little bit late, but Optimus (https://github.com/jenssegers/optimus) does exactly what is here asked for.
$encoded = $optimus->encode(20); // 1535832388
$original = $optimus->decode(1535832388); // 20
Only the initial setup is a bit weird (generate primenumbers)
Probably the simplest way is to just generate a long random string and use it instead of the auto-increment ID. Or maybe use it alongside the auto-increment ID. If the string is long enough and random enough, it will be unique for every record (think of GUIDs). Then you can display these to the user and not worry about anything.
Can it help?
echo hexdec(uniqid());
Off course you should store this value at db, at the same row with order id.
Just converting a ID into something like HEX might not give you the result what you like. Moreover its still easy 'guessable'
I would a a extra ID column (i.e. order_id). Set a unqi. index. Then on_creation use one of the following mysql functions:
SHA1(contcat('ORDER', id))
MD5(contcat('ORDER', id))
SHA1(contcat('ORDER', id, customer_id))
MD5(contcat('ORDER', id, customer_id))
UUID()
// try this in your mysql console
SELECT UUID(), SHA(CONCAT('ORDER',10)), SHA1(1);
You could (as in the example), add a simple text prefix like 'order'. Or even combine them. However i think UUID() would be easiest.
Implementation depends a bit on what you prefer you could use a stored procedure) or incorporate it in your model.
I have a web app with tons of documents. User can enter id (valid MongoId / ObjectId), but if user doesn't enter it I have to retrieve object with last id:
I'm concerned about speed for searching last object. I'm currently doing it like this:
db.docs.find({"status": 1}).sort({"_id": -1}).limit(1);
//Or in php:
$docs->find(array('status' => 1))->sort(array('_id' => -1))->limit(1)->getNext();
Isn't this a bit slow? First is looking for all docs with status 1 then sort them and limit then. Is there any better way for getting last document with status 1?
To make this performant you'd likely need to add an index on { status: 1, _id: -1 }.
You can also use findOne instead of find with a limit to simplify the syntax:
db.docs.findOne({"status": 1}).sort({"_id": -1});
Perhaps just store another value in another table, or in an in-memory cache perhaps, indicating the highest id value the system has as status=1. It would require a small bit of logic to be added when inserting/updating objects in the database, to compare the id value of objects with status=1 against the current cached id value, updating if the value is higher. You could then access the latest file directly using this cached value.
It is a little clunky, but would probably perform much better than the find.sort.limit operation you are currently doing as your number of objects grows.
I currently have about 4 different database tables which output to html tables. Each of these tables uses a count query to calculate data from a 5th table.
That's no problem, but what about when I want to sort and order the data, and paginate etc (like with zend). If it were a one page table, I could probably sort an array.
My thought was, to use a ticker. But that would require a new column in all 4 tables and seems like overkill or like there could be a better way.
Sadly, I can't find much info on it (likely because I don't know what to search for).
Advice?
..and please take it easy, I'm new and learning.
Assuming youre using Zend_Db_Table_Row and that you dont need to persist any modifications you might make to these rowsets then you can just append the virtual columns to the row object and have them be accessible via array notation. So if youre doing it all in one query now just use that same query, and the column should be there.
OTOH, if youre using a Data Mapper pattern then simply adjust your hydration to look for this "virtual column" and hydrate it if it exists in the result data. Then in your getter for this property have it see if the property is null or some other negative specification, and if it is, to execute a calculation query on that single object or return the already calculated result.