A component of a website I'm building requires a user to enter a unique code in a form which is then verified as being a valid code by comparing it against a predetermined list of several million generated codes. How can I do this to provide instant feedback to the user?
Originally I was planning to check all of the submissions in a batch job overnight, but the client would like users to be informed whether they have a valid code immediately.
Is it more efficient to build a trie structure, store the generated codes in a database table, or use some other approach?
Using a MySQL table for this purpose is absolutely the right thing to do. MySQL has no problem looking up an entry in a few milliseconds from a multi-million-row table if you index it correctly. In fact, a MySQL table index is pretty close to a pre-programmed trie structure, with all the hard work, concurrency, edge cases, and debugging done for you.
If you build some other kind of lookup system, you'll have to sort out such things as persistence, maintenance of the data, and all.
Related
I am building a web application that uses PHP and MySQL on the backend. I want to store some user data -- basically a set of objects in JSON format that detail the user's "favorites" info for the application. I don't want to store this JSON data in a single MySQL field in my user database table because it doesn't seem efficient.
So, I am thinking to just store the JSON data in a flat file on the server with a unique identifier that I can use to know which user the file is associated with. My questions is: would this be a scalable solution for upwards of 10,000 users?
This is likely to cause you lots of headaches, both in terms of technical aspects and in terms of security. And no, it's not very scalable. Think about the problems it will cause: What happens when you need to add a server? How will you sync the files? What if you want to do something involving multiple users, like seeing how many people have XYZ as a favorite?
A much better option is to do one of the following:
Normalize your database (do this part regardless) and put the favorites in their own table or
Save the favorites in a JSON column (probably the wrong answer, but makes sense in some contexts)
If you're worried about speed, you can implement some caching using Redis, memcached, or some other system. But do not do this yet - that's premature optimization. Do it when you need it.
Im using functions for logging in a user, When they login but fail either by no captcha sent, failed captcha or failed login it will give there IP a Try. When they reach 5 tries they get blocked from the login page for approximately 1 hour. I have a function that updates the MySQL Column to increment there try count and last try date. But from looking at PHP's documents it states:
Note: The increment/decrement operators only affect numbers and
strings. Arrays, objects and resources are not affected.
My function gets the try count from the Database and then tries updating it. My SQL result for fetching the Try count is by default an Array because of how PDO works. So how can I efficiently increment an array?
I was thinking of doing a foreach condition and use the .=opperator to save it to a string and from there increment. But is that really the most efficient way?
Thank you.
P.S: I'm not showing any example code e.t.c because this question is simple enough. I have searched around on here and couldn't find a proper answer.
To understand why your question is wrong, you have to understand what an array is.
An array is just a "bag" that holds other variables. so, your question sounds like "How can I pay for a two beers with my pocket?". The thing is, you can't pay with a pocket. you have to take the cash out of the pocket and then use that cash.
Exactly the same thing goes with arrays: you have to extract the returned data from array, and then you are free to perform any operation on its contents. On the contents, remember, not on the bag.
But for the efficient solution, go for the other answer, which solved your initial problem the right way - without the need of selecting any arrays at all.
And just a side note
MySQL result for fetching the Try count is by default an Array because of how PDO works.
As a matter of fact, PDO can work in many different ways. For example, it can return scalar values all right.
You can increment it in an update query directly. When you want to add a try, simply:
UPDATE `tries` SET `tries` = `tries` + 1 WHERE `ip` = '127.0.0.1';
Just replace the IP with the actual IP.
Just to add..
IMO you should be using a separate table for incorrect login attempts. There are many reasons for this, but one of the important is that any attack is likely to rotate usernames and not only passwords in the attempt.
Having a separate table that records all incorrect logins allows you to much more easily query for an amount of incorrect logins in xx time. Incorrect logins attached to a user limits your ability to detect DoS and brute force attacks coming from scripted sources as you can only look at the username attempted if it actually existed in the first place.
However, you can relate a field in the table to the users ID, so that you can track users independently, then on successful login, the records that relate to that user could be deleted.
To give you a working example. I have built in the following functionality into the commercial Symfony project that I work on on a daily basis.
table example
userID --- foreign key (not mandatory)
IP --- mandatory
timestamp --- mandatory
we query the data like this:
Overall failed attempts for a particular subdomain (we have lots of them in use using the same system)
the system is used in schools, so we have to cater for naughty students!
Overall failed attempts in the last minute
system sleeps for a random time based on a base value x the amount. (a bit of a hacky way to try to trip up script attacks)
Overall attempts for a particular user
similar to your example.. compares to preconfigured amounts then warns/disables users accordingly. If it blocks sends an email to the helpdesk team.
this is by no means a suggested list, or an example of what should be done.. its merely what we decided on our applications circumstances.
The point is, without a separate table much of this wouldn't be possible.
I have a program that creates logs and these logs are used to calculate balances, trends, etc for each individual client. Currently, I store everything in separate MYSQL tables. I link all the logs to a specific client by joining the two tables. When I access a client, it pulls all the logs from the log_table and generates a report. The report varies depending on what filters are in place, mostly date and category specific.
My concern is the performance of my program as we accumulate more logs and clients. My intuition tells me to store the log information in the user_table in the form of a serialized array so only one query is used for the entire session. I can then take that log array and filter it using PHP where as before, it was filtered in a MYSQL query (using multiple methods, such as BETWEEN for dates and other comparisons).
My question is, do you think performance would be improved if I used serialized arrays to store the logs as opposed to using a MYSQL table to store each individual log? We are estimating about 500-1000 logs per client, with around 50000 clients (and growing).
It sounds like you don't understand what makes databases powerful. It's not about "storing data", it's about "storing data in a way that can be indexed, optimized, and filtered". You don't store serialized arrays, because the database can't do anything with that. All it sees is a single string without any structure that it can meaningfully work with. Using it that way voids the entire reason to even use a database.
Instead, figure out the schema for your array data, and then insert your data properly, with one field per dedicated table column so that you can actually use the database as a database, allowing it to optimize its storage, retrieval, and database algebra (selecting, joining and filtering).
Is serialized arrays in a db faster than native PHP? No, of course not. You've forced the database to act as a flat file with the extra dbms overhead.
Is using the database properly faster than native PHP? Usually, yes, by a lot.
Plus, and this part is important, it means that your database can live "anywhere", including on a faster machine next to your webserver, so that your database can return results in 0.1s, rather than PHP jacking 100% cpu to filter your data and preventing users of your website from getting page results because you blocked all the threads. In fact, for that very reason it makes absolutely no sense to keep this task in PHP, even if you're bad at implementing your schema and queries, forget to cache results and do subsequent searches inside of those cached results, forget to index the tables on columns for extremely fast retrieval, etc, etc.
PHP is not for doing all the heavy lifting. It should ask other things for the data it needs, and act as the glue between "a request comes in", "response base data is obtained" and "response is sent back to the client". It should start up, make the calls, generate the result, and die as fast as it can again.
It really depends on how you need to use the data. You might want to look into storing with mongo if you don't need to search that data. If you do, leave it in individual rows and create your indexes in a way that makes them look up fast.
If you have 10 billion rows, and need to look up 100 of them to do a calculation, it should still be fast if you have your indexes done right.
Now if you have 10 billion rows and you want to do a sum on 10,000 of them, it would probably be more efficient to save that total somewhere. Whenever a new row is added, removed or updated that would affect that total, you can change that total as well. Consider a bank, where all items in the ledger are stored in a table, but the balance is stored on the user account and is not calculated based on all the transactions every time the user wants to check his balance.
I have an application where every user gets their own database. This lends itself to scale horizontally pretty well because it is easy to spin up another database server.
The part I am struggling with is how to identify the database server that a user's database/account is attached to.
Here are the possible solutions I have come up with:
Create a lookup table that maps username --> db server on one of the database servers. Make sure this query is indexed to negate any performance issues.
PROS: Easy to implement
CONS: Still have 1 point of failure. If the database server holding the username -->database relationship goes down, then the application stops working for all users
Create a text file that maps username --> database names
PROS: Pretty easy to implement and pretty fast.
CONS: Problems with keeping it in sync across web servers. Concurrency issues with writing to the file while being read. A small error in writing, could cause everything to go down
What is the best practice for something like this?
Why not keep the first solution and export that regularly scheduled to the second option?
That way if everything goes down you have a means to get back up and get operational to your client base.
I'm working with a Postgres database that I have no control over the administration of. I'm building a calendar that deals with seeing if resources (physical items) were online or offline on a specific day. Unfortunately, if they're offline I can only confirm this by finding the resource name in a text field.
I've been using
select * from log WHERE log_text LIKE 'Resource Kit 06%'
The problem is that when we're building a calendar using LIKE 180+ times (at least 6 resources per day) is slow as can be. Does anybody know of a way to speed this up (keep in mind I can't modify the database). Also, if there's nothing I can do on the database end, is there anything I can do on the php end?
I think, that some form of cache will be required for this. As you cannot change anything in database, your only chance is to pull data from it and store it in some more accessible and faster form. This is highly dependent on frequency of data inserted into table. If there are more inserts than selects, it will not probably help much. Other way there is slight chance of improved performance.
Maybe you can consider using Lucene search engine, which is capable of fulltext indexing. There is implementation from Zend and even Apache has some http service. I haven't opportunity to test it however.
If you don't use something that robust, you can write your own caching mechanism in php. It will not be as fast as postgres, but probably faster than not indexed LIKE queries. If your queries need to be more sofisticated (conditions, grouping, ordering...), you can use SQLite database, which is file based and doesn't need extra service running on server.
Another way could be using triggers in database, which could on insert data store required information to some other table in more indexed manner. But without rights to administer database, it is probably dead end.
Please be more specific with your question, if you want more specific information.