I have a table where I store almost any English word. This table is for a Scrabble type word game currently I am working on. Here is the syntax,
create table words(
`word` varchar(50),
primary key `word`
)
This table will be very big. And I have to check every time if the given word exists when gamer makes a move.
I am using mysql. Currently I have stored ENABLE words there. My question is when I start adding more words and gamers start to play wont it be performing low? If so, is there any way I can optimize it? Does NO-SQL has anything to do with this scenario?
You should have no performance problems but if you are worried about performance you can keep this in mind:
Using LIKE instead of = will cause slower queries if you have a lot of rows. (but you must have an extremely large amount of rows for a noticeable difference)
Also, you might do some testing to see which performs better on large tables, select count or select * or select word.
Related
I need to perform a large SELECT WHERE IN query on MySQL and I need it to run quickly. I have a table with more than 100 million rows with the primary key on a varchar 127 (and it has to be that way).
I am performing a SELECT col1 FROM table WHERE col1 IN ($in) where $in has 5000 values. I essentially just need to find which of the 5,000 values are in the table in the primary key col1.
The query takes between 1 and 10 seconds generally but is usually about 7 or 8 seconds.
Is there a more optimal, fast way of performing SELECTS with large IN clauses on huge tables indexed by a varchar?
I am using InnoDB on a dedicated server with PHP and PDO. Thanks for the suggestions.
This is a bit long for a comment.
I am guessing that you already have an index on table(col1), otherwise the query would probably take longer than 10 seconds. If this is not true, then add a column. Better yet, make the column the primary key.
I have a suspicion that the index doesn't fit into memory. For this, you will need to find a MySQL DBA (which you should have if you have such a large table) or learn about the memory options for MySQL. An index not fitting into memory would exhibit this type of behavior.
If this is true, then the behavior should be pretty linear. So, if you have a list of 500 ids, it should take about one second or a bit less. If you have 50 ids, then a tenth of a second or so.
It is possible that sorting the list of ids would help in this case. However, that is really just speculation on my part.
I am creating a database for keeping track of water usage per person for a city in South Florida.
There are around 40000 users, each one uploading daily readouts.
I was thinking of ways to set up the database and it would seem easier to give each user separate a table. This should ease the download of data because the server will not have to sort through a table with 10's of millions of entries.
Am I false in my logic?
Is there any way to index table names?
Are there any other ways of setting up the DB to both raise the speed and keep the layout simple enough?
-Thank you,
Jared
p.s.
The essential data for the readouts are:
-locationID (table name in my idea)
-Reading
-ReadDate
-ReadTime
p.p.s. during this conversation, i uploaded 5k tables and the server froze. ~.O
thanks for your help, ya'll
Setting up thousands of tables in not a good idea. You should maintain one table and put all entries in that table. MySQL can handle a surprisingly large amount of data. The biggest issue that you will encounter is the amount of queries that you can handle at a time, not the size of the database. For instances where you will be handling numbers use int with attribute unsigned, and instances where you will be handling text use varchar of appropriate size (unless text is large use text).
Handling users
If you need to identify records with users, setup another table that might look something like this:
user_id INT(10) AUTO_INCREMENT UNSIGNED PRIMARY
name VARCHAR(100) NOT NULL
When you need to link a record to the user, just reference the user's user_id. For the record information I would setup the SQL something like:
id INT(10) AUTO_INCREMENT UNSIGNED PRIMARY
u_id INT(10) UNSIGNED
reading Im not sure what your reading looks like. If it's a number use INT if its text use VARCHAR
read_time TIMESTAMP
You can also consolidate the date and time of the reading to a TIMESTAMP.
Do NOT create a seperate table for each user.
Keep indexes on the columns that identify a user and any other common contraints such as date.
Think about how you want to query the data at the end. How on earth would you sum up the data from ALL users for a single day?
If you are worried about primary key, I would suggest keeping a LocationID, Date composite key.
Edit: Lastly, (and I do mean this in a nice way) but if you are asking these sorts of questions about database design, are you sure that you are qualified for this project? It seems like you might be in over your head. Sometimes it is better to know your limitations and let a project pass by, rather than implement it in a way that creates too much work for you and folks aren't satisfied with the results. Again, I am not saying don't, I am just saying have you asked yourself if you can do this to the level they are expecting. It seems like a large amount of users constantly using it. I guess I am saying that learning certain things while at the same time delivering a project to thousands of users may be an exceptionally high pressure environment.
Generally speaking tables should represent sets of things. In your example, it's easy to identify the sets you have: users and readouts; there the theoretical best structure would be having those two tables, where the readouts entries have a reference to the id of the user.
MySQL can handle very large amounts of data, so your best bet is to just try the user-readouts structure and see how it performs. Alternatively you may want to look into a document based NoSQL database such as MongoDB or CouchDB, since storing readouts reports as individual documents could be a good choice aswell.
If you create a summary table that contains the monthly total per user, surely that would be the primary usage of the system, right?
Every month, you crunch the numbers and store the totals into a second table. You can prune the log table on a rolling 12 month period. i.e., The old data can be stuffed in the corner to keep the indexes smaller, since you'll only need to access it when the city is accused of fraud.
So exactly how you store the daily readouts isn't that big of a concern that you need to be freaking out about it. Giving each user his own table is not the proper solution. If you have tons and tons of data, then you might want to consider sharding via something like MongoDB.
I am simulating several instruction queues using a mysql table. There is a 'mode' column which is the name for each queue and once items are taken from the queue they are deleted right afterwards. Typical queries look like
SELECT * FROM queue_table WHERE mode='queue1' LIMIT 50.
I am currently using a MYISAM table for this but there is a lot of overhead with all the deleting and optimization takes a long time. I was just wondering if there was a more efficient way to do this and if maybe the database should be INNODB.
InnoDB are useful if your are implying Foreign key constraints .
so one thing for optimizing you Query..
create index onmode column and also dont use * ( may be your table have a lots of columns), write column name which you only need to retrive.
I cannot say I completely understand your problem but I am sure that if you will make your fileds of fixed size (i.e. no text fields, char instead of varchar) the table records become extremely reusable and require no optimization at all.
I have a table with 1 million unique keywords in all languages stored in utf_unicode format. Lately I have been having problems with selects with each select taking up to 1 second. This is really causing a slowdown in the queries.
The structure for the keyword table is (keyword_id, keyword, dirty) -> The keyword_id is the primary key, keyword has unique index and dirty has a simple index. keyword has a varchar type with 20 chars max. The dirty is a boolean.
The problems are being faced when selecting with "keyword" in the where field. How can I speed this table up.
I am using MySQL with PHP.
SAMPLE QUERY
SELECT k_id
FROM table
WHERE keyword = "movies"
Have you considered using a memory table instead of myisam in my experience is goes 10 times faster then myisam. You'll just need another table to rebuild from if the server crashes. Also instead of varchar use char 20. This will make the table a fixed format and mysql will be able to find it's result much faster.
If you have unique keywords and if you aren't doing any similarity/"like" queries, then you can create a hash index. That would guarantee a single row lookup.
The minor disadvantage this may have is that creating a hash index may take up more space than a regular index(btree based).
Refrences:
https://dev.mysql.com/doc/refman/8.0/en/index-btree-hash.html
https://dev.mysql.com/doc/refman/8.0/en/create-index.html
Using PHP, I am building an application that is MySQL database resource heavy, but I also need it's data to be very flexible. Currently there are a number of tables which have an array of different columns (including some text, longtext, int, etc), and in the future I would like to expand on the number of columns of these tables, whenever new data-groups are required.
My question is, if I have a table with, say, 10 columns, and I expand this to 40 columns in the future, would a SQL query (via PHP) be slowed down considerably?
As long as the initial, small query that is only looking up the initial 10 columns is not a SELECT-all (*) query, I would like to know if more resources or processing is used because the source table is now much larger.
Also, will the database in general run slower or be much larger due to many columns now constantly remaining as NULL values (eg, whenever a new entry that only requires the first 10 columns is inserted)?
MyISAM and InnoDB behave differently in this regard, for various reasons.
For instance, InnoDB will partition disk space for each column on disk regardless of whether it has data in it, while MyISAM will compress the tables on disk. In a case where there are large amounts of empty columns, InnoDB will be wasting a lot of space. On the other hand, InnoDB does row-level locking, which means that (with caveats) concurrent read / writes to the same table will perform better (MyISAM does a table-level lock on write).
Generally speaking, it's probably not a good idea to have many columns in one table, particularly for volatility reasons. For instance, in InnoDB (possibly MyISAM also?), re-arranging columns or changing types of columns (i.e. varchar 128 -> varchar 255) in the middle of a table requires that all data in columns to the right be moved around on disk to make (or remove) space for the altered column.
With respect to your overall database design, it's best to aim for as many columns as possible to be not null, which saves space (you don't need the null flag on the column, and you don't store empty data) and also increases query and index performance. If many records will have a particular column set to null, you should probably move it to a foreign key relationship and use a JOIN. That way disk space and index overhead is only incurred for records that are actually holding information.
Likely, the best solution would be to create a new table with the additional fields and JOIN the tables when necessary. The original table remains unchanged, keeping it's speed, but you can still get to the extra fields.
Optimization is not a trivia question. Nothing can be predicted.
In general short answer is: yes, it will be slower (because DBMS at least need to read from the disk and send more data, obviously).
But, it is very dependent on each particular case how much slower it will be. You can either even don't see the difference, or get it 10x times slower.
In all likelihood, no it won't be slowed down considerably.
However, a better question to ask is: Which method of adding more fields results in a more elegant, understandable, maintainable, cost effective solution?
Usually the answer is "It depends." It depends on how the data is accessed, how the requirements will change, how the data is updated, and how fast the tables grow.
you can divide one master table into multiple TRANSACTION tables so you will get much faster result than you getting now. and also make the primary key as UNIQUE KEY also in all the transaction as well as master tables. its really help you to make your query faster.
Thanks.