I have a Reporting table where i store description
tableA
sno | Project |name | description | mins |
1 | prjA |nameA |ABC -10% task done| 30 |
...
3000 | prjA |nameB |ABC -70% task done| 70 |
i want to query the description field and save in another table
tableB
id | valueStr | total_mins | last_sno
1 | ABC | 100 | 3000
if there is no entry in second table , i create a entry with default values
if there is and entry in second table , i update 2nd table , with the total_mins and increment the last_sno to that value say 3300 , so that the next time i query this table i get values from second table and based on the last_sno
Query
SELCT last_sno FROM tableB where valueStr ='ABC'
the first 3 characters in the description field
SELECT max(sno), sum(mins) FROM tableA
where sno > last_sno and description like 'ABC%'
Since the first table has million of rows so,
i search the first table with sno > last_sno , so that should help performance right ?
but the explain shows that it scans the same no of rows , when i query the first table from the first sno
The use of the index may not help you, because MySQL still has to scan the index from the last_sno to the end of the data. You would be better off with an index on TableA(description), because such an index can be used for description like 'ABC%'.
In fact, this might be a case where the index can hurt you. Instead of sequentially reading the pages in the table, the index reads them randomly -- which is less efficient.
EDIT: (too long for comment)
Try running the query with an ignore index hint to see if you can run the query without it. It is possible that the index is actually making things worse.
However, the "real" solution is to store the prefix you are interested in as a separate column. You can then add an index on this column and the query should work efficiently using basic SQL. You won't have to spend your time trying to optimize a simple process, because the data will be stored correctly for it.
Related
I have 2 table like this:
1-private_messages table:
messageId | message
--------------------
1 | text1
4 | text4
2-public_messages table:
messageId | message
----------------------
2 | text2
3 | text3
5 | text5
in two table , messageId column is primaryKey
now I want that these two column be auto increment and has a unique Id in two table like shown above.
now when I want to insert a row in one of tables , I had to find max Id of each table and compare them to find max of them. then increase that and insert new row.
I want know, is there any better or automatic way that when I insert new row, database do that automatically?
thanks
You can obtain unique numbers in MySQL with a programming pattern like the following.
First create a table for the sequence. It has an auto-increment field and nothing else.
CREATE TABLE sequence (
sequence_id INT NOT NULL AUTO_INCREMENT,
PRIMARY KEY (`sequence_id`)
)
Then when you need to insert a unique number into one of your tables, use something like these queries:
INSERT INTO sequence () VALUES ();
DELETE FROM sequence WHERE sequence_id < LAST_INSERT_ID();
INSERT INTO private_messages (messageID, message)
VALUES (LAST_INSERT_ID(), 'the message');
The second INSERT is guaranteed to use a unique sequence number. This guarantee holds even if you have dozens of different client programs connected to your database. That's the beauty of AUTO_INCREMENT.
The second query (DELETE) keeps the table from getting big and wasting space. We don't care about any rows in the table except for the most recent one.
Edit. If you're using php, simply issue the three queries one after the other using three calls to mysqli_query() or the equivalent method in the MySQL interface you have chosen for your program.
All that being said, beware of false economy. Don't forget that storage on Amazon S3 costs USD 0.36 per year per gigabyte. And that's the most expensive storage. The "wasted" storage cost for putting your two kinds of tables into a single table will likely amount to a few dollars. Troubleshooting a broken database app in production will cost thousands of dollars. Keep it simple!
Use flag like 1 for private messages and 0 for public in a single table , so it is easy to insert and easy to fetch and compare....
messageId | message | flag
---------------------------
1 | text1 | 1
2 | text2 | 0
3 | text3 | 0
4 | text4 | 1
5 | text5 | 0
There is no way to do this automatically that I'm aware of.
You might be able to write a function in the DB to make it happen, I don't recommend it.
Mark Baker's suggestion, to have a single messages table and a public/private flag sounds like the best way to go if you absolutely need IDs to be unique across both types of messages.
I have a table which contains a standard auto-incrementing ID, a type identifier, a number, and some other irrelevant fields. When I insert a new object into this table, the number should auto-increment based on the type identifier.
Here is an example of how the output should look:
id type_id number
1 1 1
2 1 2
3 2 1
4 1 3
5 3 1
6 3 2
7 1 4
8 2 2
As you can see, every time I insert a new object, the number increments according to the type_id (i.e. if I insert an object with type_id of 1 and there are 5 objects matching this type_id already, the number on the new object should be 6).
I'm trying to find a performant way of doing this with huge concurrency. For example, there might be 300 inserts within the same second for the same type_id and they need to be handled sequentially.
Methods I've tried already:
PHP
This was a bad idea but I've added it for completeness. A request was made to get the MAX() number for the item type and then add the number + 1 as part of an insert. This is quick but doesn't work concurrently as there could be 200 inserts between the request for MAX() and that particular insert leading to multiple objects with the same number and type_id.
Locking
Manually locking and unlocking the table before and after each insert in order to maintain the increment. This caused performance issues due to the number of concurrent inserts and because the table is constantly read from throughout the app.
Transaction with Subquery
This is how I'm currently doing it but it still causes massive performance issues:
START TRANSACTION;
INSERT INTO objects (type_id,number) VALUES ($type_id, (SELECT COALESCE(MAX(number),0)+1 FROM objects WHERE type_id = $type_id FOR UPDATE));
COMMIT;
Another negative thing about this approach is that I need to do a follow up query in order to get the number that was added (i.e. searching for an object with the $type_id ordered by number desc so I can see the number that was created - this is done based on a $user_id so it works but adds an extra query which I'd like to avoid)
Triggers
I looked into using a trigger in order to dynamically add the number upon insert but this wasn't performant as I need to perform a query on the table I'm inserting into (which isn't allowed so has to be within a subquery causing performance issues).
Grouped Auto-Increment
I've had a look at grouped auto-increment (so that the number would auto-increment based on type_id) but then I lose my auto-increment ID.
Does anybody have any ideas on how I can make this performant at the level of concurrent inserts that I need? My table is currently InnoDB on MySQL 5.5
Appreciate any help!
Update: Just in case it is relevant, the objects table has several million objects in it. Some of the type_id can have around 500,000 objects assigned to them.
Use transaction and select ... for update. This will solve concurrency conflicts.
In Transaction with Subquery
Try to make index on column type_id
I think by making index on column type_id it will speed up your subquery.
DROP TABLE IF EXISTS my_table;
CREATE TABLE my_table
(id INT NOT NULL AUTO_INCREMENT PRIMARY KEY
,type_id INT NOT NULL
);
INSERT INTO my_table VALUES
(1,1),(2,1),(3,2),(4,1),(5,3),(6,3),(7,1),(8,2);
SELECT x.*
, COUNT(*) rank
FROM my_table x
JOIN my_table y
ON y.type_id = x.type_id
AND y.id <= x.id
GROUP
BY id
ORDER
BY type_id
, rank;
+----+---------+------+
| id | type_id | rank |
+----+---------+------+
| 1 | 1 | 1 |
| 2 | 1 | 2 |
| 4 | 1 | 3 |
| 7 | 1 | 4 |
| 3 | 2 | 1 |
| 8 | 2 | 2 |
| 5 | 3 | 1 |
| 6 | 3 | 2 |
+----+---------+------+
or, if performance is an issue, just do the same thing with a couple of #variables.
Perhaps an idea to create a (temporary) table for all rows with a common "type_id".
In that table you can use auto-incrementing for your num colomn.
Then your num shoud be fully trustable.
Then you can select your data and update your first table.
I have a table like this:
ID build1 build2 test status
1 John ram test1 pass
2 john shyam test2 fail
3 tom ram test1 fail
The problem that I am facing is - on one of my webpage, only the values from the column "uild1" are available to me. Now in table there are 2 entries corresponding to "John". so, even if the user selects different "John", its showing the values for other values from the row only. On my webpage, in the drop down list, user can see 2 "John" but since query has been made using "John" condition, on both occasions, its showing the results from the first row only.
Try this:
SELECT t1.*
FROM Table1 t1
WHERE t1.build1 NOT IN(SELECT t2.build1
FROM table1 t2
GROUP BY t2.build1
HAVING COUNT(t2.build1) > 1);
SQL Fiddle Demo
This will give you only:
| ID | BUILD1 | BUILD2 | TEST | STATUS |
-----------------------------------------
| 3 | tom | ram | test1 | fail |
Since, it is the only row that has no duplicate build1.
If I'm understanding your question correctly, given a web page with 2 johns available to click on, how can you get each result accordingly? Unfortunately, there is no way of doing this with just SQL.
In your PHP code, if you can pass a parameter to your SQL code with either the ID or a counter/row number, then you could query the database to return a corresponding unique record.
Good luck.
You build1 is not unique or primary key so it is picking all the row matching your condition. You should use primary key or unique key to find the result. In your select drop-down your option value should be uniq/primary key so when you select particular "John" it will get result of that john.
select * from table_name where id=params[:id] ;
If you post some more information. It will be helpful to write better code for you.
select * from yourtable where build1 == 'john' limit 1;
I'm working in a commenting application and i would like some feedback on the method that i am using to keep track of the number of replies or likes that a comment has. Comments and replies are stored in the same table, to determine if a comment is a reply i use the field parent_id if it is anything other than 0 the comment is a reply.
Please note that i wont be including all the columns of the table below:
cid | parent_id | replies | likes
-----+-----------+---------+-------
2 | 0 | 3 | 0
3 | 2 | 0 | 0
4 | 2 | 0 | 2
5 | 2 | 0 | 0
In the table above comments with id (cid) [3,4,5] are replies of comment #2. The columns replies and likes are integer that hold the count of replies and likes accordingly. The integrity and accuracy of these columns is maintain and updated through the PHP code, for example if another reply for comment #2 is added than the replies column would be increased by one or decreased by one if deleted.
Im also aware that i could dynamically calculate the replies count in the SQL query that fetches the comments but i thought it would add more stress to the SQL server. This query would look something like these:
SELECT cid, parent_id, (
SELECT count(*)
FROM comments as SC
WHERE RC.parent_id = C.cid
) AS replies
FROM comments AS C
WHERE thread = {thread_id}
Am i doing it right by storing the replies and likes in an actual column in the table? or am i exaggerating about the stress that a query such as the one above would have in the MySql server and i should use such complex query instead?
Any feedback would be appreciated, thanks
I dont think you need the column called 'replies'. Just occupies additional unwanted space.
Do a combined Index on cid and parentId. That should be good enough. Queries should be fast.
By having the column, you are adding more stress to app code & mysql. (App code for maintaining integrity and mysql coz 2 writes in the place of 1 write - when a comment is entered).
But if you are talking about millions of rows, i wouldnt choose mysql for it, rather mongo, the data can be constructed as a beautiful JSON and dumped in mongo.
my problem is caused when i try to get the COUNT of a consult return a big amount of records (example 500.000):
Example:
$limit = ' LIMIT 0,30';
$ResultQuery = mysql_query("
SELECT id, name, description, date
FROM products
WHERE name
LIKE 'dog%'".$limit);
$CountQuery = mysql_query("
SELECT COUNT(id)
FROM products
WHERE name LIKE 'dog%'");
while ($Product = mysql_fetch_assoc($ResultQuery)) { [...]
NOTE: The Use of COUNT(id) its more fast (in my case) than use mysql_num_rows of $ResultQuery.
If i see what is doing the server using the MySQL Administrator, i see 3 seconds make the 1º Query (the limit), one second "sending data", and 143 seconds "sending data" of the 2º Query.
I read more articles about this problem its caused because for get the count of the query, need to scan ALL ROWS (without the limit) of the Query.
Not exists any method or mode to bypass this problem? Any method to extract the count of rows with big number of results?
Thanks for the help.
My best bet is you're not having set your indexes up correctly. By the looks of it, you haven't set up a proper index for your name field and that causes MySQL to go through every row (more than 22 000 000) to look for 'dog%'.
I suggest you try using a regular index and benchmark the results
CREATE INDEX `idx_name` ON products (name)
id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra
1 | SIMPLE | products | ALL | name | NULL | NULL | NULL | 22160980 | Using where
You can see from the above that the index "name" is not being used. And the reason for that is the following.
You are using Full Text index on the name column which is only useful for Full-text search queries
http://dev.mysql.com/doc/refman/5.5/en/fulltext-search.html
If you do not do any full-text searches, delete that index and create another index as suggested above
CREATE INDEX idx_name ON products (name)
This will create a BTREE index on name which will then be used in your search query. Note that this index will not be useful if you do a like query with '%' at the beginning. For example
SELECT count(id) FROM products WHERE name LIKE '%dogs'
will not use the index.
$count = mysql_num_rows($ResultQuery);