my problem is caused when i try to get the COUNT of a consult return a big amount of records (example 500.000):
Example:
$limit = ' LIMIT 0,30';
$ResultQuery = mysql_query("
SELECT id, name, description, date
FROM products
WHERE name
LIKE 'dog%'".$limit);
$CountQuery = mysql_query("
SELECT COUNT(id)
FROM products
WHERE name LIKE 'dog%'");
while ($Product = mysql_fetch_assoc($ResultQuery)) { [...]
NOTE: The Use of COUNT(id) its more fast (in my case) than use mysql_num_rows of $ResultQuery.
If i see what is doing the server using the MySQL Administrator, i see 3 seconds make the 1º Query (the limit), one second "sending data", and 143 seconds "sending data" of the 2º Query.
I read more articles about this problem its caused because for get the count of the query, need to scan ALL ROWS (without the limit) of the Query.
Not exists any method or mode to bypass this problem? Any method to extract the count of rows with big number of results?
Thanks for the help.
My best bet is you're not having set your indexes up correctly. By the looks of it, you haven't set up a proper index for your name field and that causes MySQL to go through every row (more than 22 000 000) to look for 'dog%'.
I suggest you try using a regular index and benchmark the results
CREATE INDEX `idx_name` ON products (name)
id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra
1 | SIMPLE | products | ALL | name | NULL | NULL | NULL | 22160980 | Using where
You can see from the above that the index "name" is not being used. And the reason for that is the following.
You are using Full Text index on the name column which is only useful for Full-text search queries
http://dev.mysql.com/doc/refman/5.5/en/fulltext-search.html
If you do not do any full-text searches, delete that index and create another index as suggested above
CREATE INDEX idx_name ON products (name)
This will create a BTREE index on name which will then be used in your search query. Note that this index will not be useful if you do a like query with '%' at the beginning. For example
SELECT count(id) FROM products WHERE name LIKE '%dogs'
will not use the index.
$count = mysql_num_rows($ResultQuery);
Related
I have an PHP/MySQL application which is connected to a database featuring 2 tables called 'displays' and 'display_substances'. The structure of these tables is as follows:
mysql> DESCRIBE displays;
+----------+----------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+----------+----------------------+------+-----+---------+----------------+
| id | smallint(5) unsigned | NO | PRI | NULL | auto_increment |
| label | varchar(255) | NO | | NULL | |
+----------+----------------------+------+-----+---------+----------------+
mysql> DESCRIBE display_substances;
+--------------+-----------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+--------------+-----------------------+------+-----+---------+----------------+
| id | mediumint(8) unsigned | NO | PRI | NULL | auto_increment |
| display_id | smallint(5) unsigned | NO | MUL | NULL | |
| substance_id | mediumint(8) unsigned | NO | MUL | NULL | |
| value | text | NO | | NULL | |
+--------------+-----------------------+------+-----+---------+----------------+
There is also a 'substances' table and the foreign key display_substances.substance_id is associated with substances.id.
The 'displays' table contains exactly 400 rows and the 'display_substances' approx 1.5 million rows.
What I'm trying to do is output all 400 'displays.label' into a HTML table, and then in a second column show the 'display_substances.value'. Since the page only shows 1 substance at a time it also needs to be based on the 'display_substances.substance_id'.
The issue I'm having is that records only exist in 'display_substances' when we have data available for the appropriate 'displays'. However, the output has to show all records from 'displays' and then put the text "Not Listed" next to anything where there is no corresponding record in 'display_substances'.
I've done the following - which gives me the output I want - but is flawed (see "The Problem" section below).
Select all of the records in the "displays" table: SELECT label FROM displays ORDER BY label ASC
Select all display_substances.display_id for the substance currently being shown: SELECT display_id FROM display_substances WHERE substance_id = 1 (assuming 1 is the current substance ID). Store the ID's in an array called $display_substances
Loop through (1) and use in_array() to see if (2) exists:
foreach ($displays as $display) { // step (1)
$display_substances = // See step (2)
if (in_array($display['id'], $display_substances)) { // step (3)
$display_value = // See step (4)
} else {
$display_value = 'Not listed';
}
$output[] = ['display_label' => $display['label'], 'display_value' => $display_value]; // See step (5)
}
If the in_array() condition is true then I make a query to select the corresponding row of "display_substances": SELECT value FROM display_substances WHERE display_id = $display['id'] AND substance_id = 1
The $output variable buffers all the data and then it gets output into a HTML table later. The output I get is exactly as I want.
The problem
Although the output is correct I want to do this all as 1 query (if possible) because I need to add features to search by either displays.label or display_substances.value - or a combination of both. The first part of this is fairly trivial because I can amend the query in (1) to:
SELECT label FROM displays WHERE label LIKE '% Foo %' ORDER BY label ASC
However, this won't make display_substances.value searchable because by the time we get to step (3) we're dealing with a single row of display_substances not the whole table. I can't see how to write it differently though since we need to know which records exist in that table for the loaded substance.
I have also written the following query - but this will not work because it misses anything that's "Not Listed":
SELECT displays.label, display_substances.`value` FROM displays JOIN display_substances ON displays.id = display_substances.display_id WHERE display_substances.substance_id = 1
I have read How do I get the records from the left side of a join that do not exist in the joined table? but that didn't help.
For further clarification
Let's say there are 120 rows in display_substances that correspond to substance ID 1 (WHERE display_substances.substance_id = 1). The output of the query should always have 400 rows. In this example, 120 should have display_substances.value next to them, and 280 should have the text "Not Listed".
You need a left join & a group_concat to get all records on the left table along with group by.
But keep in mind that group_concat has a limit so you might not get all associated records, as it's usually used for small fields but since you have a 'text' field for your value there's a high probability you'd hit the limit
Anyway here's the query
SELECT d.*, GROUP_CONCAT(ds.value) `substances`
FROM displays `d`
LEFT JOIN display_substances `ds` ON `d`.`id` = `ds`.`display_id`
GROUP BY `d`.`id`
Something like this might work then if I understand correctly
SELECT d.*, IFNULL((SELECT GROUP_CONCAT(value) FROM display_substances `ds` WHERE `ds`.`display_id` = `d`.`id` GROUP BY `ds`.`display_id`), 'Not Listed') `substances`
FROM displays `d`
You can update the where & add AND substance_id = 1
My understanding is that you want the following:
Your page limits the results by substance ID
You want one substance per row
If there are displays with no substances, they should still show in the page with "Not Listed" as the substance value
I believe this should work for you:
SELECT
d.id AS display_id,
d.label AS display_label,
IFNULL(ds.value, 'Not Listed') AS substance_value
FROM displays AS d
LEFT JOIN display_substances AS ds ON (ds.display_id = d.id)
WHERE ds.substance_id = 1 OR ds.substance_id IS NULL;
I've realized that Ramy has about 98% of the solution.
FWIW this problem is just a variation on one that occurs all the time.
You will find other answers on SO when you search for 'left outer join with where clause' -- that address the problem. One example is this question.
Ultimately, you have a many to many resolution table (display_substances) that resolves the many to many relationship between substances and displays. You are just looking for an outer join from one of the 2 parent tables, but also requiring that you filter the results by a specific substance.
SELECT
d.id AS display_id,
d.label AS display_label,
IFNULL(ds.value, 'Not Listed') AS substance_value
FROM displays AS d
LEFT JOIN display_substances AS ds ON (ds.display_id = d.id AND ds.substance_id = 1);
This query does not generate a value of 'not listed' but it does generate NULL columns for those display rows where there is no corresponding display_substance value. You could embelish it with the IF_NULL() function demonstrated by ahmad, but as you are using PHP to go through the result set, you can just as easily handle that in the procedural loop you'll use to fetch the results.
I'm posting what I've used as a solution. I don't think it's possible to do this in MySQL as one query. Several people answered but none of the answers worked.
For even further clarification - although obvious from the question - the output in the application is a table with 2 columns. The first of these columns should include all 400 rows from displays:
displays.label | display_substances.value
------------------|--------------------------
Display 1
------------------|--------------------------
...
------------------|--------------------------
Display 400
------------------|--------------------------
This is fairly trivial since at this point it's just SELECT * FROM displays.
The challenge begins when we want to populate the second column of the table with display_substances.value. The data for a given substance (assume substance ID is 1) might look like this:
id | display_id | substance_id | value
-----|----------------|-----------------|-------------
206 | 1 | 1 Foo
-----|----------------|-----------------|-------------
361 | 3 | 1 Bar
-----|----------------|-----------------|-------------
555 | 5 | 1 Baz
-----|----------------|-----------------|-------------
The problem: In this case we only have 3 records for substance ID 1. So if we do a JOIN query, it will only return 3 rows. But the table we are displaying in the application needs to show all 400 rows from displays and put the text "Not listed" on any row where there is no corresponding row in display_substances. So in the example above, when it encounters display_id 2, 4, 6...400 it should say "Not listed" (because we only have data for three display_substances.display_id [1,3,5] for substance ID 1).
Both columns also need to be searchable.
My solution
I don't think it's possible to do this in MySQL so I resorted to using PHP. The logic is as now follows:
If the user is doing a search on column 2 (display_substances.value): SELECT display_id FROM display_substances WHERE value LIKE '% Search term column 2 %'. Store this as an array, $ds.
Select all 400 records from displays. If the user is performing a search on column 1 (displays.label) then that must form part of the query: SELECT * FROM displays WHERE label LIKE '% Search term column 1 %'. Critically - if the $ds array from step (1) is not empty then the following must become part of query: WHERE displays.id IN (2, 4, 6...400). Store this as an array, $displays
Get all of the display_id's associated with the substance being viewed: SELECT display_id FROM display_substances WHERE substance_id = 1
Do a loop as per point (3) of the original question.
The result is that the page loads in <2 seconds, each column is searchable.
The SQL queries given in answers took - at best - around 15-20 seconds to execute and never gave all 400 rows.
If anyone can improve on this or has a pure SQL solution please post.
I have a Reporting table where i store description
tableA
sno | Project |name | description | mins |
1 | prjA |nameA |ABC -10% task done| 30 |
...
3000 | prjA |nameB |ABC -70% task done| 70 |
i want to query the description field and save in another table
tableB
id | valueStr | total_mins | last_sno
1 | ABC | 100 | 3000
if there is no entry in second table , i create a entry with default values
if there is and entry in second table , i update 2nd table , with the total_mins and increment the last_sno to that value say 3300 , so that the next time i query this table i get values from second table and based on the last_sno
Query
SELCT last_sno FROM tableB where valueStr ='ABC'
the first 3 characters in the description field
SELECT max(sno), sum(mins) FROM tableA
where sno > last_sno and description like 'ABC%'
Since the first table has million of rows so,
i search the first table with sno > last_sno , so that should help performance right ?
but the explain shows that it scans the same no of rows , when i query the first table from the first sno
The use of the index may not help you, because MySQL still has to scan the index from the last_sno to the end of the data. You would be better off with an index on TableA(description), because such an index can be used for description like 'ABC%'.
In fact, this might be a case where the index can hurt you. Instead of sequentially reading the pages in the table, the index reads them randomly -- which is less efficient.
EDIT: (too long for comment)
Try running the query with an ignore index hint to see if you can run the query without it. It is possible that the index is actually making things worse.
However, the "real" solution is to store the prefix you are interested in as a separate column. You can then add an index on this column and the query should work efficiently using basic SQL. You won't have to spend your time trying to optimize a simple process, because the data will be stored correctly for it.
I have a table like this:
ID build1 build2 test status
1 John ram test1 pass
2 john shyam test2 fail
3 tom ram test1 fail
The problem that I am facing is - on one of my webpage, only the values from the column "uild1" are available to me. Now in table there are 2 entries corresponding to "John". so, even if the user selects different "John", its showing the values for other values from the row only. On my webpage, in the drop down list, user can see 2 "John" but since query has been made using "John" condition, on both occasions, its showing the results from the first row only.
Try this:
SELECT t1.*
FROM Table1 t1
WHERE t1.build1 NOT IN(SELECT t2.build1
FROM table1 t2
GROUP BY t2.build1
HAVING COUNT(t2.build1) > 1);
SQL Fiddle Demo
This will give you only:
| ID | BUILD1 | BUILD2 | TEST | STATUS |
-----------------------------------------
| 3 | tom | ram | test1 | fail |
Since, it is the only row that has no duplicate build1.
If I'm understanding your question correctly, given a web page with 2 johns available to click on, how can you get each result accordingly? Unfortunately, there is no way of doing this with just SQL.
In your PHP code, if you can pass a parameter to your SQL code with either the ID or a counter/row number, then you could query the database to return a corresponding unique record.
Good luck.
You build1 is not unique or primary key so it is picking all the row matching your condition. You should use primary key or unique key to find the result. In your select drop-down your option value should be uniq/primary key so when you select particular "John" it will get result of that john.
select * from table_name where id=params[:id] ;
If you post some more information. It will be helpful to write better code for you.
select * from yourtable where build1 == 'john' limit 1;
I have MySQL Table with an Order table in a DB this table has an auto-incremental id. Up-till now we have general numeric Order-ID likewise 1,2,3,4,5... From now onwards I have to append the series A20 to my id like A20103, A20104, A20105 and so on and when the last three digits reaches 999 the series appended should get changed to A21001, A21002, A21003 and so on, the same series has to be added in the previously added orders..
How can i achieve this task? please guide
Altering an existing auto_increment column does not sound like a good idea - do you really have to do this? Instead, why not just modify your select query to return a suitably formatted id? By doing so, you maintain referential integrity, and you are also free to change the order id format at any time in the future, without having to update your database.
SELECT id, CONCAT('A2', LPAD(id, 4, '0')) AS order_id FROM <table>;
Example output:
+------+----------+
| id | order_id |
+------+----------+
| 1 | A20001 |
| 2 | A20002
...
| 999 | A20999 |
| 1000 | A21000 |
| 1001 | A21001 |
+------+----------+
something along the lines of:
"AR2" . str_pad((int) $ordernumber, 4, "0", STR_PAD_LEFT);
jim
[edit] - i'm assuming this is for display purposes as stated elsewhere, the ID field on the DB is integer!!
You can't have an auto-increment which is not a numeric field. You will better keep the current auto-incrementing column, and add a new one which you will compute manually according to your rules.
You'll probably want to use the MAX() function to get the latest order and generate the next value: remember to do it within a transaction.
You could create a function or a trigger, to do the insert cleanly for you.
You can't add prefixes directly to the database. However, when selecting it you can prepend it.
SELECT concat('A', id) as id FROM table
To get the effect of starting from 20000 you can set the auto increment starting value.
I am trying to search MySQL database with a search key entered by the user. My data contain upper case and lower case. My question is how to make my search function not case sensitive. ex:data in mysql is BOOK but if the user enters book in search input. The result is not found....Thanks..
My search code
$searchKey=$_POST['searchKey'];
$searchKey=mysql_real_escape_string($searchKey);
$result=mysql_query("SELECT *
FROM product
WHERE product_name like '%$searchKey%' ORDER BY product_id
",$connection);
Just uppercase the search string and compare it to the uppercase field.
$searchKey= strtoupper($_POST['searchKey']);
$searchKey=mysql_real_escape_string($searchKey);
$result=mysql_query("SELECT * FROM product
WHERE UPPER(product_name) like '%$searchKey%' ORDER BY product_id
",$connection);
If possible, you should avoid using UPPER as a solution to this problem, as it incurs both the overhead of converting the value in each row to upper case, and the overhead of MySQL being unable to use any index that might be on that column.
If your data does not need to be stored in case-sensitive columns, then you should select the appropriate collation for the table or column. See my answer to how i can ignore the difference upper and lower case in search with mysql for an example of how collation affects case sensitivity.
The following shows the EXPLAIN SELECT results from two queries. One uses UPPER, one doesn't:
DROP TABLE IF EXISTS `table_a`;
CREATE TABLE `table_a` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`value` varchar(255) DEFAULT NULL,
INDEX `value` (`value`),
PRIMARY KEY (`id`)
) ENGINE=InnoDB;
INSERT INTO table_a (value) VALUES
('AAA'), ('BBB'), ('CCC'), ('DDD'),
('aaa'), ('bbb'), ('ccc'), ('ddd');
EXPLAIN SELECT id, value FROM table_a WHERE UPPER(value) = 'AAA';
+----+-------------+---------+-------+---------------+-------+---------+------+------+--------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+---------+-------+---------------+-------+---------+------+------+--------------------------+
| 1 | SIMPLE | table_a | index | NULL | value | 258 | NULL | 8 | Using where; Using index |
+----+-------------+---------+-------+---------------+-------+---------+------+------+--------------------------+
EXPLAIN SELECT id, value FROM table_a WHERE value = 'AAA';
+----+-------------+---------+------+---------------+-------+---------+-------+------+--------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+---------+------+---------------+-------+---------+-------+------+--------------------------+
| 1 | SIMPLE | table_a | ref | value | value | 258 | const | 2 | Using where; Using index |
+----+-------------+---------+------+---------------+-------+---------+-------+------+--------------------------+
Notice that the first SELECT which uses UPPER has to scan all the rows, whereas the second only needs to scan two - the two that match. On a table this size, the difference is obviously imperceptible, but with a large table, a full table scan can seriously impact the speed of your query.
This is an easy way to do it:
$searchKey=strtoupper($searchKey);
SELECT *
FROM product
WHERE UPPER(product_name) like '%$searchKey%' ORDER BY product_id
First of all, try to avoid using * as much as possible. It is generally considered a bad idea. Select the columns using column names.
Now, your solution would be -
$searchKey=strtoupper($_POST['searchKey']);
$searchKey=mysql_real_escape_string($searchKey);
$result=mysql_query("SELECT product_name,
// your other columns
FROM product
WHERE UPPER(product_name) like '%$searchKey%' ORDER BY product_id
",$connection);
EDIT
I will try to explain why it is a bad idea to use *. Suppose you need to change the schema of the product table(adding/deleting columns). Then, the columns that are being selected through this query will change, which may cause unintended side effects and will be hard to detect.
According to the MySQL manual, case-sensitivity in searches depends on the collation used, and should be case-insensitive by default for non binary fields.
Make sure you have the field types and the query right (maybe there's an extra space or something). If that doesn't work, you can convert the string to upper case in PHP (ie: $str = strtoupper($str)) and do the same on the MySQL side (#despart)
EDIT: I posted the article above (^). AndI just tested it. Searches on CHAR, VARCHAR, and TEXT fields are case-insensitive (collation = latin1)