When trying to execute this query my mysql server cpu usage goes to 100% and the page just stalls. I setup an index on (Client_Code, Date_Time, Time_Stamp, Activity_Code, Employee_Name, ID_Transaction) it doesn't seem to help. What steps can I go about next to fix this issue? Also there is already one index on the database if that matters any. Thanks
Here is what this query does
Database info
ID_Transaction | Client_Code | Employee_Name | Date_Time |Time_Stamp| Activity_Code
1 | 00001 | Eric | 11/15/10| 7:30AM | 00023
2 | 00001 | Jerry | 11/15/10| 8:30AM | 00033
3 | 00002 | Amy | 11/15/10| 9:45AM | 00034
4 | 00003 | Jim | 11/15/10| 10:30AM | 00063
5 | 00003 | Ryan | 11/15/10 | 12:00PM | 00063
6 | 00003 | bill | 11/14/10 | 1:00pm | 00054
7 | 00004 | Jim | 11/15/10 | 1:00pm | 00045
8 | 00005 | Jim | 11/15/10| 10:00 AM| 00045
The query takes the info above and counts it like so. By the most recent entry for each client_code. In this case the query would look like this. After php.
Jerry = 1
2 | 00001 | Jerry | 11/15/10| 8:30AM | 00033
Amy = 1
3 | 00002 | Amy | 11/15/10| 9:45AM | 00034
Ryan = 1
5 | 00003 | Ryan | 11/15/10 | 12:00PM | 00063
Jim = 2
7 | 00004 | Jim | 11/15/10 | 1:00pm | 00045
8 | 00005 | Jim | 11/15/10| 10:00 AM| 00045
$sql = "SELECT m.Employee_Name, count(m.ID_Transaction)
FROM ( SELECT DISTINCT Client_Code FROM Transaction)
md JOIN Transaction m ON
m.ID_Transaction = ( SELECT
ID_Transaction FROM Transaction mi
WHERE mi.Client_Code = md.Client_Code AND Date_Time=CURdate() AND Time_Stamp!='' AND
Activity_Code!='000001'
ORDER BY m.Employee_Name DESC, mi.Client_Code DESC, mi.Date_Time DESC,
mi.ID_Transaction DESC LIMIT 1 )
group by m.Employee_Name";
Is there a better way to write this query so it doesnt bog down my system? The query works fine with 10 database entries but it locks my server up when the database has 300,000 entries.
Thanks
Eric
+----+--------------------+-------------+--------+------------------------+--------------+---------+----------------+------+----------+----------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+--------------------+-------------+--------+------------------------+--------------+---------+----------------+------+----------+----------------------------------------------+
| 1 | PRIMARY | <derived2> | ALL | [NULL] | [NULL] | [NULL] | [NULL] | 8 | 100.00 | Using temporary; Using filesort |
| 1 | PRIMARY | m | index | [NULL] | search index | 924 | [NULL] | 21 | 100.00 | Using where; Using index; Using join buffer |
| 3 | DEPENDENT SUBQUERY | mi | ref | search index,secondary | search index | 18 | md.Client_Code | 3 | 100.00 | Using where; Using temporary; Using filesort |
| 2 | DERIVED | Transaction | index | [NULL] | secondary | 918 | [NULL] | 21 | 38.10 | Using index |
+----+--------------------+-------------+--------+------------------------+--------------+---------+----------------+------+----------+----------------------------------------------+
What about going with multiple GROUP BY's instead of the all the sub queries to simplify things.... something like:
SELECT * FROM Transaction WHERE Date_Time=CURdate() AND Time_Stamp!='' AND Activity_Code != '000001' GROUP BY Client_Code, Employee_Name
If I'm understanding your query correctly then something like this would solve the issues and prevent the need for sub queries.
You'll definitely want to do a join instead of a sub select.
Also, how many records are you viewing? Is pagination and using limit out of the question?
If you set up your initial query modified with inner/outer joins as a view and it doesn't crash, you'll be one step closer. Once the view is set up, you'll be able to use a much less complicated select statement - potentially paginated.
Related
I have a table of words used in the title of articles. I want to find which words which are used the least in the set or article titles.
Example:
Titles:
"Congressman Joey of Texas does not sign bill C1234."
"The pretty blue bird flies at night in Texas."
"Congressman Bob of Arizona is the signs bill C1234."
The table would contain the following.
Table WORDS_LIST
----------------------------------------------------
| INDEX ID | WORD | ARTICLE ID |
----------------------------------------------------
| 1 | CONGRESSMAN | 1234 |
| 2 | JOEY | 1234 |
| 3 | SIGN | 1234 |
| 4 | BILL | 1234 |
| 5 | C1234 | 1234 |
| 6 | TEXAS | 1234 |
| 7 | PRETTY | 1235 |
| 8 | BLUE | 1245 |
| 9 | BIRD | 1245 |
| 10 | FLIES | 1245 |
| 11 | NIGHT | 1245 |
| 12 | TEXAS | 1245 |
| 13 | CONGRESSMAN | 1246 |
| 14 | BOB | 1246 |
| 15 | ARIZONA | 1246 |
| 16 | SIGNS | 1246 |
| 17 | BILL | 1246 |
| 18 | C1234 | 1246 |
----------------------------------------------------
In this case, the words "pretty,blue, flies, night" would be the used in the least number of articles.
I would appreciate any ideas on how to best create this query. So far below is what I started with. I can also write something in PHP but figured a query would be faster.
SELECT distinct a1.`word`, count(a1.`word`)
FROM mmdb.words_list a1
JOIN mmdb.words_list b1
ON a1.id = b1.id AND
upper(a1.word) = upper(b1.word)
where date(a1.`publish_date`) = '2017-06-09'
group by `word`
order by count(a1.`word`);
I don't see why a self-join is necessary. Just do something like this:
select wl.word, count(*)
from mmdb.words_list wl
where date(wl.`publish_date`) = '2017-06-09'
group by wl.word
order by count(*);
You can add a limit to get a fixed number of words. If publish_date is already a date, you should do the comparison as:
where publish_date = '2017-06-09'
If it has a time component:
where publish_date >= '2017-06-09' and publish_date < '2017-06-10'
This expression allows MySQL to use an index.
Try this. It's a bit more simple and should return the correct results:
SELECT `WORD`,
COUNT(*) as `num_articles`
FROM `WORDS_LIST`
WHERE date(`publish_date`) = '2017-06-09'
GROUP BY `WORD`
ORDER BY COUNT(*) ASC;
I have a table with concatenated values within both rows, I am therefore uniquely retrieve ranking for each row in the tables.
UPDATE
The other tables has been added to question
NamesTable
NID | Name |
1 | Mu |
2 | Ni |
3 | ices |
GroupTable
GID | GName |
1 | GroupA |
2 | GroupB |
3 | GroupC |
MainTable
| NID | Ages | Group |
| 1 | 84 | 1 |
| 2 | 64 | 1 |
| 3 | 78 | 1 |
| 1 | 63 | 2 |
| 2 | 25 | 2 |
| 3 | 87 | 2 |
| 1 | 43 | 3 |
| 2 | 62 | 3 |
| 3 | 37 | 3 |
Now the first Name is equated to the first age in the table, I am able to equate them using php and foreach statements, Now the problem is with the ranking of the ages per each group. I am ranking the names uniquely on each row or group.
Results which is expected
| Names | Ages | Group | Ranking |
| Mu,Ni,ices | 84,64,78 | 1 | 1,3,2 |
| Mu,Ni,ices | 63,25,87 | 2 | 2,3,1 |
| Mu,Ni,ices | 43,62,37 | 3 | 2,1,3 |
In my quest to solving this, I am using GROUP_CONCAT, and I have been able to come to this level in the below query
SELECT
GROUP_CONCAT(Names) NAMES,
GROUP_CONCAT(Ages) AGES,
GROUP_CONCAT(Group) GROUPS,
GROUP_CONCAT( FIND_IN_SET(Ages, (
SELECT
GROUP_CONCAT( Age ORDER BY Age DESC)
FROM (
SELECT
GROUP_CONCAT(Ages ORDER BY Ages DESC ) Age
FROM
`MainTable` s
GROUP by `Group`
) s
)
)) rank
FROM
`MainTable` c
GROUP by `Group`
This actually gives me the below results.
| Names | Ages | Group | Ranking |
| 1,2,3 | 84,64,78 | 1 | 7,9,8 |
| 1,2,3 | 63,25,87 | 2 | 5,6,4 |
| 1,2,3 | 43,62,37 | 3 | 2,1,3 |
The only problem is that the ranking Values increase from 1 to 9 instead of ranking each row uniquely. Its there any idea that can help me cross over and fix this? I would be grateful for your help. Thanks
As a developer, I know a good DB guy is worth their weight in gold. I often find myself using seriously inefficient ways to tackle non critical problems, but in this current case, I need speed over 'just make it work'. I won't even show what I've done so far as it's so embarrassing, but lets just say that I had sub queries within my main PHP while loop. Sorry.
I have several tables that I need to join together for a json request to an indexing engine (Apache SOLR).
tbl_contacts
+----+--------------+---------------+
| ID | FirstName | LastName |
+----+--------------+---------------+
| 1 | Joe | Blogs |
+----+--------------+---------------+
| 2 | Jane | Baker |
+----+--------------+---------------+
| 3 | John | Doe |
+----+--------------+---------------+
tbl_attributes_map
+----+--------------+---------------+
| ID | ContactID | AttributeID |
+----+--------------+---------------+
| 1 | 1 | 1 |
+----+--------------+---------------+
| 2 | 1 | 3 |
+----+--------------+---------------+
| 3 | 2 | 2 |
+----+--------------+---------------+
tbl_attributes
+----+---------------+---------------+
| ID | AttributeType | Attribute |
+----+---------------+---------------+
| 1 | Lower | Shoe |
+----+---------------+---------------+
| 2 | Upper | T-Shirt |
+----+---------------+---------------+
| 3 | Upper | Vest |
+----+---------------+---------------+
tbl_notes
+----+---------------+---------------+
| ID | ContactID | Note |
+----+---------------+---------------+
| 1 | 1 | Big feet |
+----+---------------+---------------+
| 2 | 2 | Showoff |
+----+---------------+---------------+
| 3 | 2 | Sweaty |
+----+---------------+---------------+
tbl_appointment
+----+---------------+---------------+--------------------+-------------------+
| ID | ContactID | Location | TimeFrom | TimeTo |
+----+---------------+---------------+--------------------+-------------------+
| 1 | 1 | Big Ben | 2015-12-16 08:00:00|2015-12-16 08:30:00|
+----+---------------+---------------+--------------------+-------------------+
| 1 | 2 | London | 2015-12-17 10:00:00|2015-12-17 11:00:00|
+----+---------------+---------------+--------------------+-------------------+
| 1 | 2 | New York | 2015-12-16 12:00:00|2015-12-16 12:30:00|
+----+---------------+---------------+--------------------+-------------------+
I need to run a query which essentially allows me to print an array structure such as:
Array(
[FirstName] => Joe
[LastName] => Blogs
[Upper] => Array(
Vest
)
[Lower] => Array(
Shoe
)
[Notes] => Array(
Big Feet
)
[Location] => Array(
Big Ben
)
[ApptFrom] => Array(
2015-12-16 08:00:00
)
[ApptTo] => Array(
2015-12-16 08:30:00
)
If I can get to a stage where I can run a query to get the following output, I can run a delimiter to implode on the fields that I need to build an array for. EG:
+----+------------+-----------+---------+---------+-----------------+-------------------+-----------------------------------------+-----------------------------------------+
| ID | FirstName | LastName | Upper | Lower | Notes | Location | ApptFrom | ApptTo |
+----+------------+-----------+---------+---------+-----------------+-------------------+-----------------------------------------+-----------------------------------------+
| 2 | Jane | Baker | T-Shirt | | Show off,Sweaty | London,New York | 2015-12-17 10:00:00,2015-12-16 12:00:00 | 2015-12-17 11:00:00,2015-12-16 12:30:00 |
+----+------------+-----------+---------+---------+-----------------+-------------------+-----------------------------------------+-----------------------------------------+
My script currently works, but at a high performance penalty. It takes around 3 hours to churn through 80,000 contacts :-/
Thanks in advance.
I have exactly similar situation when I start indexing Elasticsearch (use Apache Lucene as SOLR) 50 million of data, now it only take me couple of hours I think what you can do are following:
put explain on your query to see if your query are using the proper index
try to use more sub select instead of join because mysql have problem to select with index for million of data (you can use force index maybe)
start muti-thread to do your indexing
I have got this mySQL table:
id | url | origin | categroy | date
1 | url.com/1 | US | news | 2015-10-01 12:39:19
2 | url.com/1 | | | 2015-10-01 13:59:34
3 | url.com/2 | CN | news | 2015-09-01 12:45:26
4 | url.com/3 | | | 2015-08-12 09:10:10
5 | url.com/3 | US | news | 2015-09-14 04:56:36
6 | url.com/4 | US | | 2015-09-09 12:12:09
is there a way to group the entries by the url and remove the duplicates in one mySQL call without creating a new table?
desired table:
1 | url.com/1 | US | news | 2015-10-01 12:39:19
3 | url.com/2 | CN | news | 2015-09-01 12:45:26
5 | url.com/3 | US | news | 2015-09-14 04:56:36
6 | url.com/4 | US | | 2015-09-09 12:12:09
I am a beginner and I tried to solve this with a php script which failed. I guess there is a fairly easy SQL-answer to that, but I couldn't find an easy answer on Stackoverflow.
Thank you very much!
Try this query
SELECT id, url,
SUBSTRING_INDEX(GROUP_CONCAT(origin ORDER BY date DESC),',',1) as origin_value,
SUBSTRING_INDEX(GROUP_CONCAT(category ORDER BY date DESC),',',1) as category_value,
SUBSTRING_INDEX(GROUP_CONCAT(date ORDER BY date DESC),',',1) as date_value FROM tablename
GROUP BY url
ORDER BY url,date DESC
I have 2 tables - users and users_audit, need to display results from 2 separate queries into 2 columns with relation to each other. I already figured out both queries.
query for users: SELECT rowid,data FROM users WHERE columnid = 1 GROUP BY rowid;
+-------+------------+
| rowid | data |
+-------+------------+
| 1 | John |
| 2 | Abi |
| 3 | Tony |
| 4 | Gregg |
| 5 | Jon |
| 6 | Victor |
| 7 | Daniel |
query for users_audit: SELECT date_created FROM users_audit WHERE rowid = 6 ORDER BY date_created DESC, date_created DESC LIMIT 1;
Purpose of the query is to display latest date_created for particular rowid.
+---------------------+
| date_created |
+---------------------+
| 2012-07-04 09:20:12 |
+---------------------+
Table users_audit looks like this:
SELECT * FROM users_audit;
+-------+---------+--------------+-------------+---------------------+
| id | rowid | before_value | after_value | date_created |
+-------+---------+--------------+-------------+---------------------+
| 1 | 6 | 3 | 5 | 2012-06-29 15:48:11 |
| 2 | 5 | Out (0) | 2 | 2012-07-04 09:20:10 |
| 3 | 6 | 5 | 4 | 2012-07-04 09:20:12 |
| 4 | 7 | 3 | 6 | 2012-07-04 09:20:14 |
| 5 | 3 | 2 | 3 | 2012-07-04 09:20:16 |
| 6 | 15 | 6 | 5 | 2012-07-04 09:20:22 |
I need to display 2 columns - data from users and date_created from users_audit for each rowid. It means query for users_audit must be run for each rowid (in the loop?).
Expected result displayed in php is:
+------------+----------------------+
| data | date_created |
+------------+----------------------+
| John | (latest date) |
| Abi | (latest date) |
| Tony | (latest date) |
| Gregg | (latest date) |
| Jon | (latest date) |
| Victor | 2012-07-04 09:20:12 |
| Daniel | (latest date) |
How can this be achieved this in php?
Why cant you just create a view and use a join to create the table you want?
SELECT users.data, users_audit.date_created FROM users
INNER JOIN users_audit ON users.rowid = users_audit.rowid
Or am I missing something here?
EDIT
SELECT users.data, users_audit.date_created FROM users
INNER JOIN users_audit ON users.rowid = users_audit.rowid
WHERE users_audit.date_created = (SELECT MAX(date_created) from users_audit
GROUP BY rowid)
I believe this should work..? Let me know.