PIVOT table mysql with data from multiple tables - php

I have a school management system, this system has to analyze data, especially student marks.
I want to be able to show student performance, in the following way;
Show a table that shows the students marks per subject as well as the student average mark and position.
Example
Position | Class | Student Average | Student Name | Geography | History | Biology | English | Math |
10 Grade11 57% John Doe 59% 40% 66% 48% 56%
11 Grade11 56% John Smith 53% 33% 56% 68% 26%
12 Grade11 55% Paul Doe 29% 30% 46% 38% 36%
This is the code I am using and it is not displaying the marks like above
SELECT
marks.student_id,
subjects.subject_name,
ROUND(AVG(mark)) AS mark
FROM
marks
INNER JOIN teaching_loads ON teaching_loads.id = marks.teaching_load_id
INNER JOIN subjects ON subjects.id = teaching_loads.subject_id
INNER JOIN grades ON grades.id=teaching_loads.class_id
WHERE
grades.stream_id = 5 AND marks.assessement_id = 1
GROUP BY
subject_id, marks.student_id
Result of above query is
The system has the following tables
Marks Table-This is the table that stores student marks
CREATE TABLE `marks` (
`id` bigint(20) UNSIGNED NOT NULL,
`teacher_id` bigint(20) UNSIGNED NOT NULL,
`student_id` bigint(20) UNSIGNED NOT NULL,
`teaching_load_id` bigint(20) UNSIGNED NOT NULL,
`assessement_id` bigint(20) UNSIGNED NOT NULL,
`mark` int(11) NOT NULL,
`created_at` timestamp NULL DEFAULT NULL,
`updated_at` timestamp NULL DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
Teaching Loads Table
This table is where we store the subject the teacher is teaching and in which class he/she is teaching that subject.
CREATE TABLE `teaching_loads` (
`id` bigint(20) UNSIGNED NOT NULL,
`teacher_id` bigint(20) UNSIGNED NOT NULL,
`subject_id` bigint(20) UNSIGNED NOT NULL,
`class_id` bigint(20) UNSIGNED NOT NULL,
`session_id` bigint(20) UNSIGNED NOT NULL,
`created_at` timestamp NULL DEFAULT NULL,
`updated_at` timestamp NULL DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
Subjects Table
CREATE TABLE `subjects` (
`id` bigint(20) UNSIGNED NOT NULL,
`subject_name` varchar(255) COLLATE utf8mb4_unicode_ci NOT NULL,
`subject_type` enum('core','elective','non-value','passing_subject') COLLATE utf8mb4_unicode_ci NOT NULL,
`created_at` timestamp NULL DEFAULT NULL,
`updated_at` timestamp NULL DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
Users Table
CREATE TABLE `users` (
`id` bigint(20) UNSIGNED NOT NULL,
`name` varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,,
`password` varchar(255) COLLATE utf8mb4_unicode_ci NOT NULL,
`lastname` varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`middlename` varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

You'll have to join the marks, teaching_loads, and subjects tables multiple times, once for each subject, and calculate the row average on each row (The AVG aggregate function calculates a value across rows for the specified column). Here's an example with the first two subjects to show what I mean (this assumes all subjects for a given student are the same class):
SELECT
geog_load.class_id
,(geog.mark + hist.mark) / 2
,student.name || ' ' || student.lastname
,geog.mark
,hist.mark
FROM
users AS student
INNER JOIN marks AS geog ON geog.student_id = student.id
INNER JOIN teaching_loads AS geog_load ON geog_load.id = geog.teaching_load_id
INNER JOIN
subjects as geog_subject ON geog_subject.id = geog_load.subject_id
AND geog_subject.subject_name = 'Geography'
INNER JOIN marks AS hist ON hist.student_id = student.id
INNER JOIN teaching_loads AS hist_load ON hist_load.id = hist.teaching_load_id
INNER JOIN
subjects as hist_subject ON hist_subject.id = hist_load.subject_id
AND hist_subject.subject_name = 'History'
Then you can filter how you like with a WHERE clause, and use RANK() to show Position numbers, or whatever.

Related

Optimize this query for 1000000+ rows

I need to pull the data and write it to a csv file but its taking too much time and too much ram. What is wrong with it and what can I do? Also, I feel like there's a redundancy in the query itself. I'm doing this with PHP.
Here's the query
CREATE TEMPORARY TABLE temp1 SELECT * FROM vicidial_closer_log
USE INDEX(call_date)
WHERE call_date BETWEEN '1980-01-01 00:00:00' AND '2019-03-12 23:59:59';
CREATE TEMPORARY TABLE temp2 SELECT * FROM vicidial_closer_log
USE INDEX(call_date)
WHERE call_date BETWEEN '1980-01-01 00:00:00' AND '2019-03-12 23:59:59';
SELECT a.call_date,
a.lead_id,
a.phone_number
AS customer_number,
IF(a.status != 'DROP', 'ANSWERED', 'UNANSWERED')
AS status,
IF(a.lead_id IS NOT NULL, 'inbound', 'outbound')
AS call_type,
a.USER
AS agent,
a.campaign_id
AS skill,
NULL
AS campaign,
a.status
AS disposition,
a.term_reason
AS Hangup,
a.uniqueid,
Sec_to_time(a.queue_seconds)
AS time_to_answer,
Sec_to_time(a.length_in_sec - a.queue_seconds)
AS talk_time,
Sec_to_time(a.park_sec)
AS hold_sec,
Sec_to_time(a.dispo_sec)
AS wrapup_sec,
From_unixtime(a.start_epoch)
AS start_time,
From_unixtime(a.end_epoch)
AS end_time,
c.USER
AS
transfered,
a.comments,
IF(a.length_in_sec IS NULL, Sec_to_time(a.queue_seconds),
Sec_to_time(a.length_in_sec + a.dispo_sec))
AS duration,
Sec_to_time(a.length_in_sec - a.queue_seconds + a.dispo_sec)
AS handling_time
FROM temp1 a
left outer join temp2 c
ON a.uniqueid = c.uniqueid
AND a.closecallid < c.closecallid
GROUP BY a.closecallid
I've uploaded screenshot of table structure and the indices.
Table Structure
Indices of Table
Thanks.
UPDATE:
SHOW CREATE TABLE vicidial_closer_log
vicidial_closer_log CREATE TABLE `vicidial_closer_log` (
`closecallid` int(9) unsigned NOT NULL AUTO_INCREMENT,
`lead_id` int(9) unsigned NOT NULL,
`list_id` bigint(14) unsigned DEFAULT NULL,
`campaign_id` varchar(20) COLLATE utf8_unicode_ci DEFAULT NULL,
`call_date` datetime DEFAULT NULL,
`start_epoch` int(10) unsigned DEFAULT NULL,
`end_epoch` int(10) unsigned DEFAULT NULL,
`length_in_sec` int(10) DEFAULT NULL,
`status` varchar(6) COLLATE utf8_unicode_ci DEFAULT NULL,
`phone_code` varchar(10) COLLATE utf8_unicode_ci DEFAULT NULL,
`phone_number` varchar(18) COLLATE utf8_unicode_ci DEFAULT NULL,
`user` varchar(20) COLLATE utf8_unicode_ci DEFAULT NULL,
`comments` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`processed` enum('Y','N') COLLATE utf8_unicode_ci DEFAULT NULL,
`queue_seconds` decimal(7,2) DEFAULT 0.00,
`user_group` varchar(20) COLLATE utf8_unicode_ci DEFAULT NULL,
`xfercallid` int(9) unsigned DEFAULT NULL,
`term_reason` enum('CALLER','AGENT','QUEUETIMEOUT','ABANDON','AFTERHOURS','HOLDRECALLXFER', 'HOLDTIME','NOAGENT','NONE','MAXCALLS') COLLATE utf8_unicode_ci DEFAULT 'NONE',
`uniqueid` varchar(20) COLLATE utf8_unicode_ci NOT NULL DEFAULT '',
`agent_only` varchar(20) COLLATE utf8_unicode_ci DEFAULT '',
`queue_position` smallint(4) unsigned DEFAULT 1,
`called_count` smallint(5) unsigned DEFAULT 0,
`nopaperform` varchar(5) COLLATE utf8_unicode_ci NOT NULL DEFAULT 'NO',
`park_sec` int(3) DEFAULT 0,
`dispo_sec` int(3) DEFAULT 0,
`record_file` text COLLATE utf8_unicode_ci DEFAULT NULL,
PRIMARY KEY (`closecallid`),
KEY `lead_id` (`lead_id`),
KEY `call_date` (`call_date`),
KEY `campaign_id` (`campaign_id`),
KEY `uniqueid` (`uniqueid`),
KEY `phone_number` (`phone_number`),
KEY `date_user` (`call_date`,`user`),
KEY `closecallid` (`closecallid`)
) ENGINE=MyISAM AUTO_INCREMENT=1850672 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
EXPLAIN QUERY(On third query only):
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE a ALL NULL NULL NULL NULL 664640 Using temporary; Using filesort
1 SIMPLE c ALL NULL NULL NULL NULL 662480 Using where; Using join buffer (flat, BNL join)
UPDATE(Updated Query):
SELECT a.call_date,
a.lead_id,
a.phone_number
AS customer_number,
IF(a.status != 'DROP', 'ANSWERED', 'UNANSWERED')
AS status,
IF(a.lead_id IS NOT NULL, 'inbound', 'outbound')
AS call_type,
a.user
AS agent,
a.campaign_id
AS skill,
NULL
AS campaign,
a.status
AS disposition,
a.term_reason
AS Hangup,
a.uniqueid,
Sec_to_time(a.queue_seconds)
AS time_to_answer,
Sec_to_time(a.length_in_sec - a.queue_seconds)
AS talk_time,
Sec_to_time(a.park_sec)
AS hold_sec,
Sec_to_time(a.dispo_sec)
AS wrapup_sec,
From_unixtime(a.start_epoch)
AS start_time,
From_unixtime(a.end_epoch)
AS end_time,
c.user
AS
transfered,
a.comments,
IF(a.length_in_sec IS NULL, Sec_to_time(a.queue_seconds),
Sec_to_time(a.length_in_sec + a.dispo_sec))
AS duration,
Sec_to_time(a.length_in_sec - a.queue_seconds + a.dispo_sec)
AS handling_time
FROM vicidial_closer_log a
LEFT OUTER JOIN vicidial_closer_log c
ON a.closecallid <> c.closecallid
AND a.uniqueid = c.uniqueid
AND a.closecallid < c.closecallid
WHERE a.call_date BETWEEN '2018-01-01 00:00:00' AND '2019-03-13 23:59:59'
EXPLAIN on updated query:
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE a ALL call_date,date_user NULL NULL NULL 662829 Using where
1 SIMPLE c ref PRIMARY,uniqueid,closecallid uniqueid 62 aastell_bliss.a.uniqueid 1 Using where
Updated Query Execution Result:
Number of rows present between given time range: 155016 rows
Time taken: 0.0149 secs
It works!
Summary of comments that lead to an answer:
CREATE TEMPORARY TABLE ... SELECT doesn't create indexes on the temporary table
Explicit use of a temporary table, particularly of a large size, will rarely give a performance gain.
Using table aliases in a join allows for a self join
Group by Primary Key on the left side of a join doesn't add much as its already unique and the JOIN had no aggregate expressions. GROUP BY adds an implicit ORDER BY so you expression could end up slower if a secondary index was used to join the table.
While the date range of the query was large, preparing for it to be a significant filter when small would make the call_date more favourable as an index. To make this more favorable, the join key is added to the end of the index so most of the work of the join can happen by just looking at the index.
When PK is on a column, a secondary index on the same column isn't needed.

Optimization Needed For Dual Left Join Query

I've always struggled with mysql joins but have started incorporating more but struggling to understand despite reading dozens of tutorials and mysql manual.
My situation is I have 3 tables:
/* BASICALLY A TABLE THAT HOLDS FAN RECORDS */
CREATE TABLE `fans` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`first_name` varchar(255) DEFAULT NULL,
`middle_name` varchar(255) DEFAULT NULL,
`last_name` varchar(255) DEFAULT NULL,
`email` varchar(255) DEFAULT NULL,
`join_date` datetime DEFAULT NULL,
`twitter` varchar(255) DEFAULT NULL,
`twitterCrawled` datetime DEFAULT NULL,
`twitterImage` varchar(255) DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `email` (`email`)
) ENGINE=MyISAM AUTO_INCREMENT=20413 DEFAULT CHARSET=latin1;
/* A TABLE OF OUR TWITTER FOLLOWERS */
CREATE TABLE `twitterFollowers` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`screenName` varchar(25) DEFAULT NULL,
`twitterId` varchar(25) DEFAULT NULL,
`customerId` int(11) DEFAULT NULL,
`uniqueStr` varchar(50) DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `unique` (`uniqueStr`)
) ENGINE=InnoDB AUTO_INCREMENT=13426 DEFAULT CHARSET=utf8;
/* TABLE THAT SUGGESTS A LIKELY MATCH OF A TWITTER FOLLOWER BASED ON THE EMAIL / SCREEN NAME COMPARISON OF THE FAN vs OUR FOLLOWERS
IF SOMEONE (ie. a moderator) CONFIRMS OR DENIES THAT IT'S A GOOD MATCH THEY PUT A DATESTAMP IN `dismissed` */
CREATE TABLE `contentSuggestion` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`userId` int(11) DEFAULT NULL,
`fanId` int(11) DEFAULT NULL,
`twitterAccountId` int(11) DEFAULT NULL,
`contentType` varchar(50) DEFAULT NULL,
`contentString` varchar(255) DEFAULT NULL,
`added` datetime DEFAULT NULL,
`dismissed` datetime DEFAULT NULL,
`uniqueStr` varchar(255) DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `unstr` (`uniqueStr`)
) ENGINE=InnoDB AUTO_INCREMENT=2 DEFAULT CHARSET=utf8;
What I'm trying to get is:
SELECT [fan columns]
WHERE fan screen name IS IN twitterfollowers
AND WHERE fan screen name IS NOT IN contentSuggestion (with a datestamp in dismissed)
My attempts so far:
~33 seconds
SELECT fans.id, tf.screenName as col1, tf.twitterId as col2 FROM fans
LEFT JOIN twitterFollowers tf ON tf.screenName = fans.emailUsername
LEFT JOIN contentSuggestion cs ON cs.contentString = tf.screenName WHERE dismissed IS NULL
GROUP BY(fans.id) HAVING col1 != ''
~14 seconds
SELECT id, emailUsername FROM fans WHERE emailUsername IN(SELECT DISTINCT(screenName) FROM twitterFollowers) AND emailUsername NOT IN(SELECT DISTINCT(contentString) FROM contentSuggestion WHERE dismissed IS NULL) GROUP BY (fans.id);
9.53 seconds
SELECT fans.id, tf.screenName as col1, tf.twitterId as col2 FROM fans
LEFT JOIN twitterFollowers tf ON tf.screenName = fans.emailUsername WHERE tf.uniqueStr NOT IN(SELECT uniqueStr FROM contentSuggestion WHERE dismissed IS NULL)
I hope there is a better way. I've been struggling to really use JOINS outside of a single LEFT JOIN which has already helped me speed up other queries by a significant amount.
Thanks for any help you can give me.
I would go with a variation of the second method. Instead of IN, use EXISTS. Then add the correct indexes and remove the aggregation:
SELECT f.id, f.emailUsername
FROM fans f
WHERE EXISTS (SELECT 1
FROM twitterFollowers tf
WHERE f.emailUsername = tf.screenName
) AND
NOT EXISTS (SELECT 1
FROM contentSuggestion cs
WHERE f.emailUsername = cs.contentString AND
cs.dismissed IS NULL
) ;
Then be sure you have the following indexes: twitterFollowers(screenName) and contentSuggestion(contentString, dismissed).
Some notes:
When using IN, don't use SELECT DISTINCT. I'm not 100% sure that MySQL is always smart enough to ignore the DISTINCT in the subquery (it is redundant).
Historically, EXISTS was faster than IN in MySQL. The optimizer has improved in recent versions.
For performance, you need the correct indexes.
Then be sure you have the following indexes: twitterFollowers(screenName) and contentSuggestion(contentString, dismissed).
Assuming that fan.id is unique (a very reasonable assumption), you don't need the final group by.

i want to use left and inner join in 3 tables in mysql?

these are my tables. first one is appusers table.
CREATE TABLE IF NOT EXISTS `appusers` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`email` varchar(50) NOT NULL,
`is_active` tinyint(2) NOT NULL DEFAULT '0',
`zip` varchar(20) NOT NULL,
`city` text NOT NULL,
`country` text NOT NULL,
`created` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=23 ;
second table is stickeruses table.
CREATE TABLE IF NOT EXISTS `stickeruses` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`user_id` int(11) NOT NULL,
`sticker_id` int(11) NOT NULL,
`count` int(11) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=24 ;
Third table is Devices
CREATE TABLE IF NOT EXISTS `devices` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`user_id` int(11) NOT NULL,
`regid` varchar(300) NOT NULL,
`imei` varchar(50) NOT NULL,
`device_type` tinyint(2) NOT NULL,
`notification` tinyint(2) NOT NULL DEFAULT '1',
`is_active` tinyint(2) NOT NULL DEFAULT '0',
`activationcode` int(6) NOT NULL,
`created` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=28 ;
I Want to find the Sum(stickeruses.count) and COUNT(devices.id) for all appusers.
Here is my query.
SELECT `Appuser`.`id`, `Appuser`.`email`, `Appuser`.`country`, `Appuser`.`created`,
`Appuser`.`is_active`, SUM(`Stickeruse`.`count`) AS total, COUNT(`Device`.`id`)
AS tdevice
FROM `stickerapp`.`appusers` AS `Appuser`
LEFT JOIN `stickerapp`.`stickeruses` AS `Stickeruse`
ON (`Stickeruse`.`user_id`=`Appuser`.`id`)
INNER JOIN `stickerapp`.`devices` AS `Device`
ON (`Device`.`user_id`=`Appuser`.`id`)
WHERE `Appuser`.`is_active` = 1
GROUP BY `Appuser`.`id`
LIMIT 10
When I am applying each join separately the results are right, but I want to combine both joins. And when I am doing it then results are wrong. please help.
When mixing JOIN and LEFT JOIN it is a good idea to use parentheses to make it clear what your intent is.
I don't know what you need, but these syntaxes might give you different results:
FROM a LEFT JOIN ( b JOIN c ON b..c.. ) bc ON a..bc..
FROM ( a LEFT JOIN b ON a..b.. ) ab JOIN c ON ab..c..
Also, you can rearrange them do FROM a JOIN c LEFT JOIN b (plus parentheses) or any of several other arrangements. Granted, some pairs rearrangements are equivalent.
Also, beware; aggregates (such as SUM()) get inflated values when JOINing. Think of it this way: first the JOINs get all appropriate combinations of rows from the tables, then the SUM adds them up. With that in mind, see if this works better:
SELECT a.`id`, a.`email`, a.`country`, a.`created`, a.`is_active`,
( SELECT SUM(`count`)
FROM stickerapp.stickeruses
WHERE user_id = a.id
) AS total,
( SELECT COUNT(*)
FROM stickerapp.devices
WHERE user_id = a.id
) AS tdevice
FROM stickerapp.`appusers` AS a
WHERE a.`is_active` = 1
GROUP BY a.`id`
LIMIT 10

Improve speed of MySQL query [closed]

It's difficult to tell what is being asked here. This question is ambiguous, vague, incomplete, overly broad, or rhetorical and cannot be reasonably answered in its current form. For help clarifying this question so that it can be reopened, visit the help center.
Closed 10 years ago.
EDIT:
I reworked the query as follows:
SELECT
a.title, count(*),at.search_code
FROM
`qitz3_attributes_type` at
left join
qitz3_attributes a
on
a.attribute_type_id = at.id
left join
qitz3_attributes_property ap
on
ap.attribute_id = a.id
left join
qitz3_helloworld h
on
h.id = ap.property_id
where
at.id in (1,2,8,9,11)
and
a.search_filter = 1
and
h.area=506
and
h.expiry_date >= '2013-02-20 13:28:04'
group by
a.title
order by search_code
This seems much faster but I am still getting a using temporary and using filesort on the explain...
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE at range PRIMARY PRIMARY 4 NULL 5 Using where; Using temporary; Using filesort
1 SIMPLE a ref PRIMARY,Attribute type ID,Search filter,Attribute ... Attribute type ID 4 password.at.id 6 Using where
1 SIMPLE ap ref Property ID,Attribute ID,Attribute Property Attribute Property 4 password.a.id 142 Using where; Using index
1 SIMPLE h eq_ref PRIMARY,Area indexes,Expiry date PRIMARY 4 password.ap.property_id 1 Using where
ENDEDIT
I am working on a search component for a site I am developing and while it is all working there are a couple of queries that I would like to see running a touch faster.
In order to populate a set of search filters (display the count of the number of properties that have each facility or are of a particular type) I am using the following two queries. The first will get a list of IDs that I then plug into the second query as follows:
Based on the following info can anyone suggest a more efficient way to do this? I would really like to speed the first query up as it seems a little slow compared to the others.
Is 150ms actually that slow? 15ms would be better...
Thanks,
Adam
Query 1 (takes around 150ms):
SELECT h.id, h.parent_id, h.level, h.title as property_title, h.area, h.region, h.department, h.city,
LEFT(h.description, 250) as description, h.thumbnail, h.occupancy, h.swimming, g.path, (single_bedrooms + double_bedrooms + triple_bedrooms + quad_bedrooms + twin_bedrooms) as bedrooms, c.title as location_title, ( select min(tariff)
from qitz3_tariffs
where id = h.id ) as price, e.title as tariff_based_on, f.title as base_currency, a.title as property_type, a2.title as accommodation_type, ( select count(*)
from qitz3_reviews
where property_id = h.id
group by h.id ) as reviews
FROM qitz3_classifications c
LEFT JOIN qitz3_helloworld h
on c.id = h.area
LEFT JOIN qitz3_attributes_property ap
ON ap.property_id = h.id
LEFT JOIN qitz3_attributes_type at
ON at.id = ap.attribute_id
LEFT JOIN qitz3_attributes a
ON a.id = ap.attribute_id
LEFT JOIN qitz3_attributes_property ap2
ON ap2.property_id = h.id
LEFT JOIN qitz3_attributes_type at2
ON at2.id = ap2.attribute_id
LEFT JOIN qitz3_attributes a2
ON a2.id = ap2.attribute_id
LEFT JOIN qitz3_attributes e
ON e.id = h.tariff_based_on
LEFT JOIN qitz3_attributes f
ON f.id = h.base_currency
LEFT JOIN qitz3_classifications g
ON g.id = h.city
WHERE a.attribute_type_id = 1
AND a2.attribute_type_id = 2
AND c.id = 506
AND h.expiry_date >= '2013-02-20 12:05:13'
AND h.id > 1
Explain:
id select_type table type possible_keys key key_len ref rows Extra
1 PRIMARY c const PRIMARY PRIMARY 4 const 1
1 PRIMARY h ref PRIMARY,Area indexes,Expiry date Area indexes 4 const 615 Using where
1 PRIMARY ap ref Property ID,Attribute ID Property ID 4 password.h.id 21 Using where
1 PRIMARY at eq_ref PRIMARY PRIMARY 4 password.ap.attribute_id 1 Using index
1 PRIMARY a eq_ref PRIMARY,Attribute type ID PRIMARY 4 password.ap.attribute_id 1 Using where
1 PRIMARY ap2 ref Property ID,Attribute ID Property ID 4 password.ap.property_id 21 Using where
1 PRIMARY at2 eq_ref PRIMARY PRIMARY 4 password.ap2.attribute_id 1 Using index
1 PRIMARY a2 eq_ref PRIMARY,Attribute type ID PRIMARY 4 password.ap2.attribute_id 1 Using where
1 PRIMARY e eq_ref PRIMARY PRIMARY 4 password.h.tariff_based_on 1
1 PRIMARY f eq_ref PRIMARY PRIMARY 4 password.h.base_currency 1
1 PRIMARY g eq_ref PRIMARY PRIMARY 4 password.h.city 1
3 DEPENDENT SUBQUERY qitz3_reviews ref Property ID Property ID 4 password.h.id 1 Using index; Using temporary; Using filesort
2 DEPENDENT SUBQUERY qitz3_tariffs ref Property ID Property ID 4 password.h.id 2
Query 2 (takes around 30ms):
SELECT a.id,count(attribute_id) as count, a.title AS attribute, a.published, at.title as facility_type, at.search_code
FROM qitz3_attributes AS a
LEFT JOIN qitz3_attributes_type at
on at.id = a.attribute_type_id
LEFT JOIN qitz3_attributes_property ap
on ap.attribute_id = a.id
WHERE search_filter = 1
AND a.published = 1
AND property_id in (152843,103180,152845,4628,5653,3865,107553,155945,106029,107575,149052,837,104264,98635,98636,98637,98638,3667,106838,3672,157278,155743,157791,157792,151153,151155,100725,106109,157569,157576,107145,150666,103310,5780,3480,102041,154016,3490,154018,932,153518,151991,154041,154042,4288,5832,149451,157646,102094,148444,153822,157407,153839,151536,157393,157395,157428,157429,102236,104770,157378,157380,157381,157382,157383,157654,104768,103746,153683,150175,4618,101050,104942,157229,157230,98515,104771,104944,3612,148721,104212,5307,3432,156676,102706,404,5518,156359,5252,102697,5271,98979,101827,149524,102676,107551,5685,101736,156538,152703,4730,151881,95838,3759,149769,5269,98429,153729,5233,703,5579,98943,157433,3157,155661,107347,147545,147547,5216,106345,156533,93833,158091,96403,3491,968,158195,158196,157580,104148,3030,94686,154725,150582,103027,99062,102462,4384,5634,153874,157974,101669,47,105285,102481,102234,5749,156793,153748,96404,151467,154292,147645,97471,100551,102090,4841,3563,155643,4656,98424,157243,150601,157415,4701,102283,100719,100738,5643,98425,98972,103261,531,3105,98108,150592,5719,150616,157532,3974,3212,157581,97469,97470,149183,157638,149730,102114,156395,153621,102560,102913,94684,5609,157578,98423,98971,102151,146734,150585,104287,155296,104956,94592,102433,147575,156325,106344,101766,107058,106560,103026,157848,98973,4303,5620,149767,150563,4407,104268,97876,156784,156786,149922,701,154317,153821,102480,348,102778,102779,102780,102781,102479,352,103025,98677,5254,98697,3995,156322,100305,98532,3833,5374,150172,151435,102368,102380,157228,103171,147740,152870,103579,3870,104037,103016,4995,105104,157605,5811,955,147643,156648,107802,101502,94685,3569,148755,150293,4122,157013,157297,98676,156794,102848,157635,157640,95717,98980,102764,102777,102782,36,101765,154373,149829,154955,107683,158176,102557,157552,103163,5760,104627,153561,266,151335,151176,147620,147379,3085,155760,106339,151424,106759,5145,104990,97877,155495,5241,156407,156625,3236,152782,96066,147617,3860,4614,3497,147883,158207,102985,104622,101816,157275,149037,4792,149226,3496,101825,102538,150571,105015,97874,157391,158192,102562,155032,5383,102558,104194,156740,101446,147615,5815,107081,155992,97473,148817,945,101751,158074,4249,101792,4532,152828,104316,157319,156071,157508,157510,148836,4745,153823,157942,3859,157442,100017,102555,147629,149272,157845,151256,101481,154735,154737,157652,106232,97991,4660,3309,157597,407,157658,154152,157374,153385,148037,158214,100452)
GROUP BY a.id
This query gives a count of the number of properties with each attribute (e.g. Golf, Air conditioning, property type etc). The good thing about this is that only attributes are shown that have properties. So as the user drills down attributes with no properties are not shown. This is basically down to the first query where I get a list of the properties matching a particular set of attributes.
The tables are as follows:
--
-- Table structure for table qitz3_attributes
CREATE TABLE IF NOT EXISTS `qitz3_attributes` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`attribute_type_id` int(11) NOT NULL,
`title` varchar(75) NOT NULL,
`ordering` int(11) NOT NULL,
`state` tinyint(4) NOT NULL DEFAULT '1',
`published` int(11) NOT NULL,
`search_filter` bit(1) NOT NULL,
`language_code` varchar(6) NOT NULL DEFAULT 'en-GB',
PRIMARY KEY (`id`),
KEY `Attribute type ID` (`attribute_type_id`),
KEY `Search filter` (`search_filter`),
KEY `Published` (`published`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=1522 ;
--
-- Table structure for table qitz3_attributes_property
CREATE TABLE IF NOT EXISTS `qitz3_attributes_property` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`property_id` int(11) NOT NULL,
`attribute_id` int(11) NOT NULL,
PRIMARY KEY (`id`),
KEY `Property ID` (`property_id`),
KEY `Attribute ID` (`attribute_id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=66261 ;
--
-- Table structure for table qitz3_attributes_type
CREATE TABLE IF NOT EXISTS `qitz3_attributes_type` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`title` varchar(75) NOT NULL,
`language_code` varchar(6) NOT NULL,
`field_name` varchar(25) NOT NULL,
`search_code` varchar(25) NOT NULL,
`state` int(11) NOT NULL DEFAULT '1',
`published` int(11) NOT NULL DEFAULT '1',
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=34 ;
--
-- Table structure for table qitz3_classifications
CREATE TABLE IF NOT EXISTS `qitz3_classifications` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`parent_id` int(10) unsigned NOT NULL DEFAULT '0',
`lft` int(11) NOT NULL DEFAULT '0',
`rgt` int(11) NOT NULL DEFAULT '0',
`level` int(10) unsigned NOT NULL DEFAULT '0',
`title` varchar(255) NOT NULL,
`description` text NOT NULL,
`path` varchar(255) NOT NULL DEFAULT '',
`alias` varchar(255) NOT NULL,
`access` tinyint(3) unsigned NOT NULL DEFAULT '0',
`published` int(11) NOT NULL,
`longitude` float(10,6) NOT NULL,
`latitude` float(10,6) NOT NULL,
PRIMARY KEY (`id`),
KEY `idx_left_right` (`lft`,`rgt`),
KEY `Alias index` (`alias`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 AUTO_INCREMENT=158052 ;
--
-- Table structure for table qitz3_helloworld (Property table)
CREATE TABLE IF NOT EXISTS `qitz3_helloworld` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`parent_id` int(10) NOT NULL DEFAULT '0',
`lft` int(11) NOT NULL DEFAULT '0',
`rgt` int(11) NOT NULL DEFAULT '0',
`level` int(10) unsigned NOT NULL,
`alias` varchar(250) NOT NULL DEFAULT '',
`access` tinyint(3) unsigned NOT NULL DEFAULT '0',
`path` varchar(255) NOT NULL DEFAULT '',
`title` varchar(120) NOT NULL,
`area` int(11) NOT NULL DEFAULT '0',
`region` int(11) NOT NULL DEFAULT '0',
`department` int(11) NOT NULL DEFAULT '0',
`city` int(11) NOT NULL DEFAULT '0',
`params` text NOT NULL,
`created_by` int(10) NOT NULL DEFAULT '0',
`created_on` datetime NOT NULL,
`modified` datetime DEFAULT NULL,
`expiry_date` date DEFAULT NULL,
`availability_last_updated_on` datetime DEFAULT NULL,
`modified_by` int(11) DEFAULT NULL,
`lang` varchar(5) NOT NULL DEFAULT 'en-GB',
`description` mediumtext NOT NULL COMMENT 'The summary and description for this accommodation',
`internal_facilities_other` varchar(1000) NOT NULL,
`external_facilities_other` varchar(1000) NOT NULL,
`activities_other` varchar(5000) NOT NULL,
`location_details` varchar(5000) NOT NULL,
`getting_there` varchar(5000) NOT NULL,
`thumbnail` varchar(150) NOT NULL,
`occupancy` int(11) DEFAULT NULL,
`single_bedrooms` int(11) NOT NULL,
`double_bedrooms` int(11) NOT NULL,
`triple_bedrooms` int(11) DEFAULT NULL,
`quad_bedrooms` int(11) DEFAULT NULL,
`twin_bedrooms` int(11) DEFAULT NULL,
`childrens_beds` int(11) DEFAULT NULL,
`cots` int(11) DEFAULT NULL,
`extra_beds` int(11) DEFAULT NULL,
`bathrooms` int(11) NOT NULL,
`toilets` int(11) DEFAULT NULL,
`swimming` int(11) NOT NULL,
`latitude` decimal(10,7) DEFAULT NULL,
`longitude` decimal(10,7) DEFAULT NULL,
`nearest_town` varchar(50) DEFAULT NULL,
`distance_to_coast` int(11) DEFAULT NULL,
`additional_price_notes` varchar(3000) DEFAULT NULL,
`base_currency` int(11) DEFAULT NULL,
`tariff_based_on` int(11) DEFAULT NULL,
`linen_costs` varchar(250) DEFAULT NULL,
`changeover_day` int(11) DEFAULT NULL,
`published` tinyint(4) NOT NULL DEFAULT '0',
`video` tinyint(4) NOT NULL DEFAULT '0',
PRIMARY KEY (`id`),
KEY `idx_left_right` (`lft`,`rgt`),
KEY `Area indexes` (`area`,`region`,`department`),
KEY `Expiry date` (`expiry_date`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 AUTO_INCREMENT=158249 ;
--
-- Table structure for table qitz3_reviews
CREATE TABLE IF NOT EXISTS `qitz3_reviews` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`property_id` int(11) NOT NULL,
`title` varchar(150) NOT NULL,
`review_text` varchar(4000) NOT NULL,
`date` date NOT NULL,
`rating` int(11) NOT NULL,
`guest_name` varchar(75) NOT NULL,
`guest_email` varchar(150) NOT NULL,
`state` tinyint(3) NOT NULL DEFAULT '0',
`published` tinyint(1) NOT NULL DEFAULT '0',
`created` datetime NOT NULL,
`created_by` int(11) NOT NULL,
PRIMARY KEY (`id`),
KEY `Property ID` (`property_id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=921 ;
--
-- Table structure for table qitz3_tariffs
CREATE TABLE IF NOT EXISTS `qitz3_tariffs` (
`tariff_id` int(11) NOT NULL AUTO_INCREMENT,
`id` int(11) NOT NULL COMMENT 'Denotes the property listing ID',
`start_date` date NOT NULL,
`end_date` date NOT NULL,
`tariff` int(11) NOT NULL COMMENT 'Price per booking period between the dates specified. dated spec',
PRIMARY KEY (`tariff_id`),
KEY `Property ID` (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=9267 ;
Quick play, but trying to move the subselects out from the field list.
SELECT h.id, h.parent_id, h.level, h.title as property_title, h.area, h.region, h.department, h.city, LEFT(h.description, 250) as description, h.thumbnail, h.occupancy, h.swimming, g.path, (single_bedrooms + double_bedrooms + triple_bedrooms + quad_bedrooms + twin_bedrooms) as bedrooms, c.title as location_title,
Sub1.price,
e.title as tariff_based_on, f.title as base_currency, a.title as property_type, a2.title as accommodation_type,
Sub2.reviews
FROM qitz3_classifications c
LEFT JOIN qitz3_helloworld h on c.id = h.area
LEFT JOIN qitz3_attributes_property ap ON ap.property_id = h.id
LEFT JOIN qitz3_attributes_type at ON at.id = ap.attribute_id
LEFT JOIN qitz3_attributes a ON a.id = ap.attribute_id
LEFT JOIN qitz3_attributes_property ap2 ON ap2.property_id = h.id
LEFT JOIN qitz3_attributes_type at2 ON at2.id = ap2.attribute_id
LEFT JOIN qitz3_attributes a2 ON a2.id = ap2.attribute_id
LEFT JOIN qitz3_attributes e ON e.id = h.tariff_based_on
LEFT JOIN qitz3_attributes f ON f.id = h.base_currency
LEFT JOIN qitz3_classifications g ON g.id = h.city
LEFT JOIN ( SELECT id, MIN(tariff) AS price FROM qitz3_tariffs GROUP BY id) Sub1 ON Sub1.Id = h.id
LEFT JOIN ( SELECT property_id, COUNT(*) AS reviews FROM qitz3_reviews GROUP BY property_id ) as Sub2 ON Sub2.property_id = h.id
WHERE a.attribute_type_id = 1
AND a2.attribute_type_id = 2
AND c.id = 506
AND h.expiry_date >= '2013-02-20 12:05:13'
AND h.id > 1
When you need more speed in searching data you should take a look at Solr or Sphinx. With this index servers you can index your MySQL-Data and query them.
Its much more faster then MySQL.
There's quite a lot wrong in your first query:
The first 7 LEFT JOINs should be INNER JOINs. For qitz3_attributes you should consider normalizing the data less aggressively (use multiple columns instead of multiple rows to describe the data.

SQL select query taking long time

I am using the below SQL query but it takes more than 180 sec to execute. Is there a way to speed it up ? This SQL give me the pic_id and of all the females.
SELECT pic_id, small
FROM picture
WHERE hide =0
AND userhide =0
AND `fbid`
IN (
SELECT fbid
FROM user
WHERE gender = "female"
)
ORDER BY `picture`.`pic_id` ASC
LIMIT 1500 , 200
The Explain SQL
id select_type table type possible_keys key key_len ref rows Extra
1 PRIMARY picture index NULL PRIMARY 4 NULL 1700 Using where
2 DEPENDENT SUBQUERY user ALL NULL NULL NULL NULL 7496 Using where
--- Result of explain statement for Tim's sql answer --
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE picture ALL NULL NULL NULL NULL 41443 Using where; Using temporary; Using filesort
1 SIMPLE user ALL NULL NULL NULL NULL 7501 Using where; Using join buffer
-- Structure ---
CREATE TABLE `user` (
`sid` int(11) NOT NULL AUTO_INCREMENT,
`name` varchar(255) CHARACTER SET utf8 NOT NULL,
`first_name` varchar(255) CHARACTER SET utf8 NOT NULL,
`username` varchar(255) CHARACTER SET utf8 NOT NULL,
`birthday` date NOT NULL,
`location` varchar(255) CHARACTER SET utf8 NOT NULL,
`gender` varchar(6) CHARACTER SET utf8 NOT NULL,
`created` date NOT NULL,
`fbid` bigint(50) NOT NULL,
`token` varchar(255) CHARACTER SET utf8 NOT NULL,
`relationship_status` varchar(20) CHARACTER SET utf8 NOT NULL,
`smallest` varchar(255) CHARACTER SET utf8 NOT NULL,
`email` varchar(40) CHARACTER SET utf8 NOT NULL,
`ref` varchar(15) NOT NULL,
PRIMARY KEY (`sid`),
KEY `gender` (`gender`),
KEY `fbid` (`fbid`)
) ENGINE=MyISAM AUTO_INCREMENT=7595 DEFAULT CHARSET=latin
---- structure of picture table ---
CREATE TABLE `picture` (
`fbid` bigint(50) NOT NULL,
`pic_id` int(11) NOT NULL AUTO_INCREMENT,
`pic_location` varchar(255) NOT NULL,
`hide` int(1) NOT NULL,
`small` varchar(255) NOT NULL,
`userhide` int(1) NOT NULL,
`likes` int(10) NOT NULL,
`hot` int(1) NOT NULL,
PRIMARY KEY (`pic_id`),
UNIQUE KEY `pic_location` (`pic_location`),
UNIQUE KEY `small` (`small`),
KEY `fbid` (`fbid`),
KEY `hide` (`hide`),
KEY `userhide` (`userhide`)
) ENGINE=MyISAM AUTO_INCREMENT=42749 DEFAULT CHARSET=latin1
try something like this:
SELECT pic_id, small
FROM picture
INNER JOIN user ON ( picture.fbid = user.fbid and user.gender='female' )
WHERE hide =0
AND userhide =0
ORDER BY `picture`.`pic_id` ASC
LIMIT 1500 , 200
I put gender in the join because a query will not return rows that don't have a match on an inner join.
You should also read this stack overflow topic
EDIT:
make sure you have indexed the following fields:
picture.fbid
user.fbid
user.gender
picture.hide
picture.userhide
Try using a join instead:
SELECT p.pic_id, p.small
FROM picture p
INNER JOIN fbid f USING ( fbid )
WHERE p.hide =0
AND p.userhide =0
AND f.gender = 'female'
ORDER BY `picture`.`pic_id` ASC
LIMIT 1500 , 200

Categories