Speed up query (~1 million rows) on a complex table - php

first of all thanks for your time reading this.
I'm working on a PHP web application that deals with the following data, as of right now there is noticeable time when trying to retrieve data, especially when drilling down the options. The primary options right now are VehicleType, Make, Model, Year, InternetPrice and Mileage. There will eventually be even more columns being used. Other columns from this list are being used for displaying data throughout the page, as 10 records are displayed at one instance.
Table structure:
CREATE TABLE `vehicles` (
`id` int(12) NOT NULL AUTO_INCREMENT,
`DealerID` int(6) DEFAULT NULL,
`VIN` varchar(17) DEFAULT NULL,
`StockNumber` varchar(10) DEFAULT NULL,
`Status` varchar(1) DEFAULT NULL,
`VehicleType` int(1) DEFAULT NULL,
`Year` int(4) DEFAULT NULL,
`Make` varchar(13) DEFAULT NULL,
`Model` varchar(24) DEFAULT NULL,
`Trim` varchar(35) DEFAULT NULL,
`Body` varchar(25) DEFAULT NULL,
`VehicleClass` varchar(50) DEFAULT NULL,
`VehicleCategory` varchar(6) DEFAULT NULL,
`Mileage` int(6) DEFAULT NULL,
`Transmission` varchar(24) DEFAULT NULL,
`EngineDisplacement` varchar(7) DEFAULT NULL,
`EngineSize` varchar(15) DEFAULT NULL,
`Induction` varchar(25) DEFAULT NULL,
`DriveTrain` varchar(3) DEFAULT NULL,
`FuelType` varchar(9) DEFAULT NULL,
`FuelEconomyCity` int(2) DEFAULT NULL,
`FuelEconomyHighway` int(2) DEFAULT NULL,
`FuelEconomyCombined` int(1) DEFAULT NULL,
`Doors` int(1) DEFAULT NULL,
`OEMColorCodeExterior` varchar(10) DEFAULT NULL,
`OEMColorCodeInterior` varchar(10) DEFAULT NULL,
`OEMColorNameExterior` varchar(49) DEFAULT NULL,
`OEMColorNameInterior` varchar(10) DEFAULT NULL,
`GenericColorExterior` varchar(35) DEFAULT NULL,
`GenericColorInterior` varchar(38) DEFAULT NULL,
`InternetPrice` int(6) DEFAULT NULL,
`ComparisonPrice` int(6) DEFAULT NULL,
`WholeSalePrice` varchar(10) DEFAULT NULL,
`MSRP` varchar(10) DEFAULT NULL,
`InternetSpecial` varchar(1) DEFAULT NULL,
`OemModelCode` varchar(12) DEFAULT NULL,
`HasWarranty` varchar(1) DEFAULT NULL,
`CertificationWarranty` int(3) DEFAULT NULL,
`WarrantyMonth` int(1) DEFAULT NULL,
`WarrantyMiles` int(1) DEFAULT NULL,
`CertificationNumber` varchar(7) DEFAULT NULL,
`ServiceContract` varchar(1) DEFAULT NULL,
`InServiceDate` varchar(19) DEFAULT NULL,
`CertificationDate` varchar(19) DEFAULT NULL,
`DateManufactured` varchar(19) DEFAULT NULL,
`DateCreated` varchar(19) DEFAULT NULL,
`DateUpdated` varchar(19) DEFAULT NULL,
`DateRemoved` varchar(19) DEFAULT NULL,
`DatePhotosUpdated` varchar(19) DEFAULT NULL,
`Photos` int(2) DEFAULT NULL,
`SuperSizePhotos` int(2) DEFAULT NULL,
`AddendumDetails` varchar(10) DEFAULT NULL,
`DepartmentComments` varchar(239) DEFAULT NULL,
`VehicleComments` varchar(1987) DEFAULT NULL,
`Options` varchar(2264) DEFAULT NULL,
`PurchasePayment` decimal(5,2) DEFAULT NULL,
`PurchaseDownPayment` decimal(6,2) DEFAULT NULL,
`PurchaseTerm` int(2) DEFAULT NULL,
`PurchaseDisclosure` varchar(10) DEFAULT NULL,
`PurchaseRate` decimal(3,2) DEFAULT NULL,
`LeasePayment` decimal(2,2) DEFAULT NULL,
`LeaseDownPayment` decimal(2,2) DEFAULT NULL,
`LeaseTerm` int(1) DEFAULT NULL,
`LeaseDisclosure` varchar(10) DEFAULT NULL,
`LeaseRate` decimal(2,2) DEFAULT NULL,
`LeaseResidual` decimal(2,2) DEFAULT NULL,
`Reserved1` varchar(10) DEFAULT NULL,
`Reserved2` varchar(10) DEFAULT NULL,
`Reserved3` varchar(10) DEFAULT NULL,
`Reserved4` varchar(10) DEFAULT NULL,
`Reserved5` varchar(10) DEFAULT NULL,
`Reserved6` varchar(10) DEFAULT NULL,
`sitecert` int(11) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `VIN` (`VIN`),
KEY `Make` (`Make`),
KEY `StockNumber` (`StockNumber`),
KEY `Model` (`Model`),
KEY `Trim` (`Trim`),
KEY `Body` (`Body`),
KEY `VehicleClass` (`VehicleClass`),
KEY `Transmission` (`Transmission`),
KEY `DealerID` (`DealerID`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 AUTO_INCREMENT=46527428 ;
And an example query:
SELECT *
FROM vehicles
WHERE Make = 'Audi'
AND VehicleType = '0'
AND Model = 'A4'
AND InternetPrice BETWEEN 0 AND 999999
AND Year BETWEEN 1983 AND 2015
AND Mileage > 0
AND DealerID IN ('AA'
,'156553'
,'229602'
,'160483'
,'2953'
,'161712'
,'228576'
,'228921'
,'7590'
,'20813'
,'158105'
,'160286'
,'164479'
,'164487'
,'182543'
,'158860'
,'186479'
,'227170'
,'226327'
)
ORDER BY sitecert DESC
,InternetPrice DESC
LIMIT 0, 10
Would anyone have any advice to optimize this query or table? Of course there is other code throughout the page but I've timed it down to this query that is causing the delay.
Thanks for your help!
UPDATE:
MySQL Explain:
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE vehicles ref Make,Model,DealerID Model 75 const 1675 Using where; Using filesort
Year was updated to INT

Depending on your database, you should add indexes on the following fields:
Make
VehicleType
Model
InternetPrice
Year
Mileage
DealerId
Year should be DATE type, not text (INT also makes sense).
This should solve your problem.

My advices :
use a group by make
partition your table, columns might be of use (make,Year)
store your table in inndb engine instead of myisam and build your
self some clustered indexes to attend your predicates !

For such a case you have to read EXPLAIN results and tweak indexes accordingly.
However, for only 1600 rows to sort I wouldn't bother at all. If you still want to make it faster, it seems you need a combined key that would include a Model and a sitecert followed by price, both in a reverse order.
Means you have to have two additional columns where negative values of sitecert and price are stored. then buind an index on (Model,nsitecert, nInternetPrice) and order without DESC to eliminate a filesort
If you have search queries that involve no Model but any combination of parameters - then it's time to think of an external search engine like Sphinx Search.

Related

How do i make my database auto-generate unique Certificate number that will also be auto-incremented for example 00001

I am creating a database that will store 'graduate details'. I have been able to set the ID to auto increment, Unique and also my primary key that start from 1...
I also want to make the certificate number field unique and auto-increment value in an increasing order that will start from 00001.
Please any help on how to go about this will be very appreciated. Thank you.
This is the MYSQL code for the database below;
CREATE TABLE IF NOT EXISTS `graduates` (
`id` int(10) NOT NULL AUTO_INCREMENT,
`f_name` varchar(25) NOT NULL,
`l_name` varchar(25) NOT NULL,
`gender` varchar(6) DEFAULT NULL,
`address` varchar(100) DEFAULT NULL,
`city` varchar(15) DEFAULT NULL,
`region` varchar(30) DEFAULT NULL,
`phone` varchar(15) DEFAULT NULL,
`email` varchar(50) DEFAULT NULL,
`certificate_number` int(50) NOT NULL AUTO_INCREMENT,
`programme` varchar(50) CHARACTER SET utf8mb4 NOT NULL,
`marks` int(10) NOT NULL,
`college_name` varchar(50) CHARACTER SET utf8mb4 NOT NULL,
`date_of_birth` date DEFAULT NULL,
`created_by` int(10) UNSIGNED NOT NULL DEFAULT 0,
`created_at` date DEFAULT NULL,
`updated_by` int(10) UNSIGNED NOT NULL DEFAULT 0,
`updated_at` timestamp NULL DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `certificate_number` (`certificate_number`)
) ENGINE=InnoDB AUTO_INCREMENT=90 DEFAULT CHARSET=utf8;

Using PHPExcel to export Excel to MySql

I've found information on how to export data from MySQL to Excel, but not upside down, from Excel to MySQL.
Also, I'm asking using PHPExcel because a friend suggested to do it with it, but if there is an easiest way to do it, it's welcome.
I know I can export from excel to MySql (with static field) using following syntax:
LOAD DATA LOCAL INFILE '\\path' INTO TABLE test FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY '\r\n'
But not all Excel fields will be imported into MySQL or in the same order.
This is the table structure that I have:
CREATE TABLE cliente (
clienteId int(11) NOT NULL AUTO_INCREMENT,
clienteIdentidadTipo varchar(45) DEFAULT NULL,
clienteIdEntidadNumero varchar(45) DEFAULT NULL,
clienteNombres varchar(100) DEFAULT NULL,
clienteApellidosPrimero varchar(45) DEFAULT NULL,
clienteApellidosSegundo varchar(45) DEFAULT NULL,
clienteDireccion varchar(45) DEFAULT NULL,
clienteCelular varchar(45) DEFAULT NULL,
clienteTelefono varchar(45) DEFAULT NULL,
clienteGenero varchar(45) DEFAULT NULL,
clienteFechaNacimiento date DEFAULT NULL,
clienteEstadoCivil varchar(45) DEFAULT NULL,
clienteCargo varchar(45) DEFAULT NULL,
clienteOcupacion varchar(45) DEFAULT NULL,
clienteRecibeMensajesEmail tinyint(1) DEFAULT NULL,
clienteEmail varchar(45) DEFAULT NULL,
clienteClave varchar(100) DEFAULT NULL,
clienteRecibeMensajesSms tinyint(1) DEFAULT NULL,
clienteObjetivoVermeMejor tinyint(1) DEFAULT NULL,
clienteObjetivoSalud tinyint(1) DEFAULT NULL,
clienteObjetivoDesempeno tinyint(1) DEFAULT NULL,
clientePracticaDeporte tinyint(1) DEFAULT NULL,
clienteServicioCardio tinyint(1) DEFAULT NULL,
clienteServicioPersonalizado tinyint(1) DEFAULT NULL,
clienteServicioGrupales tinyint(1) DEFAULT NULL,
clienteServicioPesas tinyint(1) DEFAULT NULL,
clienteServicioFisioterapia tinyint(1) DEFAULT NULL,
clienteServicioFuncional tinyint(1) DEFAULT NULL,
clienteServicioOtro varchar(45) DEFAULT NULL,
clienteMedioConocioGym varchar(45) DEFAULT NULL,
clienteEmergenciaNombre varchar(45) DEFAULT NULL,
clienteEmergenciaTelefono varchar(45) DEFAULT NULL,
clienteEps varchar(45) DEFAULT NULL,
clienteIps varchar(45) DEFAULT NULL,
clienteCiudadId int(11) NOT NULL,
clienteEmpresaId int(11) NOT NULL,
clienteGrupoId int(11) NOT NULL,
clienteCertificaConcentimientoInformado tinyint(1) DEFAULT NULL,
clienteCertificaCuestionamientoAntecedente tinyint(1) DEFAULT NULL,
clienteCongelacion int(11) DEFAULT NULL,
clienteDiasTolerancia int(11) DEFAULT NULL,
clienteFechaPago int(11) DEFAULT NULL,
clienteTiquetera tinyint(1) DEFAULT NULL,
clienteUsuario varchar(45) DEFAULT NULL,
historiaClinicaId int(11) NOT NULL,
planId int(11) NOT NULL,
clientePracticaDeporteCual varchar(45) DEFAULT NULL,
clienteCupoCredito int(7) NOT NULL,
clienteImagen varchar(500) NOT NULL,
clienteValidacion int(11) NOT NULL,
clienteRh varchar(45) DEFAULT NULL,
PRIMARY KEY (clienteId,clienteCiudadId,clienteEmpresaId,clienteGrupoId,historiaClinicaId,planId),
UNIQUE KEY clienteId_UNIQUE (clienteId),
KEY fk_cliente_ciudad_idx (clienteCiudadId),
KEY fk_cliente_empresa1_idx (clienteEmpresaId),
KEY fk_cliente_grupo1_idx (clienteGrupoId),
KEY fk_cliente_historiaClinica1_idx (historiaClinicaId),
KEY fk_cliente_plan1_idx (planId),
CONSTRAINT fk_cliente_ciudad FOREIGN KEY (clienteCiudadId) REFERENCES ciudad (ciudadId) ON DELETE NO ACTION ON UPDATE NO ACTION,
CONSTRAINT fk_cliente_empresa1 FOREIGN KEY (clienteEmpresaId) REFERENCES empresa (empresaId) ON DELETE NO ACTION ON UPDATE NO ACTION,
CONSTRAINT fk_cliente_grupo1 FOREIGN KEY (clienteGrupoId) REFERENCES grupo (grupoId) ON DELETE NO ACTION ON UPDATE NO ACTION,
CONSTRAINT fk_cliente_historiaClinica1 FOREIGN KEY (historiaClinicaId) REFERENCES historiaclinica (historiaClinicaId) ON DELETE NO ACTION ON UPDATE NO ACTION,
CONSTRAINT fk_cliente_plan1 FOREIGN KEY (planId) REFERENCES plan (planId) ON DELETE NO ACTION ON UPDATE NO ACTION
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
And this is the Excel file that I want to import into the database:
This will probably help:
Import an excel file into a MySQL table with PHPExcel
Also, in the future, most of the questions you will have with programming have already been answered. So you can usually find the answer in 1-2 google searches.

Exclude title and content that are NULL before output in SQL

I have the following SQL query and want to exclude title and content that is null. I have tried with WHERE title IS NOT NULL, but obviously that didn't work for me and returns error: Syntax error or access violation.
The current query
SELECT * FROM entries ORDER BY id DESC LIMIT $postnumbers OFFSET $offset
What i tried
SELECT * FROM entries WHERE title IS NOT NULL ORDER BY id DESC LIMIT $postnumbers OFFSET $offset
What is the correct answer?
Table definition
entries
CREATE TABLE entries (
id int(11) NOT NULL AUTO_INCREMENT,
title varchar(150) DEFAULT NULL,
content longtext,
category int(11) DEFAULT NULL,
status int(11) DEFAULT NULL,
keyword1 varchar(45) DEFAULT NULL,
keyword2 varchar(45) DEFAULT NULL,
likes varchar(45) DEFAULT NULL,
image varchar(45) DEFAULT NULL,
description` varchar(45) DEFAULT NULL,
owner int(30) DEFAULT 0,
page_views int(50) DEFAULT NULL,
time int(20) DEFAULT NULL,
maincategory int(10) DEFAULT NULL,
entriescol varchar(45) DEFAULT NULL,
PRIMARY KEY (id)
) ENGINE=InnoDB AUTO_INCREMENT=73 DEFAULT CHARSET=latin1

Making join on 2 tables with millions record to be faster

I have two tables
security_stat => having 4 millions record
security_trade => having 10 millions record
I have this query running successfully but how can i OPTIMIZE this to be able to at least query 100,000 record within 10 seconds ( is it possible? ).. Currently it is very very slow.
SELECT `sec_stat_sec_name`, `sec_stat_date`, `sec_stat_market`, `sec_trade_close`, `sec_stat_date`
FROM security_stat` LEFT JOIN `security_trade`
ON `security_trade`.`sec_trade_sec_name` = `security_stat`.`sec_stat_sec_name`
and `security_trade`.`sec_trade_date` = `security_stat`.`sec_stat_date`
limit 100,000
I have INDEX on sec_trade_sec_name, sec_stat_sec_name, sec_trade_date , sec_stat_date
I tried limiting result with WHERE sec_stat_date >= 2005-01-01 but that doesn't help much. (my records range from 1975 - 2014)
EDIT
security_stat schema
CREATE TABLE `security_stat` (
`sec_stat_id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`sec_stat_date` date NOT NULL,
`sec_stat_sec_name` varchar(255) NOT NULL,
`sec_stat_sec_id` int(11) NOT NULL,
`sec_stat_market` varchar(255) NOT NULL,
`sec_stat_industry` int(11) NOT NULL,
`sec_stat_sector` int(11) NOT NULL,
`sec_stat_subsector` int(11) NOT NULL,
`sec_stat_sec_type` varchar(1) NOT NULL,
`sec_stat_status` varchar(2) NOT NULL,
`sec_stat_benefit` varchar(2) NOT NULL,
`sec_stat_listed_share` bigint(20) NOT NULL,
`sec_stat_earn_p_share` decimal(12,5) NOT NULL,
`sec_stat_value` decimal(9,2) NOT NULL,
`sec_stat_p_of_earn` int(11) NOT NULL,
`sec_stat_as_date` date NOT NULL,
`sec_stat_div_p_share` decimal(16,12) NOT NULL,
`sec_stat_p_of_div` int(11) NOT NULL,
`sec_stat_end_date_div` date NOT NULL,
`sec_stat_pe` decimal(8,2) NOT NULL,
`sec_stat_pbv` decimal(8,2) NOT NULL,
`sec_stat_div_yield` decimal(8,2) NOT NULL,
`sec_stat_par_value` decimal(16,5) NOT NULL,
`sec_stat_market_cap` decimal(20,2) NOT NULL,
`sec_stat_turn_ratio` decimal(8,2) NOT NULL,
`sec_stat_npg_flag` varchar(1) NOT NULL,
`sec_stat_acc_div` decimal(16,12) NOT NULL,
`sec_stat_acc_no_of_pay` int(11) NOT NULL,
`sec_stat_div_pay_ratio` decimal(6,2) NOT NULL,
`sec_stat_earn_date` date NOT NULL,
`created_at` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`updated_at` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`sec_stat_ev` decimal(20,2) DEFAULT NULL,
`sec_stat_ev_revenue` decimal(20,2) DEFAULT NULL,
`sec_stat_ev_ebit` decimal(20,2) DEFAULT NULL,
`sec_stat_ev_ebitda` decimal(20,2) DEFAULT NULL,
`sec_stat_earning_yield` decimal(10,5) DEFAULT NULL,
`sec_stat_ps_ratio` decimal(10,5) DEFAULT NULL,
PRIMARY KEY (`sec_stat_id`),
UNIQUE KEY `sec_stat_date_name_id_cap` (`sec_stat_date`,`sec_stat_market`,`sec_stat_sec_id`,`sec_stat_sector`),
KEY `sec_stat_date` (`sec_stat_date`),
KEY `sec_stat_sec_name` (`sec_stat_sec_name`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 AUTO_INCREMENT=3598612 ;
security_trade schema
CREATE TABLE `security_trade` (
`sec_trade_id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`sec_trade_date` date NOT NULL,
`sec_trade_sec_name` varchar(20) NOT NULL,
`sec_trade_sec_id` int(11) NOT NULL,
`sec_trade_market` varchar(1) NOT NULL,
`sec_trade_trading_method` varchar(1) NOT NULL,
`sec_trade_trade_report` varchar(1) NOT NULL,
`sec_trade_prior_date` date NOT NULL,
`sec_trade_prior` decimal(8,2) NOT NULL,
`sec_trade_open` decimal(8,2) NOT NULL,
`sec_trade_high` decimal(8,2) NOT NULL,
`sec_trade_low` decimal(8,2) NOT NULL,
`sec_trade_close` decimal(8,2) NOT NULL,
`sec_trade_last_bid` decimal(8,2) NOT NULL,
`sec_trade_last_offer` decimal(8,2) NOT NULL,
`sec_trade_transaction` int(11) NOT NULL,
`sec_trade_volume` bigint(20) NOT NULL,
`sec_trade_value` decimal(20,2) NOT NULL,
`sec_trade_avg_price` decimal(8,2) NOT NULL,
`created_at` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`updated_at` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
PRIMARY KEY (`sec_trade_id`),
UNIQUE KEY `sec_trade_close` (`sec_trade_date`,`sec_trade_sec_name`,`sec_trade_market`,`sec_trade_trade_report`,`sec_trade_trading_method`),
KEY `security_trade_sec_trade_sec_name_index` (`sec_trade_sec_name`),
KEY `security_trade_sec_trade_date_index` (`sec_trade_date`),
KEY `security_trade_sec_trade_prior_date_index` (`sec_trade_prior_date`),
KEY `security_trade_sec_trade_close_index` (`sec_trade_close`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 AUTO_INCREMENT=10019817 ;
My Final query will actually have more
WHERE sec_stat_earning_yield IS NULL
ORDER BY updated_at ASC
but because when i add this two statement into the query with LIMIT of 1,000 records it make my query even slower ( may be because i didn't have index on this two columns? )
Thanks in Advance
Taking the following as your actual query:
SELECT `sec_stat_sec_name`, `sec_stat_date`, `sec_stat_market`, `sec_trade_close`, `sec_stat_date`
FROM `security_stat` LEFT JOIN `security_trade`
ON `security_trade`.`sec_trade_sec_name` = `security_stat`.`sec_stat_sec_name`
and `security_trade`.`sec_trade_date` = `security_stat`.`sec_stat_date`
WHERE sec_stat_earning_yield IS NULL
ORDER BY updated_at ASC
limit 100,000
You filter the security_stat table in two ways:
1. Only where sec_stat_earning_yield IS NULL
2. First 100k records when ordered by updated_at
Note: I've assume you mean security_stat.updated_at, but you don't make that clear.
In order to make that as cheap as possible add an index that covers both of those fields (sec_stat_earning_yield, updated_at).
Note: Adding indexes that change a lot, especially when the order of the records changes within the index, can make a INSERTs slower. You will need to balance INSERT performance against SELECT performance.
Then you join the trades table on, and so you want that lookup to be as fast as possible, which can be achieved with an index on that table covering (sec_trade_sec_name, sec_trade_date, sec_trade_close).
- The first two fields in the index make the lookup simpler
- The last field in the index means the DBMS can avoid having to look in the table
Once done you may also be well served looking at the EXPLAIN plan, although relatively complicated it will give you key information to understand the best places to target your optimisation.
First, try createing indexes to match the join so:
security_trade (sec_trade_sec_name, sec_trade_date)
security_stat (sec_stat_sec_name, sec_stat_date)
or possibly
security_stat (sec_stat_earning_yield, sec_stat_sec_name, sec_stat_date)
And as pointed out in the comments above, your "Limit" clause may cause the result set to be not clearly defined.

MySQL Query Optimization (Takes way too long)

I use the following prepared statement:
SELECT *
FROM
c_members,c_users,c_positions,c_done_meetings
WHERE
c_positions.POS_ID=c_users.POS_ID
AND c_members.CLUB_ID = ?
AND USER_POINTS >= ?
AND USER_POINTS <= ?
AND c_users.POS_ID LIKE ?
AND MEM_ACADEMY LIKE ?
AND MEM_SEX LIKE ?
AND MEM_GRADELVL LIKE ?
AND MEM_GPA >= ?
AND MEM_GPA <= ?
AND MEM_ARCHIVE = 0
GROUP BY
c_members.MEM_ID, c_members.CLUB_ID
HAVING
SUM(c_done_meetings.MEDONE_ATTEND = 'u') >= 1
ORDER BY
USER_POINTS DESC
However this query takes 21.971405982971 seconds to load 111 records. When I remove "Having SUM(...)" clause, the performance is 100% better. Is there a way I could optimize it better?
Edit: (Table structures)
CREATE TABLE IF NOT EXISTS `c_done_meetings` (
`MEM_ID` int(11) NOT NULL,
`CLUB_ID` int(11) NOT NULL,
`MEETING_ID` int(11) NOT NULL,
`MEDONE_ATTEND` varchar(1) NOT NULL COMMENT 'E=excused, U=unexcused, P=present',
UNIQUE KEY `unique` (`MEM_ID`,`CLUB_ID`,`MEETING_ID`),
KEY `MEETING_ID` (`MEETING_ID`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
CREATE TABLE IF NOT EXISTS `c_members` (
`MEM_ID` int(11) NOT NULL,
`CLUB_ID` int(11) NOT NULL,
`MEM_FIRST` varchar(50) NOT NULL,
`MEM_MIDDLE` varchar(50) DEFAULT NULL,
`MEM_LAST` varchar(50) NOT NULL,
`MEM_SEX` tinyint(1) NOT NULL COMMENT '0-Male 1-Female',
`MEM_EMAIL` varchar(100) DEFAULT NULL,
`MEM_GRADELVL` int(11) NOT NULL,
`MEM_ACADEMY` varchar(50) DEFAULT '',
`MEM_GPA` double DEFAULT '0',
`MEM_ADDRESS` varchar(500) DEFAULT NULL,
`MEM_CITY` varchar(100) DEFAULT NULL,
`MEM_STATE` varchar(100) DEFAULT NULL,
`MEM_ZIP` int(11) DEFAULT NULL,
`MEM_TELEPHONE` varchar(25) DEFAULT NULL,
`MEM_AP` tinyint(1) NOT NULL,
`MEM_HONORS` tinyint(1) NOT NULL,
`MEM_ESOL` tinyint(1) NOT NULL,
`MEM_HISP` tinyint(1) NOT NULL,
`MEM_WHITE` tinyint(1) NOT NULL,
`MEM_MULTI` tinyint(1) NOT NULL,
`MEM_NATIVE` tinyint(1) NOT NULL,
`MEM_BLACK` tinyint(1) NOT NULL,
`MEM_ASIAN` tinyint(1) NOT NULL,
`MEM_EXTRA` varchar(10000) DEFAULT NULL,
`MEM_ARCHIVE` tinyint(1) NOT NULL DEFAULT '0',
UNIQUE KEY `MEM_ID` (`MEM_ID`,`CLUB_ID`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
CREATE TABLE IF NOT EXISTS `c_positions` (
`POS_ID` int(11) NOT NULL AUTO_INCREMENT,
`CLUB_ID` int(11) NOT NULL,
`POS_NAME` varchar(20) NOT NULL,
`POS_DESC` varchar(500) NOT NULL,
`POS_ADMIN` tinyint(1) NOT NULL,
`POS_ATN_VIEW` tinyint(1) NOT NULL,
`POS_ATN_CHKIN` tinyint(1) NOT NULL,
`POS_ATN_FINALIZE` tinyint(1) NOT NULL,
`POS_MEM_VIEW` tinyint(1) NOT NULL,
`POS_MEM_ADD` tinyint(1) NOT NULL,
`POS_MEM_EDIT` tinyint(1) NOT NULL,
`POS_POS_VIEW` tinyint(1) NOT NULL,
`POS_POS_ADD` tinyint(1) NOT NULL,
`POS_POS_EDIT` tinyint(1) NOT NULL,
`POS_MEET_VIEW` tinyint(1) NOT NULL,
`POS_MEET_ADD` tinyint(1) NOT NULL,
`POS_MEET_EDIT` tinyint(1) NOT NULL,
`POS_EVENT_VIEW` tinyint(1) NOT NULL,
`POS_EVENT_ADD` tinyint(1) NOT NULL,
`POS_EVENT_EDIT` tinyint(1) NOT NULL,
`POS_EVENT_UPDATE` tinyint(1) NOT NULL,
`POS_REPORT_VIEW` tinyint(1) NOT NULL,
`POS_ARCHIVE_VIEW` tinyint(1) NOT NULL,
`POS_ANNOUNCEMENTS` tinyint(1) NOT NULL,
`POS_WEB_CUSTOM` tinyint(1) NOT NULL,
PRIMARY KEY (`POS_ID`),
UNIQUE KEY `UNIQUE_NAME` (`CLUB_ID`,`POS_NAME`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=14 ;
CREATE TABLE IF NOT EXISTS `c_users` (
`MEM_ID` int(11) NOT NULL,
`CLUB_ID` int(11) NOT NULL,
`USER_PIN` int(11) NOT NULL,
`USER_POINTS` double NOT NULL,
`POS_ID` int(11) NOT NULL,
`USER_ARCHIVE` tinyint(1) NOT NULL DEFAULT '0',
UNIQUE KEY `MEM_ID` (`MEM_ID`,`CLUB_ID`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
Edit 2:
Yes all ids are indexed, the SUM(...) >= # calculates the number of missed meetings. # is a parameter set by the user (I just hard coded a 1 for testing)
You need to have indexes an all fields used in the WHERE clause, on all fields on which the tables are joined (you need to explicitly state your join conditions as right now, you are getting a Cartesian join), on all fields used for grouping and all fields used for sorting.
The last problem is the HAVING clause there. You are not going to be able to use an index at all for that since it is a calculated value. If this is a query you will use often in the system (i.e. not just for reporting), you might consider adding a field you can use as a flag for this filtering purpose. Whenever you set c_done_meetings.MEDONE_ATTEND = 'u' in any of your queries, you could also set this flag for the member or user or whatever this is associated with so that you have an easy field to filter on in a WHERE clause.
Outside of that, you might actually gain better performance by getting a reduced list of users or members with that value of u in a subselect and then join using that subselect as a table.
EDIT:
After seeing your actual table structure, I can clearly see where you need to add indexes. I am also wondering why you have tables c_users and c_members with the same exact primary key. Why would these not just be a single table?
Some things that pop out at me:
You use like quite a lot. Try to do something with your php code so that this isn't necessary. For example, mem_sex has a limited number of choices. Make the front end a radio button or dropdown so you send a value where you can use = instead of like.
Two, if you add a sum() to the select clause, and the appropriate group by clause, it should run faster. It's worth a shot.
You can denormalise you tables and add field to c_members that represent your
SUM(c_done_meetings.MEDONE_ATTEND = 'u') >= 1
But you need to update that field always, when updating c_done_meetings (can be done with trigger)
Also try to avoid LIKE conditions. Use = insead (it is possible at least for SEX)

Categories