Optimize this query further? - indexed but still slow - php

Question

I have a query below that joins 4 tables. I have added indexes and the explain output looks good with indexes used (see below).
Can I optimize the query further?
The modelXml is rather big on some records.For a big project where I get back 22 records, each with approximately 2.5 - 3MB of modelXml data, the query is taking long (total of 69MB of data returned).
I suspect this is the issue but not sure how to deal with it.
I was reading on adjusting internal mysql variables e.g key_buffer_size and table_cache. Would any of this help?
key_buffer_size is currently set at 8384512 (~8MB) and table_cache at 64
What should I increase it to?
What other variables should I be looking at to manage to speed up return of such big data?
Any other suggestions are welcome. I am a novice to mysql but really trying to get better.
SELECT `m`.`modelId`, `m`.`modelTypeId`, `m`.`modelXml`, `m`.`xmlSize`, `m`.`createdById`, `m`.`creationDate`, `m`.`modifiedDate`, `u`.`firstName`, `u`.`lastName` FROM `models_1` AS `m`
INNER JOIN `modelFolderAssociations_1` AS `mfa` ON m.modelId = mfa.modelIOId
INNER JOIN `modelFolders_1` AS `mf` ON mfa.folderId = mf.folderId
INNER JOIN `users_1` AS `u` ON m.createdById = u.userId
WHERE (m.projectId = 2) AND (mfa.folderId = 5) AND (mfa.modelIOType = 2) AND (m.modelTypeId = 2)
CREATE TABLE `models` (
`modelId` int(11) NOT NULL auto_increment,
`customerId` int(11) NOT NULL,
`groupId` int(11) NOT NULL,
`projectId` int(11) NOT NULL,
`createdById` int(11) NOT NULL,
`modelTypeId` int(11) NOT NULL,
`modelXml` longtext,
`modelSpecXml` longtext NOT NULL,
`xmlSize` bigint(20) NOT NULL default '0',
`creationDate` datetime NOT NULL,
`modifiedDate` datetime NOT NULL,
PRIMARY KEY (`modelId`,`customerId`)
) ENGINE=InnoDB AUTO_INCREMENT=75 DEFAULT CHARSET=utf8
CREATE TABLE `modelFolders` (
`folderId` int(11) NOT NULL auto_increment,
`customerId` int(11) NOT NULL,
`groupId` int(11) NOT NULL,
`projectId` int(11) NOT NULL,
`parentId` int(11) NOT NULL,
`folderName` varchar(64) NOT NULL,
`folderType` int(11) NOT NULL,
`editable` tinyint(1) NOT NULL default '1',
`nextDefaultNameNumber` int(11) NOT NULL default '1',
`creationDate` datetime NOT NULL,
`modifiedDate` datetime NOT NULL,
PRIMARY KEY (`folderId`,`customerId`),
KEY `parentId` (`parentId`)
) ENGINE=InnoDB AUTO_INCREMENT=16 DEFAULT CHARSET=utf8
CREATE TABLE `modelFolderAssociations` (
`associationId` int(11) NOT NULL auto_increment,
`customerId` int(11) NOT NULL,
`folderId` int(11) NOT NULL,
`projectId` int(11) NOT NULL,
`modelIOId` int(11) NOT NULL,
`modelIOType` tinyint(1) NOT NULL default '1',
`creationDate` datetime NOT NULL,
`modifiedDate` datetime NOT NULL,
PRIMARY KEY (`associationId`,`customerId`),
KEY `folderId` (`folderId`,`modelIOType`)
) ENGINE=InnoDB AUTO_INCREMENT=75 DEFAULT CHARSET=utf8
CREATE TABLE `users` (
`userId` int(11) NOT NULL auto_increment,
`customerId` int(11) NOT NULL,
`userName` varchar(50) NOT NULL,
`password` varchar(256) NOT NULL,
`firstName` varchar(50) default NULL,
`lastName` varchar(50) default NULL,
`creationDate` datetime NOT NULL,
`modifiedDate` datetime NOT NULL,
PRIMARY KEY (`userId`,`customerId`),
UNIQUE KEY `userName` (`userName`)
) ENGINE=InnoDB AUTO_INCREMENT=21 DEFAULT CHARSET=utf8
Explain output
+----+-------------+-------------------------+--------+---------------+----------+---------+---------------------------------------------------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------------------------+--------+---------------+----------+---------+---------------------------------------------------+------+-------------+
| 1 | SIMPLE | modelFolders | const | PRIMARY | PRIMARY | 8 | const,const | 1 | Using index |
| 1 | SIMPLE | modelFolderAssociations | ref | folderId | folderId | 5 | const,const | 22 | Using where |
| 1 | SIMPLE | models | eq_ref | PRIMARY | PRIMARY | 8 | xa_system.modelFolderAssociations.modelIOId,const | 1 | Using where |
| 1 | SIMPLE | users | eq_ref | PRIMARY | PRIMARY | 8 | xa_system.models.createdById,const | 1 | |
+----+-------------+-------------------------+--------+---------------+----------+---------+---------------------------------------------------+------+-------------+

Answer 1

Having large text columns, like those you're using to store XML I presume, can hurt performance regardless of how well your indexes are structured.
It can be better in these cases to move the text columns to a separate table, indexed by string length which you're already storing and CRC32.
CREATE TABLE MODEL_XML (
xmlId INT(11) unsigned NOT NULL auto_increment,
xmlSize BIGINT(20) NOT NULL default '0',
crc32 INT(11) unsigned NOT NULL,
xmlData LONGTEXT,
PRIMARY KEY (xmlId),
UNIQUE KEY (xmlSize, crc32)
)
Then the width of your columns on the table with your important indices becomes constant.
ex.
modelXmlId INT(11) unsigned NOT NULL
specXmlId INT(11) unsigned NOT NULL
It also has the benefit of being more space-efficient for redundant text (empty strings, etc) since they would all share one xmlId and thus one row in the DB.

Answer 2

You should index your foreign keys, this will really help with your joins:
CREATE INDEX IDX_MODELS_CUSTID
on models (customerId)
CREATE INDEX IDX_FLDR_ASSOC_MODELIO
modelFolderAssociations(modelIOId)
CREATE INDEX IDX_FLDR_ASSOC_FLDRID
modelFolderAssociations(folderId)
and so on.

Answer 3

Test your query after taking off the modelXml column from the SELECT clause.
If the speed is then significantly better, then the slowness comes from the volume off data to transfer, not the query itself.

Answer 4

Your indexes don't always match the queries and joins. If all of the columns in your WHERE and JOIN clauses are not in the index, you're forcing mysql to look at the underlying row, which will kill your performance, especially since your model rows are so wide.
For "modelFolderAssociations", you made the correct composite key for the where clause, but you should include modelIOId for the join to model.
For "models", you'd want a composite index on (modelId, projectId, modelTypeId, createdById ) to cover the incoming link from mfa, the two items in the where clause, and the outbound link to users.
For "modelFolders" and "users", you have the incoming join covered in your primary keys.
The engine is only going to use one index, so adding extra index single indexes (as Mike suggested) won't be as good.