I've got two tables (NewProducts and OldProducts) that are being compared. NewProducts has about 68,000 records and OldProducts about 51,000. I'm using a covering index on each table, however the query is taking 20 minutes to execute, so I'm not using it properly. Does a covering index really apply with multiple tables? What am I doing wrong? Thank you.
Here is my query code and the indexes:
$querystring = "SELECT newProducts.Id, newProducts.SKU,
newProducts.Title, oldProducts.Title, oldProducts.product_Id
FROM
newProducts, oldProducts
WHERE
trim(newProducts.SKU)=trim(oldProducts.SKU) and
trim(newProducts.Title)=trim(oldProducts.Title) and
oldProducts.Position=1 and
oldProducts.Customer=$shop";
Indexes for NewProducts:
Primary: Id
Index: SKU, Title, customer (not unique)
Indexes for OldProducts:
Primary: Id
Index: Product_id (not unique)
Index: SKU, Title, Postition, Customer (not unique)
?>
CREATE TABLE `NewProducts` (
`Id` bigint(11) NOT NULL,
`Title` varchar(120) COLLATE utf8_unicode_ci NOT NULL,
`Category` varchar(100) COLLATE utf8_unicode_ci NOT NULL,
`Office` varchar(150) COLLATE utf8_unicode_ci NOT NULL,
`Rehashed` smallint(6) NOT NULL,
`Quantity` smallint(6) NOT NULL,
`Price1` decimal(7,2) NOT NULL,
`Price2` decimal(7,2) NOT NULL,
`Price3` decimal(7,2) NOT NULL,
`Price4` decimal(7,2) NOT NULL,
`created_at` varchar(30) COLLATE utf8_unicode_ci DEFAULT NULL,
`OldQuantity` int(11) NOT NULL,
`SKU` varchar(55) COLLATE utf8_unicode_ci DEFAULT NULL,
`Source` varchar(12) COLLATE utf8_unicode_ci NOT NULL,
`customer` varchar(70) COLLATE utf8_unicode_ci NOT NULL,
PRIMARY KEY (`Id`),
UNIQUE KEY `I-T-S` (`ItemId`,`Title`,`SKU`),
KEY `customer` (`customer`),
KEY `Title` (`Title`,`Rehashed`),
KEY `SKU` (`SKU`),
KEY `Title_2` (`Title`,`SKU`,`customer`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
CREATE TABLE `OldProducts` (
`barcode` varchar(55) COLLATE utf8_unicode_ci DEFAULT NULL,
`compare_at_price` decimal(10,2) DEFAULT NULL,
`created_at` varchar(30) COLLATE utf8_unicode_ci DEFAULT NULL,
`fulfillment` varchar(35) COLLATE utf8_unicode_ci DEFAULT NULL,
`grams` decimal(10,2) DEFAULT NULL,
`id` bigint(11) NOT NULL,
`management` varchar(55) COLLATE utf8_unicode_ci DEFAULT NULL,
`policy` varchar(55) COLLATE utf8_unicode_ci DEFAULT NULL,
`size` varchar(55) COLLATE utf8_unicode_ci DEFAULT NULL,
`color` varchar(55) COLLATE utf8_unicode_ci DEFAULT NULL,
`type` varchar(55) COLLATE utf8_unicode_ci DEFAULT NULL,
`position` int(11) DEFAULT NULL,
`price` varchar(15) COLLATE utf8_unicode_ci DEFAULT NULL,
`product_id` bigint(11) NOT NULL,
`SKU` varchar(55) COLLATE utf8_unicode_ci DEFAULT NULL,
`Title` varchar(100) COLLATE utf8_unicode_ci DEFAULT NULL,
`quantity` int(11) DEFAULT NULL,
`customer` varchar(70) COLLATE utf8_unicode_ci NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `P-S-T-PO-CUST`
(`product_id`,`SKU`,`Title`,`position`,`customer`),
KEY `product_id` (`product_id`),
TRIM is the villain. When you hide an indexed column (eg, SKU) inside a function (eg, TRIM), the the index cannot be used.
Clean up your data:
Fix the insertion code to TRIM before inserting (or as it inserts).
UPDATE tbl SET SKU = TRIM(SKU), title = TRIM(title); -- for each table
Change the SELECT: TRIM(SKU) --> SKU etc.
Even Better
oldProducts should have, in this order,
`INDEX(`position`,`customer` ,`SKU`,`Title`, `product_id`)
With this, the WHERE need look only at old rows for position=1 and customer =.... (Actually, the first 2 columns can be in any order; the last 3 in any order.)
Related
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 3 years ago.
Improve this question
I have a list of contents on my website and it is getting filled with records rapidly, for example after one month, there are 300K records in it. Because of this, the list page is responding slower and slower over time.
This page has this functionalities:
Searching
Paging
Also this table Left Joins other tables and it increases the run-time.
I run two queries every time this page is loaded, one for getting 10 limited records, and the other for getting number of all records.
How do I handle this amount of data with out jeopardizing user experience?
EDIT
Here is my Query:
SELECT *,
`note`.`attached_file` AS `attached_file`,
`note`.`description` AS `description`,
`note`.`id` AS `id`,
`note_type`.`title` AS `title`,
`note_goal`.`title` AS `goal`
FROM `note`
LEFT JOIN `contact` ON `note`.`id_contact`=`contact`.`id`
LEFT JOIN `contact_activity_field` ON `contact_activity_field`.`id_contact`=`contact`.`id`
LEFT JOIN `activity` ON `contact_activity_field`.`id_activity`=`activity`.`id`
LEFT JOIN `note_type` ON `note`.`title`=`note_type`.`id`
LEFT JOIN `note_goal` ON `note`.`goal`=`note_goal`.`id`
WHERE
( `note_type`.`title` LIKE '%$q%' OR
`firstname_eng` LIKE '%$q%' OR
`lastname_eng` LIKE '%$q%' OR
`firstname_per` LIKE '%$q%' OR
`lastname_per` LIKE '%$q%' OR
`company_name` LIKE '%$q%' OR
`company_name_per` LIKE '%$q%' OR
`description` LIKE '%$q%' OR
`note_goal`.`title` LIKE '%$q%' ) AND some other condition
GROUP BY `note`.`id`
ORDER BY `note`.`id` DESC
LIMIT $start_from, 10
EDIT 2
note table
CREATE TABLE `note` (
`id` int(10) NOT NULL AUTO_INCREMENT,
`id_user` int(10) NOT NULL,
`id_contact` int(10) NOT NULL,
`title` int(10) DEFAULT NULL,
`goal` int(10) DEFAULT NULL,
`register_date` date DEFAULT NULL,
`description` text COLLATE utf8_unicode_ci,
`attached_file` varchar(500) COLLATE utf8_unicode_ci DEFAULT NULL,
`meeting_place` text COLLATE utf8_unicode_ci,
`start_time` varchar(50) COLLATE utf8_unicode_ci DEFAULT NULL,
`finish_time` varchar(50) COLLATE utf8_unicode_ci DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=3297 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
contact table
CREATE TABLE `contact` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`id_user` int(10) NOT NULL DEFAULT '0',
`id_user_registered` int(10) DEFAULT NULL,
`code` varchar(50) COLLATE utf8_unicode_ci DEFAULT NULL,
`contact_type` varchar(20) COLLATE utf8_unicode_ci DEFAULT 'legal',
`firstname_eng` varchar(1000) COLLATE utf8_unicode_ci DEFAULT NULL,
`lastname_eng` varchar(1000) COLLATE utf8_unicode_ci DEFAULT NULL,
`firstname_per` varchar(1000) COLLATE utf8_unicode_ci DEFAULT NULL,
`lastname_per` varchar(1000) COLLATE utf8_unicode_ci DEFAULT NULL,
`gender` varchar(10) COLLATE utf8_unicode_ci DEFAULT NULL,
`id_number` varchar(20) COLLATE utf8_unicode_ci DEFAULT NULL,
`national_code` varchar(20) COLLATE utf8_unicode_ci DEFAULT NULL,
`birth_date` date DEFAULT NULL,
`company_name` varchar(500) COLLATE utf8_unicode_ci DEFAULT NULL,
`company_name_per` varchar(1000) COLLATE utf8_unicode_ci DEFAULT NULL,
`recommender_eng` varchar(1000) COLLATE utf8_unicode_ci DEFAULT NULL,
`recommender_per` varchar(1000) COLLATE utf8_unicode_ci DEFAULT NULL,
`company_type` varchar(50) COLLATE utf8_unicode_ci DEFAULT NULL,
`registration_type` varchar(50) COLLATE utf8_unicode_ci DEFAULT NULL,
`registration_date` date DEFAULT '1900-01-01',
`registration_number` varchar(20) COLLATE utf8_unicode_ci DEFAULT NULL,
`national_id` varchar(20) COLLATE utf8_unicode_ci DEFAULT NULL,
`economic_code` varchar(20) COLLATE utf8_unicode_ci DEFAULT NULL,
`website` varchar(1000) COLLATE utf8_unicode_ci DEFAULT NULL,
`postal_code` varchar(100) COLLATE utf8_unicode_ci DEFAULT NULL,
`address_eng` text COLLATE utf8_unicode_ci,
`address_per` text COLLATE utf8_unicode_ci,
`phone_number` varchar(1000) COLLATE utf8_unicode_ci DEFAULT NULL,
`email` varchar(1000) COLLATE utf8_unicode_ci DEFAULT NULL,
`fax` varchar(1000) COLLATE utf8_unicode_ci DEFAULT NULL,
`activity_comment` text COLLATE utf8_unicode_ci,
`level` varchar(50) COLLATE utf8_unicode_ci DEFAULT 'Basic',
`guild` varchar(500) COLLATE utf8_unicode_ci DEFAULT NULL,
`verify_comment` text COLLATE utf8_unicode_ci,
`status` int(1) NOT NULL DEFAULT '0',
`comment` text COLLATE utf8_unicode_ci,
`submitted` int(1) NOT NULL DEFAULT '0',
`assign_date` datetime DEFAULT NULL,
`created_date` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP,
`updated_date` datetime DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
PRIMARY KEY (`id`),
KEY `id` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=57357 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
And other tables just contain a simple id and title records.
Here is the result of EXPLAIN query:
These results are on my local database.
LIKE with a leading wild card is inefficient -- and will scan the entire table, hence the "slower as it gets bigger". Switch to FULLTEXT.
Pagination via OFFSET is inefficient because it must scan all the rows befoer the few desired. See http://mysql.rjweb.org/doc.php/pagination
Both of these have been discussed on stackoverflow repeatedly. Search for other discussions.
I'm trying to add job info to my DB from an API.
One of my feeds supplies a date(posted) and an expiration date.
I'm trying to insert both fields (along with others).
The date gets entered fine, however the expiry date gets set to the default 0000-00-00 00:00:00
example data from api:
{
"jobId":30081256,
"expirationDate":"22/08/2016",
"date":"11/07/2016"
}
I parse the dates ready to be entered like this:
$job_date = $job->date;
$timestamp = strtotime(str_replace('/', '-', $job_date));
$date = date("Y-m-d H:i:s", $timestamp);
//expiry
$job_expiry = $job->expirationDate;
$expiry_timestamp = strtotime(str_replace('/', '-', $job_expiry));
$expiry_date = date("Y-m-d H:i:s", $expiry_timestamp);
when I echo the results of $date and $expiry_date, i get:
2016-07-11 00:00:00
and
2016-08-22 00:00:00
to me all looks good and the dates are formatted as sql would expect, but the expiry date does not get set.
I'm not sure if it has something to do with my query (shortened to show the fields in question - all other fields insert correctly):
"INSERT INTO
jobs2 (jobref, date, expiry)
VALUES ('"
.$jobref
."','"
.$date
."','"
.$expiry_date
."') ON DUPLICATE KEY UPDATE
`date` = VALUES(date),
`expiry` = VALUES(expiry)"
Maybe the ON DUPLICATE part? The job may already be in my database, but sometimes it has been refreshed/ updated and the posted date or expiry may be different so I want to keep them updated with the latest values from the API.
whats going on here?
table structure as requested:
CREATE TABLE `jobs2` (
`date` datetime NOT NULL,
`title` varchar(200) COLLATE latin1_general_ci NOT NULL,
`company` varchar(100) COLLATE latin1_general_ci NOT NULL,
`email` varchar(100) COLLATE latin1_general_ci NOT NULL,
`url` varchar(1200) COLLATE latin1_general_ci NOT NULL,
`salarymin` int(20) DEFAULT NULL,
`salarymax` int(20) DEFAULT NULL,
`benefits` varchar(100) COLLATE latin1_general_ci NOT NULL,
`salary` varchar(200) COLLATE latin1_general_ci DEFAULT NULL,
`jobtype` varchar(30) COLLATE latin1_general_ci DEFAULT NULL,
`full_part` varchar(30) COLLATE latin1_general_ci DEFAULT NULL,
`salary_per` varchar(20) COLLATE latin1_general_ci DEFAULT NULL,
`location` varchar(100) COLLATE latin1_general_ci DEFAULT NULL,
`country` varchar(30) COLLATE latin1_general_ci NOT NULL,
`description` varchar(3000) COLLATE latin1_general_ci NOT NULL,
`category` varchar(100) COLLATE latin1_general_ci DEFAULT NULL,
`image` varchar(150) COLLATE latin1_general_ci NOT NULL,
`latitude` float NOT NULL,
`longitude` float NOT NULL,
`town` varchar(50) COLLATE latin1_general_ci NOT NULL,
`county` varchar(50) COLLATE latin1_general_ci NOT NULL,
`location_id` varchar(255) COLLATE latin1_general_ci NOT NULL,
`jobref` varchar(100) COLLATE latin1_general_ci NOT NULL,
`partner` varchar(30) COLLATE latin1_general_ci DEFAULT NULL,
`company_id` int(20) DEFAULT NULL,
`featured` varchar(10) COLLATE latin1_general_ci DEFAULT NULL,
`solr_stored` int(1) DEFAULT NULL,
`expiry` datetime NOT NULL,
UNIQUE KEY `location_id` (`location_id`),
KEY `partner` (`partner`),
KEY `company` (`company`),
KEY `title` (`title`),
KEY `company` (`company`),
KEY `longitude` (`longitude`),
KEY `latitude` (`latitude`),
KEY `jobref` (`jobref`),
KEY `jobtype` (`jobtype`),
KEY `company_id` (`company_id`),
KEY `salarymin` (`salarymin`),
KEY `salarymax` (`salarymax`),
KEY `salary_per` (`salary_per`),
KEY `date` (`date`),
FULLTEXT KEY `title` (`title`),
FULLTEXT KEY `company` (`company`),
) ENGINE=MyISAM DEFAULT CHARSET=latin1 COLLATE=latin1_general_ci
I'm trying to update a row in MySQL, using PHP.
This is what my MySql request lookslike:
$sql2 = "UPDATE `ordre` SET `navne`='[value-1]',`adresse`='[value-2]',`postnummer`='[value-3]',`city`='[value-4]',`phone`='[value-5]',`email`='[value-6]',`note`='[value-7]',`modtag_nyhedsbrev`='[value-8]',`fragt_metode`='[value-9]',`betalings_metode`='[value-10]',`samlet_pris`='[value-11]',`ordre_modtaget`='$_POST[ordre_recieved]',`betalt`='$_POST[payment_recieved]',`afsendt`='$_POST[ordre_afsendt]',`ordrenr`='[value-15]',`ordredato`='[value-16]',`cartInfoNameID`='[value-17]',`cartInfoProductVal`='[value-18]',`cartInfoSubTotal`='[value-19]',`paypalGebyr`='[value-20]',`http_user_agent`='[value-21]',`ip_logger`='[value-22]' WHERE ordrenr = $_id";
But every single time I tries to update, it gives me this error:
Duplicate entry '0' for key 'PRIMARY'.
This is what my PHPMyAdmin setup lookslike:
CREATE TABLE `ordre` (
`navne` varchar(100) COLLATE utf8_danish_ci NOT NULL,
`adresse` text COLLATE utf8_danish_ci NOT NULL,
`postnummer` int(11) NOT NULL,
`city` text COLLATE utf8_danish_ci NOT NULL,
`phone` int(12) NOT NULL,
`email` varchar(150) COLLATE utf8_danish_ci NOT NULL,
`note` varchar(502) COLLATE utf8_danish_ci NOT NULL,
`modtag_nyhedsbrev` varchar(100) COLLATE utf8_danish_ci NOT NULL,
`fragt_metode` varchar(100) COLLATE utf8_danish_ci NOT NULL,
`betalings_metode` varchar(100) COLLATE utf8_danish_ci NOT NULL,
`samlet_pris` varchar(100) COLLATE utf8_danish_ci NOT NULL,
`ordre_modtaget` int(11) NOT NULL,
`betalt` int(11) NOT NULL,
`afsendt` int(11) NOT NULL,
`ordrenr` int(11) NOT NULL AUTO_INCREMENT,
`ordredato` varchar(10) COLLATE utf8_danish_ci NOT NULL,
`cartInfoNameID` longtext COLLATE utf8_danish_ci NOT NULL,
`cartInfoProductVal` longtext COLLATE utf8_danish_ci NOT NULL,
`cartInfoSubTotal` longtext COLLATE utf8_danish_ci NOT NULL,
`paypalGebyr` text COLLATE utf8_danish_ci NOT NULL,
`http_user_agent` text COLLATE utf8_danish_ci NOT NULL,
`ip_logger` text COLLATE utf8_danish_ci NOT NULL,
PRIMARY KEY (`ordrenr`)
) ENGINE=MyISAM AUTO_INCREMENT=519 DEFAULT CHARSET=utf8 COLLATE=utf8_danish_ci
http://i.stack.imgur.com/scsLj.jpg
ordrenr is your primary key and also AUTO_INCREMENT. This should give you trouble. Remove this column update from the query:
`ordrenr` = '[]',
While inserting first time it gives following error: "Result consisted of more than one row". When i try to insert record second time it gives error with message duplicate entry.
SQL=INSERT INTO `master_user` (`name`,`user_name`,`email`,`password`,`system_name_of_friend`,`system_no_of_friend`,`registered_from_site`,`registered_on`,`is_existing_user`) VALUES ('FirstName LastName','username','demo#mail.com','8c71eede42e38709e9e836021b0b9b9b','','','site','','1')
any one help will be appropriated and following is the table structure which will be get help to tracking this issue very easily and get the solution for this.
CREATE TABLE IF NOT EXISTS `master_user` (
`master_user_id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
`name` varchar(100) COLLATE utf8_unicode_ci NOT NULL,
`user_name` varchar(100) COLLATE utf8_unicode_ci DEFAULT NULL,
`email` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
`password` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
`system_name_of_friend` varchar(100) COLLATE utf8_unicode_ci NOT NULL,
`system_no_of_friend` varchar(50) COLLATE utf8_unicode_ci NOT NULL,
`registered_from_ip` varchar(25) COLLATE utf8_unicode_ci DEFAULT NULL,
`registered_from_site` varchar(25) COLLATE utf8_unicode_ci DEFAULT NULL,
`registered_on` datetime DEFAULT NULL,
`is_existing_user` bit(1) NOT NULL DEFAULT b'0',
PRIMARY KEY (`master_user_id`),
UNIQUE KEY `ukMasterUser_email` (`email`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci AUTO_INCREMENT=1293 ;
I have changed the few words from column Which are conflicting and resolved.
this code tested with both server and local phpmyadmin
on localhost it returned results as expected
on server it returned a good result but when passing the dr_name it returns empty result
here is the code:
SELECT * FROM doctor WHERE dr_name LIKE '%غا%' AND specialization=12 AND state=6
the only deference between server and local is that the server mysql engine is myisam and innodb on localhost
this is the create table statement:
CREATE TABLE IF NOT EXISTS `doctor` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`dr_name` varchar(65) NOT NULL,
`dr_name_e` varchar(65) CHARACTER SET utf8 COLLATE utf8_unicode_ci NOT NULL,
`unique_name` varchar(75) CHARACTER SET utf8 COLLATE utf8_unicode_ci NOT NULL,
`pass` char(40) CHARACTER SET utf8 COLLATE utf8_unicode_ci NOT NULL,
`nickname` varchar(100) CHARACTER SET utf8 COLLATE utf8_unicode_ci DEFAULT NULL,
`specialization` int(11) NOT NULL,
`state` tinyint(4) NOT NULL,
`scientific_degree` int(11) DEFAULT NULL,
`university` int(11) DEFAULT NULL,
`grad_year` smallint(6) DEFAULT NULL,
`show_degree` tinyint(1) DEFAULT NULL,
`show_univ` tinyint(1) DEFAULT NULL,
`show_grad_year` tinyint(1) DEFAULT NULL,
`appendices` varchar(1000) CHARACTER SET utf8 COLLATE utf8_unicode_ci DEFAULT NULL,
`picture` varchar(255) CHARACTER SET utf8 COLLATE utf8_unicode_ci DEFAULT NULL,
`active` tinyint(1) NOT NULL DEFAULT '0',
`views` bigint(20) DEFAULT '0',
`search_result` bigint(20) DEFAULT '0',
`gender_male` tinyint(1) DEFAULT NULL,
`agent` int(11) DEFAULT NULL,
`next_step` tinyint(4) DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `unique_name` (`unique_name`),
KEY `university` (`university`),
KEY `scientific_degree` (`scientific_degree`),
KEY `specialization` (`specialization`),
KEY `agent` (`agent`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 AUTO_INCREMENT=31 ;