Related
I created a site that gives you info about universities, every university has registration and exams, etc
so in database I have two columns
THE FIRST TABLE IS THE UNIVERSITIES:
CREATE TABLE `universities` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`title` varchar(255) NOT NULL,
`slug` varchar(255) NOT NULL,
`views` int(11) NOT NULL DEFAULT 0,
`image` varchar(255) NOT NULL,
`banner` varchar(255) NOT NULL,
`aboutUni` text NOT NULL,
`aboutCity` text NOT NULL,
`localRank` int(5) DEFAULT NULL,
`globalRank` int(3) DEFAULT NULL,
`foundedY` int(4) NOT NULL,
`lang` varchar(10) DEFAULT NULL,
`published` tinyint(1) NOT NULL DEFAULT 1,
`created_at` timestamp NOT NULL DEFAULT current_timestamp() ON UPDATE current_timestamp(),
`user_id` int(11) DEFAULT NULL,
`video` varchar(255) DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `slug` (`slug`),
UNIQUE KEY `title` (`title`),
UNIQUE KEY `slug_2` (`slug`),
KEY `universities_ibfk_1` (`user_id`),
FULLTEXT KEY `title_2` (`title`),
CONSTRAINT `universities_ibfk_1` FOREIGN KEY (`user_id`) REFERENCES `users` (`id`) ON DELETE NO ACTION ON UPDATE NO ACTION
) ENGINE=InnoDB AUTO_INCREMENT=40 DEFAULT CHARSET=utf8
THE SECOND TABLE IS THE EXAMS
CREATE TABLE `bsc` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`university_id` int(11) DEFAULT NULL,
`dates` text NOT NULL,
`date_end` date DEFAULT NULL,
`date` date DEFAULT NULL,
`papers` text NOT NULL,
`info` text NOT NULL,
`method` text NOT NULL,
`link` varchar(255) DEFAULT NULL,
`register_link` varchar(255) NOT NULL,
`links` text NOT NULL,
`sort` int(2) NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `university_id` (`university_id`),
UNIQUE KEY `university_id_2` (`university_id`),
CONSTRAINT `bsc_ibfk_1` FOREIGN KEY (`university_id`) REFERENCES `universities` (`id`) ON DELETE CASCADE ON UPDATE NO ACTION
) ENGINE=InnoDB AUTO_INCREMENT=27 DEFAULT CHARSET=utf8mb4
NOTICE!: the tables have other columns but I think it's not important
university_id has a foreign key relation with universities.id
the problem is when I want to add tow exam to a university it appears me this error:
Duplicate entry '39' for key 'university_id'
The problem you faced Duplicate entry '39' for key 'university_id' results from trying to store in university_id a value that already existed somewhere, which is not allowed according to some constraints.
as shown in the DDL part
UNIQUE KEY `university_id` (`university_id`)
This constrain is the main cause for that error, so you need to remove that constrain by
ALTER TABLE `bsc` DROP INDEX `university_id`;
Thanks for #Tangentially, who was the first one who expected that error in his comment.
It seems that these keys are unnecesary:
UNIQUE KEY university_id (university_id),
UNIQUE KEY university_id_2 (university_id),
given that keys, you should only have distinct values in that fields.
I turned on slow query monitor in MySQL config.
Below is the query and time:
Time: 160330 20:54:11
User#Host: user[user] # [xx.xx.xxx.xxx]
Query_time: 8.794170 Lock_time: 0.000141 Rows_sent: 3942 Rows_examined: 4742825
SET timestamp=1459371251;
SELECT (SELECT (CASE WHEN ce_type = 'IN' then SUM(payment_amount)
END) as debit
FROM customer_payment_options cpo
WHERE wallet_id=cw.id
AND (cpo.real_account_type='HQ')
AND cpo.source_country_id='40'
GROUP BY cpo.wallet_id)
as debit,
(SELECT SUM(payment_amount)
as credit
FROM customer_payment_options cpo
WHERE wallet_id=cw.id
AND (cpo.real_account_type='HQ')
AND cpo.tran_id IS NOT NULL
AND cpo.source_country_id='40'
GROUP BY cpo.wallet_id)
as credit
FROM customer_wallet cw
WHERE cw.company_id='1'
AND cw.currency='40'
AND cw.is_approved = '1'
AND DATE(cw.date_added) < '2016-03-30';
Indexes on customer_payment_options:
company_id
tran_id
ce_id
wallet_id
What should I do to improve it's performance?
EXPLAIN:
http://i.stack.imgur.com/iH8rt.png
SCHEMA
CREATE TABLE `customer_payment_options` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`company_id` int(11) NOT NULL,
`local_branch_id` int(11) NOT NULL,
`tran_id` bigint(11) DEFAULT NULL,
`ce_id` int(11) DEFAULT NULL,
`wallet_id` int(11) DEFAULT NULL,
`reward_credit_id` int(11) DEFAULT NULL,
`ce_invoice_id` varchar(32) DEFAULT NULL,
`ce_type` enum('IN','OUT') DEFAULT NULL,
`payment_type` enum('CASH','DEBIT','CREDIT','CHEQUE','DRAFT','BANK_DEPOSIT','EWIRE','WALLET','LOAN','REWARD_CREDIT') NOT NULL,
`payment_amount` varchar(20) NOT NULL,
`payment_type_number` varchar(100) DEFAULT NULL,
`source_country_id` int(11) NOT NULL,
`real_account_id` int(11) DEFAULT NULL,
`real_account_type` enum('LOCAL','HQ') DEFAULT NULL,
`date_added` datetime NOT NULL,
`event_type` enum('MONEY_TRANSFER','CURRENCY_EXCHANGE','WALLET') DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `company_id` (`company_id`),
KEY `real_account_type` (`real_account_type`),
KEY `tran_id` (`tran_id`),
KEY `ce_id` (`ce_id`),
KEY `wallet_id` (`wallet_id`),
CONSTRAINT `customer_payment_options_ibfk_4` FOREIGN KEY (`wallet_id`) REFERENCES `customer_wallet` (`id`),
CONSTRAINT `customer_payment_options_ibfk_1` FOREIGN KEY (`company_id`) REFERENCES `company` (`id`),
CONSTRAINT `customer_payment_options_ibfk_2` FOREIGN KEY (`tran_id`) REFERENCES `transaction` (`id`),
CONSTRAINT `customer_payment_options_ibfk_3` FOREIGN KEY (`ce_id`) REFERENCES `currency_exchange` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=412 DEFAULT CHARSET=utf8
CREATE TABLE `customer_wallet` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`wallet_unique_id` varchar(100) DEFAULT NULL,
`company_id` int(11) NOT NULL,
`branch_admin_id` int(11) DEFAULT NULL,
`emp_id` int(11) DEFAULT NULL,
`emp_type` enum('SUPER_ADMIN','ADMIN','AGENT_ADMIN','AGENT','OVER_AGENT_ADMIN','OVER_AGENT') DEFAULT NULL,
`cus_id` bigint(11) NOT NULL,
`tran_id` bigint(11) DEFAULT NULL,
`beehive_id` int(11) DEFAULT NULL,
`type` enum('DEPOSIT','WITHDRAW','TRANSACTION') NOT NULL,
`sub_type` enum('MONEY_TRANSFER','BEEHIVE_DEPOSIT') DEFAULT NULL,
`credit_in` varchar(20) DEFAULT NULL,
`credit_out` varchar(20) DEFAULT NULL,
`currency` varchar(20) NOT NULL,
`date_added` datetime NOT NULL,
`note` varchar(255) DEFAULT NULL,
`location` enum('DIRECT') DEFAULT NULL,
`is_approved` enum('0','1') NOT NULL DEFAULT '1',
`idebit_issconf` varchar(50) DEFAULT NULL,
`idebit_issname` varchar(50) DEFAULT NULL,
`idebit_isstrack2` varchar(100) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `cus_id` (`cus_id`),
KEY `company_id` (`company_id`),
KEY `branch_admin_id` (`branch_admin_id`),
KEY `emp_id` (`emp_id`),
KEY `tran_id` (`tran_id`),
KEY `beehive_id` (`beehive_id`),
CONSTRAINT `customer_wallet_ibfk_1` FOREIGN KEY (`cus_id`) REFERENCES `customers` (`id`),
CONSTRAINT `customer_wallet_ibfk_2` FOREIGN KEY (`company_id`) REFERENCES `company` (`id`),
CONSTRAINT `customer_wallet_ibfk_3` FOREIGN KEY (`tran_id`) REFERENCES `transaction` (`id`),
CONSTRAINT `customer_wallet_ibfk_4` FOREIGN KEY (`emp_id`) REFERENCES `employees` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=152 DEFAULT CHARSET=utf8
What you are doing as a correlated query on every wallet ID to get the corresponding debits and credits. It appears you are getting one record per wallet id. This is very busy. Having a join to the customer payments table on your criteria that is common (including the join per wallet id). Then, simplify the CASE as a SUM( case/when ) as respective debit / credit.
I don't know your underlying criteria of table columns, but I would even hedge to (and did) include NOT the CE_TYPE = 'IN' as that appears basis of a debit and you would not want to falsely count as part of a credit too. Again, dont know correlation of fields, trans_id, types.
Now, as stated, having individual indexes on individual fields will not help optimize this query. I would suggest the following indexes.
table index
customer_wallet ( company_id, is_approved, currency, id, date_added )
customer_payment_options ( wallet_id, account_type, country_id )
SELECT
cw.wallet_id,
SUM( case when cpo.ce_type = 'IN'
then cpo.payment_amount
ELSE 0 end ) as Debit,
SUM( case when NOT cpo.ce_type = 'IN'
AND cpo.tran_id IS NOT NULL
then cpo.payment_amount
ELSE 0 end ) as Credit
FROM
customer_wallet cw
JOIN customer_payment_options cpo
ON cw.id = cpo.wallet_id
AND cpo.real_account_type = 'HQ'
AND cpo.source_country_id = '40'
WHERE
cw.company_id = '1'
AND cw.currency = '40'
AND cw.is_approved = '1'
AND cw.date_added < '2016-03-30'
GROUP BY
cw.id
One additional comment. if your ID columns, Currency flag, country ID, approved are actually numeric values in the table structure, remove the quotes and let compare directly on the numeric value. Also, for your date_added. You had that based on DATE( date_added ). Doing a function on a column can not fully utilize the index. Since date() strips off any time portion of a date/time stamp column, and you are asking for all added less than Mar 30, then date added of March 29 # 11:59:59pm is still less than Mar 30 at 12:00:00am, so no date conversion is required.
As commented by Ivan (below), if you want ALL Wallet IDs regardless of having any payments (debit or credit), then change from a join to a LEFT JOIN.
You need to add indexes and multi-column indexes to make it fast.
Please keep in mind, that if you have large table, extra indexes will slow-down the insertions , since index file update will take more time.
If a multiple-column index exists on col1 and col2, the appropriate
rows can be fetched directly. If separate single-column indexes exist
on col1 and col2, the optimizer attempts to use the Index Merge
optimization (see Section 8.2.1.4, “Index Merge Optimization”), or
attempts to find the most restrictive index by deciding which index
excludes more rows and using that index to fetch the rows.
If the table has a multiple-column index, any leftmost prefix of the
index can be used by the optimizer to look up rows. For example, if
you have a three-column index on (col1, col2, col3), you have indexed
search capabilities on (col1), (col1, col2), and (col1, col2, col3).
Read more
Abstract:
Every client is given a specific xml ad feed (publisher_feed table). Everytime there is a query or a click on that feed, it gets recorded (publisher_stats_raw table) (Each query/click will have multiple rows depending on the subid passed by the client (We can sum the clicks together)). The next day, we pull stats from an API to grab the previous days revenue numbers (rev_stats table) (Each revenue stat might have multiple rows depending on the country of the click (We can sum the revenue together)). Been having a hard time trying to link together these three tables to find the average RPC for each client for the previous day.
Table Structure:
CREATE TABLE `publisher_feed` (
`publisher_feed_id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`alias` varchar(45) DEFAULT NULL,
`user_id` int(10) unsigned DEFAULT NULL,
`remote_feed_id` int(10) unsigned DEFAULT NULL,
`subid` varchar(255) DEFAULT '',
`requirement` enum('tq','tier2','ron','cpv','tos1','tos2','tos3','pv1','pv2','pv3','ar','ht') DEFAULT NULL,
`status` enum('enabled','disabled') DEFAULT 'enabled',
`tq` decimal(4,2) DEFAULT '0.00',
`clicklimit` int(11) DEFAULT '0',
`prev_rpc` decimal(20,10) DEFAULT '0.0000000000',
PRIMARY KEY (`publisher_feed_id`),
UNIQUE KEY `alias_UNIQUE` (`alias`),
KEY `publisher_feed_idx` (`remote_feed_id`),
KEY `publisher_feed_user` (`user_id`),
CONSTRAINT `publisher_feed_feed` FOREIGN KEY (`remote_feed_id`) REFERENCES `remote_feed` (`remote_feed_id`) ON DELETE NO ACTION ON UPDATE NO ACTION,
CONSTRAINT `publisher_feed_user` FOREIGN KEY (`user_id`) REFERENCES `user` (`user_id`) ON DELETE NO ACTION ON UPDATE NO ACTION
) ENGINE=InnoDB AUTO_INCREMENT=124 DEFAULT CHARSET=latin1$$
CREATE TABLE `publisher_stats_raw` (
`publisher_stats_id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`unique_data` varchar(350) NOT NULL,
`publisher_feed_id` int(10) unsigned DEFAULT NULL,
`date` date DEFAULT NULL,
`subid` varchar(255) DEFAULT NULL,
`queries` int(10) unsigned DEFAULT '0',
`impressions` int(10) unsigned DEFAULT '0',
`clicks` int(10) unsigned DEFAULT '0',
`filtered` int(10) unsigned DEFAULT '0',
`revenue` decimal(20,10) unsigned DEFAULT '0.0000000000',
PRIMARY KEY (`publisher_stats_id`),
UNIQUE KEY `unique_data_UNIQUE` (`unique_data`),
KEY `publisher_stats_raw_remote_feed_idx` (`publisher_feed_id`)
) ENGINE=InnoDB AUTO_INCREMENT=472 DEFAULT CHARSET=latin1$$
CREATE TABLE `rev_stats` (
`rev_stats_id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`date` date DEFAULT NULL,
`remote_feed_id` int(10) unsigned DEFAULT NULL,
`typetag` varchar(255) DEFAULT NULL,
`subid` varchar(255) DEFAULT NULL,
`country` varchar(2) DEFAULT NULL,
`revenue` decimal(20,10) DEFAULT NULL,
`tq` decimal(4,2) DEFAULT NULL,
`finalized` int(11) DEFAULT '0',
PRIMARY KEY (`rev_stats_id`),
KEY `rev_stats_remote_feed_idx` (`remote_feed_id`),
CONSTRAINT `rev_stats_remote_feed` FOREIGN KEY (`remote_feed_id`) REFERENCES `remote_feed` (`remote_feed_id`) ON DELETE NO ACTION ON UPDATE NO ACTION
) ENGINE=InnoDB AUTO_INCREMENT=58 DEFAULT CHARSET=latin1$$
Context:
Each remote_feed has a specific subid/typetag given to it. So we need to match up the both the remote_feed_id and the subid columsn from the publisher_feed table to the remote_feed_id and typetag columns in the revenue stats table.
My current, non working, implementation:
SELECT
pf.publisher_feed_id, psr.date, sum(clicks), sum(rs.revenue)
FROM
xml_network.publisher_feed pf
JOIN
xml_network.publisher_stats_raw psr
ON
psr.publisher_feed_id = pf.publisher_feed_id
JOIN
xml_network.rev_stats rs
ON
rs.remote_feed_id = pf.remote_feed_id
WHERE
pf.requirement = 'tq'
AND
pf.subid = rs.typetag
AND
psr.date <> date(curdate())
GROUP BY
psr.date
ORDER BY
psr.date DESC
LIMIT 1;
The above keeps pulling the wrong data out of the rev_stats table (pulls the sum of the correct stats, but repeats it over because of a join). Any help with how I would be able to properly pull the correct data would be greatly helpful ( I could use multiple queries and PHP to get the correct results, but what's the fun in that!)
Figured out a way to get this accomplished. Its def not a fast method by any means, needing 4 selects to get it done, but it works flawlessly =)
SELECT
pf.publisher_feed_id,
round(
(
SELECT
SUM(rs.revenue)
FROM
xml_network.rev_stats rs
WHERE
rs.remote_feed_id = pf.remote_feed_id
AND
rs.typetag = pf.subid
AND
rs.date = subdate(current_date, 1)
),10)as revenue,
(
SELECT
MAX(rs.tq)
FROM
xml_network.rev_stats rs
WHERE
rs.remote_feed_id = pf.remote_feed_id
AND
rs.typetag = pf.subid
AND
rs.date = subdate(current_date, 1)
) as tq,
(
SELECT
SUM(psr.clicks)-SUM(psr.filtered)
FROM
xml_network.publisher_stats_raw psr
WHERE
psr.publisher_feed_id = pf.publisher_feed_id
AND
psr.date = subdate(current_date, 1)
) as clicks
FROM
xml_network.publisher_feed pf
WHERE
pf.requirement = 'tq';
I have the following mysql tables:
CREATE TABLE `video` (
`video_id` int(11) unsigned NOT NULL auto_increment,
`title` varchar(255) NOT NULL default '',
`description` text NOT NULL,
PRIMARY KEY (`video_id`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8;
CREATE TABLE `video_categories` (
`cat_id` int(11) unsigned NOT NULL auto_increment,
`name` varchar(255) NOT NULL default '',
PRIMARY KEY (`cat_id`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8;
CREATE TABLE `video_category` (
`video_id` int(11) unsigned NOT NULL default '0',
`cat_id` int(11) unsigned NOT NULL default '0',
KEY `video_id` (`video_id`),
KEY `cat_id` (`cat_id`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8;
CREATE TABLE `video_tags` (
`tag_id` int(11) unsigned NOT NULL auto_increment,
`video_id` int(11) unsigned NOT NULL default '0',
`name` varchar(255) NOT NULL default '',
KEY `video_id` (`video_id`),
KEY `name` (`name`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8;
I created a sphinx configuration file and i can search from PHP. The problem is when i want to search for related videos, a related video must be in the same category as the video i'm searching for. I can do this with MVA and and SetFilter('categories', array(3)) for example, however the total number of matches results is the global one (i need total to display pagination via ajax) not the one in the category.
Any ideas how i can search through videos (documents in sphinx) that are only in a specified category?
Thanks,
Adrian.
You can define for the category ID an integer attribute in Sphinx:
sql_attr_uint = cat_id
And then just add #cat_id=12345 to your query.
i am building a real estate application where in it will store the properties and search it. the property will have different categories like (residential, commercial, industrial or agricultural). based upon the category i want to serailize each and every property listing . for example the property with id 1 belongs to resedential will have the serial code rs_SOMERANDOMUNIQUENUMBER. and for commercial it can be cm_SOMERANDOMUNIQUENUMBER and so on. for this my database table looks like this.
CREATE TABLE IF NOT EXISTS `propSerials` (
`id` bigint(20) NOT NULL auto_increment,
`serial` varchar(50) NOT NULL,
`property_id` int(10) UNIQUE NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
what would be the best possible format to store the serial with the prefix according to category?
thank you
Why dont you add another column that holds category_id and in category table add column with prefixes for that category.
CREATE TABLE IF NOT EXISTS `propSerials` (
`id` bigint(20) NOT NULL auto_increment,
`serial` varchar(50) NOT NULL,
`property_id` int(10) UNIQUE NOT NULL,
`category_id` int(10) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
CREATE TABLE IF NOT EXISTS `propCategories` (
`id` bigint(20) NOT NULL auto_increment,
`category` varchar(50) NOT NULL,
`property_prefix` char(3) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
In query you can:
SELECT CONCAT('prefix_', 'serial');