MySql Properly Join Complex Data/Tables - php

Abstract:
Every client is given a specific xml ad feed (publisher_feed table). Everytime there is a query or a click on that feed, it gets recorded (publisher_stats_raw table) (Each query/click will have multiple rows depending on the subid passed by the client (We can sum the clicks together)). The next day, we pull stats from an API to grab the previous days revenue numbers (rev_stats table) (Each revenue stat might have multiple rows depending on the country of the click (We can sum the revenue together)). Been having a hard time trying to link together these three tables to find the average RPC for each client for the previous day.
Table Structure:
CREATE TABLE `publisher_feed` (
`publisher_feed_id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`alias` varchar(45) DEFAULT NULL,
`user_id` int(10) unsigned DEFAULT NULL,
`remote_feed_id` int(10) unsigned DEFAULT NULL,
`subid` varchar(255) DEFAULT '',
`requirement` enum('tq','tier2','ron','cpv','tos1','tos2','tos3','pv1','pv2','pv3','ar','ht') DEFAULT NULL,
`status` enum('enabled','disabled') DEFAULT 'enabled',
`tq` decimal(4,2) DEFAULT '0.00',
`clicklimit` int(11) DEFAULT '0',
`prev_rpc` decimal(20,10) DEFAULT '0.0000000000',
PRIMARY KEY (`publisher_feed_id`),
UNIQUE KEY `alias_UNIQUE` (`alias`),
KEY `publisher_feed_idx` (`remote_feed_id`),
KEY `publisher_feed_user` (`user_id`),
CONSTRAINT `publisher_feed_feed` FOREIGN KEY (`remote_feed_id`) REFERENCES `remote_feed` (`remote_feed_id`) ON DELETE NO ACTION ON UPDATE NO ACTION,
CONSTRAINT `publisher_feed_user` FOREIGN KEY (`user_id`) REFERENCES `user` (`user_id`) ON DELETE NO ACTION ON UPDATE NO ACTION
) ENGINE=InnoDB AUTO_INCREMENT=124 DEFAULT CHARSET=latin1$$
CREATE TABLE `publisher_stats_raw` (
`publisher_stats_id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`unique_data` varchar(350) NOT NULL,
`publisher_feed_id` int(10) unsigned DEFAULT NULL,
`date` date DEFAULT NULL,
`subid` varchar(255) DEFAULT NULL,
`queries` int(10) unsigned DEFAULT '0',
`impressions` int(10) unsigned DEFAULT '0',
`clicks` int(10) unsigned DEFAULT '0',
`filtered` int(10) unsigned DEFAULT '0',
`revenue` decimal(20,10) unsigned DEFAULT '0.0000000000',
PRIMARY KEY (`publisher_stats_id`),
UNIQUE KEY `unique_data_UNIQUE` (`unique_data`),
KEY `publisher_stats_raw_remote_feed_idx` (`publisher_feed_id`)
) ENGINE=InnoDB AUTO_INCREMENT=472 DEFAULT CHARSET=latin1$$
CREATE TABLE `rev_stats` (
`rev_stats_id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`date` date DEFAULT NULL,
`remote_feed_id` int(10) unsigned DEFAULT NULL,
`typetag` varchar(255) DEFAULT NULL,
`subid` varchar(255) DEFAULT NULL,
`country` varchar(2) DEFAULT NULL,
`revenue` decimal(20,10) DEFAULT NULL,
`tq` decimal(4,2) DEFAULT NULL,
`finalized` int(11) DEFAULT '0',
PRIMARY KEY (`rev_stats_id`),
KEY `rev_stats_remote_feed_idx` (`remote_feed_id`),
CONSTRAINT `rev_stats_remote_feed` FOREIGN KEY (`remote_feed_id`) REFERENCES `remote_feed` (`remote_feed_id`) ON DELETE NO ACTION ON UPDATE NO ACTION
) ENGINE=InnoDB AUTO_INCREMENT=58 DEFAULT CHARSET=latin1$$
Context:
Each remote_feed has a specific subid/typetag given to it. So we need to match up the both the remote_feed_id and the subid columsn from the publisher_feed table to the remote_feed_id and typetag columns in the revenue stats table.
My current, non working, implementation:
SELECT
pf.publisher_feed_id, psr.date, sum(clicks), sum(rs.revenue)
FROM
xml_network.publisher_feed pf
JOIN
xml_network.publisher_stats_raw psr
ON
psr.publisher_feed_id = pf.publisher_feed_id
JOIN
xml_network.rev_stats rs
ON
rs.remote_feed_id = pf.remote_feed_id
WHERE
pf.requirement = 'tq'
AND
pf.subid = rs.typetag
AND
psr.date <> date(curdate())
GROUP BY
psr.date
ORDER BY
psr.date DESC
LIMIT 1;
The above keeps pulling the wrong data out of the rev_stats table (pulls the sum of the correct stats, but repeats it over because of a join). Any help with how I would be able to properly pull the correct data would be greatly helpful ( I could use multiple queries and PHP to get the correct results, but what's the fun in that!)

Figured out a way to get this accomplished. Its def not a fast method by any means, needing 4 selects to get it done, but it works flawlessly =)
SELECT
pf.publisher_feed_id,
round(
(
SELECT
SUM(rs.revenue)
FROM
xml_network.rev_stats rs
WHERE
rs.remote_feed_id = pf.remote_feed_id
AND
rs.typetag = pf.subid
AND
rs.date = subdate(current_date, 1)
),10)as revenue,
(
SELECT
MAX(rs.tq)
FROM
xml_network.rev_stats rs
WHERE
rs.remote_feed_id = pf.remote_feed_id
AND
rs.typetag = pf.subid
AND
rs.date = subdate(current_date, 1)
) as tq,
(
SELECT
SUM(psr.clicks)-SUM(psr.filtered)
FROM
xml_network.publisher_stats_raw psr
WHERE
psr.publisher_feed_id = pf.publisher_feed_id
AND
psr.date = subdate(current_date, 1)
) as clicks
FROM
xml_network.publisher_feed pf
WHERE
pf.requirement = 'tq';

Related

Database Schema for Recipe/Ingredient/Measurement/Amount

I'm creating a recipe app to help my wife with her cake hobby. The idea is to create a recipe database to hold all of her cake recipes.
Each recipe would have multiple ingredients. Each ingredient would have a measurement (gm, ml, teaspoons etc) and then a quantity.
I understand how to create the 'recipes' and 'ingredients' tables and how to link the 2 with a junction table 'recipe_ingredients, however I am struggling with how to then implement the measurement and amount fields.
Can anyone with a bit more database experience suggest a database scheme or have any tips to handle this?
You have some options for this, and like most things, you can go the easy route (Nick Coons posted a good example as I'm typing this) or progressively more involved routes. Here are some questions to ask yourself about how you see this working:
Do you want uniformity in your measurements? (Do you want to always show "tsp" for teaspoon, or can it be freeform as in Nick's example)
How often will you need to add units? Will you need to add the dram or hogshead, or whatever as units as time goes on, or will you probably just stick to the basics?
A good middle ground would be something like
CREATE TABLE `recipe` (
`recipe_id` INT(10) UNSIGNED NOT NULL AUTO_INCREMENT,
`name` VARCHAR(128) DEFAULT NULL,
`description` TEXT,
`instructions` TEXT,
PRIMARY KEY (`recipe_id`)
)
CREATE TABLE `ingredient` (
`ingredient_id` INT(10) UNSIGNED NOT NULL AUTO_INCREMENT,
`recipe_id` INT(10) UNSIGNED NOT NULL,
`ingredient` VARCHAR(64) DEFAULT NULL,
`amount` DECIMAL(4, 2) DEFAULT NULL,
`unit` ENUM ('tsp', 'tbsp', 'oz', 'g', 'lb', 'cup', 'gallon', 'pinch') DEFAULT NULL,
PRIMARY KEY (`ingredient_id`)
)
This satisfies #1 by enforcing a set of units, which is nice. The downside is that you have to alter your table to update the units. It also may be more difficult to keep your front end up to date with the valid choices.
Next, you could add a table for units and reference it via foreign key from the ingredients table like so:
CREATE TABLE `unit` (
`unit_id` INT(10) UNSIGNED NOT NULL AUTO_INCREMENT,
`label` VARCHAR(64) DEFAULT NULL,
`sort` INT(10) UNSIGNED NOT NULL DEFAULT 0,
PRIMARY KEY (`unit_id`),
UNIQUE KEY `unit_label_uk` (`label`)
)
CREATE TABLE `ingredient` (
`ingredient_id` INT(10) UNSIGNED NOT NULL AUTO_INCREMENT,
`unit_id` INT(10) UNSIGNED NOT NULL,
`recipe_id` INT(10) UNSIGNED NOT NULL,
`ingredient` VARCHAR(64) DEFAULT NULL,
`amount` DECIMAL(4, 2) DEFAULT NULL,
`sort` INT(10) UNSIGNED NOT NULL DEFAULT 0,
PRIMARY KEY (`ingredient_id`)
)
This satisfies #1 and #2, allowing you to easily manage your units and access the list for use in your front end, so you don't have to alter your front end when you change units.
From there you could spin off into space coming up with ways to handle unit conversion, etc. but that is probably overkill for what you're trying to do.
EDIT:
Per your comment, I would set it up like this:
CREATE TABLE `recipe` (
`recipe_id` INT(10) UNSIGNED NOT NULL AUTO_INCREMENT,
`name` VARCHAR(128) NOT NULL,
`description` TEXT,
`instructions` TEXT,
PRIMARY KEY (`recipe_id`)
)
CREATE TABLE `ingredient` (
`ingredient_id` INT(10) UNSIGNED NOT NULL AUTO_INCREMENT,
`label` VARCHAR(64) NOT NULL,
`sort` INT(10) UNSIGNED NOT NULL DEFAULT 0,
PRIMARY KEY (`ingredient_id`)
UNIQUE KEY `ingredient_label_uk` (`label`)
)
CREATE TABLE `unit` (
`unit_id` INT(10) UNSIGNED NOT NULL AUTO_INCREMENT,
`label` VARCHAR(64) DEFAULT NULL,
`sort` INT(10) UNSIGNED NOT NULL DEFAULT 0,
PRIMARY KEY (`unit_id`),
UNIQUE KEY `unit_label_uk` (`label`)
)
CREATE TABLE `recipe_ingredient` (
`recipe_ingredient_id` INT(10) UNSIGNED NOT NULL AUTO_INCREMENT,
`recipe_id` INT(10) UNSIGNED NOT NULL,
`ingredient_id` INT(10) UNSIGNED NOT NULL,
`unit_id` INT(10) UNSIGNED NOT NULL,
`amount` DECIMAL(4, 2) DEFAULT NULL,
`sort` INT(10) UNSIGNED NOT NULL DEFAULT 0,
PRIMARY KEY (`recipe_ingredient_id`)
)
Your recipe_ingredient table is doing the bulk of the work here, tying everything together.
Something like this should work:
CREATE TABLE `recipe` (
`recipe_id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`name` varchar(128) DEFAULT NULL,
`description` text,
`instructions` text,
PRIMARY KEY (`recipe_id`)
)
CREATE TABLE `ingredient` (
`ingredient_id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`ingredient` varchar(64) DEFAULT NULL,
PRIMARY KEY (`ingredient_id`)
)
CREATE TABLE `xref_recipe_ingredient` (
`recipe_id` int(10) unsigned NOT NULL,
`ingredient_id` int(10) unsigned NOT NULL,
`amount` decimal(4,2) DEFAULT NULL,
`unit` varchar(8) DEFAULT NULL,
INDEX (`recipe_id`)
)
This contains a recipe table (one entry per recipe) to hold the name of the recipe, a description, and a set of instructions; an ingredient table to hold a list of possible ingredients including the name of the ingredient; and a cross-reference table to link recipes to ingredients, including the amount and unit of the ingredient for the recipe.

Mysql LEFT Join on big data table

I have a table called "tables_data_info" where is stored some of the data relative to different tables. Data like "created time", "editing time", "editing user", "create by user id" etc. I'm using this table because there was a dynamic php script that generate it automatically.
But, when i have a huge number of record ( 15k in this case ) the query getting very very very slow, and take "minutes" to do his job! But i'm not selecting all 15k records, i'm limiting to select 10 records at all!
A simple query:
SELECT pd.id, pd.title, pd.sell_price, pd.available_qt, tdi.createtime, tdi.lastupdatetime, tdi.create_member_id, tdi.create_group_id, tdi.last_update_member_id, tdi.last_update_group_id FROM zd_products AS pd LEFT JOIN zd_tables_data_info AS tdi ON ( tdi.targetid = pd.id and tdi.table_name = 'products' ) ORDER by pd.title ASC LIMIT 0, 10
How can i run this query differently but more efficiently ?
Here the table structure:
zd_products
CREATE TABLE `zd_products` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`title` varchar(256) NOT NULL DEFAULT '',
`internalcode` varchar(256) DEFAULT NULL,
`ean13_jan_code` varchar(256) DEFAULT NULL,
`upc_code` varchar(11) DEFAULT NULL,
`active` tinyint(1) NOT NULL DEFAULT '0',
`status` varchar(12) NOT NULL DEFAULT 'new',
`product_tags` longtext,
`buy_price` double NOT NULL DEFAULT '0',
`sell_price` double NOT NULL DEFAULT '0',
`fiscal_tax_id` int(11) NOT NULL DEFAULT '0',
`box_width` double NOT NULL DEFAULT '0',
`box_height` double NOT NULL DEFAULT '0',
`box_depth` double NOT NULL DEFAULT '0',
`box_weight` double NOT NULL DEFAULT '0',
`shipment_extra_price` double NOT NULL DEFAULT '0',
`available_qt` int(11) NOT NULL DEFAULT '0',
`allow_purchase_out_stock` tinyint(1) NOT NULL DEFAULT '0',
`meta_title` varchar(256) DEFAULT NULL,
`meta_description` varchar(256) DEFAULT NULL,
`meta_keywords` longtext,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=15730 DEFAULT CHARSET=utf8;
zd_tables_data_info
CREATE TABLE `zd_tables_data_info` (
`table_name` varchar(256) NOT NULL DEFAULT '',
`targetid` int(11) DEFAULT NULL,
`create_member_id` int(11) unsigned DEFAULT NULL,
`create_group_id` int(11) unsigned DEFAULT NULL,
`last_update_member_id` int(11) unsigned DEFAULT NULL,
`last_update_group_id` int(11) unsigned DEFAULT NULL,
`createtime` int(11) unsigned DEFAULT NULL,
`lastupdatetime` int(11) unsigned DEFAULT NULL,
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
PRIMARY KEY (`id`),
KEY `INDEX` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=19692 DEFAULT CHARSET=utf8;
Your data is not particularly big. Here is the query:
SELECT pd.id, pd.title, pd.sell_price, pd.available_qt,
tdi.createtime, tdi.lastupdatetime, tdi.create_member_id, tdi.create_group_id,
tdi.last_update_member_id, tdi.last_update_group_id
FROM zd_products pd LEFT JOIN
zd_tables_data_info tdi
ON tdi.targetid = pd.id and tdi.table_name = 'products'
ORDER by pd.title ASC
LIMIT 0, 10;
You can improve performance of this query with indexes. The two that come to mind are zd_products(title, id) and zd_tables_data_info(targetid, table_name). Try these and see if they help. You can create these indexes either in the create table statement (or alter table) or by using:
create index zd_products_title_id on zd_products(title, id);
create index zd_tables_data_info_targetid_table_name on zd_tables_data_info(targetid, table_name);
If not, put explain in front of your query and then edit your question with the resulting plan.

best practice db tree sort

i have table like this:
CREATE TABLE `tree` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`parent_id` int(11) DEFAULT NULL,
`version` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
`title` varchar(255) NOT NULL,
`sort` int(11) NOT NULL,
`rating` double NOT NULL DEFAULT '0',
`global_sort` int(11) NOT NULL DEFAULT '0',
PRIMARY KEY (`id`),
KEY `IDX_22359CF6727ACA70` (`parent_id`),
CONSTRAINT `FK_22359CF6727ACA70` FOREIGN KEY (`parent_id`) REFERENCES `tree` (`id`) ON DELETE SET NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
in each level items can be sorted by: version, sort or rating.
I need to return all items in correct order.
as for now i use php recursion for each level & updating field global_sort concerning to level sorting.
any better ideas?
EDIT1
and as variant: if correct order must be only inside one level (no worries for global)?
EDIT2
i mean somshing like this
http://joxi.ru/WmYPUxjKTJAfVs5nCE4
when some items must be sorted by rating ask and some rating desc

perform calculations on multiple users-own records in mysql

I need help extending a functional update query that performs calculations on one record to be able to perform calculations not only on one record in the database, but on all records associated with a particular user #.
Functionally, I need to extend an "edit this record" to "reevaluate all records of user#?"
Current calculations make use of 3 tables, sum a column, divide by the sum of another, and then creates a variable from that result(there are two columns summed separately to create two variables). Then I have a simple UPDATE query to update the record with the values of those variables. Each record has different values, and the sums will be different for every id#
#lastid is the unique record id.(allinfsds.id1)
I need to have the calculations done on all records that = a particular owners id (allinfsds.own_id)[i.e. WHERE allinfsds.own_id= usernum]
Any ideas???
Thanks ahead of time,
Nat
CREATE TABLE `allingred` (
`id6` int(8) NOT NULL auto_increment,
`usernum` varchar(255) default NULL,
`fsdsnum` int(8) unsigned zerofill NOT NULL,
`mfdfsds` varchar(255) default NULL,
`maybe` decimal(2,1) NOT NULL default '1.0',
`amount` float(10,2) default NULL,
`unit` int(6) default NULL,
`name` varchar(255) default NULL,
`wgt` int(9) NOT NULL,
PRIMARY KEY (`id6`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 AUTO_INCREMENT=38 ;
CREATE TABLE `weight` (
`NDB_No2` int(8) unsigned zerofill NOT NULL,
`Seq` smallint(6) NOT NULL,
`amt2` decimal(5,3) NOT NULL,
`Msre_Desc` varchar(80) NOT NULL,
`Gm_Wgt` decimal(7,1) NOT NULL,
`Num_Data_Pts` tinyint(4) default NULL,
`Std_Dev` decimal(7,1) default NULL,
`uni` int(7) NOT NULL auto_increment,
PRIMARY KEY (`uni`),
KEY `fb_join_NDB_No2_INDEX` (`NDB_No2`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 AUTO_INCREMENT=21731 ;
CREATE TABLE `allinnot2` (
`NDB_No` int(8) unsigned zerofill NOT NULL,
`Water` decimal(10,2) default NULL,
`Energ_Kcal` decimal(10,0) default NULL
CREATE TABLE `allinfsds` (
`id1` int(8) unsigned zerofill NOT NULL,
`own_id` int(11) NOT NULL
UNIQUE KEY `id` (`id1`),
KEY `fb_groupbyorder_item_number_INDEX` (`item_number`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8;
SET #cal = (SELECT SUM( Energ_Kcal * allingred.amount * Gm_Wgt) / SUM( allingred.amount * Gm_Wgt ) AS nut100
FROM `allingred`
LEFT JOIN weight ON allingred.unit = weight.uni
LEFT JOIN allinnot2 ON allingred.mfdfsds = allinnot2.NDB_No
LEFT JOIN allinfsds ON allingred.fsdsnum = allinfsds.own_id
WHERE fsdsnum = #lastid)
SET #prot = (SELECT SUM(Protein * allingred.amount * Gm_Wgt) / SUM( allingred.amount * Gm_Wgt ) AS nut100
FROM `allingred`
LEFT JOIN weight ON allingred.unit = weight.uni
LEFT JOIN allinnot2 ON allingred.mfdfsds = allinnot2.NDB_No
WHERE fsdsnum = #lastid)
UPDATE `allinnot2` SET
`Energ_Kcal` = #cal,
`Protein` = #prot
WHERE `NDB_No` = #lastid
How to update multiple tuples at once
If you have a list of Ids you want to update, use
UPDATE `myTable` SET `myColumn` = 'newValue'
WHERE `userId` IN (
/*list of relevant Ids for instance: */ 15, 20, 63, 987
)
or if you dont have this list, but you can query the database for this list, use
UPDATE `myTable` SET `myColumn` = 'newValue'
WHERE `userId` IN (
SELECT `userId` FROM `myOtherTable` WHERE `relevantColumn` = 'value'
)
Beware that you are not allowed to use the same table as both the update target and source of ids in the subselect, so myTable != myOtherTable.

Sphinx PHP One-to-Many Search

I have the following mysql tables:
CREATE TABLE `video` (
`video_id` int(11) unsigned NOT NULL auto_increment,
`title` varchar(255) NOT NULL default '',
`description` text NOT NULL,
PRIMARY KEY (`video_id`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8;
CREATE TABLE `video_categories` (
`cat_id` int(11) unsigned NOT NULL auto_increment,
`name` varchar(255) NOT NULL default '',
PRIMARY KEY (`cat_id`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8;
CREATE TABLE `video_category` (
`video_id` int(11) unsigned NOT NULL default '0',
`cat_id` int(11) unsigned NOT NULL default '0',
KEY `video_id` (`video_id`),
KEY `cat_id` (`cat_id`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8;
CREATE TABLE `video_tags` (
`tag_id` int(11) unsigned NOT NULL auto_increment,
`video_id` int(11) unsigned NOT NULL default '0',
`name` varchar(255) NOT NULL default '',
KEY `video_id` (`video_id`),
KEY `name` (`name`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8;
I created a sphinx configuration file and i can search from PHP. The problem is when i want to search for related videos, a related video must be in the same category as the video i'm searching for. I can do this with MVA and and SetFilter('categories', array(3)) for example, however the total number of matches results is the global one (i need total to display pagination via ajax) not the one in the category.
Any ideas how i can search through videos (documents in sphinx) that are only in a specified category?
Thanks,
Adrian.
You can define for the category ID an integer attribute in Sphinx:
sql_attr_uint = cat_id
And then just add #cat_id=12345 to your query.

Categories