I have the following table townResources in which I store every resource value for every town ID. I am a bit reserved about performance impact for a large amount of users. I am thinking for moving the balance for resources to the towns table, and the general value of an resource to store it in a .php file.
Here you have the townresources table:
CREATE TABLE IF NOT EXISTS `townresources` (
`townResourcesId` int(10) NOT NULL AUTO_INCREMENT,
`userId` int(10) NOT NULL,
`resourceId` int(10) NOT NULL,
`townId` int(10) NOT NULL,
`balance` decimal(8,2) NOT NULL,
`resourceRate` decimal(6,2) NOT NULL,
`lastUpdate` datetime NOT NULL,
PRIMARY KEY (`resourceId`,`townId`,`townResourcesId`,`userId`),
KEY `townResources_userId_users_userId` (`userId`),
KEY `townResources_townId_towns_townId` (`townId`),
KEY `townResourcesId` (`townResourcesId`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 COMMENT='Stores Town Resources' AUTO_INCREMENT=9 ;
What is the best option in my case?
Your best option is to test first. How much users & towns do you want to support? Triple that.. create the test data and see whether the performance is within bounds.
If you run into trouble with performance you should look into caching the data with redis or memcache.
Related
I have a mysql table MAINLIST.
CREATE TABLE `MAINLIST` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`NAME` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`email` tinyint(1) unsigned DEFAULT NULL,
`contact` tinyint(1) unsigned DEFAULT NULL,
PRIMARY KEY (`id`),
) ENGINE=InnoDB AUTO_INCREMENT=17 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
Every day I select a subset of these and perform some operations. Right now I do this within the MAINLIST table, but I think it would be helpful for organization, readability and debugging to create a second table daily import the selected records, do the operations and then send the records back to the Mainlist table and destroy the daily table.
What is the best way to do this with mysql, or are there other ways to approach this problem? Perhaps I should not be doing this at all. I am wondering what best practices are since I'm not experienced with Db design. I am using the redbean ORM and php.
In my application whenever a user upload a wallpaper,i need to crop that wallpaper into
3 different sizes and store all those paths(3 paths for cropped images and 1 for original upload wallpaper) into my database.
I also need to store the tinyurl of the original wallpaper(one which is uploaded by user).
While solving the above described problem i come up with following table structure.
CREATE TABLE `wallpapermaster` (
`wallpaperid` int(10) unsigned NOT NULL AUTO_INCREMENT,
`userid` bigint(20) NOT NULL,
`wallpaperloc` varchar(100) NOT NULL,
`wallpapertitle` varchar(50) NOT NULL,
`wallpaperstatus` tinyint(4) DEFAULT '0' COMMENT '0-Waiting,1-approved,2-disapproved',
`tinyurl` varchar(40) NOT NULL
) ENGINE=MyISAM
wallpaperloc is a comma separated field consisting of original wallpaper location plus locations of all cropped instances.
I know using comma separated field considered to be a bad design in the world of relational database,So Would you like to suggest some other neat and efficient ways?
Use a 1:n relationship between the wallpapermaster and a location table.
Something like this:
CREATE TABLE wallpapermaster (
wallpaperid int unsigned NOT NULL AUTO_INCREMENT,
userid bigint NOT NULL,
wallpaperloc varchar(100) NOT NULL,
wallpapertitle varchar(50) NOT NULL,
wallpaperstatus tinyint DEFAULT '0' COMMENT '0-Waiting,1-approved,2-disapproved',
primary key (wallpaperid)
) ENGINE=InnoDB;
CREATE TABLE wallpaperlocation (
wallpaperid int unsigned NOT NULL,
location varchar(100) NOT NULL,
tinyurl varchar(40),
constraint fk_loc_wp
foreign key (wallpaperid)
references wallpapermaster (wallpaperid),
primary key (wallpaperid, location)
) ENGINE=InnoDB;
The primary key in wallpaperlocation ensures that the same location cannot be inserted twice.
Note that int(10) does not define any datatype constraints. It is merely a hint for client application to indicate how many digits the number has.
Usually you use a fixed location (maybe out of a config), fix extension (usually jpg) and a special filename formats like [name]-1024x768.jpg. This way you only the the name
In my opinion using ; or , in siple application is quite good solution even in relational databases.
You should propably think about amout of splitted images count. If there will be less than 5 wallpapers I would not take overhead complex solutions.
It's easy to maintain in database and application. You will use string splitting/joining methods
No need to adding extra additional tables which you will use join to retreive values.
Using simple varchar rather xml is better because you don't have to rely on application database access engine. When you use ORM or JDBC you have extra additional work to do to handle more complex datatypes.
In more complex systems I would make XML column.
While thumbnails are generated automatically from the single uploaded file, you don't need to store paths to cropped/resized files at all.
Instead you can just use normalized filenames for thumbnails and then find them in filesystem - something that KingCrunch suggested: photo1.jpg, photo1-medium.jpg etc.
Anyway, my 2cc: for avoiding traversing your image library (and created thumbnails) with some harvesters, it's good idea to encrypt name of each thumbnail even with just MD5 + some secret key programmatically, so only your program which knows the key can create proper path to the thumbnails basing on the original name/path. For other clients, naming sequence will be just random.
CREATE TABLE `wallpapermaster` (
`wallpaperid` int(10) unsigned NOT NULL AUTO_INCREMENT,
`userid` bigint(20) NOT NULL,
`wallpapertitle` varchar(50) NOT NULL,
`wallpaperstatus` tinyint(4) DEFAULT '0' COMMENT '0-Waiting,1-approved,2-disapproved',
`tinyurl` varchar(40) NOT NULL
) ENGINE=MyISAM
Create a new table which will create relationship with "wallpapermaster" table
create wallpapermaster_mapper(
`id` unsigned NOT NULL AUTO_INCREMENT,
`wallpapermaster_id` int(10) //this will be foreign key with id of wallpapermaster table
`wallpaper_path1` varchar(100) NOT NULL,
`wallpaper_path2` varchar(100) NOT NULL,
`wallpaper_path3` varchar(100) NOT NULL,
)
I have a table with a list of 2.5 million doctors. I also have tables for accepted insurance, languages spoken, and for specialties (taxonomy) provided. The doctor table is like:
CREATE TABLE `doctors` (
`doctor_id` int(10) NOT NULL AUTO_INCREMENT,
`city_id` int(10) NOT NULL DEFAULT '0',
`d_gender` char(1) NOT NULL DEFAULT 'U',
`s_insurance` int(6) NOT NULL DEFAULT '0',
`s_languages` int(6) NOT NULL DEFAULT '0',
`s_taxonomy` int(6) NOT NULL DEFAULT '0',
PRIMARY KEY (`doctor_id`)
) ENGINE=InnoDB;
The other information is stored as such:
CREATE TABLE `doctors_insurance` (
`assoc_id` int(10) NOT NULL AUTO_INCREMENT,
`doctor_id` int(10) NOT NULL DEFAULT '0',
`insurance_id` int(10) NOT NULL DEFAULT '0',
PRIMARY KEY (`assoc_id`)
) ENGINE=InnoDB;
CREATE TABLE `doctors_languages` (
`assoc_id` int(10) NOT NULL AUTO_INCREMENT,
`doctor_id` int(10) NOT NULL DEFAULT '0',
`language_id` int(10) NOT NULL DEFAULT '0',
PRIMARY KEY (`assoc_id`)
) ENGINE=InnoDB;
CREATE TABLE `doctors_taxonomy` (
`assoc_id` int(10) NOT NULL AUTO_INCREMENT,
`doctor_id` int(10) NOT NULL DEFAULT '0',
`taxonomy_id` int(10) NOT NULL DEFAULT '0',
PRIMARY KEY (`assoc_id`)
) ENGINE=InnoDB;
Naturally each doctor supports various different insurance plans, maybe speaks multiple languages, and some doctors can have several different specialties (taxonomy). So I opted to have separate tables for indexing, this way need I add new indices or drop old ones, I can simply remove the tables and not have to wait the long time it takes to actually do it the old fashioned way.
Also because of other scaling techniques to consider in the future, classic JOINs make no difference to me right now, so I'm not worried about it.
Indexing by name was easy:
CREATE TABLE `indices_doctors_names` (
`ref_id` int(10) NOT NULL AUTO_INCREMENT,
`doctor_id` int(10) NOT NULL DEFAULT '0',
`practice_id` int(10) NOT NULL DEFAULT '0',
`name` varchar(120) NOT NULL DEFAULT '',
PRIMARY KEY (`ref_id`),
KEY `name` (`name`)
) ENGINE=InnoDB;
However when I wanted to allow people to search by the city, specialties, insurance, language, and gender and other demographics, I created his:
CREATE TABLE `indices_doctors_demos` (
`ref_id` int(10) NOT NULL AUTO_INCREMENT,
`doctor_id` int(10) NOT NULL DEFAULT '0',
`city_id` int(10) NOT NULL DEFAULT '0',
`taxonomy_id` int(6) NOT NULL DEFAULT '0',
`insurance_id` int(6) NOT NULL DEFAULT '0',
`language_id` int(6) NOT NULL DEFAULT '0',
`gender_id` char(1) NOT NULL DEFAULT 'U',
PRIMARY KEY (`ref_id`),
KEY `index` (`city_id`,`taxonomy_id`,`insurance_id`,`language_id`,`gender_id`)
) ENGINE=InnoDB;
The idea is that there will be an entry for each change in specialty, insurance, or language primarily, though others will still the same. This creates an obvious problem. If a doctor has 3 specialties, supports 3 insurance providers, and speaks 3 languages, this alone means this specific doctor has 27 entries. So 2.5 million entries easily balloons into far more.
There has to be a better approach to do this, but how can it be done? Again, I'm not interested in moving to classic indexing techniques and using JOINs because it will quickly become too slow, I need a method that can scale out easily.
I know this is not the answer you're looking for, but you've now taken the things that a RDBMs do well and tried implementing it yourself, using the same mechanism that the RDBMs could use to actually make sense of your data and optimize both retrieval and querying. In practice you've decided to drop using proper indexes to create your own half-way-there-solution, which will try to implement indexes by itself (by actually using the indexing capability of the RDBMs with the KEY).
I'd suggest to actually try to just use the database the way you've already structured it. 2.5m rows isn't that many rows, and you should be able to make it work fast and within your constraints using both JOINs and indexes. Use EXPLAIN and add proper indexes to support your the queries you want answered. If you ever run into an issue (and I'd doubt it regarding the amount of data you're querying here), decide to solve the bottle neck then when you actually know what could be the issue instead of trying to solve a problem you've only imagined so far. There might be other technologies than MySQL that can be helpful - but you'll need to know what's actually hurting your performance first.
The normal way to deal with the explosion of rows in a denormalized table like "indices_doctors_demos" is to normalize to 5NF. Try to keep in mind that normalizing has nothing at all to do with the decision to use id numbers as surrogate keys.
In the scenario you described, normalizing to 5NF seems practical. You wouldn't have any table with more than about 7 million rows. The table "indices_doctors_demos" vanishes entirely, the four "doctors" tables all become narrower, and all of them would end up with highly selective indexes.
If you worked for me, I'd require you to prove that 5NF can't work before I'd let you take a different approach.
Since you already have all the data, it makes sense to build it and test it, paying close attention to the query plans. It shouldn't take you more than one afternoon. Guessing at some table names, I'd suggest you load data into these tables.
-- You're missing foreign keys throughout. I've added some of them,
-- but not all of them. I'm also assuming you have a way to identify
-- doctors besides a bare integer.
CREATE TABLE `doctors` (
`doctor_id` int(10) NOT NULL AUTO_INCREMENT,
`city_id` int(10) NOT NULL DEFAULT '0',
`d_gender` char(1) NOT NULL DEFAULT 'U',
PRIMARY KEY (`doctor_id`)
) ENGINE=InnoDB;
CREATE TABLE `doctors_insurance` (
`doctor_id` int(10) NOT NULL DEFAULT '0',
`insurance_id` int(10) NOT NULL DEFAULT '0',
PRIMARY KEY (`doctor_id`, `insurance_id`),
FOREIGN KEY (`doctor_id`) REFERENCES `doctors` (`doctor_id`),
FOREIGN KEY (`insurance_id`) REFERENCES `insurance` (`insurance_id`)
) ENGINE=InnoDB;
CREATE TABLE `doctors_languages` (
`doctor_id` int(10) NOT NULL DEFAULT '0',
`language_id` int(10) NOT NULL DEFAULT '0',
PRIMARY KEY (`doctor_id`, `language_id`),
FOREIGN KEY (`doctor_id`) REFERENCES `doctors` (`doctor_id`),
FOREIGN KEY (`language_id`) REFERENCES `languages` (`language_id`)
) ENGINE=InnoDB;
CREATE TABLE `doctors_taxonomy` (
`doctor_id` int(10) NOT NULL DEFAULT '0',
`taxonomy_id` int(10) NOT NULL DEFAULT '0',
PRIMARY KEY (`doctor_id`, `taxonomy_id`),
FOREIGN KEY (`doctor_id`) REFERENCES `doctors` (`doctor_id`),
FOREIGN KEY (`taxonomy_id`) REFERENCES `taxonomies` (`taxonomy_id`)
) ENGINE=InnoDB;
I have ~38 columns for a table.
ID, name, and the other 36 are bit-sized settings for the user.
The 36 other columns are grouped into 6 "settings", e.g. Setting1_on, Setting1_colored, etc.
Is this the best way to do this?
Thanks.
If it must be in one table and they're all toggle type settings like yes/no, true/false, etc... use TINYINT to save space.
I'd recommend creating a separate table 'settings' with 36 records one for each option. Then create a linking table to the user table with a value column to record the user settings. This creates a many-to-many link for the user settings. It also makes it easy to add a new setting--just add a new row to the 'settings' table. Here is an example schema. I use varchar for the value of the setting to allow for later setting which might not be bits, but feel free to use TINYINT if size is an issue. This solution will not use as much space as the one table with the danger of a large sparsely populated set of columns.
CREATE TABLE `user` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`name` varchar(64) DEFAULT NULL,
`address` varchar(64) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
CREATE TABLE `setting` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`name` varchar(64) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
CREATE TABLE `setting_user` (
`user_id` int(11) NOT NULL DEFAULT '0',
`setting_id` int(11) unsigned NOT NULL,
`value` varchar(32) DEFAULT NULL,
PRIMARY KEY (`user_id`,`setting_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
All depends on how you want to access them. If you want to (or must) just select one of them, then go with the #Ray solution. If they can be functionally grouped (really, not some pretend grouping for all those that start with F) ie. you'll always need number of them for a function and reading and writing them doesn't make sense as an individual flag, then perhaps storing them as ints and using logic operaoprs on them might be a goer.
Saying that, unless you are doing a lot of read and writes to the db during a session, bundling them up into ints gives you very little performance wise, it would save some space on the DB, if all the options had to exist. If doesn't exist = false, it could be a toss up.
So all things being unequal, I'd go with Mr Ray.
MySQL has a SET type that could be useful here. Everything would fit into a single SET, but six SETs might make more sense.
http://dev.mysql.com/doc/refman/5.5/en/set.html
I have inherited a PHP project and the client is wanting to add some functionality to their CMS, basically the CMS allows them to create some news, all the news starts with the same content, and that is saved in one table, the actually news headline and articles are saved in another table, and the images for the news are saved in another, basically if the base row for the news is deleted I need all the related rows to be deleted, the database is not setup to work with foreign keys so I cannot use cascade deletion, so how can I delete the all the content I need to, when I only what the ID of the base news row is?
Any help would be very helpful I am sorry I cannot give you much more help, here is this the original SQL of tables scheme if that helps?
--
-- Table structure for table `mailers`
--
CREATE TABLE IF NOT EXISTS `mailers` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`mailer_title` varchar(150) NOT NULL,
`mailer_header` varchar(60) NOT NULL,
`mailer_type` enum('single','multi') NOT NULL,
`introduction` varchar(80) NOT NULL,
`status` enum('live','dead','draft') NOT NULL,
`flag` enum('sent','unsent') NOT NULL,
`date_mailer_created` int(11) NOT NULL,
`date_mailer_updated` int(10) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=13 ;
-- --------------------------------------------------------
--
-- Table structure for table `mailer_content`
--
CREATE TABLE IF NOT EXISTS `mailer_content` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`headline` varchar(60) NOT NULL,
`content` text NOT NULL,
`mailer_id` int(11) NOT NULL,
`position` enum('left','right','centre') DEFAULT NULL,
`created_at` int(10) NOT NULL,
`updated_at` int(10) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=18 ;
-- --------------------------------------------------------
--
-- Table structure for table `mailer_images`
--
CREATE TABLE IF NOT EXISTS `mailer_images` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`title` varchar(150) NOT NULL,
`filename` varchar(150) NOT NULL,
`mailer_id` int(11) NOT NULL,
`content_id` int(11) DEFAULT NULL,
`date_created` int(10) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=15 ;
It is worth noting that the schema cannot be changed nor can I change to the DB to MYISAM so that I can use foreign keys.
Add foreign key to table mailer_content
FOREIGN KEY (mailer_id)
REFERENCES mailers(id)
ON DELETE CASCADE
Add foreign key to table mailer_images
FOREIGN KEY (content_id)
REFERENCES mailer_content(id)
ON DELETE CASCADE
http://dev.mysql.com/doc/refman/5.1/en/innodb-foreign-key-constraints.html
It is worth noting that the schema cannot be changed nor can I change to the DB to MYISAM so that I can use foreign keys.
Why can't the schema be changed? You designed the app, didn't you? Even if you didn't, adding the proper keys is just a matter of adding the right indexes and then altering the right columns. #Michael Pakhantosv's answer has what looks to be the right bits of SQL.
Further, it's InnoDB that does foreign keys, not MyISAM. You're fine there already.
If you could change the schema, making the appropriate IDs actual, real Foreign Keys and using ON DELETE CASCADE would work. Or maybe triggers. But that's just asking for it.
Now, for some reason, ON DELETE CASCADE isn't liked very much around here. I disagree with other people's reasons for not liking it, but I don't disagree with their sentiment. Unless your application was designed to grok ON DELETE CASCADE, you're in for a world of trouble.
But, given your requirement...
basically if the base row for the news is deleted I need all the related rows to be deleted
... that's asking for ON DELETE CASCADE.
So, this might come as a shock, but if you can't modify the database, you'll just have to do your work in the code. I'd imagine that deleting a news article happens in only one place in your code, right? If not, it'd better. Fix that first. Then just make sure you delete all the proper rows in an appropriate order. And then document it!
If you can not change the schema then triggers are not an option.
InnoDB supports transactions, so deleting from two tables should not be an issue, what exactly is your problem?
P.S. It would be worth noting which version of the server are you using.