For a simple web application the main requirement is to process around 30 (10m * 3 tables) million records as fast as possible. I haven't worked with such amount of data before so would like some suggestions/advise from experienced people.
The database will be holding details of businesses. Around 25 attributes will describe a single business; name, address etc. Table structure is as follows.
CREATE TABLE IF NOT EXISTS `businesses` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`type` int(2) NOT NULL,
`organisation` varchar(40) NOT NULL,
`title` varchar(12) NOT NULL,
`given_name` varchar(40) NOT NULL,
`other_name` varchar(40) NOT NULL,
`family_name` varchar(40) NOT NULL,
`suffix` varchar(5) NOT NULL,
`reg_date` date NOT NULL,
`main_trade_name` varchar(150) NOT NULL,
`son_address_l1` varchar(50) NOT NULL,
`son_address_l2` varchar(50) NOT NULL,
`son_address_suburb` int(3) NOT NULL,
`son_address_state` int(2) NOT NULL,
`son_address_postcode` varchar(10) NOT NULL,
`son_address_country` int(3) NOT NULL,
`bus_address_l1` varchar(50) NOT NULL,
`bus_address_l2` varchar(50) NOT NULL,
`bus_address_suburb` int(3) NOT NULL,
`bus_address_state` int(2) NOT NULL,
`bus_address_postcode` varchar(10) NOT NULL,
`bus_address_country` int(3) NOT NULL,
`email` varchar(165) DEFAULT NULL,
`phone` varchar(12) NOT NULL,
`website` varchar(80) NOT NULL,
`employee_size` int(4) NOT NULL,
PRIMARY KEY (`id`),
KEY `type` (`type`),
KEY `phone` (`phone`),
KEY `reg_date` (`reg_date`),
KEY `son_address_state` (`son_address_state`),
KEY `bus_address_state` (`bus_address_state`),
KEY `son_address_country` (`son_address_country`),
KEY `bus_address_country` (`bus_address_country`),
FULLTEXT KEY `title` (`title`),
FULLTEXT KEY `son_address_l1` (`son_address_l1`),
FULLTEXT KEY `son_address_l2` (`son_address_l2`),
FULLTEXT KEY `bus_address_l1` (`bus_address_l1`),
FULLTEXT KEY `bus_address_l2` (`bus_address_l2`)
) ENGINE=MyISAM;
There going to be 2 other tables like this, reason being each business details will be presented in 3 sources (for comparison purposes). Only one table is going to have writes.
About the app usage,
Few writes, loads of reads.
10 * 3 million of data will not be inserted overtime, its going to be inserted initially.
App is not going to have lots of requests, <10 requests per second.
After the initial data load, users will be updating these details. Comparing one table's data with other 2 and update the data in first table.
There will be lots of searches, mainly by name, by address, by phone and state. Single search will go through all the 3 tables. Searching needs to be fast.
Planing to build it using PHP
My Questions are,
Is it worth to handle 3 sources within one table rather than having 3 tables?
Can MySQL provide a good solution?
Will MongoDB able to handle the same scenario using less hardware resources?
What's the best way to setup a sample database for testing? I purchased a Amazon RDS (large) and inserted 10000 records and doubled them until I get 10 million records.
Any good reading about this subject?
Thank You.
I cannot answer to your direct question, but I have experience of working with large datasets.
First thing I would work out is what the majority use case (in your case search) operations woud be, and then consider data storage/partitioning based on the that.
Next thing is measure, measure, and measure again. Some database systems will work well with one kind of operation and others with others. As the amount of data increases and operational complexity increases, things that worked well may start to degrade. This is why you measure - don't try to design this without good evidence of how the db systems you're using work under these loads.
And then work iteratively to the add more operations.
Don't try to deisgn a best fit for all. As your design and research is distilled youll see places where optimisations may be needed or availble. You may also find as we've done in in the past, that different type of caching and indexing may beeded at different times.
Good luck - sounds like an interesting project.
Related
I cleaned the question a little bit because it was getting very big and unreadable.
Running on my localhost.
As you can see in the image below, the query takes 755.15 ms when selecting from the table Job that contains 15000 rows (with the where conditions returning 6650)
The table Company contains 1000 rows.
The table geo__name contains 84300 rows approx and is not giving me any problem, so I believe the problem is the database structure or something.
The structure of these 2 tables is the following:
Table Job is:
CREATE TABLE `job` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`created_at` datetime NOT NULL,
`updated_at` datetime NOT NULL,
`company_id` int(11) NOT NULL,
`activity_sector_id` int(11) DEFAULT NULL,
`status` int(11) NOT NULL,
`active` datetime NOT NULL,
`contract_type_id` int(11) NOT NULL,
`salary_type_id` int(11) NOT NULL,
`workday_id` int(11) NOT NULL,
`geoname_id` int(11) NOT NULL,
`title` varchar(255) COLLATE utf8mb4_unicode_ci NOT NULL,
`minimum_experience` int(11) DEFAULT NULL,
`min_salary` decimal(7,2) DEFAULT NULL,
`max_salary` decimal(7,2) DEFAULT NULL,
`zip_code` int(11) DEFAULT NULL,
`vacancies` int(11) DEFAULT NULL,
`show_salary` tinyint(1) NOT NULL,
PRIMARY KEY (`id`),
KEY `created_at` (`created_at`,`active`,`status`) USING BTREE,
CONSTRAINT `FK_FBD8E0F823F5422B` FOREIGN KEY (`geoname_id`) REFERENCES `geo__name` (`id`),
CONSTRAINT `FK_FBD8E0F8398DEFD0` FOREIGN KEY (`activity_sector_id`) REFERENCES `activity_sector` (`id`),
CONSTRAINT `FK_FBD8E0F85248165F` FOREIGN KEY (`salary_type_id`) REFERENCES `job_salary_type` (`id`),
CONSTRAINT `FK_FBD8E0F8979B1AD6` FOREIGN KEY (`company_id`) REFERENCES `company` (`id`),
CONSTRAINT `FK_FBD8E0F8AB01D695` FOREIGN KEY (`workday_id`) REFERENCES `workday` (`id`),
CONSTRAINT `FK_FBD8E0F8CD1DF15B` FOREIGN KEY (`contract_type_id`) REFERENCES `job_contract_type` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=15001 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
The table company is:
CREATE TABLE `company` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`name` varchar(255) COLLATE utf8mb4_unicode_ci NOT NULL,
`logo` varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`created_at` datetime NOT NULL,
`updated_at` datetime NOT NULL,
`website` varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`user_id` int(11) NOT NULL,
`phone` varchar(255) COLLATE utf8mb4_unicode_ci NOT NULL,
`cifnif` varchar(255) COLLATE utf8mb4_unicode_ci NOT NULL,
`type` int(11) NOT NULL,
`subscription_id` int(11) DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `UNIQ_4FBF094FA76ED395` (`user_id`),
KEY `IDX_4FBF094F9A1887DC` (`subscription_id`),
KEY `name` (`name`(191)),
CONSTRAINT `FK_4FBF094F9A1887DC` FOREIGN KEY (`subscription_id`) REFERENCES `subscription` (`id`),
CONSTRAINT `FK_4FBF094FA76ED395` FOREIGN KEY (`user_id`) REFERENCES `user` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=1001 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
The query is the following:
SELECT
j0_.id AS id_0,
j0_.status AS status_1,
j0_.title AS title_2,
j0_.min_salary AS min_salary_3,
j0_.max_salary AS max_salary_4,
c1_.id AS id_5,
c1_.name AS name_6,
c1_.logo AS logo_7,
a2_.id AS id_8,
a2_.name AS name_9,
g3_.id AS id_10,
g3_.name AS name_11,
j4_.id AS id_12,
j4_.name AS name_13,
j5_.id AS id_14,
j5_.name AS name_15,
w6_.id AS id_16,
w6_.name AS name_17
FROM
job j0_
INNER JOIN company c1_ ON j0_.company_id = c1_.id
INNER JOIN activity_sector a2_ ON j0_.activity_sector_id = a2_.id
INNER JOIN geo__name g3_ ON j0_.geoname_id = g3_.id
INNER JOIN job_salary_type j4_ ON j0_.salary_type_id = j4_.id
INNER JOIN job_contract_type j5_ ON j0_.contract_type_id = j5_.id
INNER JOIN workday w6_ ON j0_.workday_id = w6_.id
WHERE
j0_.active >= CURRENT_TIMESTAMP
AND j0_.status = 1
ORDER BY
j0_.created_at DESC
When executing the above query I have these results:
In MYSQL Workbench: 0.578 sec / 0.016 sec
In Symfony profiler: 755.15 ms
The question is: Is the duration of this query correct? if not, how can I improve the speed of the query? it seems too much.
The Symfony debug toolbar if it helps:
As you can see in the below image, I'm only getting the data I really need:
The explain query:
The timeline:
The MySQL server can't handle the load being placed on it. This could be due to resource contention, or because it has not been appropriately tuned and it could also be a problem with your hard drive.
First, I would start your performance by adding MySQL keyword "STRAIGHT_JOIN" which tells MySQL to query the data in the order I have provided, dont try to think the relationships for me. However, on your dataset being so small, and already 1/2 second, don't know if that will help as much, but on larger datasets I have known it to SIGNIFICANTLY improve performance.
Next, you appear to be getting lookup descriptions based on the PK/FK relationship results. Not seeing the indexes on those tables, I would suggest doing covering indexes which contain both the key and description so the join can get the data from the index pages it uses for the JOIN instead of use index page, find the actual data pages to get the description and continue.
Last, your job table with the index on (created_at,active,status), might perform better if the index had the index as ( status, active, created_at ).
With your existing index, think of it this way, each day of data is put into a single box. Within each day box that is sorted by an active timestamp (even if simplified by active date), THEN the status.
So, for each day CREATED, you open a box. Look at secondary boxes, one for each "Active" timestamp (ex: by day). Within each Active timestamp (day), only now can you see if the "Status = 1" records. So open each active timestamp day, assess Status = 1, then close each created day box and go to the next created day box and repeat. So look at the labor intensive of open each box per day, each active box within that day.
Now, under the suggested index starting with status. You now have a very finite number of boxes, one for each status. Open only the 1 box for status = 1 These are the only ones you want to consider... All the others you don't care. Inside that, you have the actual records based on ACTIVE Timestamp and that is sub-sorted. From that, you can jump directly to those at the current timestamp. From the first record and the rest within the box, you now have all the records that qualify. Done. Since these records (index) ALSO has the Created_at as part of the index, it can optimize that with the descending sort order.
For ensuring "covering indexes" for the other lookup tables if they do not yet exist, I suggest the following.
table index
company ( id, name, logo )
activity_sector (id, name )
geo__name ( id, name )
job_salary_type ( id, name )
job_contract_type ( id, name )
workday ( id, name )
And the MySQL Keyword...
SELECT STRAIGHT_JOIN (rest of query...)
There are several reasons as to why Symfony is slow.
1. Server fault
First, it could be the server fault. Server performances may hinder your query time.
2. Data size and defered rendering
Then comes the data size. As you can see on the image below, the query on one of my project have a 50Mb data size (currently about 20k rows).
Parsing 50Mb in HTML can take some time, mostly because of loops.
Still, there are solutions about this, like defered rendering.
Defered rendering is quite simple, instead of parsing data in your twig you,
send all data to a javascript varaible, and use javascript to parse/render data once the DOM is loaded.
3. Query optimisation
As I wrote in comment, you can check the following question, on which I explained why custom queries are important.
Are Doctrine relations affecting application performance?
In this question, you will read that order matter... It's in fact the most important thing.
While static data in your databases are often inserted in the right order,
it's rarely the case for dynamic data (data provided by user during the website life)
Which is why, using ORDER BY in your query will often speed up the page rendering,
as doctrine won't be doing extra queries on it's own.
As exemple, One of my site have about 700 entries diplayed on the index.
First, here is the query count while using findAll() :
It show 254 query (253 duplicates) in 144ms, plus 39 render time.
Next, using the second parameter of findBy(), ORDER BY, I get this result :
You can see the full query here (sreenshot is big)
Much better, 1 query only in 8ms, and about the same render time.
But, here, I don't use any fields from associations.
From the moment I will do it, doctrine qui do some extra query, and query count and time will skyrocket.
In the end, it will turn back to something like findAll()
And last, this is the custom query :
In this custom query, the query time went from 8ms to 38ms.
But, unlike the previous query, I got way more data in my result,
which will prevent doctrine from doing extra query.
Again, ORDER BY() matter in this query. Without it, I skyrocket back to 84 queries.
4. Partials
When you do custom query, you can load partials objects instead of full data.
As you said in your question, description field seems to slow down your loading speed,
with partials, you can avoid to load some fields from the table, which will speed up query speed.
First, instead of your regular syntax, this is how you will create the query builder :
$em=$this->getEntityManager();
$qb=$em->createQueryBuilder();
Just in case, I prefer to keep $em as a separate variable (if I want to fetch some class repository for example).
Then you can start your partial select. Careful, first select can't include any association fields :
$qb->select("partial job.{id, status, title, minimum_experience, min_salary, max_salary, zip_code, vacancies")
->from(Job::class, "job");
Then you can add your associations :
$qb->addSelect("company")
->join("job.company", "company");
Or even add partial association in case you don't need all the data of the association :
$qb->addSelect("partial activitySector.{id}")
->join("job.activitySector", "activitySector");
$qb->addSelect("partial job.{id, company_id, activity_sector_id, status, active, contract_type_id, salary_type_id, workday_id, geoname_id, title, minimum_experience, min_salary, max_salary, zip_code, vacancies, show_salary");
5. Caches
You could also use various caches, like Zend OPCache for PHP, which you will find some advices in this question: Why Symfony3 so slow?
There is also the SQL cache Varnish.
This round up about everything I can share to lower your loading time.
Hope it will prove useful and you will be able to solve your problem.
So many keys , try to minimize the number of keys.
I'm working on a project and right now I'm implementing a leaderboard. Before I start working on it, I need some advices for better practice of my leaderboard's structure.
First of all the leaderboard will be displayed on two pages, the one is on the home page of each player's which will contain the first 10 teams (same 10 teams for all players) and the other leaderboard will be in the leaderboard's page, which there, will have all the teams with sorting functionalities.
The structure of the leaderboard of each row is the following:
• ranking position
• team name
• team value
• total of the games the team won
• total of the games the team defeated
• total of the games the team had draw
• sum of the goals the team has made
• sum of the goals the team has conceded
• the last 4 game results of the team
Below is my database's tables
challenges table
CREATE TABLE `challenges` (
id` int(10) unsigned NOT NULL AUTO_INCREMENT,
'challenge_date` datetime NOT NULL,
`status` varchar(20) COLLATE utf8_unicode_ci NOT NULL,
`created_at` timestamp NULL DEFAULT NULL,
`updated_at` timestamp NULL DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `challanges_id_index` (`id`),
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
challenges results
CREATE TABLE `challenges_results` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`challenge_id` int(11) NOT NULL,
`team_id` int(11) NOT NULL,
`goals` int(11) NOT NULL,
`result` char(1) DEFAULT NULL,
`challenge_date` datetime NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
On challenges results result column can be W for wins, D for draws and L for defeats
team values
CREATE TABLE `team_values` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`team_id` int(11) DEFAULT NULL,
`value` double(15,8) DEFAULT '1500.00000000',
`created_at` datetime DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
team
CREATE TABLE `teams` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`name` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
`avatar` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
`founded` date NOT NULL,
`residense_city_id` int(10) unsigned NOT NULL,
`slug` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
`primary_color` char(10) COLLATE utf8_unicode_ci NOT NULL,
`secondary_color` char(10) COLLATE utf8_unicode_ci NOT NULL,
`status` varchar(20) COLLATE utf8_unicode_ci NOT NULL,
`created_at` timestamp NULL DEFAULT NULL,
`updated_at` timestamp NULL DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `teams_slug_unique` (`slug`),
KEY `teams_id_index` (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
One team can have many values (teams_values) but only the recent will be displayed.
One team can be in many challenges.
One team can have many results from different challenges.
The leaderboard will work as follow. The teams will be sorted with the highest values from teams_values table. That value is calculated and stored every time the team is having a challenge.
In case where two or more teams have the same value we need to apply the following three rules. The rules also needs to be executed one by one, for example if I run the first rule and still there are teams which are equal also on value and goals scored then I will apply the second rule and so on.
• Best offense (higher Number of goals scored)
• Best Defense (Less Number of goals conceded)
• The team with the most wins in the games between them
So I came with three solutions which still I don't know which one is the better and if there is a better from the three.
The first option that I though is to use options like inner join, union etc to collect the information from the tables and apply the rules on the same SQL query. So every time that I want to view the leaderboard, I will execute this SQL. The problem with this solution is that I don't know how effective will be in case that we want the leaderboard to be always up to date with the latest results. Because imaging having 10k visitors per day and everyone executing this query.
Second option is to collect the information and in case of duplicate values, I will use PHP to get the duplicate teams, apply the rules and then based on the results of the rules swipe the teams in the array. From performance site I don't know how effective is this option.
Third solution is to create another table called leaderboard which I will store all this information in case the team doesn't exist or I will update the record if exist based on the results of the latest challenge e.g increasing the goals if the team scored. Then I will use only the leaderboard table for filtering the data and printing the ranking of the teams. I believe this option is better because I need to deal only with one table and I will update the record only when a team finished a challenge.
We will use cache, but for now we are thinking that the leaderboard should be always up to date and not updating it once a day.
Which one is better solution and why and in case of a better solution I'm open for suggestions. Thanks
Since you're running on a shared account on a virtual private server, the chances are very going you're going to theoretically run into cases where you contend for the use of server resources, disk usage, memory usage, cpu processing power.
First and foremost, try do all your database calculations in MySQL, and only return the data to PHP once you've completed all operations on them. MySQL is optimised for the job, whereas PHP is better at general computing problems.
Your one option to take some load off the server would be to use PHP to create a webpage that is viewable in the browser, every single time the leaderboard is updated. That way, you run through calculations only once every time the leaderboard needs to be updated.
If I was building the system and the system was never going to reach enterprise-grade level, but instead remain small and functional, I would write a PHP script early on, because you can save a lot of processing power that way alone.
For what it's worth, if the server is well configured, you shouldn't be worried about getting 10k user requests a day, unless your code is really terribly written.
EDIT: As an afterthought, you can install a program like https://memcached.org/, which caches your SQL data in RAM. Sites like LiveJournal and Wordpress use it, but you'd need to configure it in a way that works for the rest of the vps users unless the box is really high spec.
A couple of years ago I designed a reward system for 11-16yo students in PHP, JavaScript and MySQL.
The premise is straightforward:
Members of staff issue points to students under various categories ("Positive attitude and behaviour", "Model citizen", etc)
Students accrue these points then spend them in our online store (iTunes vouchers, etc)
Existing system
The database structure is also straightforward (probably too much so):
Transactions
239,189 rows
CREATE TABLE `transactions` (
`Transaction_ID` int(9) NOT NULL auto_increment,
`Datetime` date NOT NULL,
`Giver_ID` int(9) NOT NULL,
`Recipient_ID` int(9) NOT NULL,
`Points` int(4) NOT NULL,
`Category_ID` int(3) NOT NULL,
`Reason` text NOT NULL,
PRIMARY KEY (`Transaction_ID`),
KEY `Giver_ID` (`Giver_ID`),
KEY `Datetime` (`Datetime`),
KEY `DatetimeAndGiverID` (`Datetime`,`Giver_ID`),
KEY `Recipient_ID` (`Recipient_ID`)
) ENGINE=InnoDB AUTO_INCREMENT=249069 DEFAULT CHARSET=latin1
Categories
34 rows
CREATE TABLE `categories` (
`Category_ID` int(9) NOT NULL,
`Title` varchar(255) NOT NULL,
`Description` text NOT NULL,
`Default_Points` int(3) NOT NULL,
`Groups` varchar(125) NOT NULL,
`Display_Start` datetime default NULL,
`Display_End` datetime default NULL,
PRIMARY KEY (`Category_ID`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1
Rewards
82 rows
CREATE TABLE `rewards` (
`Reward_ID` int(9) NOT NULL auto_increment,
`Title` varchar(255) NOT NULL,
`Description` text NOT NULL,
`Image_URL` varchar(255) NOT NULL,
`Date_Inactive` datetime NOT NULL,
`Stock_Count` int(3) NOT NULL,
`Cost_to_User` float NOT NULL,
`Cost_to_System` float NOT NULL,
PRIMARY KEY (`Reward_ID`)
) ENGINE=InnoDB AUTO_INCREMENT=91 DEFAULT CHARSET=latin1
Purchases
5,889 rows
CREATE TABLE `purchases` (
`Purchase_ID` int(9) NOT NULL auto_increment,
`Datetime` datetime NOT NULL,
`Reward_ID` int(9) NOT NULL,
`Quantity` int(4) NOT NULL,
`Student_ID` int(9) NOT NULL,
`Student_Name` varchar(255) NOT NULL,
`Date_DealtWith` datetime default NULL,
`Date_Collected` datetime default NULL,
PRIMARY KEY (`Purchase_ID`)
) ENGINE=InnoDB AUTO_INCREMENT=6133 DEFAULT CHARSET=latin1
Problems
The system ran perfectly well for a period of time. It's now starting to slow-down massively on certain queries.
Essentially, every time I need to access a students' reward points total, the required query takes ages. Here is a few example queries and their run-times:
Top 15 students, excluding attendance categories, across whole school
SELECT CONCAT( s.Firstname, " ", s.Surname ) AS `Student` , s.Year_Group AS `Year Group`, SUM( t.Points ) AS `Points`
FROM frog_rewards.transactions t
LEFT JOIN frog_shared.student s ON t.Recipient_ID = s.id
WHERE t.Datetime > '2013-09-01' AND t.Category_ID NOT IN ( 12, 13, 14, 26 )
GROUP BY t.Recipient_ID
ORDER BY `Points` DESC
LIMIT 0 , 15
Run-time: 44.8425 sec
SELECT Recipient_ID, SUM(points) AS Total_Points FROMtransactionsGROUP BY Recipient_ID
Run-time: 9.8698 sec
Now I appreciate that, especially with the second query, I shouldn't ever be running a call which would return such a vast quantity of rows but the limitations of the framework within which the system runs meant that I had no other choice if I wanted to display students' total reward points for teachers/tutors/year managers/leadership to view and analyse.
Time for a solution
Fortunately the framework we've been forced to use is changing. We'll now be using oAuth rather than a horrible, outdated JavaScript widget format.
Unfortunately - or, I guess, fortunately - it means we'll have to rewrite quite a lot of the system.
One of the main areas I intend to look at when rewriting the system is the database structure. As time goes on it will only get bigger, so I need to do a bit of future-proofing.
As such, my main question is this: what is the most efficient and effective way of storing students' point totals?
The only idea I can come up with is to have a separate table called totals with Student_ID and Points fields. Every time a member of staff gives out some points, it adds a row into the transactions table but also updates the totals table.
Is that efficient? Would it be efficient to also have a Points_Since_Monday type field? How would I update/keep on top of that?
On top of the main question, if anyone has suggestions for general improvement with regard to optimisation of the database table, please let me know.
Thanks in advance,
Duncan
There is nothing particularly wrong with your design which should make it as slow as you have reported. I'm thinking there must be other factors at work, such as the server it is running on being overloaded or slow, for example. Only you will be able to find out if that is the case.
In order to test your design I recreated it on the 2008 SQL Server I have running on my desktop computer. I have a standard computer, single hard disc, not SSD, not raid etc. so on a proper database server the results should be even better. I had to make some changes to the design as you are using MySQL but none of the changes should affect performace, it's just so I can run it on my database.
Here's the table structure I used, I had to guess at what you would have in the Student and Staff tables as you do not descibe those. I also took the liberty of changing the field names in the Transaction table for Giver_ID and Receiver_ID as I assume only staff give points and students receive them.
I generated random data to fill the tables with the same number of rows as you said you have in your database
I ran the two queries you said are taking a long time, I've changed them to suit my design but I (hope) the result is the same
SELECT TOP 15
Firstname + ' ' + Surname
,Year_Group
,SUM(Points) AS Points
FROM points.[Transaction]
INNER JOIN points.Student ON points.[Transaction].Student_ID = points.Student.Student_ID
WHERE [Datetime] > '2013-09-01'
AND Category_ID NOT IN ( 12, 13, 14, 26 )
GROUP BY Firstname + ' ' + Surname
,Year_Group
ORDER BY SUM(Points) DESC
SELECT Student_ID
,SUM(Points) AS Total_Points
FROM points.[Transaction]
GROUP BY Student_ID
Both queries returned results in about 1s. I have not created any additional indexes on the tables other than the CLUSTERED indexes generated by default on the primary keys. Looking at the execution plan the query processor estimates that implementing the following index could improve the query cost by 81.0309%
CREATE NONCLUSTERED INDEX [<Name of Missing Index>]
ON [points].[Transaction] ([Datetime],[Category_ID])
INCLUDE ([Student_ID],[Points])
As others have commented I would look elsewhere for bottlenecks before spending a lot of time redesigning your database.
Update:
I realised I never actually addressed your specific question:
what is the most efficient and effective way of storing students'
point totals?
The only idea I can come up with is to have a separate table called
totals with Student_ID and Points fields. Every time a member of staff
gives out some points, it adds a row into the transactions table but
also updates the totals table.
I would not recommend keeping a separate point total unless you have explored every other possible way to speed up the database. A separate tally can become out of sync with the transactions and then you have to reconcile everything and track down what went wrong, and what the correct total should be.
You should always focus on maintaining the correctness and consistency of the data before trying to increase speed. Most of the time a correct (normalised) data model will operate quickly enough.
In one place I worked we found the most cost effective way to speed up our database was simply to upgrade the hardware; much quicker and cheaper than spending many man-hours redesigning the database :)
I have the following table townResources in which I store every resource value for every town ID. I am a bit reserved about performance impact for a large amount of users. I am thinking for moving the balance for resources to the towns table, and the general value of an resource to store it in a .php file.
Here you have the townresources table:
CREATE TABLE IF NOT EXISTS `townresources` (
`townResourcesId` int(10) NOT NULL AUTO_INCREMENT,
`userId` int(10) NOT NULL,
`resourceId` int(10) NOT NULL,
`townId` int(10) NOT NULL,
`balance` decimal(8,2) NOT NULL,
`resourceRate` decimal(6,2) NOT NULL,
`lastUpdate` datetime NOT NULL,
PRIMARY KEY (`resourceId`,`townId`,`townResourcesId`,`userId`),
KEY `townResources_userId_users_userId` (`userId`),
KEY `townResources_townId_towns_townId` (`townId`),
KEY `townResourcesId` (`townResourcesId`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 COMMENT='Stores Town Resources' AUTO_INCREMENT=9 ;
What is the best option in my case?
Your best option is to test first. How much users & towns do you want to support? Triple that.. create the test data and see whether the performance is within bounds.
If you run into trouble with performance you should look into caching the data with redis or memcache.
I am working on a social network type site in PHP, I have done this once before and the site outgrew my coding ability to keep up, this was a couple years back and now I am wanting to tackle this project again.
Basicly on my network there is a friend_friend mysql table that keeps track of who is who's friend, for every confirmed friend, there are 2 entries into the DB
here is that table:
CREATE TABLE IF NOT EXISTS `friend_friend` (
`autoid` int(11) NOT NULL AUTO_INCREMENT,
`userid` int(10) DEFAULT NULL,
`friendid` int(10) DEFAULT NULL,
`status` enum('1','0','3') NOT NULL DEFAULT '0',
`submit_date` datetime NOT NULL DEFAULT '0000-00-00 00:00:00',
`alert_message` enum('yes','no') NOT NULL DEFAULT 'yes',
PRIMARY KEY (`autoid`),
KEY `userid` (`userid`),
KEY `friendid` (`friendid`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 AUTO_INCREMENT=1657259 ;
I then have a user table with all users info called friend_reg_user
Then a table for bulletins that users post, the object is to only show bulletins from users who you are friends with.
Here is bulletins table
CREATE TABLE IF NOT EXISTS `friend_bulletin` (
`auto_id` int(11) NOT NULL AUTO_INCREMENT,
`user_id` int(10) NOT NULL DEFAULT '0',
`bulletin` text NOT NULL,
`subject` varchar(255) NOT NULL DEFAULT '',
`color` varchar(6) NOT NULL DEFAULT '000000',
`submit_date` datetime NOT NULL DEFAULT '0000-00-00 00:00:00',
`status` enum('Active','In Active') NOT NULL DEFAULT 'Active',
`spam` enum('0','1') NOT NULL DEFAULT '1',
PRIMARY KEY (`auto_id`),
KEY `user_id` (`user_id`),
KEY `submit_date` (`submit_date`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 AUTO_INCREMENT=455144 ;
Ok so to do this I would either run a query on the friend_friend table to get all friends of a user and add them to a string like this 1,2,3,4,5,6 those would be friend ID numbers and then select from bulletin table where bulletin author ID is in my friend ID list
The second method is to use JOINS to get all this data at once.
My quest now finally, once the site gets very large, when there are millions of friends records and bulletins in the DB this all slows down, what are my options to speed things up? Is there a better way to do this? Also I am planning on changing bulletins to include more then just bulletins but do more of user actions like the big sites do now so it will show status updates and blogs and bulletins and all
What you are looking to do can likely be done in a number of ways. You can have a summary rollup table that combines all of the associated data (friends in this instance) for a given member.
That is a pretty basic approach but it can become much more sophisticated.
Summary rollups act as a persistent caching mechanism. You'll have to keep this up to date by some method - a cron job, MapReduce, etc. You dont want to compute all that data every time you need it - instead, compute it at regular intervals so that it is ready quickly.
Memcache is a great tool for caching but that caches data that has to be computed at some point anyway. Unfortunately, Memcache is not persistent. That means that if the memcached servier or service dies, so does your data.
You can explore some advanced cutting edge technologies such as MongoDB, CouchDB, Project Voldemort and neo4j for some even more efficient tools.
Id also recommend looking at the source code for the open source PHP based social network Elgg at http://www.elgg.org/
Facebook uses memcached to store SQL databases as distributed hash tables. That's probably your best bet.