I am using PHP, MySQL to develop a website to be used in an educational institution.
One function of this is to allocate lectures when creating a batch. To allocate lectures to the starting batch, system will prompt available lectures based on their availability and qualifications. Then the course coordinator will take the decision.
My problem is how to check the availability of the lecture on a particular week day given time slot.(time slot is varying not fixed durations)
I am planning to keep lecture schedules in a table where it shows lecture_Id, Batch_Id, day, start_time, end_time, start_day , end_day.
Then when availability checking I need to write a complex query to check the available lecture_Id's. I couldn't figure out it yet.
Is there any other smart ways to do this?
Thanks
The way you're proposing to store the info is the first way I'd think of doing it - the complex query isn't so bad though.
So if I understand correctly, you have lectures stored in another table, with a one - to -many relationship with your lecture_schedules table?
If so, this will get the lecture details for a given time range.
Something like this:
SELECT * FROM lecture_schedules
INNER JOIN lectures USING(lecture_id)
WHERE start_day<=DAY(yourstarttime) AND end_day>=DAY(yourendtime)
AND start_time<=yourstarttime AND end_time>=yourendtime;
Note you will need to edit table names and column names to reflect your actual schema, and replace yourstarttime and yourendtime with the time range values.
Hope this helps
EDIT:
This query makes several assumptions about the columns and datatypes of those columns in your schema - don't just copy & paste and expect it to work first time :)
Related
I have a question as to a better way of doing this as I have a very large database with a lot symbols. "Hence a, aa... etc"
I would like to know if I can actually query every table also desc order would be nice. In one line. Otherwise I will have to type thousands of unions and it will be a pain later as the database will be changed often. As a table is erased and another joins it place.
Every table has the Date column and would like to search based on a date.
Thank you in advance.
I.E.
SELECT * from a where Date = '2017-07-31' union
SELECT * from aa where Date = '2017-07-31' union
SELECT * from aaap where Date = '2017-07-31' union
SELECT * from aabvf where Date = '2017-07-31' union
I mean, you COULD....
SELECT * FROM a,aa,aaap,aabvf WHERE date='2017-07-21'
Ahmed helped me out. As to why my data structure is like that. Well. If you have better suggested I'm opening to it. So.
Why.
Basically I have data in the form of symbols
I.E. A, AA that are stock tickers
They have dates that are unique keys to open, high, low, various other stock measurements.
So why I would want to grab just a single date. It's basically the top date or "today" to display and chart. So I can do various other things with the data.
If you have another method of storing I'm open.
I written a java program (not normally a web developer) that mines the data and that form and stores how I suggested. Which I could change, if you have a better way. I would love to hear. Also. If you have opinion on how to store data faster with MySQL I would love to hear. Currently I have few hundred threads that basically store data. Each thread handles a symbol. It creates a table if it doesn't exist with the ticker name and puts its data in columns separated date (unique key) open, high, etc... also various other operations the incoming data and stores that. Thank you for the answer and thank you if you have a better method !
Ps sorry I didn't mean chart. I display the top date as a table with corresponding data attached!
I have a table with 8000 Vendors (Think a Seller on Amazon).
Each Vendor has around 100 items in their purchase history with columns such as price, tax, date.
Each Vendor has columns such as State, Region, Type.
I am creating a Html Table to display Vendors Purchases History by month. However, I also want to compare it to Average by State, Region, and Overall.
My question is, how should I store these aggregate averages. Should I create tables for each one, store them in cache (Redis), calculate in mysql, or calculate in php?
I am using Laravel on the backend. The main problem i have with storing them in their own table is a don't want a bunch of tables like purchase_histroy_state with columns date, state_id, and total. Currently, the server times out when I try to calculate with php.
8,000 vendors x 100 items is 800,000 items that you're asking PHP to trawl through, calculate and store in memory every time it is ran.
If you're wanting a quick and dirty solution - then consider setting up a SQLite database with a denormalized structure, and appropriate indexes. Databases are designed to do this sort of stuff - even SQLite.
I wouldn't recommend this as a long term solution though as it's not really scalable - but it might help in the early development stages to figure out what you need.
I am having the query with database.
In my case each user will have daily 5 records to save in a table.
So in 10 days, their will be 50 records for one user.. I have 50000 users the count of record goes to 50000*5=250000 records per day.
If I want to retrieve a particular record for a particular day for a particular user, I have to traverse through these many records.Is it a right practice?
If not, What is the solution for this?
I would suggest you to create indexes in the user and date columns, you can see details in the link suggested by tausif. Also I would recommend you to avoid your queries with "select * from ...", you should specify the columns you need in each query rather than a start (*) to retrieve all the columns.
Can you provide more details of the scenario you have? Is the database in a server? which database technology are you using? Is the data saved for a long period of time?
I would suggest you to start looking to indexes one or two of you columns, but maybe is not the right approach for your particular solution.
I need to store about 73,200 records per day consisting of 3 points of data: id, date, and integer.
Some members of my team suggest creating tables using month's as the table name (september_2010), while others are suggesting having one table with lots of data in it...
Any suggestions on how to deal with this amount of data? Thanks.
========== Thank you to all the feedback.
I recommend against that. I call this antipattern Metadata Tribbles. It creates multiple problems:
You need to remember to create a new table every year or else your app breaks.
Querying aggregates against all rows regardless of year is harder.
Updating a date potentially means moving a row from one table to another.
It's harder to guarantee the uniqueness of pseudokeys across multiple tables.
My recommendation is to keep it in one table until and unless you've demonstrated that the size of the table is becoming a genuine problem, and you can't solve it any other way (e.g. caching, indexing, partitioning).
Seems like it should be just fine holding everything in one table. It will make retrieval much easier in the future to maintain 1 table, as opposed to 12 tables per year. At 73,200 records per day it will take you almost 4 years to hit 100,000,000 which is still well within MySQLs capabilities.
Absolutely not.
It will ruin relationship between tables.
Table relations being built based on field values, not table names.
Especially for this very table that will grow by just 300Mb/year
so in 100 days you have 7.3 M rows, about 25M a year or so. 25M rows isn't a lot anymore. MySQL can handle tables with millions of rows. It really depends on your hardware and your query types and query frequency.
But you should be able to partition that table (if MySQL supports partitioning), what you're describing is an old SQL Server method of partition. After building those monthly tables you'd build a view that concatenates them together to look like one big table... which is essentially what partitioning does but it's all under-the-covers and fully optimized.
Usually this creates more trouble than it's worth, it's more maintenance , your queries need more logic, and it's painful to pull data from more than one period.
We store 200+ million time based records in one (MyISAM) table, and queries are blazingly still fast.
You just need to ensure there's an index on your time/date column and that your queries makes use of the index (e.g. a query that messes around with DATE_FORMAT or similar on a date column will likely not use an index. I wouldn't put them in separate tables just for the sake of retreival performance.
One thing that gets very painful with such a large number of records is when you have to delete old data, this can take a long time (10 minutes to 2 hours for e.g. wiping a month worth of data in tables with hundreds of mullions rows). For that reason we've partitioning the tables, and use a time_dimension(see e.g. the time_dimension table a bit down here) relation table for managing the periods instead of simple date/datetime columns or strings/varchars representing dates.
Some members of my team suggest creating tables using month's as the table name (september_2010), while others are suggesting having one table with lots of data in it...
Don't listen to them. You're already storing a date stamp, what about different months makes it a good idea to split the data that way? The engine will handle the larger data sets just fine, so splitting by month does nothing but artificially segregate the data.
My first reaction is: Aaaaaaaaahhhhhhhhh!!!!!!
Table names should not embed data values. You don't say what the data means, but supposing for the sake of argument it is, I don't know, temperature readings. Just imagine trying to write a query to find all the months in which average temperature increased over the previous month. You'd have to loop through table names. Worse yet, imagine trying to find all 30-day periods -- i.e. periods that might cross month boundaries -- where temperature increased over the previous 30-day period.
Indeed, just retrieving an old record would go from a trivial operation -- "select * where id=whatever" -- would become a complex operation requiring you to have the program generate table names from the date on the fly. If you didn't know the date, you would have to scan through all the tables searching each one for the desired record. Yuck.
With all the data in one properly-normalized table, queries like the above are pretty trivial. With separate tables for each month, they're a nightmare.
Just make the date part of the index and the performance penalty of having all the records in one table should be very small. If the size of table really becomes a performance problem, I could dimply comprehend making one table for archive data with all the old stuff and one for current data with everything you retrieve regularly. But don't create hundreds of tables. Most database engines have ways to partition your data across multiple drives using "table spaces" or the like. Use the sophisticated features of the database if necessary, rather than hacking together a crude simulation.
Depends on what searches you'll need to do. If normally constrained by date, splitting is good.
If you do split, consider naming the tables like foo_2010_09 so the tables will sort alphanumerically.
what is your DB platform?
In SQL Server 2K5+ you can partition on date.
My bad, I didnt notice the tag. #thetaiko is right though and this is well within MySQL capabilities to deal with this.
I would say it depends on how the data is used. If most queries are done over the complete data, it would be an overhead to always join the tables back together again.
If you most times only need a part of the data (by date), it is a good idea to segment the tables into smaller pieces.
For the naming i would do tablename_yyyymm.
Edit: For sure you should then also think about another layer between the DB and your app to handle the segmented tables depending on some date given. Which can then get pretty complicated.
I'd suggest dropping the year and just having one table per month, named after the month. Archive your data annually by renaming all the tables $MONTH_$YEAR and re-creating the month tables. Or, since you're storing a timestamp with your data, just keep appending to the same tables. I assume by virtue of the fact that you're asking the question in the first place, that segregating your data by month fits your reporting requirements. If not, then I'd recommend keeping it all in one table and periodically archiving off historical records when performance gets to be an issue.
I agree with this idea complicating your database needlessly. Use a single table. As others have pointed out, it's not nearly enough data to warrent extraneous handling. Unless you use SQLite, your database will handle it well.
However it also depends on how you want to access it. If the old entries are really only there for archival purposes, then the archive pattern is an option. It's common for versioning systems to have the infrequently used data separated out. In your case you'd only want everything >1 year to move out of the main table. And this is strictly an database administration task, not an application behavior. The application would only join the current list and the _archive list, if at all. Again, this highly depends on the use case. Are the old entries generally needed? Is there too much data to process regularily?
Say you've got a database like this:
books
-----
id
name
And you wanted to get the total number of books in the database, easiest possible sql:
"select count(id) from books"
But now you want to get the total number of books last month...
Edit: but some of the books have been
deleted from the table since last month
Well obviously you cant total for a month thats already past - the "books" table is always current and some of the records have already been deleted
My approach was to run a cron job (or scheduled task) at the end of the month and store the total in another table, called report_data, but this seems clunky. Any better ideas?
Add a default column that has the value GETDATE(), call it "DateAdded". Then you can query between any two dates to find out how many books there were during that date period or you can just specify one date to find out how many books there were before a certain date (all the way into history).
Per comment: You should not delete, you should soft delete.
I agree with JP, do a soft delete/logical delete. For the one extra AND statement per query it makes everything a lot easier. Plus, you never lose data.
Granted, if extreme size becomes an issue, then yeah, you'll potentially have to start physically moving/removing rows.
My approach was to run a cron job (or scheduled task) at the end of the month and store the total in another table, called report_data, but this seems clunky.
I have used this method to collect and store historical data. It was simpler than a soft-delete solution because:
The "report_data" table is very easy to generate reports/graphs from
You don't have to implement special soft-delete code for anything that needs to delete a book
You don't have to add "and active = 1" to the end of every query that selects from the books table
Because the code to do the historical reporting is isolated from everything else that uses books, this was actually the less clunky solution.
If you needed data from the previous month then you should not have deleted the old data. Instead you can have a "logical delete."
I would add a status field and some dates to the table.
books
_____
id
bookname
date_added
date_deleted
status (active/deleted)
From there you would be able to query:
SELECT count(id) FROM books WHERE date_added <= '06/30/2009' AND status = 'active'
NOTE: It my not be the best schema, but you get the idea... ;)
If changing the schema of the tables is too much work I would add triggers that would track the changes. With this approach you can track all kinds of things like date added, date deleted etc.
Looking at your problem and the reluctance in changing the schema and the code, I would suggest you to go with your idea of counting the books at the end of each month and storing the count for the month in another table. You can use database scheduler to invoke a SP to do this.
You have just taken a baby step down the road of history databases or data warehousing.
A data warehouse typically stores data about the way things were in a format such that later data will be added to current data instead of superceding current data. There is a lot to learn about data warehousing. If you are headed down that road in a serious way, I suggest a book by Ralph Kimball or Bill Inmon. I prefer Kimball.
Here's the websites: http://www.ralphkimball.com/
http://www.inmoncif.com/home/
If, on the other hand, your first step into this territory is the only step you plan to take, your proposed solution is good enough.
The only way to do what you want is to add a column to the books table "date_added". Then you could run a query like
select count(id) from books where date_added <= '06/30/2009';