I finished my meeting site design and now I should create database. But I am not sure which method to use.I am using Apache, PHP and MySQL. I have users, each user has input and output mail box and each user has a friends list. This is the basic scenario. And here I have three questions.
For in box and out box should I create a new table for each person? Or will 1 big in box and out box table and identifying for each mail who is sender and receiver be enough? If there will be 1 million message for each user searching and showing his mail boxes will cause a slow down?
For creating friend list , also creating one table that stores all both user relationship as a one row data who is a friend with whom ? also searching table for each user friends will cause slow down?
And I am using Mysql, but says that Oracle most power full. For these reason which one should I choose?
Maybe you can think how many person should register to my site and force to database but who knows ))
I'll take a couple of your questions:
For in box and out box should I create for each person new table? or 1 big in box and out box table and identifying for each mail who is sender and receiver will be enough?
Use one table. This has many advantages, such as being able to easily query all users' messages in a single query. For example to search for all users that have sent more than x messages you can use GROUP BY(userid) and COUNT each group. If you have all the data in one table then it is simple, but if you have a table for each user you will have to generate the SQL for this query dynamically.
If the table grows too large you can partition it, for example you could partition on userid, or year of post, or whatever else makes sense for your data distribution.
If there will be 1 million message for each user searching and showing his mail boxes will cause a slow down?
It depends. If you index the table and search using the indexes then it will be fine. If you write LIKE '%foo%' then it will be slow. Make sure that all your searches use indexes.
I'll leave the other questions for other people to answer.
I think you need to firstly identify all the information you want in the database (names, email addresses, mail messages...) and apply the rules of normalisation to identify your entities and attributes.
Data modeling tools such as Oracle's SQL Developer Data Modeler can help progress that into an implementation.
Don't be scared of 'large' data volumes. Databases cope very well with tables of millions of rows (and orders of magnitude above that).
Oracle vs MySQL depends a lot of money and implementation. You'll find a lot of cheap mySQL hosting. Oracle will be trickier to find and more expensive.
Related
I'm building a aweber-like list management system (for phone numbers, not emails).
There are campaigns. A phone number is associated with each campaign. Users can text to a number after which they will be subscribed.
I'm building "Create a New Campaign" page.
My current strategy is to create a separate table for each campaign (campaign_1,campaign_2,...,campaign_n) and store the subscriber data in it.
It's also possible to just create a single table and add a campaign_id column to it.
Each campaign is supposed to have 5k to 25k users.
Which is a better option? #1 or #2?
Option 2 makes more sense and is widely used approach.
I suppose it really depends on the amount of campaigns you're going to have. Let's give you some pros/cons:
Pros for campaign_n:
Faster queries
You can have each instance run with its own code and own database
Cons for campaign_n:
Database modifications are harder (you need to sync all tables)
You get a lot of tables
Personally I'd go for option 2 (campaign_id field), unless you have a really good reason not to.
I'm working on making a consumer CRM system for my boot-strapped startup, where we'll use MySQL. We're moving from an old paper and pen method of tracking leads and referrals, to a digital method for our dealers.
The database will have a standard fields, like lead name, spouse, jobs, referral type, referrer, and lead dealer. This is easy, almost child's play.
Now is the part I'm having a hard time figuring it out. I want to track all the attempted contact dates and responses, and appointments that have been set or reset. The system is going to be web-based, with the front-end in PHP.
I thought about doing nested tables, but I don't want to use Oracle or PostgreSQL, as I like the familiar setup of MySQL.
For the sake of feasibility, say I have 4,000 leads, and each lead is going to be called on average 30 times. So I'll have 120,000 data points to track.
Would it be advisable to:
Make a two dimensional PHP array in the field, to keep track of these metrics.
Have a contact table with all 120k in it, that the application pulls when these metrics are needed
Have a contact table for each lead, which keeps track of all needed metrics
I would make one table for contacts. Add a column to record whether the contact was successful or not.
I would also use MySQL's table partitioning by lead, if many of the queries will be to report on specific leads.
But I second the comment from #Bryan Agee that you should consider carefully before implementing a CRM system from scratch on your weekends.
Start with the table of just the leads. Ideally, it should be filterable and searchable and sortable. Look into the jquery datatables plugin. You can have a table that's paged and pulls its data using AJAX from the server. That way you only need to query and return a few records at a time.
Then create a second table that pops up when the user clicks on the contact. This one is also AJAX and displays the contact history for that particular contact.
This way you never have to query and return the full list, especially if you have 4000, which would be a pain not only for the server but for the people using the system.
Have a contact table for each lead, and add data to it every time action(Contact) is made. It will also give you count and other metrics and it will be easy to implement and track.
I see many implementations such as the Facebook like, forum karma, mark as read on forum posts and other simple options and selections available to multiple users on a given item.
I know I can implement this in mysql by creating a table which links say post IDs to liker user IDs for say, a like system.
My problem is, on a page with lots of posts, I will have to make a lookup for every post. I use prepared statements so that makes it faster for me.
Is there another way to implement these systems, if not, are there optimisations like database types or other tweaks that can make this faster?
Basically, is there a powerful, fast implementation of a many to many database interaction.
*EDIT***
I'm using opera mini and so I have issues with the ajax and js for commenting
Right now, I have a table with two columns. One for user id and the other for post id. Both are indexed and are used in foreign key constraints.
I'm thinking of making a compound primary key across the two.
My main issue is for the karma. I allow users to vote on each post. The problem is, for each post, I need to get the total votes, determine if a user has voted to either allow the user to or not to vote.
My site allows many users to host their own sites and so I need to seriously optimize this.
Someone suggested I use memory tables for this.
NOTE**
I can't use memcached.
I strongly suggest using something else than a MySQL db. I've written an opensocial app which had both heavy writes and reads to a database. It all started with a MySQL DB, I even switched to a dedicated master slave replication setup. But to no avail, it was expensive and it didn't scale very well.
The final solution was to use a NoSQL db which made the most out of RAM. My decision was mongoDB which has an activy community and solved my problem very well. MongoDB proofed to be highly scalable.
Still a little hazy about what you got so far, but I'll start it off and keep adding stuff if need be:
Make sure you're Indexing
Minimum lookups -so you get the list
of posts that will pop up, use that
list to match the like's, if they've viewed the article etc.
Using numbers - make sure all your
comparisons are with numbers
If you're running queries, don't run a single query for each post
Is there a limit in your query? - make sure you use that
De-normalization is not a sin
You can partition your databases to decrease lookups (e.g. if data is older than 60 days and barely touched, move it to a secondary database/table, so the size of your table is not huge)
e.g. SELECT * FROM user_liked WHERE post_id IN (1,2,3)
instead of
SELECT * FROM user_liked WHERE post_id = 1
Philipp Keller wrote a bunch of articles on tag systems based on MYSQL a few years ago. Just as Like-ing, Tagging is establishing a many-to-many relationship between a thing (tag, article being liked) and a user. The logic in his articles should be directly applicable to your problem as well.
Check out the comments as well.
http://www.pui.ch/phred/archives/2005/04/tags-database-schemas.html
Database Schemas for Tagging solutions
http://www.pui.ch/phred/archives/2005/05/tags-with-mysql-fulltext.html
Abusing the MySQL FULLTEXT indices for tagging and tag search (requires MyISAM, I'd not go there).
http://www.pui.ch/phred/archives/2005/06/tagsystems-performance-tests.html
Performance Tests of tagging systems
I have been creating a web app and am looking to expand. In my web app I have a table for users which includes privileges in order to track whether a user is an administrator, a very small table for a dynamic content section of a page, and a table for tracking "events" on the website.
Being not very experienced with web application creation, I'm not really sure about how professionals would create systems of databases and tables for a web application. In my web app, I plan to add further user settings for each member of the website and even a messaging system. I currently use PHP with a MySQL database that I query for all of my commands, but I would be willing to change any of this if necessary. What would be the best wat to track content such as messages that are interpersonal and also specific user settings for each user. Would I want to have multiple databases at any point? Would I want to have multiple tables for each user, perhaps? Any information on how this is done or should be done would be quite helpful.
I'm sorry about the broadness of the question, but I've been wanting to reform this web app since I feel that my ideas for table usage are not on par with those that experienced programmers have.
Here's my seemingly long, hopefully not too convoluted answer to your question. I think I've covered most, if not all of your queries.
For your web app, you could have a table of users called "Users", settings table called "UserSettings" or something equally as descriptive, and messages in "PrivateMessages" table. Then there could be child tables that store extra data that is required.
User security can be a tricky thing to design and implement. Do you want to do it by groups (if you plan on having many users, making it easier to manage their permissions), or just assign individually due to a small user base? For security alone, you'd end up with 4 tables:
Users
UserSettings
UserGroups
UserAssignedGroups
That way you can have user info, settings, groups they can be assigned to and what they ARE assigned to separated properly. This gives you a decent amount of flexibility and conforms to normalization standards (as mentioned above by DrSAR).
With your messages, don't store them with the username, but rather the User ID. For instance, in your PrivateMessages table, you would have a MessageID, SenderUserID, RecipientUserID, Subject, Body and DateSent to store the most basic info. That way, when a user wants to check their received messages, you can query the table saying:
SELECT * FROM PrivateMessages WHERE RecipientUserID = 123556
A list of tables for your messages could be as such:
PrivateMessages
MessageReplies
The PrivateMessages table can store the parent message, and then the MessageReplies table can store the subsequent replies. You could store it all in one table, but depending on traffic and possibly writing recursive functions to retrieve all messages and replies from one table, a two table approach would be simplest I feel.
If I were you, I'd sit down with a pencil and paper, and write down/draw what I want to track in my database. That way you can then draw links between what you want to store, and see how it will come together. It helps me when I'm trying to visualise things.
For the scope of your web app you don't need multiple databases. You do need, however, multiple tables to store your data efficiently.
For user settings, always use a separate table. You want your "main" users table as lean as possible, since it will be accessed (= searched) every time a user will try to log in. Store IDs, username, password (hashed, of course) and any other field that you need to access when authenticating. Put all the extra information in a separate table. That way your login will only query a smaller table and once the user is authenticated you can use its ID to get all other information from the secondary table(s).
Messages can be trickier because they're a bigger order of magnitude - you might have tens or hundreds for each user. You need to design you table structure based on your application's logic. A table for each user is clearly not a feasible solution, so go for a general messages table but implement procedures to keep it to a manageable size. An example would be "archiving" messages older than X days, which would move them to another table (which works well if your users aren't likely to access their old messages too often). But like I said, it depends on your application.
Good luck!
Along the lines of Cristian Radu's comments: you need to split your data into different tables. The lean user table will (in fact, should) have one unique ID per user. This (unique) key should be repeated in the secondary tables. It will then be called a foreign key. Obviously, you want a key that's unique. If your username can be guaranteed to be unique (i.e. you require user be identified by their email address), then you can use that. If user names are real names (e.g. Firstname Sirname), then you don't have that guarantee and you need to keep a userid which becomes your key. Similarly, the table containing your posts could (but doesn't have to) have a field with unique userids indicating who wrote it etc.
You might want to read a bit about database design and the concept of normalization: (http://dev.mysql.com/tech-resources/articles/intro-to-normalization.html) No need to get bogged down with the n-th form of normalization but it will help you at this stage where you need to figure out the database design.
Good luck and report back ;-)
I have a pretty large social network type site I have working on for about 2 years (high traffic and 100's of files) I have been experimenting for the last couple years with tweaking things for max performance for the traffic and I have learned a lot. Now I have a huge task, I am planning to completely re-code my social network so I am re-designing mysql DB's and everything.
Below is a photo I made up of a couple mysql tables that I have a question about. I currently have the login table which is used in the login process, once a user is logged into the site they very rarely need to hit the table again unless editing a email or password. I then have a user table which is basicly the users settings and profile data for the site. This is where I have questions, should it be better performance to split the user table into smaller tables? For example if you view the user table you will see several fields that I have marked as "setting_" should I just create a seperate setting table? I also have fields marked with "count" which could be total count of comments, photo's, friends, mail messages, etc. So should I create another table to store just the total count of things?
The reason I have them all on 1 table now is because I was thinking maybe it would be better if I could cut down on mysql queries, instead of hitting 3 tables to get information on every page load I could hit 1.
Sorry if this is confusing, and thanks for any tips.
alt text http://img2.pict.com/b0/57/63/2281110/0/800/dbtable.jpg
As long as you don't SELECT * FROM your tables, having 2 or 100 fields won't affect performance.
Just SELECT only the fields you're going to use and you'll be fine with your current structure.
should I just create a seperate setting table?
So should I create another table to store just the total count of things?
There is not a single correct answer for this, it depends on how your application is doing.
What you can do is to measure and extrapolate the results in a dev environment.
In one hand, using a separate table will save you some space and the code will be easier to modify.
In the other hand you may lose some performance ( and you already think ) by having to join information from different tables.
About the count I think it's fine to have it there, although it is always said that is better to calculate this kind of stuff, I don't think for this situation it hurt you at all.
But again, the only way to know what's better your you and your specific app, is to measuring, profiling and find out what's the benefit of doing so. Probably you would only gain 2% of improvement.
You'll need to compare performance testing results between the following:
Leaving it alone
Breaking it up into two tables
Using different queries to retrieve the login data and profile data (if you're not doing this already) with all the data in the same table
Also, you could implement some kind of caching strategy on the profile data if the usage data suggests this would be advantageous.
You should consider putting the counter-columns and frequently updated timestamps in its own table --- every time you bump them the entire row is written.
I wouldn't consider your user table terrible large in number of columns, just my opinion. I also wouldn't break that table into multiple tables unless you can find a case for removal of redundancy. Perhaps you have a lot of users who have the same settings, that would be a case for breaking the table out.
Should take into account the average size of a single row, in order to find out if the retrieval is expensive. Also, should try to use indexes as while looking for data...
The most important thing is to design properly, not just to split because "it looks large". Maybe the IP or IPs could go somewhere else... depends on the data saved there.
Also, as the socialnetworksite using this data also handles auth and autorization processes (guess so), the separation between login and user tables should offer a good performance, 'cause the data on login is "short enough", while the access to the profile could be done only once, inmediately after the successful login. Just do the right tricks to improve DB performance and it's done.
(Remember to visualize tables as entities, name them as an entity, not as a collection of them)
Two things you will want to consider when deciding whether or not you want to break up a single table into multiple tables is:
MySQL likes small, consistent datasets. If you can structure your tables so that they have fixed row lengths that will help performance at the potential cost of disk space. One thing that from what I can tell is common is taking fixed length data and putting it in its own table while the variable length data will go somewhere else.
Joins are in most cases less performant than not joining. If the data currently in your table will normally be accessed all at the same time then it may not be worth splitting it up as you will be slowing down both inserts and quite potentially reads. However, if there is some data in that table that does not get accessed as often then that would be a good candidate for moving out of the table for performance reasons.
I can't find a resource online to substantiate this next statement but I do recall in a MySQL Performance talk given by Jay Pipes that he said the MySQL optimizer has issues once you get more than 8 joins in a single query (MySQL 5.0.*). I am not sure how accurate that magic number is but regardless joins will usually take longer than queries out of a single table.