I'm working on making a consumer CRM system for my boot-strapped startup, where we'll use MySQL. We're moving from an old paper and pen method of tracking leads and referrals, to a digital method for our dealers.
The database will have a standard fields, like lead name, spouse, jobs, referral type, referrer, and lead dealer. This is easy, almost child's play.
Now is the part I'm having a hard time figuring it out. I want to track all the attempted contact dates and responses, and appointments that have been set or reset. The system is going to be web-based, with the front-end in PHP.
I thought about doing nested tables, but I don't want to use Oracle or PostgreSQL, as I like the familiar setup of MySQL.
For the sake of feasibility, say I have 4,000 leads, and each lead is going to be called on average 30 times. So I'll have 120,000 data points to track.
Would it be advisable to:
Make a two dimensional PHP array in the field, to keep track of these metrics.
Have a contact table with all 120k in it, that the application pulls when these metrics are needed
Have a contact table for each lead, which keeps track of all needed metrics
I would make one table for contacts. Add a column to record whether the contact was successful or not.
I would also use MySQL's table partitioning by lead, if many of the queries will be to report on specific leads.
But I second the comment from #Bryan Agee that you should consider carefully before implementing a CRM system from scratch on your weekends.
Start with the table of just the leads. Ideally, it should be filterable and searchable and sortable. Look into the jquery datatables plugin. You can have a table that's paged and pulls its data using AJAX from the server. That way you only need to query and return a few records at a time.
Then create a second table that pops up when the user clicks on the contact. This one is also AJAX and displays the contact history for that particular contact.
This way you never have to query and return the full list, especially if you have 4000, which would be a pain not only for the server but for the people using the system.
Have a contact table for each lead, and add data to it every time action(Contact) is made. It will also give you count and other metrics and it will be easy to implement and track.
Related
I have a database containing many tables : [followers, favorites, posts ...etc)
These tables define the different activities a user can achieve, he can send posts, add other people to favorites and follow others.
What I want to do.. is to extract data from these tables, and build a real-time news feed .
I have two options:
1- Creating a separate table for the notifications (so that I won't have to get data from multiple tables, then using a Javascript timer to return results every x seconds.
2- using XMPP server...that sends (or pushes) notifications every x second, without actually sending any ajax queries.
And for these two options, I don't know whether I should connect to these tables to get news feed, or just create a separate table for notifications.
I searched in the subject, but I didn't find something really helpful yet, Any links will be appreciated.
If your data is normalized, you should be able to pull all the data with one query (using JOINs), or you could try creating a View if you want to query from one table. It's always best to keep your data in the appropriate tables to avoid having to duplicate data.
The push notifications are easier on the server, since they're not receiving requests from each client. Depending on your load, you could probably get away with simple AJAX requests.
The request for news feed will be very frequently. So you must keep your code run fast and take as less resource (CPU-time, database query) as possible.
I suggest you to take the first option. It meets your requirement and is simple enough.
Because you have many tables, all of them will grow bigger day by day. Every time you connect them to get news feed will take a long time and increase the load of your database. In the other hand, you query sql will be complex.
Like #Curtis Mattoon said: avoid having to duplicate data, but sometime, we need spend more space for less time.
So I suggest to create a new table to store the notification data. You even can delete the old data from this table periodically.
At the same time, your sql and php code for news feed will be simple and run fast.
I'm new to web development and database design, and I'm kind of stumped as how to best accomplish a simple review system for items.
In the current database schema I have a table, call it tbl_item, that has columns for different properties of items. I want users to be able to review items and associate each review in the tbl_reviews to a particular item.
Of course I have a foreign key set up referencing an id column in tbl_item but I do not know where to go from here. Basically my question is: What should calculate the review average?
Should the application make a SQL call every time a review score is requested for a particular item, where the DB would have to then search through all the tbl_reviews rows to find those with a particular item_id?
(That seems wrong.) Should the DB get involved and have some type of calculated field or view or stored procedure that does the same?
Should I have a new column in tbl_item that has the average score in it and is updated whenever any new review corresponding to a particular item is CRUD'ded?
If it matters, I'm using Yii (PHP) and MySQL.
Basically you're asking about efficiency and math.
Here's what I would do:
Your DB is relational. Good, you got that. Each review has a numerical value? Like 1 - 10?
Say it does for this example.
I would say that upon each review, the review itself is set in the DB as well as a queue in an action table. Something that has the item id and a type of action. In this case review.
You then have a cron running in the background every minute or so checking that action queue and in the event of a new review or set of reviews, you run an algorithm for each applicable item that collects all of the data available on the review and returns an educated number based on the standard deviation of the collective data.
This way the math is not run in realtime by the user or when a review is sent. For all we know you have tons of items and tons of reviews, so real time would be bad if your intelligence script is heavy.
As for standard deviation, I check a large variety of things for anti-spam. I store all userdata, IP, datetime, and anything else I can to make sure it's not just one guy logging in with different accounts reviewing his own things with a 10 rating each time. Can't fall for that.
Plus, if you get 100 10 reviews that look legit and 1 review with a score of 1 you can discount it as a hater and just ignore it in the results.
You have to understand your request is enormous, so code snippets are out of the question here.
What I just explained was like 4 months of work for a huge client and a serious anti-spam calculator.
good luck though
I am currently working on a big management system (in PHP using Zend Framework, but that is not really revelant of the solution to this question) in which I have to manage multiple entries. Each entry has many fields and span on two tables in a 1-to-many relationship (through a single foreign key). There is roughly 50 fields in the first table and 30 fields in the second one.
I am now at the stage to implement a history tracking of the different modifications made by users (and some automated tasks). Each entry might enventually be rolled back partially or totally to a previous value.
I was thinking about using a system similiar to the one present in the CMS Typo3. One table to manage the whole history with the following fields
history_id
entry_id
entry_table
last_modifcation_timestamp
last_modification_user
data
The data would be "serialized" in a json or xml format.
My concerns through this method is that overtime, the content of the history table would grow exponentially. To overcome this issue, I was thinking that i could make a new database to manage this history every years and then show history data by years to the users.
I am looking for advice about ways to improve this solution and ease the implementation. Any advice or documentation to help me will be welcome
I'd add a threshold and remove or dump to an external file all entries older than a certain period of time.
I finished my meeting site design and now I should create database. But I am not sure which method to use.I am using Apache, PHP and MySQL. I have users, each user has input and output mail box and each user has a friends list. This is the basic scenario. And here I have three questions.
For in box and out box should I create a new table for each person? Or will 1 big in box and out box table and identifying for each mail who is sender and receiver be enough? If there will be 1 million message for each user searching and showing his mail boxes will cause a slow down?
For creating friend list , also creating one table that stores all both user relationship as a one row data who is a friend with whom ? also searching table for each user friends will cause slow down?
And I am using Mysql, but says that Oracle most power full. For these reason which one should I choose?
Maybe you can think how many person should register to my site and force to database but who knows ))
I'll take a couple of your questions:
For in box and out box should I create for each person new table? or 1 big in box and out box table and identifying for each mail who is sender and receiver will be enough?
Use one table. This has many advantages, such as being able to easily query all users' messages in a single query. For example to search for all users that have sent more than x messages you can use GROUP BY(userid) and COUNT each group. If you have all the data in one table then it is simple, but if you have a table for each user you will have to generate the SQL for this query dynamically.
If the table grows too large you can partition it, for example you could partition on userid, or year of post, or whatever else makes sense for your data distribution.
If there will be 1 million message for each user searching and showing his mail boxes will cause a slow down?
It depends. If you index the table and search using the indexes then it will be fine. If you write LIKE '%foo%' then it will be slow. Make sure that all your searches use indexes.
I'll leave the other questions for other people to answer.
I think you need to firstly identify all the information you want in the database (names, email addresses, mail messages...) and apply the rules of normalisation to identify your entities and attributes.
Data modeling tools such as Oracle's SQL Developer Data Modeler can help progress that into an implementation.
Don't be scared of 'large' data volumes. Databases cope very well with tables of millions of rows (and orders of magnitude above that).
Oracle vs MySQL depends a lot of money and implementation. You'll find a lot of cheap mySQL hosting. Oracle will be trickier to find and more expensive.
I have a site where the users can view quite a large number of posts. Every time this is done I run a query similar to UPDATE table SET views=views+1 WHERE id = ?. However, there are a number of disadvantages to this approach:
There is no way of tracking when the pageviews occur - they are simply incremented.
Updating the table that often will, as far as I understand it, clear the MySQL cache of the row, thus making the next SELECT of that row slower.
Therefore I consider employing an approach where I create a table, say:
object_views { object_id, year, month, day, views }, so that each object has one row pr. day in this table. I would then periodically update the views column in the objects table so that I wouldn't have to do expensive joins all the time.
This is the simplest solution I can think of, and it seems that it is also the one with the least performance impact. Do you agree?
(The site is build on PHP 5.2, Symfony 1.4 and Doctrine 1.2 in case you wonder)
Edit:
The purpose is not web analytics - I know how to do that, and that is already in place. There are two purposes:
Allow the user to see how many times a given object has been shown, for example today or yesterday.
Allow the moderators of the site to see simple view statistics without going into Google Analytics, Omniture or whatever solution. Furthermore, the results in the backend must be realtime, a feature which GA cannot offer at this time. I do not wish to use the Analytics API to retrieve the usage data (not realtime, GA requires JavaScript).
Quote : Updating the table that often will, as far as I understand it, clear the MySQL cache of the row, thus making the next SELECT of that row slower.
There is much more than this. This is database killer.
I suggest u make table like this :
object_views { object_id, timestamp}
This way you can aggregate on object_id (count() function).
So every time someone view the page you will INSERT record in the table.
Once in a while you must clean the old records in the table. UPDATE statement is EVIL :)
On most platforms it will basically mark the row as deleted and insert a new one thus making the table fragmented. Not to mention locking issues .
Hope that helps
Along the same lines as Rage, you simply are not going to get the same results doing it yourself when there are a million third party log tools out there. If you are tracking on a daily basis, then a basic program such as webtrends is perfectly capable of tracking the hits especially if your URL contains the ID's of the items you want to track... I can't stress this enough, it's all about the URL when it comes to these tools (Wordpress for example allows lots of different URL constructs)
Now, if you are looking into "impression" tracking then it's another ball game because you are probably tracking each object, the page, the user, and possibly a weighted value based upon location on the page. If this is the case you can keep your performance up by hosting the tracking on another server where you can fire and forget. In the past I worked this using SQL updating against the ID and a string version of the date... that way when the date changes from 20091125 to 20091126 it's a simple query without the overhead of let's say a datediff function.
First just a quick remark why not aggregate the year,month,day in DATETIME, it would make more sense in my mind.
Also I am not really sure what is the exact reason you are doing that, if it's for a marketing/web stats purpose you have better to use tool made for that purpose.
Now there is two big family of tool capable to give you an idea of your website access statistics, log based one (awstats is probably the most popular), ajax/1pixel image based one (google analytics would be the most popular).
If you prefer to build your own stats database you can probably manage to build a log parser easily using PHP. If you find parsing apache logs (or IIS logs) too much a burden, you would probably make your application ouput some custom logs formated in a simpler way.
Also one other possible solution is to use memcached, the daemon provide some kind of counter that you can increment. You can log view there and have a script collecting the result everyday.
If you're going to do that, why not just log each access? MySQL can cache inserts in continuous tables quite well, so there shouldn't be a notable slowdown due to the insert. You can always run Show Profiles to see what the performance penalty actually is.
On the datetime issue, you can always use GROUP BY MONTH( accessed_at ) , YEAR( accessed_at) or WHERE MONTH(accessed_at) = 11 AND YEAR(accessed_at) = 2009.