Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
I don't know, if this type of question is already asked or not. Actually I don't know what to search for. Am asking at the right place?
Just as an example, I always wonder how the social media giants like Facebook manages their user settings module... What would be the database design and how they manage to hide the user updates on his friends' timeline if he has chosen not to show his updates on that particular friends timeline. I mean if I had been programming there then I would have loaded all the settings value in an array and there would be many conditional statements to check each and every user setting and accordingly printed data.
But I think this would make that code unmanageable because there would be so many conditions which could lead to undesired results.
So my question is, is there any better approach to do this?
I don't know I am making any sense here, but I tried to explain my question.
Facebook's data is maintained in document repository (Nosql) and efficient indexing is used to quickly find the tags and searches. This approach of search and data storage is markedly different from relational database based data storage and search.
Google also uses similar scheme to map the entire web and promptly give you back the result.
So in simpler terms you data is stored and indexed the way Google indexes messages, only difference is, the data is also lying with Facebook.
The related technologies are Bigdata, Mongodb, Apache Hadoop. And one of the leading index management and search algorithm is Lucene. Apache Elasticsearch is an user friendly package around Lucene.
So facebook treats these security critaria like tag (in simple language) and does google like search and presents you in a pleasing frontend, not sounding like a search engine.
While setting up your system, you can use elasticsearch to have faster search. Elasticsearch is makes implementation of lucene easier. It definitely will have some learning curve. Elasticsearch can also be used along with rdbms, in this case your data is saved in database but indexes also maintained to faster search. Definitely the cost would be disk-space. It makes it possible to have many criteria but still being able to get result quicker.
A quick tutorial on elasticsearch.
There would be many conditions to evaluate, that is correct. But in a SELECT statement you can easily compose all of those conditions in a WHERE clause which is very efficient.
Essentially, as long as you're comparing on equality, the database can easily optimize that, allowing it to quickly search for posts that fit the desired constraints. Even though there are a lot conditions, they don't really affect performance when compared to the fact that there are millions of entries in a table to be searched.
What your asking for is a result of really tough planing.. whenever you need to develop something that has a good potential to be complex you'll have to plan (Engineering) it well using known methodologies.
Usually the DB has many polymorphic relationships with entities, there are guys who are responsible of writing Query Procedures that should retrieve the wanted data fairly for the developers.
It's really not something you could come up with easy solution, the key here is planning, and planning good. there's no one right answer.
If your application is fairly small, you could just implement it your way, then you'll see what can be upgraded.. It's pretty much your only way to go. (BTW that's what most of statups are doing)
I wish you the best of luck.
Regarding facebook's db schema's and how it works and why its a good design, here are some articles that would explain to you why:
The power of the graph
This is posted by facebook and it explains how they are managing data. They use TAO data model and through the application of graph theory and other complicated algorithms and advanced memoray caching and data handling, they can efficiently manage lots of user data..
but regarding to your question: What would be the database design and how they manage to hide the user updates on his friends' timeline if he has chosen not to show his updates on that particular friends timeline?
I think this post would give you some insights on what kind of db structure facebook has and what would be the functionality of it for every user: Social Network Friends Relationship Database Design
Usually, the hiding of user updates on your friends' timeline if you have not shown your update to that particular friend is managed by storing values in the database.. you can create a view_type table in db and that would determine what kind of view the user can see, then issue a where condition in your sqls based on the view the user has selected.. there are still many ways to handle this and a good database structure is needed for this and of course planning for a good and efficient database is a very important and strict procedure..
Related
I'm having a hard time implementing a search feature for a web based system I’m working on, I first use MySQL Like with %wildcards%, but it not searching what I want to display, then I come upon Full Text index search, it search very good but has an issue on displaying joined multiple tables with foreign key which I don’t know workarounds, then I came along with MySQL with sphinx,
may I ask for any advice the best way/technologies to implement a search feature to search a Complex database tables
Check Apache Solr search server
Apache Solr official website
this technology will solve all your searching related problems
I guess the general answer here is you want a 'search index' - an index specifically for running searches. A repository that has all the required data to answer queries.
A RDBMS (like MySQL) is very good for Normalizing data, setting data up in a compact and easy to update format (ie minimise duplicate) - thats great for storage. But queries suffer as they have to do much more work to 'join' all the required data back.
... but for searching a denormalizaed structre may be best. (bigger, but easier - therefore quicker to 'search'.
There are many ways of doing that.
A materialized view as noted in your other thread php mysql full text search multiple table joined by id - keeps it all in mysql.
Using a external application. There are many examples, Lucene (variants include Solr and ElasticSearch), SphinxSearch, and many more.
This generally work in a similar way - setting up a dedicated copy of the data to make queries easier.
Use an external provider. Ther are many 'search as a service' systems (basically wrappers around the software mentioned in previous posts)
Building your own! Its possible to build a system yourself using just normal mysql tables. Basically an implementation of an inverted index will probably be the easiest.
Which you use is down to personal preference (eg, an external app is more work to setup, but overall is more powerful)
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
Greetings,
I would really love to hear about your broad knowledge about application design on this one!
I'm new to Phalcon, and now building my first MySQL & RESTful-API based application with it, but encountering some challenges along the way.
First, about my application design, the concept is as follows:
An API as a core for the application, wrapped by a "UI shell" of pages and views that utilize it.
The API is supposed to be composed of a set of Phalcon models that represent the DB and business logic, and over them, a component that acts as a layer that makes those models accessible as "HTTP services" - generally by translating requests to model names, and the HTTP verb to the appropriate model action (e.g: GET => $account->find()/findFirst(), PUT => $account->update([params]), etc.).
I was sure that the Phalcon models are gonna rid me of having to write most of my SQL, however, soon I came across some pretty common scenarios that the models couldn't handle the way I would expect:
You have entities like messages for example, and you wish to query them using a column of some other, related entity (like the FIRST NAME of the user that owns those messages). A model can't do this in a single operation.
I want to show a list of messages, each attached with the details of the user that sent it. In Phalcon, the first thought that comes to mind is taking advantage of the model relations feature, but thinking further I came to realize that will perform a full query for every message rendered, which is a disaster performance-wise, rather than retrieving them all together with their user details in some single, JOINed query.
I want to show a list of users, each with a total messages count. Found no other way to achieve that rather than a full query that includes a COUNT() field & GROUP BY or a sub-query.
I tried to look up such use cases and others, for most of them there hasn't seemed to be any elegant solution.
The Point:
What I want to achieve is a API-based architecture that makes sense, scalable and easily customizable to real world scenarios (being able to handle obvious situations like demonstrated above).
What would you do in Phalcon to handle the problems I encounter?
What do you think about the design concept I took? Is it somewhat standard and makes sense?
Most importantly, how would you design a full & flexible API without repeating cumbersome SQL queries everywhere, if at all?
Do you have any references or examples of known companies' approach on this?
That's a big question and any help will be huge!
Thanks!
Dor.
Little disclaimer first: There's no straightforward answer for this question(s), so Stackoverflow isn't the place for them. But since you have put it nicely I'll try to help :)
Almost any modern CMS API has a very similar approach that is very successful and flexible, you can follow their guidelines to implement your own API. I'd recommend storageroom and contentful documentation pages, they have a series of gotchas and patterns that I've successfully added to my own projects eventually.
Phalcon is a good choice for a RESTfull API and a proof of that is the PhalconEye project, a cms built using Phalcon. Once you have designed your API, go check the PhalconEye source code to get some samples on how to implement such things with Phalcon.
Good luck!
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
I've been using Symfony for close to 2 years now, so far, each project I build is deployed specifically for each client (i.e one client, one codebase, one db).
Lets say I have a Project Management app that I want to deploy for many clients. Assuming the clients will go with whatever features I build into the system, if I deploy a different codebase (therefore, a different db) for each client, here are the problems I foresee:
Pushing bug fixes and upgrades will be painful. I need to push it to every repository that I have deployed. It won't scale well if I have 50 clients using that same app.
Management is painful. How do I build an admin system for myself where I can pull ALL the projects into one HTML table? After all, each client has their own database, right? For me to do anything meaningful with all the records of all my clients, I need a way to look through all their databases at one go, which.. I don't think Symfony allows. (I'm not sure)
User account issues. If a user happens to work for multiple companies, all of them using my Project Management app, that user has to sign up multiple times. (I know this can be circumvented if I use oauth, but I'm trying not to go there if I can)
Here are the solutions I have thought up and tried to a certain extent.
Solution 1
One database and one codebase for ALL my clients. Projects will go under one table, Invoices go under one table, all marked by their own client_id. Users can be assigned to Projects so there is no need to sign up multiple times.
This is not that hard to create. But, what happens if different clients need different columns for their Invoices? My Invoice table will keep expanding (with different fields that different clients want), and each row can potentially contain many null fields. Not to mention, my Invoice entity will grow in file size, and I will have to update the database schema every time a new customization comes in.
Solution 2
One database where each client has their own table prefix. So for Client A, I could use clientA_projects, clientA_invoices, clientA_configuration etc.
This is ideal if each client wants to customize their fields. But, does this mean I need to create new entity and form classes for each new client that comes into the system? It looks like with this solution, I need to update the database schema with every new client I get.
Currently, I am experimenting with schema-less databases (mongo and couch), hoping that without needing to specify the table schema upfront, I can implement Solution 1 without hassle. But I'm still experimenting, and there are ways to go before I dare to deploy a production-ready app, being unfamiliar with the issues of mongo and couch with Symfony.
So, this is where I am stuck at. Being a self-trained programmer, I feel I have a lot of holes in my knowledge that requires filling (as opposed to someone from a CS background). There aren't many places on the web talking about Symfony 2 and multi-tenancy (maybe I'm looking for the wrong thing). If anyone of can point me to a clearer direction, maybe best practices, example projects, I will really appreciate it!
Btw, I plan to execute this in the latest version of Symfony (2.3.2 at this moment).
Thanks in advance guys.
I'm also using Symfony2 for similar amount of time (since one of BETAs) and I advise you to go with solution #1. If you are going SaaS you can't give out code to clients for the reasons you wrote (problems with updates / upgrades mainly). The whole hassle will be on the user management - which user has access to which data, belongs to which group, company and so on. All other things if done properly will be coded the same user-agnostic way. What should you do with different requirements for different companies? Make such features configurable. You can implement this on various levels:
simple entity attributes: have an attributes field in each table and save everything as JSON, YAML or other dynamic-structurable content,
general configuration: have one place where entity base configuration is stored (in the means I wrote above) and allow users to manage new features from there, all changes are propagated to simple entities,
implement something what I call Entity Parameters Pattern - design database tables which would contain parameter types, parameter values and relation to other entities on different levels and then make generic configurable parameter types which can be applied to any place with predefined meaning. For example "preferred_season" being a parameter of type "choice_string" containing configuration "spring,summer,autumn,winter" and when attached to given entity would always render a <select> field with choices and save selected value with relation to both entity and parameter type.
Also the solution #1 has one unbeatable advantage - it could handle more that one company even if you wanted to give out the code at the end. You'd just need to mask the ability to add more. :)
This question is tagged Symfony2 but it really shouldn't. It doesn't matter what framework you're using, you should abstract your application design from code and then use frameworks as a mere tool for doing the job smoothly. I'd like to say that even taking previous sentence into account, I'm absolutely in love with Symfony2. :)
I know this is an older question but can be usefull for others.
I agree with #Tomasz and solution #1 with one database – all tenants in one database. The biggest problem here is proper database design to solve further security issues: access for resources must be controlled by application to prevent unauthorized access between tenants. On the other side we have ease implementation as we are implementing single application with only one database.
Nice article about Symfony2 and moving to SaaS model:
http://www.browserlondon.com/blog/2015/01/moving-to-a-saas-model-with-symfony2/
Also "must read" article about designing database in SaaS platform - patterns that are platform independent:
http://labs.octivi.com/database-design-in-saas-platforms/
Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 11 years ago.
Improve this question
For the past weeks I can't stop thinking about the architecture of a CMS I have to develop shortly. First of all I want to answer to the possible first questions that you could ask me.
Q: Have you read the similar questions on StackOverflow?
A: Yes, I have. Unfortunately none applies to what I have to ask.
Q: Why would you want to code another CMS???
A: Well, this is a bit of a long answer. Long story short: it has to be closed-source and it has to meet the requirements I need (it won't be 100% a CMS, it's a much more delicate subject - any project developed with it will be somewhere between 60-70% CMS and the rest will be custom code for that project's specific needs)
Tools of the trade:
PHP
Zend Framework (my personal choice; I'm very familiar with it and I won't change it for this task whatsoever)
What I need this "CMS" to be
Developer oriented
Since it won't be a pure 100% CMS it has to be developer oriented - easy to maintain, easy to develop against (more like the feeling when developing on an enterprise level framework)
User/Editor oriented
While working on a project like this one, any programmer might find himself going in a wrong way and not thinking the person who'll work as an editor using this CMS is not in fact a very technical person. This is the part where conflicts happen - if it's not simple to use, yet powerful enough, you will have a lot of problems to deal with.
i18n & l10
I'm almost certain it will be kind of a difficult task to code something both developer and user oriented and if I won't achieve this, I would like to give more importance to the developer instead of the editor. I am aware it's not a good deal to ignore the actual user of the software, but this CMS has to be fast to develop against.
Possible architecture patterns
1. General object
The first architectural design that got me thinking was the following:
I define a general object. The user/admin goes in the administration area and defines the data object he needs. Easy one.
The object page (for example) has a title field, a body field and a slug. It was just defined by the user and now he can create content based on this "data structure". Seems pretty neat, but I still didn't solve some of this architecture's problems.
How will those dynamic objects will be stored in the database? Should I use a dataTypes table, an objects table and link them many to many via objectProperties table?
Or maybe should I serialize them and store everything in the objects table?
Should I give the user the possibility to create new dataType properties and add them to its objects?
How will I i18n all of this?
Isn't too complicated to force the user to define its own data structures?
Will it be too messy if it will have a lot of data structures defined, multiple languages? Would it be still manageable?
2. Extreme developer oriented - scaffold data structures
Today I found myself thinking about this possibility. It would be pretty neat to define the data structure of an object in a yaml or ini file and then scaffold it's database table, model and CRUD. Also mention it's relations to other "data structure" objects to get the right CRUD implementation (think about page data structure and category data structure, for example).
Unfortunately this made me also think about several possible problems.
Will I be able to code such a scaffolding tool on top of Zend Framework? (which is known to be lacking scaffolding if we except the 2 or 3 suggestions made by the community, like this one)
Is too stupid to create 2 tables for each object so I can internationalize it?
Would it be too strict on the user if he can't define new data structures without the programmer?
Conclusion
I'm still very confused on how to approach this subject and what architecture to choose. At this very moment both are pretty challenging in terms of development.
My questions are pretty straight-forward.
If you would have to choose one of this approaches, which would it be and why?
Can you maybe suggest a more interesting pattern to follow, considering my needs?
Just some general advice, if you are really trying to manage free-form content then I would stay away from relational databases for storing your data and go with an XML solution. A relational database has too much structure for something that is purely content oriented. Think of a home page... You have info displayed like: welcome notice, about us, who we are. That doesn't really map well to a table / collection of tables especially when you start adding / removing some of those items. Something with a defined structure, like stack overflow, does map well to a relation datbase however.
Take a look at Day CMS, Apache Sling, Java Content Repository for some ideas.
http://www.day.com/day/en.html
http://sling.apache.org/site/index.html
http://en.wikipedia.org/wiki/Content_repository_API_for_Java
From my point of view more options are always a problem. Users are mostly ignorant, when it comes to complex systems. Therefore I'd stick with the developer-oriented solution. Developer will decide what kind of content can be displayed. Optionally I would allow some kind of "open" content - for power users, allowing complex CSS/HTML/JS. Complex content like photo galleries, user profiles, etc. should not be designed by BFUs.
So to sum up - main feature - creating pages that can be dropped anywhere in the structure (that should be very flexible). If they want user profiles, you can create a new type of page. But at the end of the day, BFUs can do anything with enough time. It depends on the price/time scale. If they can pay for it and need it fast, they will make you create a new user profile page type, taht will be easy to fill. If they are kinda poor, they'll choose to setup all by themselves using normal page and WYSIWYG :D
I do a bit of HTML/CSS for fun but I am looking at picking up some programming skills.
I have been reading up on PHP and MySQL. So far I have not found it too hard understanding the concepts such as loops, condition statements etc but I want to get stuck in and start developing an app before I get too bored reading and giving up completely.
My idea...
I happen to be studying for an exam at the moment and have these practice exams on paper. I thought why not put this into an app so I can take the exam on the computer.
Features:
The app can hold multiple exams
It can randomise questions or display them in order
Have the option to display answer for each question
Time the duration to complete the exam
Admin page to add new exams and questions/answers
Ok, you guys can stop laughing now, I know this is one step above Hello World but thought I'd make a start somewhere!
I will develop this in PHP/MYSQL or maybe RoR if you guys think its better for a beginner. I think I will be ok reading/writing to the DB but I'm not so sure on site structure, DB design and generally best to way to do it.
If I have an exam made up of 50 questions, each given an ID. If I delete one question how do I update the rest with new IDs? I.e. if I delete question 3, all following questions need to shift their ID back by 1.
EDIT:
How would I represent an instance of one exam in my database schema? The exam and question relationship will be one to many, since questions will be unique to a single exam. But since I am aiming to have multiple exams, how do I represent that?
How would I represent an instance of one exam in my database schema? The exam and question relationship will be one to many, since questions will be unique to a single exam. But since I am aiming to have multiple exams, how do I represent that?
OK, first of all, welcome :)
Second of all: yours is not a stupid idea at all, in fact, you have chosen the best way to learn how to program: do something useful and fun.
To get to your question: why do you need consecutive IDs for the questions? Database-wise, you just need to have different IDs per row.
If it is a display requirement, why don't you assign a number over a loop during display? So, for example when you do a foreach to display the questions you can keep an index variable which you increment at each loop, and use that to display the ordinal.
Hope this is what you are looking for, good luck!
Your biggest thing here, in my opinion, will be the structure of your database. You're going to want to read up a bit about db-design, and normalisation. Basically you'll want to logically plan out the different entities that exist in your application:
Exams
Exam Takers
Questions
Answers
And some of these will be tied to eachother:
Questions -> Exams
Takers -> Exams
Answers -> Questions
Whenever you have situations like this, you need to determine whether the relationship is one-to-one, one-to-many, or many-to-many. Questions could belong to one exam, or many. That's up for you to decide. If a Question belongs to only a single Exam, you can create a field in your Question table to hold an ExamID. If a Question can belong to many exams, you'll create a new table with two fields: QuestionID, ExamID. This builds your many-to-many relationship.
In all honesty, this could be a very large and difficult undertaking for somebody who has very little programming experience. In my opinion, you should start with something much simpler, like a guestbook or a mini-blog.
Regardless what application you decide to build, SO is here for you all along the way.
If I have an exam made up of 50 questions, each given an ID. If I delete one question how do I update the rest with new IDs? I.e. if I delete question 3, all following questions need to shift their ID back by 1.
Primary IDs shouldn't ever change. When you delete one row in your table, the corresponding primary ID just disappears, but there should be no renumbering of the other row primary keys.
If you decide to go with Rails, check out Ryan Bates' excellent screencast site Railscasts.
A specific application is an excellent way to learn programming, even if such applications already exist (and they do - but I would avoid looking at those, just create and learn!).
One of your specifications is that the questions can be displayed in any order. Another is that they can be removed. So, it's clear that the numbering is not directly related to the question - it should be generated dynamically once the set of questions, and their order in the test, has been established. You need to think through these requirements, which provide the basis for design of the components. The user interface should be insulated from the program logic, which should be insulated from the database of questions. Try to pass information between these parts in the most primitive form possible. That way, when you want to redesign one of the layers, you won't have to change the whole thing.
I agree with all the other answers, implementing an useful application is a great way to learn web development. But, as Jonathan said, it's not a very simple application that you are planning to build. That said, here are my thoughts:
Ruby on Rails is a framework for Ruby. It can also utilize MySQL. PHP is a programming language (as ruby is for RoR), and provides no framework. There are quite a few for PHP though, CodeIgniter, Zend Framework, Symfony or CakePHP to name a few.
Frameworks implement often-used aspects of an application in a flexible way, so they can be used for various applications. Some of these aspects may be authentication, database access, caching, template rendering and a lot more.
Frameworks are usually written to speed up application development, if you choose php, you might want to try to write your own framework, if your primary goal is learning. You should still take a look at the other frameworks and try them out to see what they are and how they work.
I would recommend to follow through a few tutorials for one or two frameworks, and to research everything you don't already know (MVC, Coding patterns, etc.). I find the english wikipedia very helpful.
The other part of your application will be the database. Some frameworks offer some kind of ORM, which will "shield" the database access from you, once you have set up your schema. I would still recommend to take a look at how to query a database, because you then know what the framework does when it retrieves data from the database.
This seems to be a good resource for an intro to the mysql-server, and contains some links to sql-tutorials (for querying the db-server). And I would recommend this tutorial on how to create your database schema.
I hope some of these resources are useful to you and wish you good luck.