Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 6 years ago.
Improve this question
I'm rewriting a big website, that needs very solid architecture, here are my few questions, and pardon me for mixing apples and oranges and probably kiwi too:) I did a lot of research and ended up totally confused.
Main question: Which approach would you take in building a big website expected to grow in every way?
Single entry point, pages data in the database, pulled by associating GET variable with database entry (?pageid=whatever)
Single entry point, pages data in separate files, included based on GET variable (?pageid=whatever would include whatever.php)
MVC (Alright guys, I'm all for it, but can't grasp the concept besides checking all tutorials and frameworks out there, do they store "view" in database? Seems to me from examples that if you have 1000 pages of same kind they can be shaped by 1 model, but I'll still need to have 1000 "views" files?)
PAC - this sounds even more logical to me, but didn't find much resources - if this is a good way to go, can you recommend any books or links?
DAL/DAO/DDD - i learned about these terms by diligently reading through stack overflow before posting question. Not sure if it belongs to this list
Sit down and create my own architecture (likely to do if nobody enlightens me here:)
Something not mentioned...
Thanks.
Scalability/availability (iow. high-traffic) for websites is best addressed by none of the items you mention. Especially points 1 and 2; storing the page definitions in a database is an absolute no-no. MVC and other similar patterns are more for code clarity and maintenance, not for scalability.
An important piece of missing information is what kind of concurrent hits/sec are you expecting? Sometimes, people who haven't built high-traffic websites are surprised at the hit rates that actually constitute a "scalability nightmare".
There are books on how to design scalable architectures, so an SO post will not be able to the topic justice, but some very top-level concepts, in no particular order, are:
Scalability is best handled first by looking at hardware-based solutions. A beefy server with an array of SSD disks can go a long way.
Make static anything that can be static. Serve as much as you can from the web server, not the DB. For example, a lot of pages on websites dynamically generate data lists out of databases from data stores that very rarely or never really change.
Cache output that changes infrequently, and tune the cache refresh.
Build dynamic pages to be stateless or asynchronous. Look into CQRS and Event Sourcing for patterns that favor/facilitate scaling.
Tune your queries. The DB is usually the big bottleneck since it is a shared resource. Lots of web app builders use ORMs that create poor queries.
Tune your database engine. Backups, replication, sweeping, logging, all of these require just a little bit of resource from your engine. Tuning it can lead to a faster DB that buys you time from a scale-out.
Reduce the number of HTTP requests from clients. Each HTTP connect has overhead. Check your pages and see if you can increase the payload in each request so as to reduce the overall number of individual requests.
At this point, you've optimized the behavior on one server, and you have to "scale out". Now, things get very complicated very fast. Load-balancing scenarios of various types (sharding, DNS-driven, dumb balancing, etc), separating read data from write data on different DBs, going to a virtualization solution like Google Apps, offload static content to a big CDN service, use a language like Erlang or Scala and parallelize your app, etc...
Single entry point, pages data in the
database, pulled by associating GET
variable with database entry
(?pageid=whatever)
Potential nightmare for maintenance. And also for development if you have team of more than 2-3 people. You would need to create a set of strict rules for everyone to adhere to - effort that would be much better spent if using MVC. Same goes for 2.
MVC (Alright guys, I'm all for it, but
can't grasp the concept besides
checking all tutorials and frameworks
out there, do they store "view" in
database? Seems to me from examples
that if you have 1000 pages of same
kind they can be shaped by 1 model,
but I'll still need to have 1000
"views" files?)
It depends how many page layouts are there. Most MVC frameworks allow you to work with structured views (i.e. main page views, sub-views). Think of a view as HTML template for the web page. How many templates and sub-templates inside you need is exactly how many view's you'll have. I believe most websites can get away with up to 50 main views and up to 100 subviews - but those are very large sites. Looking at some sites I run, it's more like 50 views in total.
DAL/DAO/DDD - i learned about these
terms by diligently reading through
stack overflow before posting
question. Not sure if it belongs to
this list
It does. DDD is great if you need meta-views or meta-models. Say, if all your models are quite similar in structure, but differ only in database tables used and your views almost map 1:1 to models. In that case, it is a good time for DDD. A good example is some ERP software where you don't need a separate design for all the database tables, you can use some uniform way to do all the CRUD operations. In this case you could probably get away with one model and a couple of views - all generated dynamically at run-time using meta-model that maps database columns, types and rules to logic of programming language. But, please note that it does take some time and effort to build a quality DDD engine so that your application doesn't look like hacked-up MS Access program.
Sit down and create my own
architecture (likely to do if nobody
enlightens me here:)
If you're building a public-facing website, you're most likely going to do it well with MVC. A very good starting point is to look at CodeIgniter video tutorials. It helped me understand what MVC really is and how to use it way better than any HOWTO or manual I read. And they only take 29minutes altogether:
http://codeigniter.com/tutorials/
Enjoy.
I'm a fan of MVC because I've found it easier to scale your team when everything has a place and is nice and compartmentalized. It takes some getting used to, but the easiest way to get a handle on it is to dive in.
That said definitely check your local library to see if they have the O'Reilley book on scaling: http://oreilly.com/catalog/9780596102357 which is a good place to start.
If you're creating a "big" website and don't fully grasp MVC or a web framework then a CMS might be a better route since you can expand it with plugins as you see fit. With this route you can worry more about the content and page structure rather than the platform. As long as you pick the appropriate CMS.
I would suggest to create a mock app with some of the web mvc frameworks in the wild and pick one, with which your development was smooth enough. Establishing your code on a solid basis is fundamental, if you want to grasp concepts of mvc and be ready to add new functionality to your web easily.
Related
Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 11 years ago.
Improve this question
For the past weeks I can't stop thinking about the architecture of a CMS I have to develop shortly. First of all I want to answer to the possible first questions that you could ask me.
Q: Have you read the similar questions on StackOverflow?
A: Yes, I have. Unfortunately none applies to what I have to ask.
Q: Why would you want to code another CMS???
A: Well, this is a bit of a long answer. Long story short: it has to be closed-source and it has to meet the requirements I need (it won't be 100% a CMS, it's a much more delicate subject - any project developed with it will be somewhere between 60-70% CMS and the rest will be custom code for that project's specific needs)
Tools of the trade:
PHP
Zend Framework (my personal choice; I'm very familiar with it and I won't change it for this task whatsoever)
What I need this "CMS" to be
Developer oriented
Since it won't be a pure 100% CMS it has to be developer oriented - easy to maintain, easy to develop against (more like the feeling when developing on an enterprise level framework)
User/Editor oriented
While working on a project like this one, any programmer might find himself going in a wrong way and not thinking the person who'll work as an editor using this CMS is not in fact a very technical person. This is the part where conflicts happen - if it's not simple to use, yet powerful enough, you will have a lot of problems to deal with.
i18n & l10
I'm almost certain it will be kind of a difficult task to code something both developer and user oriented and if I won't achieve this, I would like to give more importance to the developer instead of the editor. I am aware it's not a good deal to ignore the actual user of the software, but this CMS has to be fast to develop against.
Possible architecture patterns
1. General object
The first architectural design that got me thinking was the following:
I define a general object. The user/admin goes in the administration area and defines the data object he needs. Easy one.
The object page (for example) has a title field, a body field and a slug. It was just defined by the user and now he can create content based on this "data structure". Seems pretty neat, but I still didn't solve some of this architecture's problems.
How will those dynamic objects will be stored in the database? Should I use a dataTypes table, an objects table and link them many to many via objectProperties table?
Or maybe should I serialize them and store everything in the objects table?
Should I give the user the possibility to create new dataType properties and add them to its objects?
How will I i18n all of this?
Isn't too complicated to force the user to define its own data structures?
Will it be too messy if it will have a lot of data structures defined, multiple languages? Would it be still manageable?
2. Extreme developer oriented - scaffold data structures
Today I found myself thinking about this possibility. It would be pretty neat to define the data structure of an object in a yaml or ini file and then scaffold it's database table, model and CRUD. Also mention it's relations to other "data structure" objects to get the right CRUD implementation (think about page data structure and category data structure, for example).
Unfortunately this made me also think about several possible problems.
Will I be able to code such a scaffolding tool on top of Zend Framework? (which is known to be lacking scaffolding if we except the 2 or 3 suggestions made by the community, like this one)
Is too stupid to create 2 tables for each object so I can internationalize it?
Would it be too strict on the user if he can't define new data structures without the programmer?
Conclusion
I'm still very confused on how to approach this subject and what architecture to choose. At this very moment both are pretty challenging in terms of development.
My questions are pretty straight-forward.
If you would have to choose one of this approaches, which would it be and why?
Can you maybe suggest a more interesting pattern to follow, considering my needs?
Just some general advice, if you are really trying to manage free-form content then I would stay away from relational databases for storing your data and go with an XML solution. A relational database has too much structure for something that is purely content oriented. Think of a home page... You have info displayed like: welcome notice, about us, who we are. That doesn't really map well to a table / collection of tables especially when you start adding / removing some of those items. Something with a defined structure, like stack overflow, does map well to a relation datbase however.
Take a look at Day CMS, Apache Sling, Java Content Repository for some ideas.
http://www.day.com/day/en.html
http://sling.apache.org/site/index.html
http://en.wikipedia.org/wiki/Content_repository_API_for_Java
From my point of view more options are always a problem. Users are mostly ignorant, when it comes to complex systems. Therefore I'd stick with the developer-oriented solution. Developer will decide what kind of content can be displayed. Optionally I would allow some kind of "open" content - for power users, allowing complex CSS/HTML/JS. Complex content like photo galleries, user profiles, etc. should not be designed by BFUs.
So to sum up - main feature - creating pages that can be dropped anywhere in the structure (that should be very flexible). If they want user profiles, you can create a new type of page. But at the end of the day, BFUs can do anything with enough time. It depends on the price/time scale. If they can pay for it and need it fast, they will make you create a new user profile page type, taht will be easy to fill. If they are kinda poor, they'll choose to setup all by themselves using normal page and WYSIWYG :D
I am creating the cms for a relatively simple site - portfolio, some general content pages, custom blog etc.
What are some of the best patterns to consider before diving into the design.
I want the system to be as flexible as possible without being too complex.
I have looked for some good resources that discus cms and blog design but can't find anything too good.
My language is php but I suppose I am looking for more language independent advice.
Flexibility without complexity... nice program.
Maybe you're a genius and you will make something that feet your needs. But I think the biggest problem you will face is security and robustness. So really, take other advices on this page and have a look at wordpress, drupal, joomla and ezpublish. A lot of security stuff is already done. And not only security...
So, study some of these tools, track their flaws, check their security policy. Study how they handle caching, sessions, bootstrap, absolute & relative url managment, documents (images, videos, etc), ajax, authentification, identification, acl, user interfaces, rich-text editing, migrations, templating, page composition, content filtering (I try to remove the things you won't need, plugins, database abstraction, fine caching, css and js minification, all the extra-complex stuff not needed for a single instance simple CMS). Soon you'll have a 'picture' of the stuff they've done.
By doing this work, you'll certainly notice some big differences, and mistakes. You'll start going on irc and flaming developpers, telling them that others have done better choices. You'll start forgetting to shave. You'll maybe do some contributions. Some will be accepted, others won't. Old core devs doesn't like when someone explain why they made mistakes (and they make mistakes).
Now, comes the day you have a beard. Some of your contributions will start looking like forks. You will have ennemies, and friends, or followers. And you will start feeling the force.
And you will go on irc and tell god that the world is ugly and that you'll make the first CMS which will be flexible without being complex. And people will cry. And birds will run in circles. And you will be able to explain what are the design pattern of a CMS.
I am a user. I know what I want. Doing what I want will make user happy. I'm happy.
You shall not trust code from people with glasses
"MVC MVC MVC" : and the people responds 'that shall be done'
Seriously, There's still a place for a good CMS with disruptive innovation, the fork history has started long time ago with phpNuke (as far as I can remember). But some of the actual products are really fine for most tasks.
I'm probably risking the reputation here, but my experience shows that building your own CMS can be a very justified decision, especially when you get familiar with current opensource systems and understand what exactly they lack in terms of features, security or what not. Open-source often means a lot of backward-compatibility concerns and bad architecture decisions that cannot be easily changed.
I strongly suggest that instead of just taking on MVC you take a look at ideas that make it attractive.
One main problem with CMSes is the range of technologies involved in driving dynamic web-sites: imperative php for logic, declarative SQL for data queries, markup HTML for interface, imperative/functional javascript for dynamic interface, JSON for ajax calls etc. To keep the system manageable you have to keep these technologies in a controlled and understandable environment, but yet allow for smooth integration. Knowledge and best practices are out there. MVC is but one approach to manage this problem.
My choice at the time was to use the following principles:
Object-oriented code with static calling (php is a one-run thing, many instances of code objects are rarely justified), nothing except for one line of init code in global context
100% code-design separation with the use of XSLT and custom content processor
Custom router that can take any http request and reroute it to registered methods
Custom content processor that can take arbitrary method output and convert it into any usable format such as xhtml, xml, json etc. based on the request parameters (i.e. http://local/class/method.xhtml, http://local/class/method.json)
One copy of code for as many virtual web servers as necessary
SQL query builder (chosen for flexibility over ORM) for all database queries
Mandatory filtering of method input with filter_* functions
I believe you can choose a few that you like :) And good luck!
A good pattern to start with is the Model View Controller pattern, or MVC.
This pattern suggests to seperate your application's logic in the following layers: data logic(model), manipulation or business logic (controller) and display logic (view).
This is a good pattern to start with as you'll run into other problems (and thus patterns) along the way.
The following website explains the MVC concept quite well: MVC Principles
There is no point reinventing the wheel unless you are trying to better it in anyway.
THere are a lot of CMS available already. I personally have worked with ezpublish. There are other options such as drupal etc. This is the list of all open source cms avaliable - Click here
If you are just trying to learn then you can perhaps pick any one of the popular opensource and work on them to find its architecture and design.
Besides, I dont think anyone can give you a list of design patterns that would be best for a CMS tool. Because each design pattern solves some particular problem. And, you just have to choose a design pattern depending on a specific problem you want to solve in your project.
These days, writing your own CMS is a horrible waste of time. The usual open source solutions -- these days Joomla, WordPress and Drupal are popular -- are written by thousands of people and while you might loose a little flexibility by using on that's ready made this is by far offset by not needing to redo everything from scratch. If you go with Drupal, you can also enjoy high quality, massively scalable etc code :)
If Your rquiremnt is portfolio, some general content pages, custom blog only, Wordpress will be simple and Better.
In PHP so many CMS available , most popular one is Joomla.
My question is in bold (see below) but I hope to provide some insight into my issues just in case it helps anyone who could answer my question.
I'm not sure which framework or language is the best for this job but I'd like to make an app that has two or three independently scrollable areas, similar to a 2 column template, that are all dynamically updated. Users can populate each list separately as they go and yah know ... I really don't want the app to move slowly in any way. It needs to be jerky with almost knee-jerk reactions to input.
I've read some interesting things about speeds for each language that I'm considering (php and python - possibly java or ruby) and well, I can't really decide for myself since I don't yet know what's going on to create any slowdowns.
The app would be very simple requiring basic information on a user and letting the user basically grow their own database of lists themselves. Some fancy things would happen on the site per the users input or list updates like color changes or pictures and numbers that change .. maybe a graph or something.
All that said, I hope my question(s) is simple and answerable:
What is a useful programming language and/or framework for making and handling user created hierarchical lists that would ultimately grow in complexity? And how does the language differ from the others for tasks like ones as described in this post?
Just trying to find the right/best/effective tool for this particular job.
By the way, I'm new to programming but have covered the basic tutorials for python (plus some django) and php via youtube mostly - I've got a few books in the queue. be gentle
I don't think you will conclusively be able to decide on a language for your task based on clear evidence of it outperforming others for similar task.
There's too many variables to factor in.
My take is, a well built app, whatever the technology would perform as well as the other one with different technologies, but built well. The difference wouldn't be much to make a fuss over.
Some things that are of note:
The client side code, HTML, CSS and Javascript would need to be top notch. You need to
Combine that with a good host, a nicely constructed and tuned database, a good lean method of communication back and forth between client and server.
Use gzip, caching, minifying and combining scripts, stylesheets, make fewer http requestes,
Architect the application with performance in mind from the get go.
If you are new to programming then the language/framework will be the least of your problems. You tried a little Python and some PHP. Which language did you find easier to grasp? Whichever one that was, pick it and just start writing.
As you work on your project and become a better programmer you can revisit the language/framework debate (although, to be honest, the programmer influences the site performance more than the framework).
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 3 years ago.
Improve this question
Me and my friend started working together as partners , we have decided to make Kick-as* website after website.
We have the ideas written down like 100's of them (yes we are choosing best and easy among them first).
My friend does the layout design and arranging things , and my part is coding and server management.
The little problem i am facing is lack of experience in planing a project. What i do is, I just start the code straight away and along with code I make DB, like when i need a table i make it.
I know this is very bad approach for a medium sized project.
Here at stackoverflow i saw lots of experienced coders. Need to learn a lot from you guys :) .
So can you plese help me on how to plan a project and what coding standard/structure/frameworks to be used (I do PHP code).
Thanks in advance.
Start by defining the scope. Write a paragraph to a page and try to describe your website. A top-down approach would be to start thinking of functionality you'd like to implement and refining it by adding more detail.
A top-down approach (is also known as step-wise design) is essentially the breaking down of a system to gain insight into its compositional sub-systems. In a top-down approach an overview of the system is first formulated, specifying but not detailing any first-level subsystems. Each subsystem is then refined in yet greater detail, sometimes in many additional subsystem levels, until the entire specification is reduced to base elements. A top-down model is often specified with the assistance of "black boxes", these make it easier to manipulate. However, black boxes may fail to elucidate elementary mechanisms or be detailed enough to realistically validate the model.
http://en.wikipedia.org/wiki/Top-down_and_bottom-up_design.
There are many other approaches, however.
http://en.wikipedia.org/wiki/Software_project_management#Software_development_process
Again, the most important step is to be able to put your idea in words; before you can adequately do that, I wouldn't even consider starting to write one line of code.
Regarding coding standard/structure/frameworks I recommend zend framework coding standard, MVC structure, and Zend Framework.
A brief guide for a MVC architecture. The idea is tho help you remember ideas (while your brain is steaming code) and have documents for future programmers.
Design the database. Make an ER diagram. Put it in a document.
Briefly describe reasons behind the design for the important issues (why you choose a polymorphic relation, why use this view, what selects you expect that are more trickey etc.). This will change as you code (and there is nothing you can do). Document the changes. To cope with the changes, use a versioning system for the database like rails migrations.
Design the structure of your website (sections, pages, links, page-flows etc.). Document it. (like: "the site 2 sections, this section is made of ..." etc.)
Based on this make a document describing your controllers and views. ( "a controller for articles, with a list view, and article view, also edit and create views but just for admins" etc).
Describe how you are going to enforce authentication and authorization (on controller and view level). Who is allowed where.
Describe how you protect against the main web-attacks (xss, csrf) where required. (example: "i escape all my view variables using htmlentities for xss protection and ...")
Design your models and side functionality (sending emails, background jobs etc.). These will be the bulk of code. Document each and describe the main issues (for example how timezones are to be handled, how this certain model is to connect to the currency service, how that model is to parse and manipulate some crone file, what algoritm you shall use to decide top 5 articles, depending on your app.) Describe what libraries you use, how, and for what purposes (example: "we are to use curl to scrap SO and make rss feed")
Describe how you protect against the main web-attacks where required (sql-injection, xss).
As you code, things change. Your knowledge about the discrete works of your system evolves and you start to improve the design based on the new found enlightenment. Document your changes.
A few thoughts from someone who loves abstractions.
Figure out what your websites will have in common. Once you have identified the main commonalities, look for frameworks or libraries that deal with as many of them as possible (besides filling your other criteria), such as the boilerplate DB code.
For the common features that you can't find any ready-to-use code (they always exist in customized websites) you'll have a chance to write a common library to be used across your websites yourself. This could be a template for your markup, a JavaScript library or a reusable server-side component, or all together.
Based on your description it sounds like you are enjoying great freedom in the creative process (versus getting a requirements spec handed to you and asked to implement it). I'd say don't over plan either, "just starting to code right away" is a lot of fun and not all bad. Experience will be your best friend, and by the time you reach website #100, you'll have lots of it. When building your second website, your experiences with the first one will help you avoid making some of the mistakes you made, and you'll discover new similarities that you did not anticipate in the planning phase. Just make sure to take the time, move the common code to a single library, and go back and edit your first website to use it. Do this a few times over and you will have learned lessons that are worth a lot.
Note: I am not into web programming, so forgive my ignorance in case the question is trivial. Also, please don't comment about "how flawed" the out-of-box comparison of these products is. The question is not about how they compete against each other, rather about the reason behind the incredible slowness of ALL of them.
Just read about a benchmark, where Joomla, Drupal, Wordpress, Plone3 & 4 had been tested. What shocked me is this: out of the box they performed around 4-14 pages/sec. How is this possible, why are they so damn slow? A CMS should just query a DB and churn out the data packed into nice templates. DBs are fast. Templates should be fast (text replacement, no big deal). Our machines are superfast and yet, these high profile CMSs could only produce a few pages/sec. How come?
A CMS should just query a DB and churn out the data packed into nice templates.
Not so much. Major, modern CMS systems are incredibly complex beasts. A typical page isn't merely body text and a title, but also dynamic category-based content queries that aggregate info across many site areas; not to mention security trimming and user-specific content zones. For example, loading http://www.volvogroup.com involves at least 7 of these queries, plus recursion through the site structure to generate navigation, and connecting to external systems to pull in news and investor relations data. Considering that, it shouldn't be such a surprise that it takes a beefy server farm to serve up several hundred hits per second.
Because it takes a alot of processing to do all of that. ITs not a matter of "query, replace, render". All of thes products are made to fit a wide range of use cases and to be extensible to some degree so really those 3 basic operations your are talking about are split up into many, many operations all of which consume time.
All things being equal - the more flexible he system the slower it will be "out of the box".
They are slow for a few reasons:
1 - Most of them are very modular,that means more files, more code and more DB queries.
2 - They largely (not wordpress so much) try to do everything, again designing a system for every possible situation makes it more complex and harder to tune.
3 - Most of them (currently) support both PHP4 and PHP5, this is again just extra work.
4 - They are allegedly made so non-technical users can use them, which means they often have to do things in a way that is not the most efficient, i.e. Drupals CCK / Views lets people that can't programme effectively create database tables and SQL queries, the flaw being these tables / queries are very general in design and are rather inefficient in comparison to custom coded efforts.
5 - They tend to use lots of DB queries, Drupal uses 40 or so for a very basic page and if you search their forums you will see reports of people claiming some pages make hundreds or even over a thousand queries.
They do of course offer caching and Drupal can get quite good performance from things like its boost module, the flaw being one of Drupal's (and Joomla's) selling points is you can make a community site, forum, Digg like site in it, all sites where caching is of limited use...
They are relatively complex systems. They allow a lot of hooks for plugins, so there's a lot of steps in the workflow from request to response.
In the real world, however, caching (whether in-application or opcode caching) is a tremendous boost to performance.
I'm not familiar with Plone, but the PHP CMSs essentially have to load and interpret almost the entire system with every single request.
Please don't take offense to this, but to preface your question by explaining your unfamiliarity with web programming, and then criticize the performance of what appears to you to be a 'simple' operation is a bit short-sighted.
I would encourage you to learn a bit more about the common problems a CMS solves, and the general theory and practice of how dynamic web pages and HTTP work. It's far from a simple I/O operation.
Also, for practical use, I would highly encourage anyone running a CMS to find a caching solution. Caching is intended to solve a lot of the 'speed' problems that arise in web technology. It should be part of any common web stack.
In my opinion :
Because CMS and frameworks think of all the things you need, that you can use:
like
Filter userinput
Create PDF,AJAX Output Template and alot more
It Depends on your need, what you realy need
Im not agree what you wrote
A CMS should just query a DB and churn out the data packed into nice templates.
A CMS Does a lot more things alot more ...
And at last but not least dont compare Desktop Software Speed with Wep Aplication.
There is a Big Difference
A CMS Does a lot more things alot more ...
And at last but not least dont compare Desktop Software Speed with Wep Aplication.
There is a Big Difference