I know this is well travelled ground but after thorough searching I still can't find a question that matches my requirements.
I'm looking to make a flexible system which allows users to define their own custom fields. E.g, some users of certain departments could require an NI number/separate passcode etc.
It currently uses MySQL (open to suggestions if this isn't ideal) a-top the CakePHP 3 Framework.
This works well with just a couple of tables so far (e.g, just users) but as the system grows many other entities will need their own custom fields too which will mean more tables which are basically the same.
I'm also concerned about data integrity - currently values go into a 'value' column which allows any data and the validation happens on the app which I appreciate might not be ideal.
There's obviously also a performance cost. Could I perhaps have an automatically created flat-file version like Magento does?
In light of this I did wonder about changing to a system where the custom data goes into a JSON array within the database but worry this might hold me back in future.
Any thoughts much appreciated.
What i would recommend is to go for a no-sql database like mongo-db. That would make it easier for handeling custom document fields. As far as fetching the data and sending to UI of the app , JSON is just fine (in case you are fetching from JS and not rendering in backend)
Related
I don't know if this question belongs here or not, someone please move it to an appropriate place if needed.
We are working on a web application using PHP and MySQL. The software is of the sort that provides a lot of pre-fed data to its users. For example, a list of questions and answers like a knowledge base. Now every user who registers into the system would have the liberty to add/update/delete this knowledge base, without affecting the data of the other users.
Now I understand that we would require to have a master copy of this pre-fed data, and would have to make a copy of this data available to users.
I was wondering how to implement this in the system without affecting the performance.
Would we have to create separate databases for each user?
Any pointers?
Thanks!
I find three approaches to this, they'd depend upon your domain requirement.
You're 'seeding' configurations and basic data for which it does make sense (to me) to localize the settings per user. I guess most of the apps follow this.
If it's domain data, when you say knowledge base (which I take to be very huge), it'd make more sense to save the per user edits and merge the master data with a user's personalized data. This is a very abstract & I wouldn't know it's implementation unless I actually see the data modeling, but then this looks a viable approach!
Save edits from all the users separately at one location (per collection or however you wish) if you want collaboration and stuff. With this, I think, it'd be easier to grow your knowledge base, although you can do the same with the previous approach with a little help from DBA!
I recently started working with Yii PHP MVC Framework. I'm looking for advice on how should I continue working with the database through the framework: should I use framework's base class CActiveRecord which deals with the DB, or should I go with the classic SQL query functions (in my case mssql)?
Obviously or not, for me it seems easier to deal with the DB through classic SQL queries, but, at some point, I imagine there has to be an advantage in using framework's way.
Some SQL queries will get pretty complex pretty often. I just can't comprehend how the framework could help me and not make things more complicated than they actually are.
Very General rule from my experience with Yii and massive databases:
Use Yii Active Record when:
You want to retrieve and post single to a few rows in the database (e.g. user changing his/her settings, updating users balance, adding a vote, getting a count of users online, getting the number of posts under a topic, checking if a model exists)
You want to rapidly design a hierarchical model structure between your tables, (e.g. $user->info->email,$user->settings->currency) allowing you to quickly adjust displayed currency/settings per use.
Stay away from Yii Active Record when:
You want to update several 100 records at a time. (too much overhead for the model)
Yii::app()->db->command()
allows you to avoid the heavy objects and retrieves data in simple arrays.
You want to do advanced joins and queries that involve multiple tables.
Any batch job!! (e.g. checking a payments table to see which customers are overdue on their payments, updating database values etc.)
I love Yii Active Record, but I interchange between the Active Record Model and plain SQL (using Yii::app()->db) based on the requirement in the application.
At the end I have the option whether I want to update a single users currency
$user->info->currency = 'USD';
$user->info->save();
or if I want to update all users currencies:
Yii::app()->db->command('UPDATE ..... SET Currency="USD" where ...');
In any language when dealing with the database a framework can help you by providing an abstraction over the database.
Here is a scenario I know I found myself in many times during my earlier development days:
I have an application that needs a database.
I write a ton of code.
I put the SQL statements in the code along with everything else.
The database changes somehow.
I'm stuck with having to go back and make 100 changes to all my SQL statements.
It's very frustrating.
Another scenario I found:
I write a ton of code against a database.
Bugs come in. Lots of bugs. I can't figure them all out.
I'm asked to write tests for my code.
This is impossible because all my code relies on a direct implementation of the database. How do you test SQL statements when they're with the actual code?
So my advice is to use the framework because it can provide an abstraction over the database. This gives you two really big advantages:
You can potentially swap out the database later and your code stays the same! If you're using interfaces/some framework, then most likely you're dealing with objects and not SQL statements directly. A given implementation might know how to write to MySQL or SQL Server, but in general your code just says "Write this object", "Read that list."
You can test your code! A good framework that deals with data will let you mock the database so you can test it easily.
Try to avoid writing SQL statements directly in the application. It'll save you pain later.
I'm unfamiliar with the database system bundled with Yii, but would advise you to use it a little bit to start with. My experience is with Propel, a popular PHP ORM. In general, ORM systems have a class per table (Propel has three per table).
Now, there'll probably be a syntax to do lookups and joins etc, but the first thing to do is to work out how to use raw SQL in your queries (for any of the CRUD operations). Put methods to do these queries in your model classes, so at least you will be benefitting from centralisation of code.
Once you've got that working, you can migrate to the recommended approach at a later time, without getting overwhelmed with the amount of material you have to learn in one go. Learning Yii (especially how to share code amongst controllers, and to write maintainable view templates) takes a while, so it may be sensible not to over-complicate it with many other things as well.
Why to use Yii:
Just imagine that you have many modules and for each module you have to write a pagination code; writing in old fashion style, will need a lot of time;
Why not use Yii ClistView widget? Oh, and this widget comes with a bonus: the data provider and the auto checking for the existance of the article that is about to be printed;
When using Yii CListView with results from ... Sphinx search engine, the widget will check if the article do really exists, because the result may not be correct
How long will it take for you to write a detection code for non existing registration?
And when you have different types of projects will you addapt the methods?
NO! Yii does this for you.
How long would it take for you to write the code in crud style ? create, read, update, delete ?
Are you going to adapt the old code from another project ?
Yii has a miracle module, called Gii, that generates models, modules, forms, controllers, the crud ... and many more
at first it might seem hard, but when you get experienced, it's easy
I would suggest you should use CActiveRecord.It will give many advantages -
You can use many widgets within yii directly as mentioned above.(For paginations,grids etc)
The queries which are generated by the Yii ORM are highly optimized.
You dont need to put the results extracted from SQLs in your VO objects.
If the tables for some reason modified(addition/deletion of column,changing data type), you just need to regenerate the models using the tool provided by yii.Just make sure you try to avoid doing any code changes in the models generated by yii, that will save your merging efforts.
If you plan to change the DB from MYSQL to other vendor in futur, it would be just config change for you.
Also you and your team would save your precious development time.
I'm staring to build a system for working with native languages, tags and such data in Yii Framework.
I already choose MongoDB for storing my data as I think it feets nicelly and will get better performance with less costs (the database will have huge amounts of data).
My question regards user authentication, payments, etc... This are sensitive bits of information and areas where I think the data is relational.
So:
1. Would you use two different db systems? Should I need them or I'm I complicating this?
2. If you recommend the two db approach how would I achieve that in Yii?
Thanks for your time!
PS: I do not intend this question to be another endless discussion between the relational vs non-relational folks. Having said that I think that my data feets mongo but if you have something to say about that go ahead ;)
You might be interested in this presentation on OpenSky's infrastructure, where MongoDB is used alongside MySQL. Mongo was utilized mainly for CMS-type data where a flexible schema was useful, and they relied upon MySQL for transactions (e.g. customer orders, payments). If you end up using the Doctrine library, you'll find that the ORM (for SQL databases) and MongoDB ODM share a similar API, which should make the experimentation process easier.
I wouldn't shy away from using MongoDB to store user data, though, as that's often a record that can benefit from embedded document storage (e.g. storing multiple billing/shipping addresses within a single user document). If anything, Mongo should be flexible enough to enable you to develop your application without worrying about schema changes due to evolving product requirements. As those requirements become more clear, you'll be able to make a decision based on the app's performance needs and types of database queries you end up needing.
There is no harm in using multiple databases (if you really need), many big websites are using multiple databases so go a head and start your project.
I have problem, I am creating quite complex form. Some parts of form are created dynamically. Lets say if you select certain option from a drop-down, extra fields gets injected to the form.
What approach would be best to store that data? I would like to try and get-away without using multiple tables. Because I makes the whole application so much more complex.
I was thinking of initializing all possible values as "0" in my model. And then overwrite them with post data, and just store the whole array in the table. Anyone see any problems with this approach?
The necessity of using multiple tables in your model doesn't depend on how much data (how many fields) you have to store - it depends on the logic of your model. So if there is a logical reason to use relationships in your model (f.e. 1:n, n:m) JUST DO IT!!!
If you will not follow the basic rules in creating your model and will try f.e. to store all the data in one table, although it should be divided into many tables, you will very soon regret it. Any change in your code in the future will cost you much more work and at some point you will not understand your own code and will have to write it again, this time following the rules ;)
And don't worry if the devoloping the right model costs a lot of work (lately I invested over two weeks in developing my model) - it really makes sense, because afterwards you can work much faster and more effectively with a well developed and planned model.
On the other hand there are situations, when storing over 100 and more fields in one table makes sense - it depends on the logic. So if you will provide some example, maybe one can say if you should work with one or more tables.
A lot depends on what you want to do with the form data later, and how often.
Serialized Single Field
In the simplest use cases you could base64_encode(serialize($data)) all the data and put that into a single column in the database.
Simple
Fast to insert
Easy to add/change input fields
Difficult AND Slow to search for values (particularly at scale)
Difficult to programmatically update should you need to make systematic changes to the data
Perfect if you always pull all of the data out of the db and never narrow your sql queries by data in the serialized string.
Metadata Table
Adding a second metadata table could offer a little more flexibility. The 2nd table would have a foreign key reference to the main form submissions, a metadata name, and the value. This allows a very flexible many to one relationship that you can easily store, search, and manipulate. You can see examples of this in wordpress.
2 tables, but still simple
Easy to add/change input fields
Much better searching via sql
Much easier to systematically update
Perfect if you don't always get all the data or have to narrow searches by the form data
And a different direction - You may also consider looking at Document based databases like MongoDB or CouchDB if you find yourself dealing with a lot of this type of data.
I do web work for a group at the university I attend. We’re having a design company redo our site and will be coding the site ourselves. We’d like to build the new site on top of a CMS so that people can easily add content. So far I’ve been looking into using Drupal. The problem I’m running into while thinking through how well Drupal will work for us is in regards to data collection. We’re overhauling our database to keep data centralized. For example, rather than create a table for every form with fields for First Name and Last Name (common fields for a lot of the forms we do) and all other fields, we’re pulling the common fields out to tables like a user table. This will help us track people’s involvement with our group much better and make our data far more useful. Is there any easy way to do this with Drupal modules? I’ve looked into CCK a bit but it doesn’t look like it’s useful for more than simple data that doesn’t have “complex” relationships between tables.
What are your recommendations? Are there some good modules I could use; do you recommend a different CMS (NOTE: needs to be php) that would better suite my needs?
I have the same exact question as you -- so far the best CMS I've come across for data collection from forms is Concrete5 (http://concrete5.org). Any time you use their "form" block, the submissions are sent to the "Reports" section on the back end of the dashboard. I don't love their organization method, but it's the best I've seen so far, and it also lets you dump to a file.
I think you want Drupal with Views and Organic Groups.
What kind of forms do you have?
You shouldn't use Drupal if you care about database structure. CCK is the method of defining content types, used on 60% of all Drupal sites and the basis of fields in Drupal 7 core.
That said, you may want to question a few of your assumptions. First, there are a lot of modules that add new field types to CCK, e.g. FileField, so it may not be as simple as you think. Second, do you really need to worry about how the database is organized, or is that just a means to some other end, e.g. reporting. If it's reporting you're after, you can generally handle that with Views, the only module used more than CCK.
If you really do need to worry about the database structure and really do need more complex relations than CCK can handle, I'd suggest you're not looking for a CMS so much as a framework. Those kinds of details are generally handled automatically in a CMS.