I need to create an application with an editable database structure. Where you can add/delete/modify tables and fields, views and structure of the database. All in production and real time.
The purpose is that the application can be adapted to the needs of the company. Allowing you to store the information that is needed, where it is needed.
I use laravel 5 and MySQL, but my question is not about my software. My questions are:
Is there a methodology, or a set of steps to follow, to achieve this functionality?
And if it exists, is there any package to apply it to laravel?
Entity-attribute-value model allows to have a DB with something like a "dynamic schema" and be able to run indexed queries on its tables (though "tables" become different from what you would have if you used the normal approach). With it you can add and remove fields and have the values indexed (unlike in a document-oriented NoSQL DB). Downsides: a lot of joins, performance might suffer; however, I've seen pretty large systems get away with it. Don't know if and how it can be applied in Laravel context, but googling gives at least some results.
You need to use Business Intelligence and reporting tools, which fulfil all your needs. They run with any db, no matter which fields you add or remove they will adjust themselves. One of the best example is:-
https://github.com/getredash/redash
Related
I recently started working with Yii PHP MVC Framework. I'm looking for advice on how should I continue working with the database through the framework: should I use framework's base class CActiveRecord which deals with the DB, or should I go with the classic SQL query functions (in my case mssql)?
Obviously or not, for me it seems easier to deal with the DB through classic SQL queries, but, at some point, I imagine there has to be an advantage in using framework's way.
Some SQL queries will get pretty complex pretty often. I just can't comprehend how the framework could help me and not make things more complicated than they actually are.
Very General rule from my experience with Yii and massive databases:
Use Yii Active Record when:
You want to retrieve and post single to a few rows in the database (e.g. user changing his/her settings, updating users balance, adding a vote, getting a count of users online, getting the number of posts under a topic, checking if a model exists)
You want to rapidly design a hierarchical model structure between your tables, (e.g. $user->info->email,$user->settings->currency) allowing you to quickly adjust displayed currency/settings per use.
Stay away from Yii Active Record when:
You want to update several 100 records at a time. (too much overhead for the model)
Yii::app()->db->command()
allows you to avoid the heavy objects and retrieves data in simple arrays.
You want to do advanced joins and queries that involve multiple tables.
Any batch job!! (e.g. checking a payments table to see which customers are overdue on their payments, updating database values etc.)
I love Yii Active Record, but I interchange between the Active Record Model and plain SQL (using Yii::app()->db) based on the requirement in the application.
At the end I have the option whether I want to update a single users currency
$user->info->currency = 'USD';
$user->info->save();
or if I want to update all users currencies:
Yii::app()->db->command('UPDATE ..... SET Currency="USD" where ...');
In any language when dealing with the database a framework can help you by providing an abstraction over the database.
Here is a scenario I know I found myself in many times during my earlier development days:
I have an application that needs a database.
I write a ton of code.
I put the SQL statements in the code along with everything else.
The database changes somehow.
I'm stuck with having to go back and make 100 changes to all my SQL statements.
It's very frustrating.
Another scenario I found:
I write a ton of code against a database.
Bugs come in. Lots of bugs. I can't figure them all out.
I'm asked to write tests for my code.
This is impossible because all my code relies on a direct implementation of the database. How do you test SQL statements when they're with the actual code?
So my advice is to use the framework because it can provide an abstraction over the database. This gives you two really big advantages:
You can potentially swap out the database later and your code stays the same! If you're using interfaces/some framework, then most likely you're dealing with objects and not SQL statements directly. A given implementation might know how to write to MySQL or SQL Server, but in general your code just says "Write this object", "Read that list."
You can test your code! A good framework that deals with data will let you mock the database so you can test it easily.
Try to avoid writing SQL statements directly in the application. It'll save you pain later.
I'm unfamiliar with the database system bundled with Yii, but would advise you to use it a little bit to start with. My experience is with Propel, a popular PHP ORM. In general, ORM systems have a class per table (Propel has three per table).
Now, there'll probably be a syntax to do lookups and joins etc, but the first thing to do is to work out how to use raw SQL in your queries (for any of the CRUD operations). Put methods to do these queries in your model classes, so at least you will be benefitting from centralisation of code.
Once you've got that working, you can migrate to the recommended approach at a later time, without getting overwhelmed with the amount of material you have to learn in one go. Learning Yii (especially how to share code amongst controllers, and to write maintainable view templates) takes a while, so it may be sensible not to over-complicate it with many other things as well.
Why to use Yii:
Just imagine that you have many modules and for each module you have to write a pagination code; writing in old fashion style, will need a lot of time;
Why not use Yii ClistView widget? Oh, and this widget comes with a bonus: the data provider and the auto checking for the existance of the article that is about to be printed;
When using Yii CListView with results from ... Sphinx search engine, the widget will check if the article do really exists, because the result may not be correct
How long will it take for you to write a detection code for non existing registration?
And when you have different types of projects will you addapt the methods?
NO! Yii does this for you.
How long would it take for you to write the code in crud style ? create, read, update, delete ?
Are you going to adapt the old code from another project ?
Yii has a miracle module, called Gii, that generates models, modules, forms, controllers, the crud ... and many more
at first it might seem hard, but when you get experienced, it's easy
I would suggest you should use CActiveRecord.It will give many advantages -
You can use many widgets within yii directly as mentioned above.(For paginations,grids etc)
The queries which are generated by the Yii ORM are highly optimized.
You dont need to put the results extracted from SQLs in your VO objects.
If the tables for some reason modified(addition/deletion of column,changing data type), you just need to regenerate the models using the tool provided by yii.Just make sure you try to avoid doing any code changes in the models generated by yii, that will save your merging efforts.
If you plan to change the DB from MYSQL to other vendor in futur, it would be just config change for you.
Also you and your team would save your precious development time.
I'm staring to build a system for working with native languages, tags and such data in Yii Framework.
I already choose MongoDB for storing my data as I think it feets nicelly and will get better performance with less costs (the database will have huge amounts of data).
My question regards user authentication, payments, etc... This are sensitive bits of information and areas where I think the data is relational.
So:
1. Would you use two different db systems? Should I need them or I'm I complicating this?
2. If you recommend the two db approach how would I achieve that in Yii?
Thanks for your time!
PS: I do not intend this question to be another endless discussion between the relational vs non-relational folks. Having said that I think that my data feets mongo but if you have something to say about that go ahead ;)
You might be interested in this presentation on OpenSky's infrastructure, where MongoDB is used alongside MySQL. Mongo was utilized mainly for CMS-type data where a flexible schema was useful, and they relied upon MySQL for transactions (e.g. customer orders, payments). If you end up using the Doctrine library, you'll find that the ORM (for SQL databases) and MongoDB ODM share a similar API, which should make the experimentation process easier.
I wouldn't shy away from using MongoDB to store user data, though, as that's often a record that can benefit from embedded document storage (e.g. storing multiple billing/shipping addresses within a single user document). If anything, Mongo should be flexible enough to enable you to develop your application without worrying about schema changes due to evolving product requirements. As those requirements become more clear, you'll be able to make a decision based on the app's performance needs and types of database queries you end up needing.
There is no harm in using multiple databases (if you really need), many big websites are using multiple databases so go a head and start your project.
I have a design discussion with a collegue.
We are implementing a PHP MySQL database application. In the first instance we have written the Insert Get Update Delete, SelectAll and Search functions for a particular table, writting the table and fieldnames in the code, with several php files, one for the object class, one to draw the HTML table for that table, one for editing a row of that table, one containing the above functions, etc.
The disagreement comes as I have now written generic functions that read/write from the database and draw the HTML taking the table name as a parameter, letting these functions discovers the fieldnames from the database or class. So now that code can be used for any table, with any fields without having to manually go in change each function that needs alteration. I understand there will be cases where more table specific functionality is needed, and this I think that should be done as requirements arise, integrating common parts where possible.
My collegue on the other hand is adamant we should keep a set of files separate for each table, i.e. around 5 php files for each table. The read/write functions written differently in each case, with the need for any changes required for all tables to be affecting 5 x number of tables amount of times.
I can certainly say there will be more than main 15 tables in the database that will at least need basic funcionality.
What approach do you think is most appropriate?
One of the important principles in programming is DRY : Don't Repeat Yourself. So, everything common to several usecases should be written once, in a single location.
Now, I've never had to develop an application where each database table had the same, generic, crud pages. If it were the case, it wouldn't be a functional application, but a database management application. Are you sure you aren't redeveloping phpMyAdmin?
If you need the same code to handle several base operations on several tables, I would say you shouldn't write that code more than once : no need to repeat yourself :
it would take more time writing the application
it would take more time (and possible cause more bugs) when maintaining it
A possible solution could be to put as much common code as possible into a single class, common to many tables ; and, then, have a specific class for each table, that would extend that common class.
This way :
The common code would be written only once
Each table could have its specific code
The common code could be overridden for a specific table, if necessary.
About that, see the Object Inheritance section of the manual.
About that common class idea, you could even go farther, and write pretty much no code at all, if your CRUD is generic enough : there are ORM frameworks that might do that for you.
For instance, you might want to take a look at Doctrine.
We're on the first stages of development, and we don't have the complete functional specifications for the web application we're developing. Yes, we know, but it's not our fault.
So, we're building some parts keeping them pretty simple and straight-forward so we can build on top of that when we have more details on what to build.
We have a section for clients, for ads, for users, ... and I wanted to keep things separate because we don't know what's coming in the future. Yes, at the moment we have only a few fields and some basic listings and editing pages, but all that will grow.
It's not that I don't want to implement some generic code that we can reuse. It's that we don't know yet what will be the limitations in the near future, and I don't want to write generic code that we'll have to parametrize intensely.
For example, Alex built a generic Update method to which you pass an object and it will create an UPDATE SQL statement and execute it. OK, that's cool, but that doesn't work for the Users section of the web app because we store the password encoded. First, it won't encode the password. Second, if you edit a user and don't enter anything on the password and password-confirmation fields, the old password will remain. So, we have a problem with the generic Update method, and as I see it there are two possible solutions:
a) Parametrize the Update method so if it is modifying a user, keep the password if the password on the object is blank. And encode the password, of course.
b) Override the Update method for the child class.
Alex's implementation didn't use inheritance and he used the generic methods in a static class he'd call this way DataAccess::Update($object);. The method takes the table name from the class name as he modified the database to make them match (I prefer "Clients" for the table and "Client" for the class). So, option b is not possible with Alex's implementation.
The way I was trying to build it was keeping separate Update methods for each table. Yes, I was repeating myself but, as I said before, we don't have a full specification, so we don't know how it's going to grow. We have an idea, but we don't have the exact details.
So, the point here is that I don't want to write generic code until we have a much more detailed specification so we can evaluate what can and what cannot be shared between the parts.
Not all sections of the web app work the same and, as JB Nizet said: "If it were the case, it wouldn't be a functional application, but a database management application."
And I can tell you for sure this is not a database management application, even though Alex would say "we're just building a database app." Well, maybe, but a database application is not only showing/modifying the tables. And now, views won't solve all problems.
Again, as JB Nizet said: "If it were the case, it wouldn't be a functional application, but a database management application."
And now I'm repeating myself again, but this time there's no reason for that.
Thanks for your time.
My question is very similar to this question but a bit more specific.
My application has multiple companies and multiple users per company. It makes the most sense to me (at this point) for each company to have a "private" set of tables. This makes security extremely simple as I don't have to worry about JOIN-ing up my structure tree to be sure I only get data for the specific company. I can also extend the mysqli database extension and have it put a prefix on the table names in the query so that I never have to worry about security while writing my queries.
One other major advantage that I can see is that if one of the companies needs a customization, I can modify their specific tables and not have to take into account everyone else. The way that my app is designed it is extremely modular and implementing custom code is very simple.
There are some disadvantages that I can see but so far it seems that the above advantages would out-weigh them. The above proposed system does sort of grate on my (possibly) hyper-normalized database schema preferences up to this point. Another obvious disadvantage is implementing schema alterations but I can script them and be safe enough. One point that I'm not sure about is performance. If I have MySQL working with so many tables, will I make bottlenecks for myself?
I look forward to your thoughts!
Your proposal sounds reasonable to me. I would suggest that instead of prefixing your tables with the company name, you store the tables for each company in a separate schema. That way you can have tables with the same name, reducing your problems in the code, and have each set of tables protected by a different username and password in a convenient manner. Backups and replication would then all be distinguishable at need.
Lookup tables could be stored in yet another schema to which all users have access.
I'm writing an application that that I'm going to provide as a service and also as a standalone application.
It's written in Zend Framework and uses MySQL.
When providing it as a service I want users to register on my site and have subdomains like customer1.mysite.com, customer2.mysite.com.
I want to have everything in one database, not creating new database for each user.
But now I wonder how to do it better.
I came up with two solutions:
1. Have user id in each table and just add it to WHERE clause on each database request.
2. Recreate tables with unique prefix like 'customer1_tablename', 'customer2_tablename'.
Which approach is better? Pros and cons?
Is there another way to separate users on the same database?
Leonti
I would stick to keeping all the tables together, otherwise there's barely any point to using a single database. It also means that you could feasibly allow some sort of cross-site interaction down the track. Just make sure you put indexes on the differentiating field (customer_number or whatever), and you should be ok.
If the tables are getting really large and slow, look at table partitioning.
It depends on what you intend to do with the data. If the clients don't share data, segmenting by customer might be better; also, you may get better performance.
On the other hand, having many tables with an identical structure can be a nightmare when you want to alter the structure.
I'd recommend using separate databases for each user. This makes your application easier to code for, and makes MySQL maintenance (migration of single account, account removal and so on.)
The only exception to this rule would be if you need to access data across accounts or share data.
This is called a multi-tenant application and lots of people run them; see
multi tenant tag
For some other peoples' questions