I am running a Symfony 2.8 web app which allows the user to store and manage several different data entities
Over time the database grew bigger and bigger where a lot of data belongs to inactive/abandoned accounts which are no longer used.
Of course it quite easy to identify these accounts and the corresponding data can simply be deleted. However, I can never be sure if the users may return to his account and re-starts using his data. Thus I would like to implement a two step process:
If the user has not access his data for 3 month the data is moved from the production database to a cold storage. This would be just another database which holds the necessary tables to save the user data.
If the user has not accessed his account for 12 month the account and the data will be deleted.
How to setup such a cold storage?
According to the Symfony docs it should be no problem to work with two different EntityManagers to connect to two different databases.
But how to manage and setup the cold storage database?
How do I tell Symfony/Doctrine to create the tables for EntityA and EntityB entities (the data) in the production database as well as in the cold storage database but the table for EntityC only in the production db (e.g. the user settings)?
How to move the entities from one database/EntityManger to the other? Is there a more effective solution to do this instead of just loading the entity from one EntityManger and persisting it in the other?
Is there any best practice to solve this?
Related
I am building a SaaS application that has a set of requirements. One of these requirements is that the application has to be multi-tenant with a seperate schema for every tenant. But there has to be a single instance (admin interface) that needs to access all those schemas for data validation.
Example:
We have x customers who will access their application through customerx.our-company.com. Every customer gets its own schema with multiple tables. These tables contain users, invoices, contracts, timesheets,... The customer has to be able to view/modify these records.
We as a company providing the service have to be able to get all the timesheets from all those customers to be able to verify them. Once verified these records will be updated. 'verified' => true
My first idea was to store a tenant_id in a customers table in the admin database to be able to change the database connection in eloquent and merge the records before returning them. This is doable, but I have some questions about performance. What if we grow to 1000+ customers. getting all those records could take a long time.
Dumping all the records in a single schema without providing a separate schema per customer cannot be done because of GDPR. The data of the customer has to be separated.
What would be the best approach for this design wise? Are there some good articles about this subject?
What I began to think while writing this question: It is possible to leverage c++, I can create a custom PHP extension that takes multiple tenant database info to get the records via async tasks std::async, merge those records and return them. std::async creates multiple threads in a pool and I can wait until it is finished. The only big deal will be relations that need to be returned.
Updating those records is a simpler task as we know where it is stored (customer_id) so we can change the DB connection on the fly while updating.
EDIT: I created a simple c++ program that creates async MySQL connections. I've tested this with 10 schemas at once retrieving 1000 records from each. This looks promising, will keep this updated.
I have a web application where companies can register their company and use a set of features. However, lets say company 1 and company 2 has registered. They are still accessing the same website. Now each of these companies are 100% independent of each other when it comes to sharing information etc. The only thing they might share, is the users/employees.
Now my question is really, what is the best practice if each of these companies are to insert, select, update and deleted about 10K rows a day, each.
It can be everything from project handling, hourlists etc. All of which are split into different tables.
Would it be best practice to have independent databases, or use the same database for all the companies, and identify them by company_id?
Also keeping in mind the web application has to easily adapt to more than 10+ companies.
You could go one of two ways:
Add a companyId column to your tables,
Create a separate database for each company.
Option 1:
This option is the most dynamic one. You can keep the data separated by adding the correct companyId identifier to the where clause of your query.
This method is good when:
You expect a large number of customers,
You expect your number of customers to increase and decrease on a regular basis,
You do not need to share your database access with your customers (they only access it through your API/GUI).
Option 2:
This option gives a better separation of data. You keep each custommers data in their own dedicated instance of the database schema. This option allows you to offload the access-control burden to the database server, instead of having to enforce it in your application logic (which is more error prone).
However, there are some downsides: whenever a new customer shows up, you need to create a new database instance for them, which implies having a user with create database and grant privileges, something not every system administrator would be overly happy about.
The other issue is that whenever something changes in the database structure, you need to apply the chance to each instance of the database.
The good thing about this option is that you can give backup copies of your database to your customers, give them direct access to the database server, if needs be, or, in a more limited form, you could give them a copy of the database structure, without the need to filter out the customerId columns (as would be the case with option 1 above).
In summary:
There is no silver bullet, it all depends on your use-case. Option 1 is more flexible, Options 2 offers a better separation of data and easier access management.
[1]Keep separate database as there is more DML operations with your database.
[2]Keep very good database maintenance plan for Statistics management, Index maintenance and Backup/Recovery,otherwise you will have performance issue or more down time in case of database crash.
I've been working on a web app for a few months now. It's a PHP and MySQL driven database app which relates objects between each other.
I'd like to add functionality so that someone could register to use it and set up a monthly subscription. When they log in the app would simply populate with data from their own database.
I've done some scouring online but I'm struggling to find a starting point for adding this sort of feature.
The app relies on half a dozen tables within a database. So I'm also not sure if creating an individual database per user is practical.
Creating a db per user is very rarely the way to go - it's complicated and has very few benefits with lots of drawbacks (eg maintaining multiple simultaneous Db connections as most libraries only connect to a single Db). What you really need to do is create a user table and then tag various records with a UserId.
When a user registers, create a new User record, then create their new entries in other tables all referencing that Id. (Actually, only the "root" of each relational object needs a UserId eg If you have Order and OrderItems only the Order table needs a UserId)
Without more detail, it's impossible to help further.
Incidentally, the only time you should consider db-per-user is if each user requires unique tables (in which case your app would have to be very complex to actually use the data) or if there are security concerns re: storing data together (more likely with multiple clients than multiple users).
I have to develop a basic social network for an academic purpose; but I need some tips for the users management..
The users are subdivided into 3 groups with different privilege: admins,analysts and standards users.
For every user should be stored into the database the following information: name,lastname,e-mail,age,password.
I'm not quite sure how I should design the database between theese two solutions:
1)one table called 'users' with the 'role' attribute that explain what a user can do and what can't do, and the permissions are managed via php
2)every application user is a database user created with the query 'CREATE ROLE' (It's a postgres database) and he has permissions on some tables granted with the 'GRANT' statement
You should take into account that the project is for a database exam..
thanks
Don't use the database's authorization mechanism to be your application's authorization system. Three main reasons:
A) You'll never be able to change to a different database without rebuilding the whole app.
B) The types of things you want to grant the users in the app might differ from what the db's ACL system allows.
And most importantly:
C) You don't want to give an application user the ability to do anything directly to your database. Ever.
So your #2 option is right out. Thus, store a user type field with each user record, and then "what that user type allows" becomes part of your business logic that is calculated in PHP.
Solution 1 every time as you don't want to restrict yourself to only assign permissions on a per-table basis. Using database users would be cumbersome and not very practical.
Go with Option 1. It will be much more flexible in the long run, probably easier to code, and you don't want to tie your application logic too closely to a specific implementation. What if you later on want to port the application to run on SQL-Server? If database users are implemented differently, Option 2 could give you serious pains.
Go with your first alternative (manage permissions with PHP). Here are the reasons:
The database does not give you enough choices and granularity in the permissions you'll need to manage (who is allowed to send emails, to what groups people are allowed access, etc.)
Typically connections to the database are rather expensive so you'll want to connect once and stay connected as long as possible (with the same database user)
All databases are not created equal in the way they handle user accounts. By building your own user system above SQL you can hope to be more database independant
In the real world the tasks of administering the database and developping programs are done by completely different people and the program does not have the right to create or alter database users
I am building a customer sales and invoicing app for a company.The app is in PHP MYSQL, but I guess that shouldn't matter much.
The app structure is as follows:
website files: .php, ,.htm, images and css
database: containing 20+ tables
The app is currently being used by the company and 2 other sister concerns(beta testing mostly). Since the user base is small, I manually copy the website files and the database to set the app up for usage by a new compnay.
I am looking for a way to make the app more 'scalable' without having to manually do the 'scaling'.(meaning I don't want to manage three different filesets and dbs manually)
Since the code is company neutral and the databases contain the company info, I will only have to recreate the database when a user requests a new company to be setup. There are multiple ways that I can create the database for a new compnay.
At runtime I can create a new databse with the 20+ tables using CREATE DATABASE
At runtime I can create additional 20+ tables with the company name as prefix for the tables using CREATE TABLES
I can add a company column to all of my tables and then continue adding info as before.
The new database method appeals to me because backup and maintenance would be easy, it would probably be a bit more secure since a hacker will only be able to access the details of one company(probably...). This option wont work on a shared hosting with a limit on number of databases.
The second option would mean I can create everything in one database. But this option is a bit more 'shared'.
I wouldn't go for the third option due to table level locking issues in MySQL (I am not using InnoDb for all my tables).
So my choices are between option 1 and 2. Developers who've managed financial apps , please advice, as once the beta testing phase is done with, the usage base will increase, and I don't wish to manually change the same thing in 10 databases and filesets. What will be the best thing to do?
From the security point of view, customers should have separate databases, which restricted access from MySQL users.
That user should only have the permissions needed by the application (often SELECT, INSERT, DELETE and UPDATE), and not administrative permissions (DROP, CREATE, GRANT, ...). In this way, you've a clear overview on databases and tables.
When you need to alter a table structure, you just executes the (thoroughly tested) SQL query on your database.
CSS, images and other static content could be put in a subdomain, or Alias (Apache)
Libraries and neutral classes should be put in one directory too, using include_path to include such a file, so you have only one fileset that needs to be changed.