We run a high transaction volume insurance system.
The goal is to update a single flat table of 200 columns on any changes in a customers profile.
The customers profile is stored in a complex model of around 400 tables.
e.g If the customer makes a single premium payment, it goes across 15+ tables but we want the flat table to update say 5 out of 200 columns to reflect some representative data off this payment for the customer
We are considering using triggers, that is not working out well in testing as the current MySQL server (AWS Aurora) is not able to deal with it.
The main application is PHP Code-igniter
Any one has any suggestions that can help?
"If the customer makes a single premium payment ..." --
That should touch about 2 tables.
A record of the payment (who, when, how much; not much more)
Some kind of status for the user.
The first of those tables is a running "log" of what happened. The second is the current state of things.
Having 400 tables or 200 columns, or touching 15 tables for a simple operation -- each of these indicate poor design.
It is OK, even normal, to have a few (not 400) JOINs to reconstruct things on the SELECT side.
Related
Although the topic is quite trivial but I am still confused what could be the best design. So I am trying build a php mysql based website. The idea is to have user registrations, as well as store items in current shopping cart into database and also store purchased items record per user. (I Plan to serialize items, maybe comma seperated string with Produt IDs and then deserialize upon retrieval from DB)
In the beginning it looks trivial, but lets imagine there are a million users.
Now i wonder how this design should scale.
My initial idea is to have a simple table with EMAIL - ID (autogenerated)
And then use this id to generate further table containing Name, Address, Phone, Password
And another table with current items as well as purchased items in the past against that ID.
So based on this ID, i can query all these tables for quick reference.
Do you think this idea would scale good lets say with a million users accounting for 3-4 million tables. Or i should try to squeeze everything in a single table??
I am working on a project, and I am trying to find a way to associate my pricing plans table with a customer.
id name description price days
1 Free the free plan 0.00 0
2 Silver the silver plan 49.99 365
3 Gold the gold plan 99.99 365
My first thought was to add the plan id as foreign key to the customers table. But i also need to know when is the expire date (based on the purchase date and the days of the plan selected).
When attempting to do something like this, it is best to keep tables abstracted from each other.
user_pricing_plans - the table name
pricing_plans_id - the id for the record you want to associate with a user
user_id - the id of the user
date_expiration - the date the plan will expire
date_purchased - the date the plan was purchased
following an approach like this will allow you alter this table if needed to add extra information. you could also follow a similar approach in the future with another table.
the key to all of this is separating your concerns ( the data ) into separate containers.
Edit: to kind of hint at what I'm getting at about why separating the tables is a good idea, I added the date_purchased as a field in the user_pricing_plans table.
a dba I know once said that "mysql is not a place for developers - they try to create tables to work with their code. tables are meant to represent data. your code should be written to work with the data and the schemas that represent them - not the other way around"
I do not know why I can not add a comment to the best answer in the post.
As #Jonathon Hibbard points few years ago, there is another reason to separate the data between pricing plan and user model.
You used to have users who have one plan and choose another later, that is called, "history". To manage that, this third table is very important.
And in my opinion, more important, one thing is your pricing table, an another one is the final price you have with every client. Yo have knownledge people, close clients that you want "special" prices, this third table gives you the oportunity to set diferent prices for one plan with fixed price and a lot of other use cases.
Think about your main plan table like a product, the user is the client, and the third party as the ticket, with "temp pricing" aplied, ocassional discounts or whatever.
I am designing an online accounting system as a college project. I have got the following simple transactions.
Sales on cash $300
Cash burrowed from Smith $250
Electricity bill paid $50
I have referred Double Entry Accounting in a Relational Database and designed the above Transaction table(minus sign means the amount is credited). The id 1 and 2 belong to transaction 1, id 3 and 4 belong to transaction 2 and so on. How can I identify each two rows as a single transaction in MySQL database.For example if the first transaction to be selected how can I do it since there is no relationship between the two rows. If this design does not sound good can you suggest an alternative.
You should drop the two line design unless it's really necessary. I would merge the rows and add an extra field where you could store if it was cash or not and the amount value would be + when credited and - if debited (or reversed, I"m not good at accounting :))
Good Job.
The only thing I might add is two number columns for debit and credit. That would look just like a journal in the text books or old bookkeeping books. It would be a little more visibly obvious and handle any number of double entries. Your current amount column could be calculated by debit - credit if you like.
I do not think you need to make it relational unless you wanted a specific chart of accounts as a table, more complicated and less elegant. But, for more flexibly you could generate real time chart of accounts from the journal.
Since you are using MySQL relational database, you can add several tables.
To answer your question, you need to create another table to hold general information for each transaction (e.g. you could call it a TRANSACTIONS). This table could include Transaction ID, and Date Column.
Now, reference the Transaction ID column of the TRANSACTIONS table in your currently existing table. This means record 1 and 2 will have one Transaction ID and so you can reference it as such in your queries.
Hopefully I can do this without writing a book... I'm working on a system for a Surveying company. They're also throwing in an accounting section, and one of the functionalities it needs is to find jobs that are not completely paid off.
A few points to be made:
A 'job' is a project, essentially. There are different types of surveys.
A job can have more than one "type" of survey in it, thus having more than one price that is calculated into the job's total price.
Payments are made on individual jobs (the client might send 30 checks if they have 30 jobs, it's an accounting thing)
Job types are dynamic (they can create/delete them from the system panel) - so I can't hardcode these things
Here's the applicable database structure:
table jobs: job_id, client_id
table job_types: type_id, type_name
table job_type_assoc: id_job, type_id, price
table payments: payment_id, job_id, amount
Note: payments aren't made on each job type, but the JOB as a whole (again, as opposed to the client's account having a "balance").
I need to somehow pull jobs where the total of price from job_type_assoc is less than the total of amount in payments.
I don't know if this is possible in mysql alone or if php would be more efficient - also, their old system has about 340,000 jobs in it. Granted that they won't have the accounting info from then, they do have a lot of work and the new entries will build quickly, AND if I somehow do it in PHP where I end up querying the jobs table a lot, it might get messy.
One possible way you can do this is with a clever combination of subqueries.
select * from
(select job_id,
(select ifnull(sum(job_type_assoc.price),0) from job_type_assoc
where jobs.job_id = job_type_assoc.id_job) as amount_due,
(select ifnull(sum(payments.amount),0) from payments
where jobs.job_id = payments.job_id) as payments
from jobs) as outstanding
where payments< amount_due;
This shouldn't be too bad a query to run, if you have properly indexed job_type_assoc and payments.
Please note that I haven't tested this exact query, so it might need tweaking. I've run the same query against tables on my local DB.
I'm actually worried about designing the perfect tables to handle transactions and commissions. especially that we really don't want tables to be out of sync in any case.
Basically this is like ebay, where merchants sell to buyers and we get a commission out of each transaction.
The app is already built using Codeigniter, and it has:
Merchants(id,name,balance)
Transactions(id,merchant_id,buyer_id,amount,transaction_id)
Now what other tables to make if needed to make us able to fetch our revenue and the seller's revenue, and what's the best DB mechanism to process a refund?
And should we make another table of each deposit to the merchant's balance rather than +x the balance row.
Whenever you are dealing with any financial tasks, always store as much information about it as possible.
There are two ways of keeping all your tables in sync.
Database level - using options such as stored procedures and triggers
Application level - Use transactions (innoDB) and make sure that the transactions are atomic. (I would suggest to go with this)
Regarding the database design -
Add a column commission to Transactions that tracks your commission. Data for this can be either calculate beforehand when pricing the item or on the fly during the transaction.
The buyer pays product cost + commission, seller gets product cost and you get commission. If your commission is a variable (not a flat rate and changes from item to item), you could have a column in the Items table which stores it. Things should be trivial from now on. You just have to maintain a History table that keeps a list of transactions with respect to the user. It would look something like
AccountHistory(id, transaction_type, transaction_id, amount)
You could split AccountHistory into two - one for incoming money and one for outgoing money.
So everytime a transaction takes place the following happens
Entry into to Transactions table.
Entry into AccountHistory for buyer
Update buyer balance
Entry into AccountHistory for seller
Update seller balance
All this should be atomic.
For rollbacks, you could add a column transaction_type to distinguish rollbacks from normal transaction or you could maintain a separate table.
Having worked with a database where the design had financial transactions scattered around a bunch of different tables, I can tell you that it is much better to have all transactions in a single table. You should add transaction type to your transaction table (and a look up table for the transaction types).
Then each transaction can have a separate line for commission and seller product cost. Transaction that are undone later would get entered in as negative amounts. It is also helpful for these types of things to add a parent transaction field so that you can easily identify which transaction it was changing.
All data entry that forms part of a transaction should be in a stored proc and have transactions so that if one part of the insert fails, the entire thing is immediately rolled back. This is critical. It needs to be in a stored proc so that no one (except systems dbas) has direct access to tables that store financial records.
When you start to think of handling financial transactions the very worst thing you can do is design for the user interface alone. You need to design for the financial reporting you will need to have. You need to consult the people who will be auditing your finances to see what information they need and what internal controls they expect you to have in place to prevent internal as well as external fraud.