I m looking for advices in report generating per client, per product, per salesman, in a given period, using mysql and php. I m currently working on an application with sales and i was wondering what is the best way to structure this function. I m planning on hosting in a shared hosting server, so i m worried about efficiency. So here are the two ways i m looking things:
1- add entries of sales then once a user specifies a period of the sales report, executing a mysql query that will sum up everything then feed it to the php to arrange it a send response back
2- fetch all entries then calculate sums and everything in php
3- on every sales, add the totals on a report table for each day, then another table for each client, then for each product then for each salesman. And once a user requires a report, fetch the totals in each table and sum up the totals (the problem here is if a sale has been edited or deleted, i have to re-adjust the totals for all tables)
I m looking for suggestions for the best way to do this, also all the data are on the same table and segregated by user_id so for example if there are 1000 users with 1000 sales, the table sales will have 1000000 rows with corresponding user_id.
Thanks in advance
Related
I have 3 tables (customers, invoices, invoice_items).
A customer can have multiple invoices and an invoice can have multiple items.
This is my current schema:
customers
id
name
timestamps
invoices
id
customer_id
timestamps
invoice_items
id
invoice_id
price
timestamps
Note: timestamps are created_at and updated_at fields. No need to worry.
as you can see, I am not storing the total amount of an invoice in the invoices table. I let the front-end calculate it.
Now, my client has 1000+ customers and they want to show all customers in the homepage along with the total sum of all invoices they have.
ID | Customer name | Total Invoice (sum of all invoices)
Right now I am using Laravel, so my query looks like this:
$customers = Customer::with('invoices.items')->get();
What's the best way to do this? Performance wise?
Should I just store the total amount of each invoice in the invoices table to avoid querying the items and calculate the sum? Something like:
invoices
id
customer_id
total
timestamps
Imagine a customer having thousand invoices and each invoice has multiple items. The query will be very slow and I have to calculate every thing in the front-end.
Also, when an invoice has been made, it can't be changed. So the total for an invoice is static.
Ofcourse, when you add a total column on invoice table then you should get a boost in the performance, as you will not have to go through all the invoice_items and add them up. This is good if you need to add items to the invoice only at once and do not need to be updated later. This way, you will need to add the total while generating the invoice at once.
It looks good apart from the timestamps.
Can you not just use one timestamp column in your invoices table?
Why do you need the same column in all 3 tables?
Also, I'm sure you can calculate total in your php script.
You don't need to store the total that you can calculate through a count() in php.
Adding total column in invoice table will be definitely good for performance and it will also help fetch result through eloquent. However, if performance is not something important here, we can do it without it. Your SQL query should be like (might need little adjustment)
SELECT c.*, i.*, sum(ii. price) as total_invoice_price
FROM customers c
INNER JOIN invoices i on i.customer_id = c.id
INNER JOIN invoice_items ii on ii.invoice_id = i.id
GROUP BY i.id
Once you have expected result through SQL, you can run it using DB::table in laravel.
This is classic case of WORM (Write Once Read Many).
Lets look into pros and cons of both the approaches:
Case 1: Not storing the total in invoices table
Pros: Simple and easy. No data redundancy.
Cons: Slow performance and consuming resources(memory and cpu) every time you compute the total
Case 2: Storing the total in invoice table
Pros: Better performance because "complex" invoice total computation
happens only once and subsequent reads are faster
Cons: Data redundancy and requires some extra storage
Now if you look into the cons of both the approaches, case 2 is obviously better because extra storage can be bought easily but performance can't be even if you increase RAM and CPU upto some extent.
I am working on a project, and I am trying to find a way to associate my pricing plans table with a customer.
id name description price days
1 Free the free plan 0.00 0
2 Silver the silver plan 49.99 365
3 Gold the gold plan 99.99 365
My first thought was to add the plan id as foreign key to the customers table. But i also need to know when is the expire date (based on the purchase date and the days of the plan selected).
When attempting to do something like this, it is best to keep tables abstracted from each other.
user_pricing_plans - the table name
pricing_plans_id - the id for the record you want to associate with a user
user_id - the id of the user
date_expiration - the date the plan will expire
date_purchased - the date the plan was purchased
following an approach like this will allow you alter this table if needed to add extra information. you could also follow a similar approach in the future with another table.
the key to all of this is separating your concerns ( the data ) into separate containers.
Edit: to kind of hint at what I'm getting at about why separating the tables is a good idea, I added the date_purchased as a field in the user_pricing_plans table.
a dba I know once said that "mysql is not a place for developers - they try to create tables to work with their code. tables are meant to represent data. your code should be written to work with the data and the schemas that represent them - not the other way around"
I do not know why I can not add a comment to the best answer in the post.
As #Jonathon Hibbard points few years ago, there is another reason to separate the data between pricing plan and user model.
You used to have users who have one plan and choose another later, that is called, "history". To manage that, this third table is very important.
And in my opinion, more important, one thing is your pricing table, an another one is the final price you have with every client. Yo have knownledge people, close clients that you want "special" prices, this third table gives you the oportunity to set diferent prices for one plan with fixed price and a lot of other use cases.
Think about your main plan table like a product, the user is the client, and the third party as the ticket, with "temp pricing" aplied, ocassional discounts or whatever.
I am developing a MySQL DB for an online jewelry shopping website. Every jewelry may have different stones, metals, … Their net prices will change everyday (e.g., every ounce of gold may be X today and Y tomorrow). Product A may be more expensive than B today, but cheaper tomorrow. Every time a user requests the products list, the website automatically calculates the price of every product.
How can users sort products by its price, considering that net price is not stored in a field in DB and every time will be calculated (using weights, net fees, …) by a function named func($id), where $id is the id of the product.
An illustration to the question:
Assuming you have all of your data in your DB, there is no reason you can't ORDER BY any kind of expression you want.
If, say, you had a table products (pid, name), productContents (pid, type, amount), and contentPrices (type, price), you could run something like
SELECT a.name, SUM(b.amount*c.price) AS totalPrice FROM products a
LEFT JOIN productContent b ON a.pid = b.pid
LEFT JOIN contentPrices c ON b.type=c.type
ORDER BY totalPrice DESC;
EDIT to work with the updated question schema:
SELECT p.title, p.metal_weight*m.fee + p.stone_weight*s.fee AS totalPrice FROM Products p
LEFT JOIN Metals m ON p.metal_id=m.id
LEFT JOIN Stones s ON p.stone_id=s.id
ORDER BY totalPrice DESC;
I would do it in two ways, but first of all, if you want a good performance, store the price value in a Field. I don't think it changes every second, does it?
So first, if you can make your Price calculator function in MySQL, create a Stored Procedure for it and run this on the table periodically (for example every day at 00:00) and store the value in a field. You can find how to schedule an event in MySQL here.
If you can't do this, stay with php, but it also should called periodically to calculate everything. You can do it with cron on Unix (if you have the rights). This can be scheduled on a remote server too, so if you have somewhere a server where you can use cron, simply call the updater.php with wget, for example:
wget -c http://jewerly-shop.com/update_prices.php && rm -rf update_prices.php
But if you need further help, we need more info (like the function that does the calculating and the table structures).
Hopefully I can do this without writing a book... I'm working on a system for a Surveying company. They're also throwing in an accounting section, and one of the functionalities it needs is to find jobs that are not completely paid off.
A few points to be made:
A 'job' is a project, essentially. There are different types of surveys.
A job can have more than one "type" of survey in it, thus having more than one price that is calculated into the job's total price.
Payments are made on individual jobs (the client might send 30 checks if they have 30 jobs, it's an accounting thing)
Job types are dynamic (they can create/delete them from the system panel) - so I can't hardcode these things
Here's the applicable database structure:
table jobs: job_id, client_id
table job_types: type_id, type_name
table job_type_assoc: id_job, type_id, price
table payments: payment_id, job_id, amount
Note: payments aren't made on each job type, but the JOB as a whole (again, as opposed to the client's account having a "balance").
I need to somehow pull jobs where the total of price from job_type_assoc is less than the total of amount in payments.
I don't know if this is possible in mysql alone or if php would be more efficient - also, their old system has about 340,000 jobs in it. Granted that they won't have the accounting info from then, they do have a lot of work and the new entries will build quickly, AND if I somehow do it in PHP where I end up querying the jobs table a lot, it might get messy.
One possible way you can do this is with a clever combination of subqueries.
select * from
(select job_id,
(select ifnull(sum(job_type_assoc.price),0) from job_type_assoc
where jobs.job_id = job_type_assoc.id_job) as amount_due,
(select ifnull(sum(payments.amount),0) from payments
where jobs.job_id = payments.job_id) as payments
from jobs) as outstanding
where payments< amount_due;
This shouldn't be too bad a query to run, if you have properly indexed job_type_assoc and payments.
Please note that I haven't tested this exact query, so it might need tweaking. I've run the same query against tables on my local DB.
Im building a classifieds website of adverts where I want to store a count of the number of views of each advert which I want to be able to display in a graph at a later date by day and month etc.. for each user and each of their adverts. Im just struggling with deciding how best to implement the mysql database to store potentially a large amount of data for each advert.
I am going to create a table for the page views as follows which would store a record for each view for each advert, for example if advert (id 1) has 200 views the table will store 200 records:
Advert_id (unique id of advert)
date_time (date and time of view)
ip_address ( unique ip address of person viewing advert)
page_referrer (url of referrer page)
As mentioned I am going to create the functionality for each member of the site to view a graph for the view statistics for each of their adverts so they can see how many total views each of their adverts have had, and also how many views their advert has had each day (between 2 given dates) and also how many views per month each advert has had. I'll do this by grouping by the date_time field.
If my site grows quite large and for example has 40,000 adverts and each advert has on average 3,000 page views, that would mean the table has 120 Million records. Is this too large ? and would the mysql queries to produce the graphs be very slow?
Do you think the table and method above is the best way to store these advert view statistics or is there a better way to do this?
Unless you really need to store all that data it would probably be better to just increment the count when the advert is viewed. So you just have one row for each advert (or even a column in the row for the advert).
Another option is to save this into a text file and then process it offline but it's generally better to process data as you get it and incorporate that into your applications process.
If you really need to save all of that data then rotating the log table weekly maybe (after processing it) would reduce the overhead of storing all of that information indefinitely.
I was working with website with 50.000 unique visitors per day, and i had same table as you.
Table was growthing ~200-500 MB/day, but i was able to clean table every day.
Best option is make second table, count visitors every day, add result to 2nd table, and flush 1st table.
first table example:
advert_id
date & time
ip address
page referrer
second table example (for graph):
advert_id
date
visitors
unique visitors
Example SQL query to count unqiue visitors:
SELECT
advert_id,
Count(DISTINCT ip_address),
SUBSTRING(Date,1,10) as Date
FROM
adverts
GROUP BY
advert_id,
Date
Problem is not even perfomance (MySQL ISAM Engine is quite smart and fast), problem is storage such big data.
90% statistics tools (even google analytics or webalyzer) is making graphs only once per day, not in real-time.
And quite good idea is store IP as INT using function ip2long()