I have a SQL table being created daily that is downloaded from a suppliers website,containing product info, that is in csv. I have everything creating alright and all the tables are identical. The problem that I am needing solved is that I need to compare the tables between today and yesterday (table names are the dates in following format mm-dd-yyyy) I need to compare a few different columns for different things.
I need to know all products that are in today's data that weren't in
yesterdays (can be checked by supplier SKU)
I need to know all product that were in yesterday's data that is no
longer in today's
I need to know when the price went up from yesterday's data
I need to know when the price has gone down from yesterday's data
I need to know when a sale has started based on yesterday's data as
well as stopped
These need to show the following labels in the table that will show the changes
regular up
regular down
miscillanious change (description change or change to a fields that aren't a priority)
promo on (discount added from supplier)
promo off (discount taken off by supplier)
delete (no record of the product in new list {probably been deleted})
new item (new record of product in new list)
out of stock
I have been searching everywhere for the answer for these issues and have found stuff that kind of shows me how to do this using union and join but I don't fully understand how to use them based on this scenario.
I have tried different PHP solutions by going through each piece of data and searching for the sku in the new table and vice versa then checking for any changes if they exist in both tables but this is taking a really long time and I have over 200 000 products in these tables. I am hoping that I can do these in less queries and by letting the sql server do more work then the php script.
Thanks for all the help!
Yesterday's Table
__________________________________________________________
| id | price | sale | description | qty | sku |
---------------------------------------------------------
| 1 | 12.50 | 0.00 | description product 1 | 12 | 12345 |
| 2 | 22.99 | 20.99 | describe the problem | 1 | 54321 |
| 3 | 192.99 | 0.00 | description ftw | 5 | 53421 |
| 4 | 543.52 | 0.00 | description | 15 | 45121 |
----------------------------------------------------------
Today's Table
__________________________________________________________
| id | price | sale | description | qty | sku |
---------------------------------------------------------
| 1 | 12.50 | 0.00 | description product 1 | 12 | 12345 |
| 2 | 22.99 | 0.00 | describe the problem | 1 | 54321 |
| 3 | 192.99 | 50.00 | description ftw | 5 | 53421 |
| 4 | 523.99 | 0.00 | description | 15 | 45123 |
----------------------------------------------------------
I need the new table to look like the following
_____________________________________________________________
| id | sku | label | description | price |
-------------------------------------------------------------
| 1 | 54321 | promo off | describe the problem | 22.99 |
| 2 | 53421 | promo on | description ftw | 192.99|
| 3 | 45123 | new item | description | 523.99|
| 4 | 45121 | delete | description | 543.52|
-------------------------------------------------------------
The following is the code I have for the deleted and new items currently. I am using int for the label/status in the example below and just signifying the different numbers.
$deleted = mysql_query("SELECT * FROM `test1`") or die(mysql_error());
while($skus= mysql_fetch_array($deleted))
{
$query = mysql_num_rows(mysql_query("SELECT * FROM `test2` WHERE SKU='".$skus['sku']."'"));
if($query < 1)
{
$tata= mysql_query("INSERT INTO `gday` (Contract_Price, SKU, status) VALUES (".$skus['price'].", ".$skus['sku'].", 1)");
}
}
$deleted = mysql_query("SELECT * FROM `test2`") or die(mysql_error());
while($skus= mysql_fetch_array($deleted))
{
$query = mysql_num_rows(mysql_query("SELECT * FROM `test1` WHERE SKU='".$skus['sku']."'"));
if($query < 1)
{
$tata= mysql_query("INSERT INTO `gday` (Contract_Price, SKU, status) VALUES (".$skus['price'].", ".$skus['sku'].", 2)");
}
}
EDIT:
The Following is the true table that all the data will be going into. I originally didn't want to muddy the water with the large table but by request I have included it.
ID
Status
DiscountDate
Price
Discount
DiscountEndDate
Desc1
Desc2
Desc3
Warranty
Qty1
Qty2
PricingUnit
PriceUpdate
Vendor
Category
UPC
Weight
WeightUnit
Because of the size of your database I could propose you an SQL solution.
When you work with a lot of data, PHP can be slow for several reason.
You can try to do some functions.
You just have to store the status data in an other table. This is the documentation to write a trigger.
http://dev.mysql.com/doc/refman/5.0/en/triggers.html
Becareful if you have a lot of operation on your table I can suggest you to use PostgresSQL instead of mysql because I found it more simple to write function, trigger, ... using PL/sql
EDIT: Just have to write a simple function
I'm working on it at the moment. Think about using select case like in this answer
EDIT: Core function
DELIMITER |
CREATE PROCEDURE export()
BEGIN
(SELECT today.id, today.sku,
CASE today.price
WHEN today.price = yesterday.price THEN 'nothing'
WHEN today.price < yesterday.price THEN 'promo on'
ELSE 'promo off' END AS label
FROM today, yesterday WHERE yesterday.sku = today.sku)
UNION
(
SELECT today.id, today.sku,
'new item' AS label
FROM today LEFT JOIN yesterday ON yesterday.sku = today.sku WHERE yesterday.sku IS NULL)
UNION
(
SELECT yesterday.id, yesterday.sku,
'delete' AS label
FROM yesterday LEFT JOIN today ON today.sku = yesterday.sku WHERE today.sku IS NULL
);
END|
DELIMITER ;
To call just do:
CALL export();
Here is an example of possible core functions. Be careful in this case id could be the same. In the function you'll have to add a personal one in the first column.
If you need performance to display it faster in PHP, think about APC cache
Related
In credits table I have
(id, user_id, process, amount, date_add, date_exp, date_redeemed, remark)
SELECT * FROM credits WHERE user_id = 2;
+----+---------+---------+--------+------------+------------+---------------+----------+
| id | user_id | process | amount | date_add | date_exp | date_redeemed | remark |
+----+---------+---------+--------+------------+------------+---------------+----------+
| 22 | 2 | Add | 200.00 | 2018-01-01 | 2019-01-01 | | Credit1 |
| 23 | 2 | Add | 200.00 | 2018-03-31 | 2019-03-31 | | Credit2 |
| 24 | 2 | Deduct | 200.00 | | | 2018-04-28 | Redeemed |
| 25 | 2 | Add | 200.00 | 2018-07-11 | 2018-10-11 | | Campaign |
| 26 | 2 | Deduct | 50.00 | | | 2018-08-30 | Redeemed |
| 27 | 2 | Add | 200.00 | 2018-10-01 | 2019-09-30 | | Credit3 |
| 28 | 2 | Deduct | 198.55 | | | 2018-10-20 | Redeemed |
+----+---------+---------+--------+------------+------------+---------------+----------+
The following query I wrote will only calculate the balance, but I don't know whether the credit is expired and used before expired.
SELECT
u.id,
email,
CONCAT(first_name, ' ', last_name) AS name,
type,
(CASE
WHEN (SUM(amount) IS NULL) THEN 0.00
ELSE CASE
WHEN
(SUM(CASE
WHEN process = 'Add' THEN amount
END) - SUM(CASE
WHEN process = 'Deduct' THEN amount
END)) IS NULL
THEN
SUM(CASE
WHEN process = 'Add' THEN amount
END)
ELSE SUM(CASE
WHEN process = 'Add' THEN amount
END) - SUM(CASE
WHEN process = 'Deduct' THEN amount
END)
END
END) AS balance
FROM
users u
LEFT JOIN
credits c ON u.id = c.user_id
GROUP BY u.id;
Or I am doing it in the wrong way? Maybe I should have done the calculation in my backend instead of SQL?
EDIT 1:
I want to calculate the balance of every user's e-wallet, but the credit will expire,
IF it was expired AND not redeemed then exclude from the balance
ELSE IF used before expired AND redeem amount < expire amount
THEN (balance - (expire amount - redeem amount))
ELSE IF used before expired AND redeem amount > expire amount
THEN the usable balance will be deducted as expire amount is not enough to deduct redeemed amount
EDIT 2:
The query above will output 351.45, my expected output is 201.45. Which will not calculate the redemption on 2018-08-30 as the redeem amount is lower than the expired amount
EDIT 3:
User table:
+----+------------+-----------+----------+----------------+----------+
| id | first_name | last_name | type | email | password |
+----+------------+-----------+----------+----------------+----------+
| 2 | Test | Oyster | Employee | test#gmail.com | NULL |
+----+------------+-----------+----------+----------------+----------+
My output:
+----+----------------+-------------+----------+---------+
| id | email | name | type | balance |
+----+----------------+-------------+----------+---------+
| 2 | test#gmail.com | Test Oyster | Employee | 351.45 |
+----+----------------+-------------+----------+---------+
Expected output:
total (200+200+200) 600
redeemed amount 448.55 (200+50+198.55)
Remaining balance is 151.45
+----+----------------+-------------+----------+---------+
| id | email | name | type | balance |
+----+----------------+-------------+----------+---------+
| 2 | test#gmail.com | Test Oyster | Employee | 151.45 |
+----+----------------+-------------+----------+---------+
There are basic structural problems with your current table. So I would propose some alterations to the table structure and subsequently application code. Table structures for Wallet systems can be very detailed; but I would suggest minimum possible changes here. I am not suggesting that it would be ideal way; but it should work. Initially, I will layout some of the problems with the current approach.
Problems:
What if there are multiple Credits available which are not yet expired ?
Out of these available Credits, some may actually have been utilized already, but not yet expired. How do we ignore them for available balance ?
Also, some may have been partially utilized. How do we account for partial utilization ?
There may be a scenario where a redemption amount spans across multiple unexpired credits. Some may get partially utilized; while some may get fully utilized.
General Practice:
We generally follow FIFO (First In First Out) approach, to give maximum benefit to the customer. So the older credits (which has higher chance of getting expired without utilization) gets utilized first.
In order to follow FIFO, we will have to effectively use a Looping technique in the query/application code every-time, in order to compute basic things, such as, "Available Wallet Balance", "Expired and Underutilized Credit", etc. Writing a query for this will be cumbersome and possibly inefficient at bigger scales
Solution:
We can add one more column amount_redeemed in your current table. It basically represent the amount which has already been redeemed against a specific credit.
ALTER TABLE credits ADD COLUMN amount_redeemed DECIMAL (8,2);
So, a filled table would look something like below:
+----+---------+---------+--------+-----------------+------------+---------------+---------------+----------+
| id | user_id | process | amount | amount_redeemed | date_add | date_exp | date_redeemed | remark |
+----+---------+---------+--------+-----------------+------------+---------------+---------------+----------+
| 22 | 2 | Add | 200.00 | 200.00 | 2018-01-01 | 2019-01-01 | | Credit1 |
| 23 | 2 | Add | 200.00 | 200.00 | 2018-03-31 | 2019-03-31 | | Credit2 |
| 24 | 2 | Deduct | 200.00 | | | | 2018-04-28 | Redeemed |
| 25 | 2 | Add | 200.00 | 0.00 | 2018-07-11 | 2018-10-11 | | Campaign |
| 26 | 2 | Deduct | 50.00 | | | | 2018-08-30 | Redeemed |
| 27 | 2 | Add | 200.00 | 48.55 | 2018-10-01 | 2019-09-30 | | Credit3 |
| 28 | 2 | Deduct | 198.55 | | | | 2018-10-20 | Redeemed |
+----+---------+---------+--------+-----------------+------------+---------------+---------------+----------+
Notice that the amount_redeemed against Credit of id = 25 is 0.00, using FIFO approach. It got a chance for redemption on 2018-10-20, but by that time, it has expired already (date_exp = 2018-10-11)
So, now once we have this setup, you can do the following things in your application code:
Populate amount_redeemed value in existing rows in the table:
This will be a one time activity. For this, formulating a single query would be difficult (that is why we are here in the first place). So I would suggest you to do it one time in your application code (eg: PHP), using Loops and FIFO approach. Look at Point 3 below, to get an idea of how to do it in application code.
Get Current Available Balance:
Query for this becomes trivial now, as we just need to calculate Sum of amount - amount_redeemed for all Add process, which are not yet expired.
SELECT SUM(amount - amount_redeemed) AS total_available_credit
FROM credits
WHERE process = 'Add' AND
date_exp > CURDATE() AND
user_id = 2
Update amount_redeemed at the time of redemption:
In this, you can firstly get all available Credits, which has amount available for redemption, and not yet expired.
SELECT id, (amount - amount_redeemed) AS available_credit
FROM credits
WHERE process = 'Add' AND
date_exp > CURDATE() AND
user_id = 2 AND
amount - amount_redeemed > 0
ORDER BY id
Now, we can loop over the above query results, and utilize the amount accordingly
// PHP code example
// amount to redeem
$amount_to_redeem = 100;
// Map storing amount_redeemed against id
$amount_redeemed_map = array();
foreach ($rows as $row) {
// Calculate the amount that can be used against a specific credit
// It will be the minimum of available credit and amount left to redeem
$amount_redeemed = min($row['available_credit'], $amount_to_redeem);
// Populate the map
$amount_redeemed_map[$row['id']] = $amount_redeemed;
// Adjust the amount_to_redeem
$amount_to_redeem -= $amount_redeemed;
// If no more amount_to_redeem, we can finish the loop
if ($amount_to_redeem == 0) {
break;
} elseif ($amount_to_redeem < 0) {
// This should never happen, still if it happens, throw error
throw new Exception ("Something wrong with logic!");
exit();
}
// if we are here, that means some more amount left to redeem
}
Now, you can use two Update queries. First one would update amount_redeemed value against all the Credit id(s). Second one would Insert the Deduct row using the sum of all individual amount_redeemed value.
SELECT `id`, `email`, `NAME`, `type`,
(
( SELECT SUM(amount) FROM credit_table AS ct1 WHERE u.id = ct1.id AND process = 'ADD' AND date_exp > CURDATE()) -
( SELECT SUM(amount) FROM credit_table AS ct2 WHERE u.id = ct2.id AND process = 'Deduct' )
) AS balance
FROM
`user_table` AS u
WHERE
id = 2;
Hope it works as you wish
My job is to cleanse the database from broken store records(records with missing data like a store name or streetname, database example below). However it turns out some of those records have employees linked to them so we can't just delete them.
Question: what kind of query do I need to make sure only the employees of those stores are shown?
I have three tables in which i need to work: user, department and store_users (store_users is a table in which all ID's are packed together to see which user_id belongs to which store_id.)
Example of code
its most likely wrong but its to give some insight in what I am trying to accomplish.(its purely mysql at the moment)
SELECT
*
FROM
gebruiker
WHERE
(SELECT
*
FROM
winkel_adresgegevens
WHERE
winkel_id IN (SELECT
winkel_gebruikers_winkel_id
FROM
winkel_gebruikers)
AND (winkel_naam = '' OR winkel_straat = ''
OR winkel_huisnummer = ''
OR winkel_postcode = ''
OR winkel_plaats = ''));
database example
+--------+---------------+----------+------------+------------------+
|store_id|store_branch_id|store_name|store_street|store_streetnumber|
|-------------------------------------------------------------------|
| 1 | 100 | Jumbo | Hilversum | 14 |
|--------+---------------+----------+------------+------------------|
| 2 | 150 | Lidl | Kerkelanden| 24 |
|--------+---------------+----------+------------+------------------|
| 3 | 105 | | Loosdrecht | |
|--------+---------------+----------+------------+------------------|
| 4 | 200 | Coop | | 14 |
+--------+---------------+----------+------------+------------------+
hence the question: only show employees from the broken records. (the ones with missing data, see store_name, store_street and store_streetnumber)
UPDATE : This is what you want
SELECT *
FROM employees
WHERE store_name IS NULL OR store_street IS NULL OR store_streetnumber IS NULL ;
Products :
--------------------------------------------
| ID | Group | Name | Sold |
--------------------------------------------
| 1 | A | Dell | 0 |
--------------------------------------------
| 2 | A | Dell | 0 |
--------------------------------------------
| 3 | B | Dell | 1 |
--------------------------------------------
| 4 | B | Dell | 1 |
--------------------------------------------
| 5 | C | Dell | 0 |
--------------------------------------------
| 6 | C | Dell | 1 |
--------------------------------------------
Hi everyone, i have a table (products) stored in MySql with many records, for now i'm using this query SELECT * FROM products WHERE sold = 0, in results i get :
--------------------------------------------
| ID | Group | Name | Sold |
--------------------------------------------
| 1 | A | Dell | 0 |
--------------------------------------------
| 2 | A | Dell | 0 |
--------------------------------------------
| 5 | C | Dell | 0 |
--------------------------------------------
i want to get only one record from each group, so the results will be like :
--------------------------------------------
| ID | Group | Name | Sold |
--------------------------------------------
| 1 | A | Dell | 0 |
--------------------------------------------
| 5 | C | Dell | 0 |
--------------------------------------------
You could easily do this by using a distinct clause and removing the id column. If you want to keep the id column you need to specify how one would chose which id to keep.
select distinct
`group`
, name
, sold
from
products
where
sold = 0;
To keep the row with the smallest id (as your example shows) something along the lines of the example below would work.
select
id
, `group`
, name
, sold
from
products
where
sold = 0
and id = (
select
min(p.id)
from
products p
where
p.`group` = products.`group`
and p.sold = 0
);
First, change your field named Group to something like Group_Name. GROUP is a reserved keyword, and if it is not causing you problems now it probably will later.
Second, you should ask yourself what you are really after. The following query should generate your desired result. It adds an additional condition where the IDs that are returned are the lowest numbered ID in each group.
SELECT * FROM products
WHERE sold = 0
AND ID IN (SELECT MIN(ID) FROM products WHERE sold = 0 GROUP BY Group_Name)
Why do you want that, though? That is not a normal desired end state. You should ask yourself why you care about the ID. It looks like your goal is to figure out which products have not sold anything. In that case, I would recommend this instead:
SELECT DISTINCT Group_Name, Name
FROM products
WHERE sold = 0
ORDER BY Group_Name, Name
I found the solution by using the statement GROUP BY,
SELECT * FROM products WHERE sold = 0 GROUP BY group
in the results now, i get only one record for each group and the minimal id without adding any other statement, and in my real table i am using product_group instead of group because it's a reserved word.
Try this:
SELECT `ID`, `Group`, `Name`, `Sold` FROM products WHERE sold = 0 GROUP BY `Group`;
My environment:
I have these tables:
Table ___Billing:
|----------|----------|----------|--------------|---------------------|
| BIL_Id | BIL_Item | BIL_Rate | BIL_Quantity | BIL_ApplicableTaxes |
|----------|----------|----------|--------------|---------------------|
| 1 | Hot-Dog | 4.50 | 2 | 3 |
| 2 | Tea | 3.25 | 3 | 3,4 |
|----------|----------|----------|--------------|---------------------|
BIL_Id = the ID for this item in this table.
BIL_Item = the id of the item (referring to the ___SalesTaxes table).
BIL_Quantity = the quantity of the item the customer used.
BIL_ApplicableTaxes = the id of the applicable taxes for this item. Each taxe id is comma separated.
Table ___SalesTaxes:
|----------|--------------|------------|
| STX_Id | STX_TaxeName | STX_Amount |
|----------|--------------|------------|
| 3 | Tax 1 | 3.50 |
| 4 | Tax 2 | 6.55 |
|----------|--------------|------------|
STX_Id = the ID for this item in this table.
STX_TaxeName = the name of the taxe.
STX_Amount = the amount in percentage of the taxe.
My question:
How is it possible from these two tables, to loop into the ___Billing table in order to get the taxes in percentage I need to applied ?
For example :
For the first row, I should have: 3.50.
For the second row, I should have : 3.50, 6.55.
Hope the question is clear.
Thanks.
If you phrase your query correctly, you can use MySQL's FIND_IN_SET() function to match tax IDs agains the CSV list you have in the BIL_ApplicableTaxes column:
SELECT t1.BIL_Id,
t1.BIL_Rate,
GROUP_CONCAT(COALESCE(t2.STX_Amount, 'NA')) AS tax_due
FROM ___Billing t1
LEFT JOIN ___SalesTaxes t2
ON FIND_IN_SET(t2.STX_Id, t1.BIL_ApplicableTaxes) > 0
GROUP BY t1.BIL_Id
However, you should seriously consider normalizing the ___Billing table and removing the CSV data. Instead, each tax entry should have its own record.
Demo here:
SQLFiddle
I have the following problem: I have a table import_data
The table is enriched something like this
| id | profile_id | sku | vendor | price | importRun |
| 1 | 39 | 123 | myVen | 2.0 | 1 |
| 2 | 39 | 456 | myVen | 2.0 | 1 |
| 3 | 39 | 123 | myVen | 3.0 | 2 |
What I need to get is an output of all elements, but only once. Every sku must be unique. To make it even worse, I need the newest data, if they are redundant.
My output should be like this:
| id | profile_id | sku | vendor | price | importRun |
| 2 | 39 | 456 | myVen | 2.0 | 1 |
| 3 | 39 | 123 | myVen | 3.0 | 2 |
Short Summary about the structure:
ID = PK
sku = a unique identifier for an article
importRun = Needed for comparision.
So, to explain it: I have an import-script, which reads a given CSV-file and imports all articles listed in it. I get this CSV-file in a regular period (once a week). I need to collect all data and save it, to create a price-evolution later on.
After every Import, I increment the number of importRun, so that no information is lost (remember, I can't use REPLACE INTO or INSERT IGNORE.
Now, when I export it, I need the newest Data, if a SKU occures multiple times. In this case, the SKU 123 is inserted 2 times on 2 different runs. That means, since my newest Run had the number 2, I need this tupel (and ignore the first one).
When I work with DISINCT, it would still output the same table, since they aren't distinct at all, because importRun differs.
I should be able to use GROUP BY, but I couldn't figure out which value will be taken, the first- or the last one? (importRun 1 or 2)
Update 1
Followed by the idea of #mitkosoft, I tried the following:
SELECT DISTINCT t1.*
FROM import_data t1
INNER JOIN import_profiles imp on t1.profile_id = imp.id
INNER JOIN (
SELECT DISTINCT sku, MAX(importRun) AS importRun
FROM import_data
GROUP BY sku ) t2
ON t1.sku = t2.sku
WHERE imp.creditornr = 73329
AND t1.vendor = 'rackmountit'
AND t1.importRun = t2.importRun
** Update 2 **
I added the complete Structures for all tables, which are relevant.
Import-Data:
|
Import-Profiles
But i still get duplicates :/
All you need to do is to determine MAX(importRun) for each sku:
SELECT
t1.*
FROM
import_data t1
INNER JOIN (
SELECT sku, MAX(importRun) AS importRun FROM import_data GROUP BY sku
) t2
ON t1.sku = t2.sku
AND t1.importRun = t2.importRun
Output is:
+----+------------+-----+--------+-------+-----------+
| id | profile_id | sku | vendor | price | importRun |
+----+------------+-----+--------+-------+-----------+
| 2 | 39 | 456 | myVen | 2.0 | 1 |
| 3 | 39 | 123 | myVen | 3.0 | 2 |
+----+------------+-----+--------+-------+-----------+
2 rows in set
Perhaps you could join on the same table where sku = sku