1- I am sorry for the title, I couldn't describe my complex situation better.
2- I have a table for a Double Accounting System where I am trying to calculate the balance at a specific date and until a specific transaction, and due to specific situations in the frond-end i need to get the result in a single query.
Table example is like that:
| id | date | amount |
| --- | ---------- | ------ |
| 93 | 2018-03-02 | -200 |
| 94 | 2018-01-23 | 250 |
| 108 | 2018-03-05 | 400 |
| 120 | 2018-01-23 | 720 |
| 155 | 2018-03-02 | -500 |
| 170 | 2018-03-02 | 100 |
And here is my simple query that I am using inside a loop of every transaction, because I want to show the new BALANCE after every transaction is made:
... for ...
Transactions::where('date', '<=', $item->date)->get()
... end ...
That query is returning the balance at the END of the day, means until the last transaction made that day, and I don't want this result.
Desired result is achieved by something like:
... for ...
Transactions::where('date', '<=', $item->date)
-> and transaction is < index of current $item
->get()
... end ...
Of course I can't use the ID because the ID is not related in this situation, as the whole ordering and calculation operations are date related.
So basically what i want is a query to get all the transactions from the age of stone until a specific date BUT exclude all the transactions made after the CURRENT one (in the loop).
For example, in the above table situation the query for:
Transaction ID # 93 should return: 93
Transaction ID # 94 should return: 94
Transaction ID # 108 should return: 94,120,93,155,170,108
Transaction ID # 120 should return: 94,120
Transaction ID # 155 should return: 94,120,155
..
...
....
The last transaction to get should be the current transaction.
I hope I could clear it well, I spend 3 days searching for a solution and I came up with this slow method:
$currentBalance = Transaction::where('date', '<=', $item->date)->get(['id']);
$array = array();
foreach ($currentBalance as $a) {
$array[] = $a->id;
}
$newBalanceA = array_slice($array, 0, array_search($item->id, $array) + 1);
$currentBalance = Transaction::whereIn('id', $newBalanceA)->sum('amount');
return $currentBalance;
It is slow and dirty, I appreciate saving me with a simple solution in 1 query if this is possible.
Related
My user_level database structure is
| user_id | level |
| 3 | F |
| 4 | 13 |
| 21 | 2 |
| 24 | 2 |
| 33 | 3 |
| 34 | 12+ |
I have another table users
| id | school_id |
| 3 | 3 |
| 4 | 4 |
| 21 | 2 |
| 24 | 2 |
| 33 | 3 |
| 34 | 1 |
What I have to achieve is that, I will have to update the level of each user based on a certain predefined condition. However, my users table is really huge with thousands of records.
At one instance, I only update the user_level records for a particular school. Say for school_id = 3, I fetch all the users and their associated levels, and then increase the value of level by 1 for those users (F becomes 1, 12+ is deleted, and all other numbers are increased by 1).
When I use a loop to loop through the users, match their user_id and then update the record, it will be thousands of queries. That is slowing down the entire application as well as causing it to crash.
One ideal thing would be laravel transactions, but I have doubts if it optimises the time. I tested it in a simple query with around 6000 records, and it was working fine. But for some reason, it doesnt work that good with the records that I have.
Just looking some recommendation on any other query optimization techniques.
UPDATE
I implemented a solution, where I would group all the records based on the level (using laravel collections), and then I would only have to issue 13 update queries as compared to hundreds/thousands now.
$students = Users::where('school_id', 21)->get();
$groupedStudents = $students->groupBy('level');
foreach ($groupedStudents as $key => $value) :
$studentIDs = $value->pluck('id');
// condition to check and get the new value to update
// i have used switch cases to identify what the next level should be ($NexLevel)
UserLevel::whereIn('userId', $studentIDs)->update(["level" => $nextLevel]);
endforeach;
I am still looking for other possible options.
First defined a relationship in your model, like:
In UserLevel model:
public function user() {
return $this->belongsTo(\App\UserLevel::class);
}
And you can just update the level without 12+ level's query, only by one query, and delete all 12+ level by one query.
UserLevel::where('level', '<=', 12)->whereHas('user', function($user) {
$user->where('school_id', 3);
})->update(['level' => DB::raw("IF(level = 'F', 1, level+1)")]);
UserLevel::whereHas('user', function($user) {
$user->where('school_id', 3);
})->where('level', '>', 12)->delete();
If your datas is too huge. you can also use chunk to split them for reduce memory consumption.
like this:
UserLevel::where('level', '<=', 12)->whereHas('user', function($user) {
$user->where('school_id', 3);
})->chunk(5000, function($user_levels) {
$user_levels->update(['level' => DB::raw("IF(level = 'F', 1, level+1)")]);
});
UserLevel::whereHas('user', function($user) {
$user->where('school_id', 3);
})->where('level', '>', 12)->delete();
Consider the following table
+-------------+---------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------------+---------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| date | date | NO | | NULL | |
| sku | varchar(10) | | | NULL |
| impressions | int(11) | NO | | NULL | |
| sales | int(11) | NO | | NULL | |
+-------------+---------+------+-----+---------+----------------+
The table gets populated daily from a bulk download of the previous days sales records.
Each days download not only contains the previous days sales data but also all data from the last 90 days (possible 50k+ records).
However the data for previous days may change since the original insert due to matters outside our control, e.g.
Day 1.
Date: 2015-01-01
SKU: ABCD
Impressions: 100
Sales: 0
Day 2.
Date: 2015-01-01
SKU: ABCD
Impressions: 100
Sales: 3
Date: 2015-01-02
SKU: ABCD
Impressions: 105
Sales: 0
So for any given record from the data download it could be
a) Already seen and the same as before - ignore
b) New - add to database
c) Already seen but new data - Update
Arguably this could be trivially solved by checking each row as so
while (!$file->eof()) {
$row = $file->fgets();
$data = explode("\t", $row);
$sku = $data[0];
$date = $data[1];
$impressions = $data[2];
$sales = $data[3];
$order = $em->getRepository('Orders')->findOneBy(['sku' => $sku, 'date' => $date]);
if($order && $order->getImpressions() != $impressions && $order->getSales() != $sales) {
$order->setImpressions($impressions);
$order->setSales($sales);
} else {
... create new model
}
$em->persist($order);
}
However the rows which will have updated data will be minimal and doing a select for each and every row would mean this job would be incredibly slow due to sheer number of rows.
So my question is what patterns could be used to solve this problem as efficiently as possible?
Any ideas welcome
I would suggest you completely replace the previous 90 days' data with the newly downloaded data.
The reasoning is simple:
The processing time to do this will be trivial. 50,000 rows is tiny in database terms. I would probably do this even if it were a million rows.
Trying to replace only the changed rows is complicated and could introduce errors.
When you say "same as before" it seems like the keys are date and sku (combined) and sales and impressions are the fields that could be updated. If that's correct, then the most efficient way to do this in MySQL is to use INSERT ... ON DUPLICATE KEY UPDATE ... query:
Create a unique key on date and sku columns.
In your php script pre-parse all data from file (or do it in batches if you'd like).
Run a query similar to this (substitute actual data from parsed values in step 1):
INSERT INTO
mytable (`date`, sku, impressions, sales)
VALUES
('2015-01-01', 'ABCD', 100, 3),
('2015-01-02','ABCD', 100, 3),
...
ON DUPLICATE KEY UPDATE
impressions = VALUES(impressions),
sales = VALUES(sales)
A couple of notes:
check out the documentation for this syntax
if the next day's data update containing previous date record was supplementary, you could do sales = sales + VALUES(sales) but I don't think that's the case for you
EXPLAIN
Let's say we have this transaction table :
+----+-------------+--------+----------------+----------------+---------------------+
| id | id_customer | amount | id_where_trans | id_money_from | date_trans |
+----+-------------+--------+----------------+----------------+---------------------+
| 1 | 10 | 10 | 100 | NULL | 2015-07-20 10:20:30 |
| 2 | 10 | -2 | 100 | NULL | 2015-07-21 09:10:11 |
| 3 | 10 | 7 | 120 | NULL | 2015-07-24 18:22:25 |
| 4 | 10 | -11 | 120 | here the magic | 2015-07-24 18:22:26 |
+----+-------------+--------+----------------+----------------+---------------------+
Read this in "human" language:
id 1 => Customer Alessandro (id_customer 10) charge 10€ on his account in shop A (id_where trans = 100)
id 2 => Customer Alessandro spent 2€ on Shop A
id 3 => Customer Alessandro charge 7€ on his account in shop B
id 4 => Customer ALessandro spend 11€ in shop B.
At the end of period (month, for example), Shop B need to receive 8€ (+10-2) from another shop, in our example shop A.
/end of human read.
You imagine that insert is a simple (for id 4, for example):
INSERT INTO transaction (id_customer, amount, id_where_trans, date) VALUES (10,-11,120,2015-07-24 18:22:26)
GOAL
My goal is set - during the insert, and if possible / simpler via PHP - the SQL splitted in two or more SELECT:
INSERT INTO transaction (id_customer, amount, id_where_trans, id_money_from, date) VALUES (10,-8,120,100,2015-07-24 18:22:26)
INSERT INTO transaction (id_customer, amount, id_where_trans, id_money_from, date) VALUES (10,-3,120,NULL,2015-07-24 18:22:27)
+----+-------------+--------+----------------+------------+---------------------+
| id | id_customer | amount | id_where_trans | id_money_from | date |
+----+-------------+--------+----------------+------------+---------------------+
| 1 | 10 | 10 | 100 | NULL | 2015-07-20 10:20:30 |
| 2 | 10 | -2 | 100 | NULL | 2015-07-21 09:10:11 |
| 3 | 10 | 7 | 120 | NULL | 2015-07-24 18:22:25 |
| 4 | 10 | -8 | 120 | 100 | 2015-07-24 18:22:26 |
| 5 | 10 | -3 | 120 | NULL | 2015-07-24 18:22:27 |
+----+-------------+--------+----------------+------------+---------------------+
If I can get a table with that informations, I can run a query that make my final job correctly.
Basically, I need to calculate the id_shop where money are taken for discharge (in this case, -8 from 100, that made previous charge, and so is NOT NULL, and -3 from 120,itself, THIS IS NULL).
Please note that FIRST I consume / discharge the previous charge (id 4 now has id_money_from 100), AFTER I will consume / discharge others amounts (id 5 in effect is NULL, because that -3 has taken from id 3)
ASSUMPTION / MANDATARY
1) id are INCREMENTAL, no concurrency. Date are INCREMENTAL, date of id 4 is >= of id 3 and so on.
2) If amount of last recharge is made from same id where transaction is done (see id 1 and id 2) insert NULL (I need mandatary NULL)
3) First charge made, first need to be zero-ed, and so on.
MY LOGIC / PSEUDOCODE
1) If a transaction is negative, make a recursive getting LAST positive charge which, summed to the negative susequential, is not zero.
getted this charge, if id_where_trans==id_money_from we need insert, simply insert, with NULL
INSERT INTO transaction (id_customer, amount, id_where_trans, id_money_from, date) VALUES (10,-2,120,NULL,2015-07-24 18:22:26)
Recursive start here
If this amount is <= of last recharge, walking in database and split negative in two or more id_where_trans (problaby in real scenario this will be impossible, but I need to think that amount from discharge (id_money_from) could be splitted by 2, 3, x id_where_trans). For example, if SHOP A charge +1, Shop B charge +1, Shop C charge +1 and SHOP D DISCHARGE -3, we need to insert 3 different rows.
$amount_to_discharge = x;
$last_positive = $query->("SELECT * FROM transaction WHERE "); // make the SUB-SUM
if ($last_positive['amount'] >= $amount_to_discharge) {
$query->("INSERT INTO......");
} else {
// start recursive
$sql = "SELECT * FROM AMOUNT WHERE amount > 0 AND id < $last_positive['id']";
$new_search = $query->sql($sql);
// how implement correctly that recursive?
}
Thank you guy. Let me know if you need others explain!
i'm using mysql to store Kwh usage of my home. I get a fault rarely and get a 0 value to get stored. When i extract the values from my table i don't want to get that 0s but the last valid value before.
SELECT unix_timestamp(dataora) as time, kwhg
FROM misure
WHERE dataora BETWEEN '2013-10-08 00:00:00.000' AND '$data_scelta 23:59:59.997'
GROUP BY date(dataora),hour(dataora)
I used the above code to get the below table:
+-------+-----------+
| time | kwhg |
+-------+-----------+
| 9 | 2 |
| 10 | 3 |
| 11 | 0 |
| 12 | 4 |
| 13 | 0 |
+-------+-----------+`
I want to obtain
+--------+----------+
| time | kwhg |
+-------+-----------+
| 9 | 2 |
| 10 | 3 |
| 11 | 3 |
| 12 | 4 |
| 13 | 4 |
+-------+-----------+`
and remove the zero with the previus value.
Any tricks to do that?
You can use MySQL user-defined variables to return either the current row's value of kwhg if it's greater than zero, or else the variable defined on the previous row.
SELECT unix_timestamp(dataora) as time, #kwhg := IF(kwhg>0, kwhg, #kwhg) AS kwhg
FROM misure
WHERE dataora BETWEEN '2013-10-08 00:00:00.000' AND '$data_scelta 23:59:59.997'
GROUP BY date(dataora),hour(dataora)
Like #OddEssay's answer, this can't come up with a nonzero value if the first entry is zero. In that case, it will return whatever the current value of #kwhg is, which is probably NULL unless you've run the query before in the current session.
If it's a small result set, you could simply loop over it and created a fixed dataset with something like:
$fixedResults = array();
$lastGoodReading = 0;
foreach($results as $row){
if($row['kwhg']){
$fixedResults[$row['time']] = $row['kwhg'];
$lastGoodReading = $row['kwhg'];
} else {
$fixedResults[$row['time']] = $lastGoodReading;
}
}
Which will work if there is multiple failed readings in a row, but will still give zero if the first result fails.
You might also want to do something a bit more advanced, like checking both the previous result, and the next result and take an average of the two.
in mysql
UPDATE mytable
SET kwhg = (#n := COALESCE(number, #n))
ORDER BY time;
#n is a MySQL user variable
I have a SQL table being created daily that is downloaded from a suppliers website,containing product info, that is in csv. I have everything creating alright and all the tables are identical. The problem that I am needing solved is that I need to compare the tables between today and yesterday (table names are the dates in following format mm-dd-yyyy) I need to compare a few different columns for different things.
I need to know all products that are in today's data that weren't in
yesterdays (can be checked by supplier SKU)
I need to know all product that were in yesterday's data that is no
longer in today's
I need to know when the price went up from yesterday's data
I need to know when the price has gone down from yesterday's data
I need to know when a sale has started based on yesterday's data as
well as stopped
These need to show the following labels in the table that will show the changes
regular up
regular down
miscillanious change (description change or change to a fields that aren't a priority)
promo on (discount added from supplier)
promo off (discount taken off by supplier)
delete (no record of the product in new list {probably been deleted})
new item (new record of product in new list)
out of stock
I have been searching everywhere for the answer for these issues and have found stuff that kind of shows me how to do this using union and join but I don't fully understand how to use them based on this scenario.
I have tried different PHP solutions by going through each piece of data and searching for the sku in the new table and vice versa then checking for any changes if they exist in both tables but this is taking a really long time and I have over 200 000 products in these tables. I am hoping that I can do these in less queries and by letting the sql server do more work then the php script.
Thanks for all the help!
Yesterday's Table
__________________________________________________________
| id | price | sale | description | qty | sku |
---------------------------------------------------------
| 1 | 12.50 | 0.00 | description product 1 | 12 | 12345 |
| 2 | 22.99 | 20.99 | describe the problem | 1 | 54321 |
| 3 | 192.99 | 0.00 | description ftw | 5 | 53421 |
| 4 | 543.52 | 0.00 | description | 15 | 45121 |
----------------------------------------------------------
Today's Table
__________________________________________________________
| id | price | sale | description | qty | sku |
---------------------------------------------------------
| 1 | 12.50 | 0.00 | description product 1 | 12 | 12345 |
| 2 | 22.99 | 0.00 | describe the problem | 1 | 54321 |
| 3 | 192.99 | 50.00 | description ftw | 5 | 53421 |
| 4 | 523.99 | 0.00 | description | 15 | 45123 |
----------------------------------------------------------
I need the new table to look like the following
_____________________________________________________________
| id | sku | label | description | price |
-------------------------------------------------------------
| 1 | 54321 | promo off | describe the problem | 22.99 |
| 2 | 53421 | promo on | description ftw | 192.99|
| 3 | 45123 | new item | description | 523.99|
| 4 | 45121 | delete | description | 543.52|
-------------------------------------------------------------
The following is the code I have for the deleted and new items currently. I am using int for the label/status in the example below and just signifying the different numbers.
$deleted = mysql_query("SELECT * FROM `test1`") or die(mysql_error());
while($skus= mysql_fetch_array($deleted))
{
$query = mysql_num_rows(mysql_query("SELECT * FROM `test2` WHERE SKU='".$skus['sku']."'"));
if($query < 1)
{
$tata= mysql_query("INSERT INTO `gday` (Contract_Price, SKU, status) VALUES (".$skus['price'].", ".$skus['sku'].", 1)");
}
}
$deleted = mysql_query("SELECT * FROM `test2`") or die(mysql_error());
while($skus= mysql_fetch_array($deleted))
{
$query = mysql_num_rows(mysql_query("SELECT * FROM `test1` WHERE SKU='".$skus['sku']."'"));
if($query < 1)
{
$tata= mysql_query("INSERT INTO `gday` (Contract_Price, SKU, status) VALUES (".$skus['price'].", ".$skus['sku'].", 2)");
}
}
EDIT:
The Following is the true table that all the data will be going into. I originally didn't want to muddy the water with the large table but by request I have included it.
ID
Status
DiscountDate
Price
Discount
DiscountEndDate
Desc1
Desc2
Desc3
Warranty
Qty1
Qty2
PricingUnit
PriceUpdate
Vendor
Category
UPC
Weight
WeightUnit
Because of the size of your database I could propose you an SQL solution.
When you work with a lot of data, PHP can be slow for several reason.
You can try to do some functions.
You just have to store the status data in an other table. This is the documentation to write a trigger.
http://dev.mysql.com/doc/refman/5.0/en/triggers.html
Becareful if you have a lot of operation on your table I can suggest you to use PostgresSQL instead of mysql because I found it more simple to write function, trigger, ... using PL/sql
EDIT: Just have to write a simple function
I'm working on it at the moment. Think about using select case like in this answer
EDIT: Core function
DELIMITER |
CREATE PROCEDURE export()
BEGIN
(SELECT today.id, today.sku,
CASE today.price
WHEN today.price = yesterday.price THEN 'nothing'
WHEN today.price < yesterday.price THEN 'promo on'
ELSE 'promo off' END AS label
FROM today, yesterday WHERE yesterday.sku = today.sku)
UNION
(
SELECT today.id, today.sku,
'new item' AS label
FROM today LEFT JOIN yesterday ON yesterday.sku = today.sku WHERE yesterday.sku IS NULL)
UNION
(
SELECT yesterday.id, yesterday.sku,
'delete' AS label
FROM yesterday LEFT JOIN today ON today.sku = yesterday.sku WHERE today.sku IS NULL
);
END|
DELIMITER ;
To call just do:
CALL export();
Here is an example of possible core functions. Be careful in this case id could be the same. In the function you'll have to add a personal one in the first column.
If you need performance to display it faster in PHP, think about APC cache