Querying sum of multiple aggregate columns (without groupBy) - php

I have a transactions table and I'm trying to get the total of each type.
To simply put it looks like this
id
type
credit_movement
1
top_up
10000
2
fee
-50
3
deduct
-1000
I am trying to get sum of each type to show as a report.
top_up: 10000
fee: 50
deduct: 1000
net_expense: 9850 [top_up - deduct - fee]
$types = [
'top_up' => ['top_up'],
'deduct' => ['deduct'],
'fee' => ['fee'],
'net_expense' => ['top_up', 'deduct', 'fee'],
];
$query = DB::table('transactions');
foreach ($types as $type => $fields) {
$query->selectSub(function ($query) use ($fields) {
return $query->selectRaw('SUM(credit_movement)')->whereIn('type', $fields);
}, $type);
};
$results = $query->get();
When I do this, I get:
1140 In aggregated query without GROUP BY, expression #1 of SELECT list contains nonaggregated column 'project.transactions.type'; this is incompatible with sql_mode=only_full_group_by..
When I change my database.mysql.strict = false, it works; however I want to make it work properly without needing to change the mysql config.
As of my understanding, this error indicates that I am only selecting aggregated columns, but in my case I don't actually want to groupBy() anything as this is just reports.
If I try to groupBy('type') it returns everything grouped by type, but the queries are only run within that group.
{
0: {
top_up: 10000,
deduct: 0,
fee: 0,
net_expense: 10000
}
1: {
top_up: 0,
deduct: -1000,
fee: 0,
net_expense: -1000
},
// etc...
}
Is there a way to obtain without changing strict to false?
{
0 => {
top_up: 10000,
deduct: -1000,
fee: -50,
net_expense: 9850
}
}

If I understand you correctly this might be very easy but again I might have not understood it right.
$result = DB::table('transactions')->selectRaw('type, SUM(credit_movement) as sum')->groupBy('status')->get();
This should return something like this:
type
sum
fee
-5656
topup
8758
deduct
-7625
For the total sum you can just do it in php which would make it easier
$net = $result->sum('sum'); // equals -5656+8758-7625
Hope this helps and let me know if I am wrong about it.

The problem with your approach is in the final column that is the sum of the other 3 so you can't use SUM because you don't have a column to group.
You could use a subquery but I think that the best solution is to add a little elaboration of the raw data that you get from a simpler query.
$query = DB::table('transactions')
->selectRaw('type, SUM(credit_movement) AS movements')
->groupBy('type');
$results = array_reduce($query->get(), function(array $res, array $value){
$res[$array['type']] = $array['movements'];
return $res;
}, []);
$results['net_expense'] = array_sum($results);

Related

Laravel Paginate method takes too much time for 1 million record

So this is my case. I have a main table payment_transactions having almost 1 million records.
I am getting the data from this table with joins and where clauses and there is Laravel paginate method,
But this method takes too much time and after investigation, I found that its count method is taking time like 4 to 5 seconds just for counting.
So how can I optimize and speed up this query, especially is there any way to improve paginate method speed?
Note: I can't use simplePaginate because there is a datatable on frontend and I need a total count for that.
So for paginate, two queries run
1 is the main query and other one is for count, and I felt that, the count query is taking much time.
Here is the count query after getQueryLog
select count(*) as aggregate from `payment_transactions`
left join `users` as `U` on `U`.`id` = `payment_transactions`.`user_id`
left join `coupons` as `C`
on `C`.`id` = `payment_transactions`.`coupon_id`
where `payment_transactions`.`refund_status` = 'NO_REFUND'
and `payment_transactions`.`transaction_type`
in ('BOOKING','SB_ANPR','QUERCUS_ANPR','CANDID_ANPR','SB_TICKET',
'ORBILITY_TICKET','TOPUP,CREDIT','DEBIT','GIFT')
and `payment_transactions`.`status` != 'INITIATED'
Here is my code example:
//Get Transactions data
public function adminTransactions(Request $request)
{
$selectableFields = [
'payment_transactions.id', 'payment_transactions.transaction_id AS transaction_id',
'payment_transactions.refund_status',
'payment_transactions.created_at', 'payment_transactions.response_data', 'payment_transactions.status',
'payment_transactions.transaction_type', 'payment_transactions.payment_mode','payment_transactions.payment_source',
'payment_transactions.subtotal', 'payment_transactions.service_fees', 'C.coupon_code','C.amount AS coupon_value',
DB::raw("IF(payment_transactions.refund_remarks='NULL','-NA-',payment_transactions.refund_remarks) as refund_remarks"),
DB::raw("IF(payment_transactions.transaction_type='TOPUP' AND payment_transactions.coupon_id IS NOT NULL
AND payment_transactions.coupon_id!=0,
payment_transactions.amount + C.amount,payment_transactions.amount) as amount"),
DB::raw("CONCAT(U.first_name,' ',U.last_name) AS username"), 'U.id AS user_id',
DB::raw("JSON_UNQUOTE(json_extract(payment_transactions.response_data, '$.description')) AS description"),
DB::raw("payment_transactions.invoice_id"),
DB::raw("JSON_UNQUOTE(json_extract(payment_transactions.response_data, '$.Data.PaymentID')) AS upay_payment_id"),
];
return PaymentTransactions::select($selectableFields)
->with('homeScreenMessages:payment_transaction_id,from_name,message,amount')
->leftJoin('users AS U', 'U.id', '=', 'payment_transactions.user_id')
->leftJoin('coupons AS C', 'C.id', '=', 'payment_transactions.coupon_id')
->where(DB::raw("CONCAT(U.first_name,' ',U.last_name)"), 'like', "%{$request->input('query')}%")
->orWhere('U.id', $request->input('query'))
->orWhere("U.phone_number", "LIKE", "%" . $request->input('query') . "%")
->orWhere("U.email", "LIKE", "%" . $request->input('query') . "%")
->orWhere('payment_transactions.id', $request->input('query'))
->orWhere('payment_transactions.transaction_id', $request->input('query'));
}
//Paginate function
public function paginationCalculate($queryObject, $request) {
$draw = $request->get('draw');
$start = $request->get("start");
$rowperpage = $request->get("length"); // Rows display per page
$columnIndex_arr = $request->get('order');
$columnName_arr = $request->get('columns');
$order_arr = $request->get('order');
$columnIndex = $columnIndex_arr[0]['column']; // Column index
$columnName = $columnName_arr[$columnIndex]['name']; // Column name
$columnSortOrder = $order_arr[0]['dir']; // asc or desc
$pageNumber = ($start + $rowperpage) / $rowperpage;
if(!empty($columnName)) {
$queryObject->orderBy($columnName, $columnSortOrder);
}
$records = $queryObject->paginate($rowperpage, ['*'], 'page', $pageNumber)->toArray();
return array(
"draw" => intval($draw),
"recordsFiltered" => $records['total'],
"recordsTotal" => $records['total'],
"data" => $records['data']
);
}
I don't think you want "LEFT". With LEFT, the COUNT will count payment transactions even if there is no matching user and/or coupon.
Or maybe there are multiple users or coupons for each payment_transaction? In this case, the COUNT will be inflated. To fix this, simply remove the two extra tables.
Regardless of the situation, add this composite (and covering) index to payment_transactions:
INDEX(refund_status, transaction_type, status, user_id, coupon_id)
refund_type must be first since it is tested with '='.
If that is not "fast enough", then I must ask you "Who cares what the exact number when there are a million rows?" Do you ever see a search engine giving an exact number of hits? Does that imprecision bother you?
That is, rethink the requirement to calculate the exact number of rows. Or even an approximate number.
You mentioned "pagination". Please show us how that is done. OFFSET is an inefficient way since it rescans the previously seen rows again and again. Instead, 'remember where you left off': Pagination
You HAVE to adding INDEX to your MySQL database. Makes it lighting fast!
If you are using Navicat, click on the Index tab, and add the Column name and Name it the same. You WILL notice a huge difference.

Codeigniter 4 Search Pagination not counting and returning page

I'm working on a simple search project where I'm returning the results. The search function appears to work however, the total and page return the wrong values. The total field returns the total number of rows inside the data, not the total number of results from the search and the page is always {}.
Here's the model->function I've created:
public function search($string)
{
$results = $this->select('*')->orLike('title', $string)->orLike('excerpt', $string);
if ( empty( $results ) )
{
return [];
} else
{
$data = [
'results' => $results->paginate(2),
'total' => $results->countAllResults(),
'page' => $this->pager,
];
return $data;
}
}
What's puzzling is if I place the total field above the results value the count works, but then the result fields returns everything in the database at paginate(2).
Ok, I managed to solve this query by adding two separate queries to the database. The processing cost appears to be minimal and it should be alright when caching the responses. As it turns out you can chain queries but only in a particular order and if you use grouping (see ->groupStart() )
$results = $this->select('title, image, categories, id, excerpt')->groupStart()->like('title', $search)->orLike('excerpt', $search)->groupEnd()->where('status','live')->paginate(2);
$total = $this->select('title, image, categories, id, excerpt')->groupStart()->like('title', $search)->orLike('excerpt', $search)->groupEnd()->where('status','live')->countAllResults();
Some may argue the inefficiency of the two queries, but this works for my use case :) Hope this helps anyone else stuck on a similar problem.

codeigniter fetch mysql table with where_in + keep order of key array in result array

I have a string with image ids (fetched from another mysql table)
and converted to an array:
$idstring = "12, 18, 3, 392, 0, 9, 44";
$idarray = explode(',', $idstring);
Based on this array of ids, I want get all the rows from my "media" mysql table.
$result = $this->db->select('*')
->from('media')
->where_in('id', $ids)
->get()->result_array();
The problem is the $result array's values are in a weird order like this:
$result's order : 44, 9 ,0 ,18 ,3 ,392 ,12 ...
But i need them to stay like in my $id string/array order...
I've tried 4 approaches to solve the issue so far:
Fetch rows in a loop without where_in() - what creates a lot of queries - but works for now ...
Reorder the $result array based on the order of the $idstring or the $idarray, though I could not manage to to find a working result and I don't get the point why this step is necessary at all
Try to get the query itself fixed. I've heard about ORDER_BY and FIND_IN_SET, $ids but I could not get it into my a working codeigniter query and don't know about the performance if this is really a help
So in conclusion, I think this should be a simple everyday task, i just want to fetch a bunch of pictures in a given order with codeigniter.
Am I missing a simple solution here?
Use Field() function of mysql
$result = $this->db->select('*')
->from('media')
->where_in('id', $ids)
->order_by("FIELD(id,".join(',',$ids).")")
->get()
->result_array()
it should be something like
FIELD(id,12, 18, 3, 392, 0, 9, 44)
Reference
Field() returns the index position of a comma-delimited list
The accepted answer should be M Khalid Junaid answer. But just in case, if you are generating from an array like I do, use it like this:
$filters = [
"order_by" => [
"title" => "DESC",
""FIELD(id,".join(',',$ids).")" => "" // The option should be empty...
]
];
foreach($filters["order_by"] as $attribute => $option){
$this->db->order_by( $attribute, $option );
}
The output will be:
...
ORDER BY `name` DESC, FIELD(id, 2017, 2031, 2032, 2034, 2035)
LIMIT 50
Just to for clarification.

MongoDB/PHP - MapReduce Max value

I used to group on mongoDB via PHP to get the max date of my items.
As i have too many items (more than 10 000), i read i must use MapReduce.
Here's my past group function :
$keys = array('ItemDate'=> true);
$initial = array('myLastDate' => 0);
$reduce = "function(obj, prev) {
if (myLastDate < obj.ItemDate) {
myLastDate = ItemDate;
}
}";
$conds = array( 'ItemID' => (int) $id );
$results = $db->items->group($keys, $initial, $reduce,
array('condition'=> $conds ) );
I've tried something but seems not to work ...
$map = new MongoCode("function() {
emit(this.ItemID,this.ItemDate);
}");
$reduce = new MongoCode("function(obj, prev) {
if(prev.myLastDate < obj.ItemDate) {
prev.myLastDate = obj.ItemDate;
}
}");
$items = $db->command(array(
"mapreduce" => "items",
"map" => $map,
"reduce" => $reduce,
"query" => array("ItemID" => $id);
$results = $db->selectCollection($items['result'])->find();
Can you please help ?
Solution
You don't need to use map/reduce for that. Provided your date field contains an ISODate, a simple query does the trick:
db.yourColl.find({},{_id:0,ItemDate:1}).sort({ItemDate:-1}).limit(1)
In order to have this query done efficiently, you need to set an index on ItemDate
db.yourColl.createIndex({ItemDate:-1})
Explanation
The query
Let us dissect the query. db.yourColl...
.find({} The default query
,{_id:0,ItemDate:1} We want only ItemDate to be returned. This is called a projection.
.sort({ItemDate:-1}) The documents returned should be sorted in descending order on ItemDate, making the document with the newest date the first to be returned.
.limit(1) And since we only want the newest, we limit the result set to it.
The index
We create the index in descending order, since this is the way you are going to use it. However, if you need to change the default query to something else, the index you create should include all fields you inspect in the query, in the exact order.

Create SQL query that orders results by which condition they meet

This is for MySQL and PHP
I have a table that contains the following columns:
navigation_id (unsigned int primary key)
navigation_category (unsigned int)
navigation_path (varchar (256))
navigation_is_active (bool)
navigation_store_id (unsigned int index)
Data will be filled like:
1, 32, "4/32/", 1, 32
2, 33, "4/32/33/", 1, 32
3, 34, "4/32/33/34/", 1, 32
4, 35, "4/32/33/35/", 1, 32
5, 36, "4/32/33/36/", 1, 32
6, 37, "4/37/", 1, 32
... another group that is under the "4/37" node
... and so on
So this will represent a tree like structure. My goal is to write a SQL query that, given the store ID of 32 and category ID of 33, will return
First, a group of elements that are the parents of the category 33 (in this case 4 and 32)
Then, a group of elements that are a child of category 33 (in this case 34, 35, and 36)
Then the rest of the "root" categories under category 4 (in this case 37).
So the following query will return the correct results:
SELECT * FROM navigation
WHERE navigation_store_id = 32
AND (navigation_category IN (4, 32)
OR navigation_path LIKE "4/32/33/%/"
OR (navigation_path LIKE "4/%/"
AND navigation_category <> 32))
My problem is that I want to order the "groups" of categories in the order listed above (parents of 33 first, children of 33 second, and parents of the root node last). So if they meet the first condition, order them first, if they meet the second condition order them second and if they meet the third (and fourth) condition order them last.
You can see an example of how the category structure works at this site:
www.eanacortes.net
You may notice that it's fairly slow. The current way I am doing this I am using magento's original category table and executing three particularly slow queries on it; then putting the results together in PHP. Using this new table I am solving another issue that I have with magento but would also like to improve my performance at the same time. The best way I see this being accomplished is putting all three queries together and having PHP work less by having the results sorted properly.
Thanks
EDIT
Alright, it works great now. Cut it down from 4 seconds down to 500 MS. Great speed now :)
Here is my code in the Colleciton class:
function addCategoryFilter($cat)
{
$path = $cat->getPath();
$select = $this->getSelect();
$id = $cat->getId();
$root = Mage::app()->getStore()->getRootCategoryId();
$commaPath = implode(", ", explode("/", $path));
$where = new Zend_Db_Expr(
"(navigation_category IN ({$commaPath})
OR navigation_parent = {$id}
OR (navigation_parent = {$root}
AND navigation_category <> {$cat->getId()}))");
$order = new Zend_Db_Expr("
CASE
WHEN navigation_category IN ({$commaPath}) THEN 1
WHEN navigation_parent = {$id} THEN 2
ELSE 3
END, LENGTH(navigation_path), navigation_name");
$select->where($where)->order($order);
return $this;
}
Then I consume it with the following code found in my Category block:
// get our data
$navigation = Mage::getModel("navigation/navigation")->getCollection();
$navigation->
addStoreFilter(Mage::app()->getStore()->getId())->
addCategoryFilter($currentCat);
// put it in an array
$node = &$tree;
$navArray = array();
foreach ($navigation as $cat)
{
$navArray[] = $cat;
}
$navCount = count($navArray);
$i = 0;
// skip passed the root category
for (; $i < $navCount; $i++)
{
if ($navArray[$i]->getNavigationCategory() == $root)
{
$i++;
break;
}
}
// add the parents of the current category
for (; $i < $navCount; $i++)
{
$cat = $navArray[$i];
$node[] = array("cat" => $cat, "children" => array(),
"selected" => ($cat->getNavigationCategory() == $currentCat->getId()));
$node = &$node[0]["children"];
if ($cat->getNavigationCategory() == $currentCat->getId())
{
$i++;
break;
}
}
// add the children of the current category
for (; $i < $navCount; $i++)
{
$cat = $navArray[$i];
$path = explode("/", $cat->getNavigationPath());
if ($path[count($path) - 3] != $currentCat->getId())
{
break;
}
$node[] = array("cat" => $cat, "children" => array(),
"selected" => ($cat->getNavigationCategory() == $currentCat->getId()));
}
// add the children of the root category
for (; $i < $navCount; $i++)
{
$cat = $navArray[$i];
$tree[] = array("cat" => $cat, "children" => array(),
"selected" => ($cat->getNavigationCategory() == $currentCat->getId()));
}
return $tree;
If I could accept two answers I would accept the first and last one, and if I could accept an answer as "interesting/useful" I would do that with the second.
:)
A CASE expression should do the trick.
SELECT * FROM navigation
WHERE navigation_store_id = 32
AND (navigation_category IN (4, 32)
OR navigation_path LIKE "4/32/33/%/"
OR (navigation_path LIKE "4/%/"
AND navigation_category <> 32))
ORDER BY
CASE
WHEN navigation_category IN (4, 32) THEN 1
WHEN navigation_path LIKE "4/32/33/%/" THEN 2
ELSE 3
END, navigation_path
Try an additional derived column like "weight":
(untested)
(IF(criteriaA,1,0)) + (IF(criteriaB,1,0)) ... AS weight
....
ORDER BY weight
Each criteria increases the "weight" of the sort.
You could also set the weights distinctly by nesting IFs and giving the groups a particular integer to sort by like:
IF(criteriaA,0, IF(criteriaB,1, IF ... )) AS weight
Does MySQL have the UNION SQL keyword for combining queries? Your three queries have mainly non-overlapping criteria, so I suspect it's best to leave them as essentially separate queries, but combine them using UNION or UNION ALL. This will save 2 DB round-trips, and possibly make it easier for MySQL's query planner to "see" the best way to find each set of rows is.
By the way, your strategy of representing the tree by storing paths from root to tip is easy to follow but rather inefficient whenever you need to use a WHERE clause of the form navigation_path like '%XYZ' -- on all DBs I've seen, LIKE conditions must start with a non-wildcard to enable use of an index on that column. (In your example code snippet, you would need such a clause if you didn't already know that the root category was 4 (How did you know that by the way? From a separate, earlier query?))
How often do your categories change? If they don't change often, you can represent your tree using the "nested sets" method, described here, which enables much faster queries on things like "What categories are descendants/ancestors of a given category".

Categories