I want to optimize my query I am executing with Laravel. I'm using MySQL for my project and I have a table named locations. It has more than 2 million records.
When I execute the code below it's too slow. How can I optimize my query to improve the speed?
foreach ($employees as $employee){
$percentage = 0;
if ($employee->position->zones->count() > 0) {
if($employee->position->zones->first()->workingzones->count() > 0) {
$workingZone = $employee->position->zones->first()->workingzones->last();
$tagCode = $employee->rfids->first()->rfid_id;
$zoneTime = DB::table('location')
->select(DB::raw('count(*) as count'))
->where('tagCode', $tagCode)
->where('xLocation', '>', $workingZone->x1)
->where('xLocation', '<', $workingZone->x3)
->where('yLocation', '>', $workingZone->y3)
->where('yLocation', '<', $workingZone->y1)
->where('locationDate', '>=',''.$currentDate.' 00:00:01')
->where('locationDate', '<=', $currentDate. ' 23:59:59')
->get();
$totalWorkedTime = DB::table('location')
->select(DB::raw('count(*) as count'))
->where('tagCode', $tagCode)
->where('locationDate', '>=',''.$currentDate.' 00:00:01')
->where('locationDate', '<=', $currentDate. ' 23:59:59')->get();
if ($zoneTime->first()->count == 0 || $totalWorkedTime->first()->count == 0) {
$percentage = 0;
}else {
$percentage = (int)ceil(($zoneTime->first()->count /12 )* 100 / ($totalWorkedTime->first()->count / 12));
}
}
}
$employee->percentage = $percentage;
}
You do a full ->get() twice and only use the ->first() result. When you just need 1, just use ->first() instead of ->get().
Also you can eagerload the position and zones while fetching the employees , and saving 2 additional queries per loop (- grouped eagerload queries on total), like this:
$employees = Employee::where(/*where foo here*/)
->with('position.zones')
->get();
And to consume less mem, chunk it.
Employee::where(/*where foo here*/)
->with('position.zones')
->chunk(200, function ($employees) {
foreach ($employees as $employee) {
// current code here with ->first() instead of ->get()
}
});
Adding indexes on the following columns should improve your performance:
xLocation
yLocation
locationDate
How to add indexes with an MySQL query:
https://dev.mysql.com/doc/refman/5.7/en/create-index.html
or if you use Laravel's migrations: https://laravel.com/docs/5.4/migrations#indexes
Edit:
Also: instead of doing the COUNT(*) on both select queries, use COUNT(`id`), so you don't count all columns, but only the id column
Related
Right now I have a subquery to get the count of payments for the current month and then getting the 4 products with the highest payment_count. This query works fine but I'm wondering if there's a more simple way to do the same since its getting difficult to read.
$latestPayments = DB::table('payments')
->select('product_id', DB::raw('COUNT(*) as payments_count'))
->whereMonth('created_at', Carbon::now()->month)
->groupBy('product_id');
$trendingProducts = DB::table('products')
->joinSub($latestPayments, 'latest_payments', function ($join) {
$join->on('products.id', '=', 'latest_payments.product_id');
})->orderBy('payments_count', 'DESC')->take(4)->get();
This did it!
$trendingProducts = Product::withCount(['payments' => function($query) {
$query->whereMonth('created_at', Carbon::now()->month);
}])->orderBy('payments_count', 'DESC')->take(4)->get();
If you are using eloquent query with relational database you can do like this:
$latestPaymentWithTrendingProduct = App\Payment::with(['products', function($product) {
$product->orderBy('payments_count', 'DESC')->take(4);
}])->whereMonth('created_at', date('m'))->get()->groupBy('product_id');
This will lessen the code but still do the same thing.
i have a simple where query that repeats in a foreach for some times that can be a lot really so here is my query :
for ($i = 0; $i < count($hasdate); $i++) {
$roomprice = RoomPricingHistory::
Where('accommodation_room_id', $hasroom[$i])
->where('from_date', '<=', $hasdate[$i])
->where('to_date', '>=', $hasdate[$i])
->get()->sortBy('created_at');
$lastget = last($roomprice);
$last_price = last($lastget);
if ($last_price) {
$final_price[] = $last_price->sales_price;
} else {
$has_not_capacity = $hasdate[$i];
}
}
so each time this runs it takes a bout 2,509.10ms in telescope and here is what the telescope shows me as the query which is running on table
select
*
from
`room_pricing_histories`
where
`accommodation_room_id` = 3
and `from_date` <= "2019-06-01 09:00:00"
and `to_date` >= "2019-06-01 09:00:00"
so any idea on how to optimize this query ??
Well, as a rule of thumb - don't run a query inside a loop.
You can use whereIn() for the querying multiple IDs
$roomIds = $hasroom // assume this has array of ids
$roomprice = RoomPricingHistory::
whereIn('accommodation_room_id', $roomIds)
->where('from_date', '<=', $fromDate)
->where('to_date', '>=', $toDate)
->get()->sortBy('created_at');
Im new to this Framework, i dont know how to optimize it using db::raw count and aliases and display it to my blade.php using #foreach
im trying to optimize my code, my goals is to count pallet_conditions and store it to my aliases, i dont want to count it one by one like what i did on this code
this is my code not optimize:
//computing the total rapairable
$repairable_total = DB::table('liip_psrm_items')
->where('psrm_items_id', '=' , $psrm_maintenance->id)
->where('pallet_condition', '=', 1)
->count();
//REPAIRABLE
//computing the total good pallets
$good_total = DB::table('liip_psrm_items')
->where('psrm_items_id', '=' , $psrm_maintenance->id)
->where('pallet_condition', '=', 0)
->count();
//GOOD
this is the code, what i wanted to learn. just to minimize, and use aliases
$result = DB::table('liip_psrm_items')
->select(DB::raw('COUNT(liip_psrm_items.pallet_condition = 0 ) AS condition_1',
'COUNT(liip_psrm_items.pallet_condition = 1 ) AS condition_2'))
->where('psrm_items_id', '=' , $psrm_maintenance->id)
->get();
You can't use single query for two different results, which has totally opposite conditions.
Case 1. You are trying to count the items where pallet_condition = 1;
Case 2. You are trying to count the items where pallet_condition = 0;
Now you want to merge these two cases into single query, which is impossible...
So, For these two cases, you have to use either separate queries ( what you did already )
or you can use single query to grab all the items and then use PHP to separate them.
Like:
$total_items = DB::table('liip_psrm_items')
->where('psrm_items_id', '=' , $psrm_maintenance->id)
->get();
$repairable_count = count(array_filter($total_items, function($item){
return (bool)$item->pallet_condition;
}));
$good_count = count(array_filter($total_items, function($item){
return !(bool)$item->pallet_condition; //just inverse of the above condition
}));
i hope this might help.
To count at multiple condition I used this approach
$lastMonthInvoices = Invoice::select(DB::raw("(COUNT(*)) as count"), DB::raw('SUM(total) as total'),'status')
->whereDate('created_at', '>', Carbon::now()->subMonth())
->groupBy('status')
->get();
i got the result with groupBy Status and in each group total number of records as count & also their sum as total
these two snaps are of one query result
You can, first group by, then get count
Like :
DB::table('liip_psrm_items')
->groupBy('pallet_condition')
->select('pallet_condition', DB::raw('count(*) as total'))
->get();
Try to pass a closure like so:
$results = DB::table('liip_psrm_items')
->where('psrm_items_id', '=' , $psrm_maintenance->id)
->where(function($query){
$query->where('pallet_condition', 1)
->orWhere('pallet_condition', 0);
})->count();
I trying to use Laravel count() function to get the count of row. I have below code to count the number of row of 2 join table.
Example 1:
$row = DB::table('log_user_login')
->select(DB::raw('log_user_login.Password as LogPassword'), 'user.*')
->join('user', 'log_user_login.Username', '=', 'user.Username')
->where('log_user_login.LoginSession', '!=', '')
->groupBy('user.ID')
->get();
$count = sizeof($row);
Example 2:
$count = DB::table('log_user_login')
->select(DB::raw('log_user_login.Password as LogPassword'), 'user.*')
->join('user', 'log_user_login.Username', '=', 'user.Username')
->where('log_user_login.LoginSession', '!=', '')
->groupBy('user.ID')
->count();
When I echo $count form Example 1, the number of $count is 15415. But $count of Example 2 return me 89. May I know why this happen and how can I get the number of row without using the get()?
Example 1 displays the sizeof array in bytes, but example 2 displays total count returned by table.
So, best way to get total count is using ->count() as done in example 2.
Hope you got your answer.
Please try it.
$row = DB::table('log_user_login')
->select(DB::raw('log_user_login.Password as LogPassword'), 'user.*')
->join('user', 'log_user_login.Username', '=', 'user.Username')
->where('log_user_login.LoginSession', '!=', '')
->groupBy('user.ID')
->get();
echo $count = count($row);
Do not use sizeof(). So many time its return amount of memory allocated.
Use count() instead of sizeof().
Say I have a user object (which belongsToMany groups) and I'm doing a whereIn with an array of their respected ids like so:
whereIn('user_id', $group->users->modelKeys())
I need to, however, set a condition where I only pull data from each array item based on a condition of the group_user pivot table, "created_at" (which is basically a timestamp of when that user was added to the group).
So I need something like this:
whereIn('user_id', $group->users->modelKeys())->whereRaw('visits.created_at > group_user.created_at')
That doesn't work though because it's not doing the whereRaw for each array item but it's doing it once for the query as a whole. I might need to do something like a nested whereIn but not sure if that'll solve it either. Thoughts?
My full query as it is now:
$ids = $group->users->modelKeys();
return DB::table('visits')->whereIn('user_id', function($query) use ($ids) {
$query->select('user_id')->from('group_user')->whereIn('group_user.user_id', $ids)->whereRaw('visits.created_at > group_user.created_at');
})->sum("views");
Ok got it to work using nested loops instead:
$visits = DB::table('visits')->whereIn('user_id', $group->users->modelKeys())->get();
$sum = 0;
foreach($group->users as $user) {
foreach($visits as $visit) {
if($visit->user_id == $user->id) {
if($visit->created_at >= $user->pivot->created_at) {
$sum += $visit->views;
}
}
}
}
return $sum;
Would still like to see if it's possible to do it in a single query, no array looping.
Solved it! The foreach loop approach was making calls take waaaay too long. Some queries had over 100k records returning (that's a lot to loop through) causing the server to hang up. The answer is in part a big help from Dharmesh Patel with his 3rd edit approach. The only thing I had to do differently was add a where clause for the group_id.
Here's the final query (returns that 100k results query in milliseconds)
//Eager loading. Has overhead for large queries
//$ids = $group->users->modelKeys();
//No eager loading. More efficient
$ids = DB::table('group_user')->where('group_id', $group->id)->lists('user_id');
return DB::table('visits')->join('group_user', function ($query) use ($ids) {
$query->on('visits.user_id', '=', 'group_user.user_id')->on('visits.created_at', '>=', 'group_user.created_at');
})->whereIn('group_user.user_id', $ids)->where('group_id', $group->id)->sum('views');
Have you considered using a foreach?
$users = whereIn('user_id', $group->users->modelKeys());
foreach ($users as $user) {
// do your comparison here
}
I guess you need to use JOINS for this query, following code may take you in right direction:
$ids = $group->users->modelKeys();
return DB::table('visits')->join('group_user', function ($query) use ($ids) {
$query->on('visits.user_id', '=', 'group_user.user_id')
->whereIn('group_user.user_id', $ids)
->whereRaw('visits.created_at > group_user.created_at');
})->sum("views");
EDIT
$ids = $group->users->modelKeys();
return DB::table('visits')->join('group_user', function ($query) use ($ids) {
$query->on('visits.user_id', '=', 'group_user.user_id');
})->whereIn('group_user.user_id', $ids)
->whereRaw('visits.created_at > group_user.created_at')->sum("views");
EDIT
$ids = $group->users->modelKeys();
return DB::table('visits')->join('group_user', function ($query) use ($ids) {
$query->on('visits.user_id', '=', 'group_user.id') // group_user.id from group_user.user_id as per the loop
->on('visits.created_at', '>=', 'group_user.created_at');
})->whereIn('group_user.user_id', $ids)
->sum("views");