How to improve eloquent nested foreach performance - php

How can I improve the performance of a query with the following characteristics (this is a simulation to exemplify):
public function getData()
{
$itens = Item::orderBy('id_item', 'DESC')->get();
foreach($itens as $item){
$item->games = $item->games()->get();
foreach( $item->games as $player){
$item->players = $player->players()->get();
}
}
return response()->json(['success'=>true, 'itens'=>$itens]);
}
In my real case, the foreach returns a lot of data with performance problems;
The Waiting (TTFB), that is, the time the browser is waiting for the
first byte of a response, takes almost 27s

class Item extends Model{
// ...
public function games(){
return this->hasMany(Game::class);
}
public function players(){
return this->hasManyThrough(Player::class ,Game::class);
}
}
public function getData()
{
$itens = Item::orderBy('id_item', 'DESC')->with('games','players')->get();
return response()->json(['success'=>true, 'itens'=>$itens]);
}
That will result in 3 queries and the output will be like:
[
// item 0 ...
[
// item data
[
// games
],
[
// players
]
],
// other items
]

Here you go
public function getData()
{
$items = Item::with('games.players')->orderBy('id_item', 'DESC')->get();
return response()->json(['success'=>true, 'items'=>$items]);
}

Code based solutions:
Looping on array is around 2 times cheaper than looping on List.
A 'for' loop is much faster than 'foreach'
a other solutio is:
Filtering unnecessary parts for running the loop

Related

can't orderBy accessor in Laravel

I cant orderBy points. Points is accessor.
Controller:
$volunteers = $this->volunteerFilter();
$volunteers = $volunteers->orderBy('points')->paginate(10);
Volunteers Model:
public function siteActivities()
{
return $this->belongsToMany(VolunteerEvent::class, 'volunteer_event_user', 'volunteer_id', 'volunteer_event_id')
->withPivot('data', 'point', 'point_reason');
}
public function getPointsAttribute(){
$totalPoint = 0;
$volunteerPoints = $this->siteActivities->pluck('pivot.point', 'id')->toArray() ?? [];
foreach ($volunteerPoints as $item) {
$totalPoint += $item;
}
return $totalPoint;
}
But I try to sortyByDesc('points') in view it works but doesn't work true. Because paginate(10) is limit(10). So it doesn't sort for all data, sort only 10 data.
Then I try to use datatable/yajra. It works very well but I have much data. so the problem came out
Error code: Out of Memory
You could aggregate the column directly in the query
$volunteers = $this->volunteerFilter();
$volunteers = $volunteers->selectRaw('SUM(pivot.points) AS points)')->orderByDesc('points')->paginate(10);

Laravel: Using external variable after processing data?

I have a function getUnfilledOrders where I get Orders from the database and then use chunk to have them go to checkStatus 10 at a time. If I have 100 orders, the flow I believe would happen is checkStatus get will get called 10 times (since there are 100 orders).
Now once that completes, I want to have access to $totalOrders in getUnfulfilledOrders. Is this possible?
protected function getUnfulfilledOrders()
{
Order::where('order_status', '!=', true)
->where('tracking_number', '!=', null)
->limit(3000)
->chunk(10, function ($unfulfilledOrders) {
$this->checkStatus($unfulfilledOrders);
});
// how to do something now with $totalOrders once ALL Orders are processed 10 at a time;
}
protected function checkStatus($unfilledOrders)
{
$totalOrders = array();
foreach ($unfulfilledOrders as $unfulfilledOrder) {
// logic here
array_push($totalOrders, $unfulFilledOrder->id);
}
}
Like this:
protected function getUnfulfilledOrders()
{
$totalOrders = [];
Order::where('order_status', '!=', true)
->where('tracking_number', '!=', null)
->limit(3000)
// Add use (&$totalOrders)
->chunk(10, function ($unfulfilledOrders) use (&$totalOrders) {
$totalOrders = array_merge($totalOrders, $this->checkStatus($unfulfilledOrders));
});
// how to do something now with $totalOrders once ALL Orders are processed 10 at a time;
}
protected function checkStatus($unfilledOrders)
{
$totalOrders = array();
foreach ($unfulfilledOrders as $unfulfilledOrder) {
// logic here
array_push($totalOrders, $unfilledOrder->id);
}
// Return the generated array
return $totalOrders;
}
Here I initiated an empty array at the start of getUnfulfilledOrders() and merged anything returned by checkStatus() into it.
More on use ($var)
More on passing by reference (&$var)

PHP best performance to delete duplicate array values

In my web application I have a function that is executed more than 100 times in a minute, and I'm trying to find a better performing approach.
public static function removeDuplicateFeeds($id, $feeds, $todo)
{
$actionsHistory = ActionsHistory::whereAccountId($id)->whereActionName($todo)->get();
if (!empty($actionsHistory)) {
foreach ($actionsHistory as $history) {
foreach ($feeds as $key => $feed) {
if ($history->action_name == $feed['pk']) {
unset($feeds[$key]);
}
}
}
}
}
I want to remove all the elements from $feeds that are in $actionsHistory as well.
UPDATE:
in this test code first index of $feeds array as "pk":"7853740779" is stored on my database and after remove duplicate this item should be removed, but i have all of $feeds items into $filteredFeeds too
$userAccountSource = InstagramAccount::with('user', 'schedule')->get();
$feeds = [
json_decode('{"pk":"7853740779","username":"teachkidss","full_name":"..."}'),
json_decode('{"pk":"7853740709","username":"teachkidss","full_name":"..."}'),
json_decode('{"pk":"7853740009","username":"teachkidss","full_name":"..."}')
];
$filteredFeeds = AnalyzeInstagramPageController::removeDuplicateFeeds($userAccountSource[0]->id, $feeds, 'like');
public function removeDuplicateFeeds($id, $feeds, $todo)
{
$feeds = collect($feeds); // If $feeds is not already a collection
$actionsHistory = ActionsHistory::whereAccountId($id)
->whereActionName($todo)
->whereIn('action_name', $feeds->pluck('pk')) // retrieves only duplicate records
->select('action_name') // reducing the select improves performance
->get(); // Should return a eloquent collection instance
if (!empty($actionsHistory)) {
return $feeds->whereNotIn('pk', $actionsHistory->pluck('action_name'));
}
return $feeds;
}
Without knowing the number of feed or database records you'll have to test these for performance against your dataset and see if these are more performant.
public static function removeDuplicateFeeds($id, $feeds, $todo)
{
$feeds = collect($feeds); // If $feeds is not already a collection
$actionsHistory = ActionsHistory::whereAccountId($id)
->whereActionName($todo)
->select('action_name') // reducing the select improves performance
->get(); // Should return an eloquent collection instance
if (!empty($actionsHistory)) {
return $feeds->whereNotIn('pk', $actionsHistory->pluck('action_name'));
}
return $feeds;
}
or if your database query returns significantly more records than you have feeds you could try leveraging mysql's faster query speed instead of using php's slower array_filter/foreach speed.
public static function removeDuplicateFeeds($id, $feeds, $todo)
{
$feeds = collect($feeds); // If $feeds is not already a collection
$actionsHistory = ActionsHistory::whereAccountId($id)
->whereActionName($todo)
->whereIn('action_name', $feeds->pluck('pk')) // retrieves only duplicate records
->select('action_name') // reducing the select improves performance
->get(); // Should return a eloquent collection instance
if (!empty($actionsHistory)) {
return $feeds->whereNotIn('pk', $actionsHistory->pluck('action_name'));
}
return $feeds;
}
If either of these works, it would be good to know how much faster this was for you. Let us know. Good luck.
You may try array_unique function from PHP. Source.
You can use build in laravel unique() function from collection, see more here :
https://laravel.com/docs/5.6/collections#method-unique
doing that you cove will maintain clean and elegant.

PHP array's: How can i filter on an argument

I have a huge JSON-string with +/-300 items. A shortened example:
[
{"DateGps":"2016-03-25T19:28:19+01:00","DateReceived":"2016-03-25T19:28:20.163+01:00","Longitude":5.85294,"Latitude":51.84475,"Speed":55,"VehicleNumber":"678","TravelNumber":"4321"},
{"DateGps":"2016-03-25T19:28:13+01:00","DateReceived":"2016-03-25T19:28:14.065+01:00","Longitude":4.8139,"Latitude":52.43844,"Speed":23,"VehicleNumber":"2335","TravelNumber":"1234"}
]
With
$array = json_decode($json,true); // i will convert this into a array.
But how can i get only the row where VehicleNumber the same is as VehicleNumber 2335?
You can simply use array_filter like as
$array = json_decode($json,true);
$resulted_array = array_filter($array,function($v){ return ($v['VehicleNumber'] == 2335);});
print_r($resulted_array);
Pretty much the same as Uchiha's answer, but using an anonymous class as a FilterIterator.
In this scenario it only adds (unnecessary) complexity (esp since json_decode(,true) returns an array and therefore the whole data is in memory anyway). But if you have a large amount of data and you can receive it step by step, iterators and/or generators might come in handy ;-)
<?php
$data = json_decode(data(), true);
$it = new class(new ArrayIterator($data)) extends FilterIterator {
public function accept() { return '2335'==$this->current()['VehicleNumber']; }
};
foreach($it as $v) {
var_export($v);
}
function data() {
return <<< eoj
[
{"DateGps":"2016-03-25T19:28:19+01:00","DateReceived":"2016-03-25T19:28:20.163+01:00","Longitude":5.85294,"Latitude":51.84475,"Speed":55,"VehicleNumber":"678","TravelNumber":"4321"},
{"DateGps":"2016-03-25T19:28:13+01:00","DateReceived":"2016-03-25T19:28:14.065+01:00","Longitude":4.8139,"Latitude":52.43844,"Speed":23,"VehicleNumber":"2335","TravelNumber":"1234"}
]
eoj;
}

How to `count` a query result and send it as an array in JSON format?

Details
I want to
Count all my distributors that I query
Send it along within the JSON file
I know that I have 89 distributors in total, I did this dd(count($distributors));
I am sure what is the best practice for this.
Here is what I have tried
I initialize $count = 0;
Increment by 1 every time the loop execute $count = $count + 1;
Send the result toArray 'count' => $count->toArray() as part of my distributors array
Here is my code
public function index()
{
$distributors = [];
$count = 0;
foreach(
// Specific Distributors
Distributor::where('type','!=','OEM')->where('type','!=','MKP')
->get() as $distributor){
// Variables
$count = $count + 1;
$user = $distributor->user()->first();
$distributors[$distributor->id] = [
'user' => $user->toArray(),
'distributor' => $distributor->toArray(),
'hq_country' => $distributor->country()->first(),
'address' => $distributor->addresses()
->where('type','=','Hq')->first()->toArray(),
'count' => $count->toArray()
];
}
return Response::json($distributors);
}
Result
The code won't run, due to my $distributor array is not exist ...
It will run, if I take 'count' => $count->toArray() off .
Updated
I am using Laravel 4.0
The code is part of my UrlController.php
It really doesn't make a lot of sense to add this kind of count to your result. It is much simpler to just send the result and let the client do the counting. Because the information on how much distributors you actually have is right in your array of distributors. It's just it's length.
Here's an example with javascript (and $.ajax although that doesn't really matter)
$.ajax({
url: 'get/distributors',
method: 'GET'
}).done(function(data){
var count = data.length;
});
Model:
class Distributor extends Eloquent {
public function country()
{
return $this->hasOne('Country');
}
public function addresses()
{
return $this->hasMany('Address');
}
public function hqAddress()
{
return $this->addresses()->where('type', 'Hq')->first();
}
public function user()
{
return $this->hasOne('User');
}
}
Controller:
$distributors = Distributor::whereNotIn('type', ['OEM', 'MKP'])
->with('country', 'user')->get();
$count = 0;
$distributors->each(function(Distributor $distributor) use(&$count) {
$distributor->count = $count;
$count++;
});
return Response::json($distributors);
Sorry, I can be wrong.
I am not laravel expert.
But what is this fragment is about?
this Index function is part of Model?
#evoque2015 is trying to put some custom array into $distributors[$distributor->id]?
if that is the goal, could you do any test with any simple value of 'count' that sholud work in your opinion?
My guess is : 'count' index is not acceptable for your Distributor::where function
(if it is acceptable - show as the value of 'count' that doesn't break your code even if return wrong result/data ).
So I would try to change the name of this parameter either to 'my_custom_count',
should it be declared somewhere in Distributor model declaration?
Found this :
Add a custom attribute to a Laravel / Eloquent model on load?
It seems to prove my guess.
So we need to change model class like:
class Distributor extends Eloquent {
protected $table = 'distributors';//or any you have already
public function toArray()
{
$array = parent::toArray();
$array['count'] = $this->count;
return $array;
}
.....
}
and probably more changes in model or just add 'count' column to the table :-)

Categories