I have a function getUnfilledOrders where I get Orders from the database and then use chunk to have them go to checkStatus 10 at a time. If I have 100 orders, the flow I believe would happen is checkStatus get will get called 10 times (since there are 100 orders).
Now once that completes, I want to have access to $totalOrders in getUnfulfilledOrders. Is this possible?
protected function getUnfulfilledOrders()
{
Order::where('order_status', '!=', true)
->where('tracking_number', '!=', null)
->limit(3000)
->chunk(10, function ($unfulfilledOrders) {
$this->checkStatus($unfulfilledOrders);
});
// how to do something now with $totalOrders once ALL Orders are processed 10 at a time;
}
protected function checkStatus($unfilledOrders)
{
$totalOrders = array();
foreach ($unfulfilledOrders as $unfulfilledOrder) {
// logic here
array_push($totalOrders, $unfulFilledOrder->id);
}
}
Like this:
protected function getUnfulfilledOrders()
{
$totalOrders = [];
Order::where('order_status', '!=', true)
->where('tracking_number', '!=', null)
->limit(3000)
// Add use (&$totalOrders)
->chunk(10, function ($unfulfilledOrders) use (&$totalOrders) {
$totalOrders = array_merge($totalOrders, $this->checkStatus($unfulfilledOrders));
});
// how to do something now with $totalOrders once ALL Orders are processed 10 at a time;
}
protected function checkStatus($unfilledOrders)
{
$totalOrders = array();
foreach ($unfulfilledOrders as $unfulfilledOrder) {
// logic here
array_push($totalOrders, $unfilledOrder->id);
}
// Return the generated array
return $totalOrders;
}
Here I initiated an empty array at the start of getUnfulfilledOrders() and merged anything returned by checkStatus() into it.
More on use ($var)
More on passing by reference (&$var)
Related
How can I improve the performance of a query with the following characteristics (this is a simulation to exemplify):
public function getData()
{
$itens = Item::orderBy('id_item', 'DESC')->get();
foreach($itens as $item){
$item->games = $item->games()->get();
foreach( $item->games as $player){
$item->players = $player->players()->get();
}
}
return response()->json(['success'=>true, 'itens'=>$itens]);
}
In my real case, the foreach returns a lot of data with performance problems;
The Waiting (TTFB), that is, the time the browser is waiting for the
first byte of a response, takes almost 27s
class Item extends Model{
// ...
public function games(){
return this->hasMany(Game::class);
}
public function players(){
return this->hasManyThrough(Player::class ,Game::class);
}
}
public function getData()
{
$itens = Item::orderBy('id_item', 'DESC')->with('games','players')->get();
return response()->json(['success'=>true, 'itens'=>$itens]);
}
That will result in 3 queries and the output will be like:
[
// item 0 ...
[
// item data
[
// games
],
[
// players
]
],
// other items
]
Here you go
public function getData()
{
$items = Item::with('games.players')->orderBy('id_item', 'DESC')->get();
return response()->json(['success'=>true, 'items'=>$items]);
}
Code based solutions:
Looping on array is around 2 times cheaper than looping on List.
A 'for' loop is much faster than 'foreach'
a other solutio is:
Filtering unnecessary parts for running the loop
In my Laravel 8 project I'm dispatching a Job which runs and collects a bunch of data from the database, the data could be any amount ranging from a few hundred rows of data to potentially thousands, so could be quite memory intensive.
Upon returning the results, they're processed and added to a database table, and I'm hoping to have some kind of progress indication as a percentage that can be reported back to the user whilst the chunking is in progress, I have two tables, a reports and a reports_data table.
I've switched by query over to Laravel's chunk method, and am splitting the data collection into smaller bits as this should improve performance, but for some reason, to use my data as a whole, as if it were a collection I'm pushing it into an empty array called $res, but I'm getting an error so my job failsError: Call to a member function groupBy() on array:
Error: Call to a member function groupBy() on array
I'm wondering what I'm missing...
/**
* Execute the job.
*
* #return void
*/
public function handle()
{
$filters = json_decode($this->report->discovery_filters);
$data = [];
// create
foreach ($filters as $findable) {
$resultData = [];
// query data
if (isset($findable->query)) {
$this->setDynamicChartOptions();
$res = [];
$chunkData = DB::table($findable->query->table)
->select($findable->query->columns)
->where($findable->query->filterBy)
->orderBy($findable->query->orderBy->field, $findable->query->orderBy->direction)
->chunk(100, function ($chunkedResults) use ($res) {
foreach ($chunkedResults as $chunk) {
// how to update some kind of progress?
array_push($res, $chunk);
var_dump(count($res));
}
});
// $res expected as a collection? Maybe I can use the `collect` method?
if (isset($findable->query->useGrouping) && $findable->query->useGrouping) {
$results = $res->groupBy(function ($item, $key) use ($findable) {
$date = Carbon::parse($item->{$findable->query->groupBy});
return $date->format($findable->query->groupByFormat);
});
$results = $results->map(function ($item, $key) {
return $item[0];
});
$resultData = $results->flatten();
}
}
$res = [
'componentID' => $findable->componentID ?? 0,
'type' => $findable->type ?? '',
'name' => $findable->name ?? '',
'labelsKey' => $findable->query->labelsKey ?? '',
'dataKey' => $findable->query->dataKey ?? '',
'data' => $resultData ?? [],
'structure' => $this->getStructure($findable, $resultData)
];
array_push($data, $res);
}
// create our report data entry
$this->createReportData($data);
}
UPDATE:
I've tried chunking and grouping, the job fails:
$res = [];
$chunkData = DB::table($findable->query->table)
->select($findable->query->columns)
->where($findable->query->filterBy)
->orderBy($findable->query->orderBy->field, $findable->query->orderBy->direction)
->chunk(100, function ($chunkedResults) use ($res) {
$res[] = $chunkedResults;
foreach($res as $chunk) {
$chunk->groupBy();
}
});
This also fails...
res = [];
$chunkData = DB::table($findable->query->table)
->select($findable->query->columns)
->where($findable->query->filterBy)
->orderBy($findable->query->orderBy->field, $findable->query->orderBy->direction)
->chunk(100, function ($chunkedResults) use ($res) {
$res[] = $chunkedResults;
});
foreach($res as $chunk) {
$chunk->groupBy();
}
And this too, still doesn't seem to work in that it doesn't give back any collection, which is what I need for the rest of my code to work:
$res = [];
$chunkData = DB::table($findable->query->table)
->select($findable->query->columns)
->where($findable->query->filterBy)
->orderBy($findable->query->orderBy->field, $findable->query->orderBy->direction)
->chunk(100, function ($chunkedResults) use ($res) {
foreach ($chunkedResults as $key => $chunk) {
array_push($res, $chunk);
}
});
$res = collect($res);
Because $res = []; is an array, not an instance of eloquent's Illuminate\Support\Collection. Therefore, you can not call $res->groupBy(), as you are trying within the first if-statement.
Remove the ->chunk() method and get your data-chunk by using slice instead for example within a loop that always takes a slice of the data.
Optionally, call collect($res) to turn the array back into a collection. However, when having a Collection already, there is no point in making it into an array first just to cast it back directly afterwords. So I would go with the slice approach.
You could also - withing your chunk callback - do the following:
->chunk(100, function ($chunkedResults) use ($res) {
$res[] = $chunkedResults;
});
And then:
foreach($res as $chunk) {
$chunk->groupBy();
}
I have the following method
public function getNextAvailableHousesToAttack(\DeadStreet\ValueObject\House\Collection $collection, $hordeSize)
{
$houses = $collection->getHouses();
$housesThatCanBeAttacked = array_filter($houses, function($house) use (&$hordeSize) {
if(!isset($house)) {
return false;
}
$house = $this->houseModel->applyMaxAttackCapacity($house, $hordeSize);
if($this->houseModel->isAttackable($house)) {
return $house;
}
return false;
});
return $housesThatCanBeAttacked;
However, this array can be huge.
I want to limit $housesThatCanBeAttacked to whatever the size of $hordeSize is set to, as I only need as many houses as there are zombies in the horde to attack this round.
However, this array $housesThatCanBeAttacked could end up containing 1 million houses, where there are only 100 in the zombie horde.
Is there a way to limit the size of this array built from the callback?
You could simply use a loop, and stop processing the array when you have enough houses.
$houses = $collection->getHouses();
housesThatCanBeAttacked[];
$i = 0;
foreach ($houses as $house) {
$house = $this->houseModel->applyMaxAttackCapacity($house, $hordeSize);
if ($this->houseModel->isAttackable($house)) {
housesThatCanBeAttacked[] = $house;
if (++$i == $hordeSize) {
break;
}
}
}
I would add a counter of houses outside of callback and use it inside callback to skip all excessive houses. Here is how a solution can look like:
public function getNextAvailableHousesToAttack(\DeadStreet\ValueObject\House\Collection $collection, $hordeSize)
{
$houses = $collection->getHouses();
$counter = $hordeSize;
$housesThatCanBeAttacked = array_filter($houses, function($house) use (&$hordeSize, &$counter) {
if($counter == 0 && !isset($house)) {
return false;
}
$house = $this->houseModel->applyMaxAttackCapacity($house, $hordeSize);
if($this->houseModel->isAttackable($house)) {
$counter--;
return $house;
}
return false;
});
return $housesThatCanBeAttacked;
This way you array_filter won't return more then $counter values.
Here's an over-simplified example that doesn't work for me. How (using this method, I know there are better ways if I were actually wanting this specific result), can I get the total number of users?
User::chunk(200, function($users)
{
return count($users);
});
This returns NULL. Any idea how I can get a return value from the chunk function?
Edit:
Here might be a better example:
$processed_users = DB::table('users')->chunk(200, function($users)
{
// Do something with this batch of users. Now I'd like to keep track of how many I processed. Perhaps this is a background command that runs on a scheduled task.
$processed_users = count($users);
return $processed_users;
});
echo $processed_users; // returns null
I don't think you can achieve what you want in this way. The anonymous function is invoked by the chunk method, so anything you return from your closure is being swallowed by chunk. Since chunk potentially invokes this anonymous function N times, it makes no sense for it to return anything back from the closures it invokes.
However you can provide access to a method-scoped variable to the closure, and allow the closure to write to that value, which will let you indirectly return results. You do this with the use keyword, and make sure to pass the method-scoped variable in by reference, which is achieved with the & modifier.
This will work for example;
$count = 0;
DB::table('users')->chunk(200, function($users) use (&$count)
{
Log::debug(count($users)); // will log the current iterations count
$count = $count + count($users); // will write the total count to our method var
});
Log::debug($count); // will log the total count of records
$regions = array();
Regions::chunk(10, function($users) use (&$regions ) {
$stickers = array();
foreach ($users as $user)
{
$user->sababu = ($user->region_id > 1)? $user->region_id : 0 ;
$regions[] = $user;
}
});
echo json_encode($regions);
Use this custom function to get return value from chunked data
function iterateRecords($qb, int $count = 15)
{
$page = 1;
do {
$results = $qb->forPage($page, $count)->get();
$countResults = $results->count();
if ($countResults == 0) {
break;
}
foreach ($results as $row) {
yield $row;
}
unset($results);
$page++;
} while ($countResults == $count);
}
How to use it
$qb = User::select();
$users = iterateRecords($qb, 100);
foreach ($users as $user) {
echo $user->id;
}
Total Users Count $totalUsersCount = $qb->count();
Right now I'm trying to write a function that would allow me to access member functions. The code in question looks a little like this:
protected $formName;
protected $formClass;
protected $formAction;
protected $formMethod;
protected $formObjArray = array(); //outputs in order. So far it should only take newLine, selectTag, inputTag, textTag.
protected $submitBtnVal;
protected $encType;
function __construct($args) {
$this->formName = $args['formName'];
$this->formAction = $args['formAction'];
if (isset($args['formClass'])) $this->formClass = $args['formClass'];
if (isset($args['encType'])) $this->encType = $args['encType'];
//default should be POST. Hell, you should never really be using GET for this..
//also, the default submit value is Submit
$this->formMethod = isset($args['formMethod']) ? $args['formMethod'] : "POST";
$this->submitBtnVal = isset($args['submitBtnVal']) ? $args['submitBtnVal'] : "Submit";
}
//get functions
function getFormName () { return $this->formName; }
function getFormAction () { return $this->formAction; }
function getFormMethod () { return $this->formMethod; }
function getSubmitBtnVal () { return $this->submitBtnVal; }
function getEncType () { return $this->encType; }
//set functions
function setFormName ($newName) { $this->fromName = $newName; }
function setFormAction ($newAction) { $this->formAction = $newAction; }
function setFormMethod ($newMethod) { $this->formMethod = $newMethod; }
function setEncType ($newEType) { $this->encType = $newEType; }
function addTag($newTag) {
if ($newTag instanceof formTag || $newTag instanceof fieldSetCont || $newTag instanceof newLine
|| $newTag instanceof noteTag)
$this->formObjArray[] = $newTag;
else throw new Exception ("You did not add a compatible tag.");
}
I'd like to be able to call $myForm->getTagByName("nameA")->setRequired(true);
How would I do that? Or would I need to do something more like..
$tagID = $myForm->getTagByName("nameA");
$myForm->tagArray(tagID)->setRequired(true);
Nothing in your code seems to be protected so you should have no trouble accessing any of it.
It looks like all your tags are in $formObjArray so it should be trivial to filter than array and return tags that match the name you've passed in. The trouble you will have is that, getTagByName really should be getTagsByName and should return an array because you can have more than one tag with the same name. Since it will return an array, you can not call setRequired on the return value, arrays don't have such a method. You'll need to do it more like:
$tags = $myForm->getTagsByName("nameA");
foreach ($tags as $tag) {
$tag->setRequired(true);
}
Exactly what are you stuck on? Maybe I don't understand the question very well.
So maybe the filtering has you stuck? Try this (if you you're using at least php 5.3)
function getTagsByName($tagname)
{
return array_filter($this->formObjArray, function($tag) use($tagname) {
return $tag->getName() == $tagname;
});
}
No ifs or switches.
Prior to 5.3, you don't have lambda functions so you need to do it differently. There are several options but this may be the simplest to understand:
function getTagsByName($tagname)
{
$out = array();
foreach ($this->formObjArray as &$tag) {
if ($tag->getName() == $tagname) {
$out[] = $tag;
}
}
return $out;
}
In your addTag method, you are storing new tags in $this->formObjArray using the [] notation, which will just append the new tag to the end of the array. If your tag objects all have a getName() method, then you can do something like this:
$this->formObjArray[$newTag->getName()] = $newTag;
Then, you can easily add a getTagByName() method:
public function getTagByName($name) {
if (array_key_exists($name, $this->formObjArray) {
return $this->formObjArray($name);
}
else {
return null;
}
}
Please beware of the solutions suggesting you to iterate through all the tags in your array! This could become very costly as your form gets larger.
If you need to use the [] construct because the order of the elements added is important, then you can still maintain a separate index by name, $this->tagIndex, that will be an associative array of name => tag. Since you are storing object references, they will not be using much space. Assuming that getTagByName will be used many times, this will save you a lot of resources over iterating the tags array on every call to getTagByName.
In that case, your addTag method would look like this:
$this->formObjArray[] = $newTag;
$this->tagIndex[$newTag->getName()] = $newTag; // it seems that you're doubling the memory needed, but you're only storing object references so this is safe
EDIT : Here is some modified code to account for the fact that multiple tags can have the same name:
In your addTag() method, do:
$this->formObjArray[] = $newTag;
$tag_name = $newTag->getName();
if (!array_key_exists($tag_name, $this->tagIndex)) {
$this->tagIndex[$tag_name] = array();
}
$this->tagIndex[$tag_name][] = $newTag
You can then rename getTagByName to getTagsByName and get the expected result.
As mentioned in the comments, this is only useful if you will call getTagsByName multiple times. You are trading a little additional memory usage in order to get quicker lookups by name.