Laravel chunk Call to a member function groupBy() on my variable - php

Question

In my Laravel 8 project I'm dispatching a Job which runs and collects a bunch of data from the database, the data could be any amount ranging from a few hundred rows of data to potentially thousands, so could be quite memory intensive.
Upon returning the results, they're processed and added to a database table, and I'm hoping to have some kind of progress indication as a percentage that can be reported back to the user whilst the chunking is in progress, I have two tables, a reports and a reports_data table.
I've switched by query over to Laravel's chunk method, and am splitting the data collection into smaller bits as this should improve performance, but for some reason, to use my data as a whole, as if it were a collection I'm pushing it into an empty array called $res, but I'm getting an error so my job failsError: Call to a member function groupBy() on array:
Error: Call to a member function groupBy() on array
I'm wondering what I'm missing...
/**
* Execute the job.
*
* #return void
*/
public function handle()
{
$filters = json_decode($this->report->discovery_filters);
$data = [];
// create
foreach ($filters as $findable) {
$resultData = [];
// query data
if (isset($findable->query)) {
$this->setDynamicChartOptions();
$res = [];
$chunkData = DB::table($findable->query->table)
->select($findable->query->columns)
->where($findable->query->filterBy)
->orderBy($findable->query->orderBy->field, $findable->query->orderBy->direction)
->chunk(100, function ($chunkedResults) use ($res) {
foreach ($chunkedResults as $chunk) {
// how to update some kind of progress?
array_push($res, $chunk);
var_dump(count($res));
}
});
// $res expected as a collection? Maybe I can use the `collect` method?
if (isset($findable->query->useGrouping) && $findable->query->useGrouping) {
$results = $res->groupBy(function ($item, $key) use ($findable) {
$date = Carbon::parse($item->{$findable->query->groupBy});
return $date->format($findable->query->groupByFormat);
});
$results = $results->map(function ($item, $key) {
return $item[0];
});
$resultData = $results->flatten();
}
}
$res = [
'componentID' => $findable->componentID ?? 0,
'type' => $findable->type ?? '',
'name' => $findable->name ?? '',
'labelsKey' => $findable->query->labelsKey ?? '',
'dataKey' => $findable->query->dataKey ?? '',
'data' => $resultData ?? [],
'structure' => $this->getStructure($findable, $resultData)
];
array_push($data, $res);
}
// create our report data entry
$this->createReportData($data);
}
UPDATE:
I've tried chunking and grouping, the job fails:
$res = [];
$chunkData = DB::table($findable->query->table)
->select($findable->query->columns)
->where($findable->query->filterBy)
->orderBy($findable->query->orderBy->field, $findable->query->orderBy->direction)
->chunk(100, function ($chunkedResults) use ($res) {
$res[] = $chunkedResults;
foreach($res as $chunk) {
$chunk->groupBy();
}
});
This also fails...
res = [];
$chunkData = DB::table($findable->query->table)
->select($findable->query->columns)
->where($findable->query->filterBy)
->orderBy($findable->query->orderBy->field, $findable->query->orderBy->direction)
->chunk(100, function ($chunkedResults) use ($res) {
$res[] = $chunkedResults;
});
foreach($res as $chunk) {
$chunk->groupBy();
}
And this too, still doesn't seem to work in that it doesn't give back any collection, which is what I need for the rest of my code to work:
$res = [];
$chunkData = DB::table($findable->query->table)
->select($findable->query->columns)
->where($findable->query->filterBy)
->orderBy($findable->query->orderBy->field, $findable->query->orderBy->direction)
->chunk(100, function ($chunkedResults) use ($res) {
foreach ($chunkedResults as $key => $chunk) {
array_push($res, $chunk);
}
});
$res = collect($res);

Answer 1

Because $res = []; is an array, not an instance of eloquent's Illuminate\Support\Collection. Therefore, you can not call $res->groupBy(), as you are trying within the first if-statement.
Remove the ->chunk() method and get your data-chunk by using slice instead for example within a loop that always takes a slice of the data.
Optionally, call collect($res) to turn the array back into a collection. However, when having a Collection already, there is no point in making it into an array first just to cast it back directly afterwords. So I would go with the slice approach.
You could also - withing your chunk callback - do the following:
->chunk(100, function ($chunkedResults) use ($res) {
$res[] = $chunkedResults;
});
And then:
foreach($res as $chunk) {
$chunk->groupBy();
}