i have a post model that will be boosted by user for some certain time and now i have a crone job to be lunched every 15 mins to run a special script to see if post is boosted or finished .
but the problem is that every time it loops all the posts and it takes some certain amount of time i wanted to know if there is any alternative and better ways to do this ?? here is my code below :
$boosted = POST::whereIn('status_id', [4, 5, 6])->where('boosted',1)->get();
foreach ($boosted as $index => $item) {
if ($item->boosted_until < Carbon::now()) {
$wish = Wish::find($item->id);
$wish->boosted = 0;
$wish->save();
}
Following my comments on your question, you can have something like but accurate:
If you have it set as relations
Wish::whereHas('posts', function($query){
$query->where('boosted_until', '<', Carbon::now());
})->where('boosted', 1)->update(['boosted' => 0]);
I think the problem is. It's a huge amount of data that can cause execution time error. I suggest you to use queues in laravel https://laravel.com/docs/6.x/queues to run it on the background. You can chunk them also into pieces. For example you are about to loop a 100k row of datas. It's best to divide them into pieces so the queue job will work smoothly.
Thankfully, we can use eager loading to reduce this operation to just 2 queries. When querying, you may specify which relationships should be eager loaded using the with method:
I assume you already have hasMany relationship between Post and Wish models
// App\Post Model
public function wishes() {
return $this->hasMany('App\Wish');
}
// Controller
//...
$boosted = POST::with('wishes')->whereIn('status_id', [4, 5, 6])->where('boosted',1)->get();
$wishesIDsToUpdate = [];
foreach ($boosted as $index => $item) {
if ($item->boosted_until < Carbon::now()) {
foreach($item->wishes as $wish) {
$wishesIDsToUpdate[] = $wish->id;
}
}
}
if(!empty($wishesIDsToUpdate)) {
Wish::whereIn('id', $wishesIDsToUpdate)->update(['boosted' => 0]);
}
Related
I have a complicated filter for my hotels and in the end i have a collection that I want to sort the parent relations by its nested relationship so here I have as below :
public function resultFilter($from_date, $to_date, $bed_count, $city_id, $stars, $type_id, $hits, $price, $accommodation_name, $is_recommended, $start_price, $end_price, $has_discount, $facility_id)
{
// $data = QueryBuilder::for(Accommodation::class)
// ->allowedFilters(['city_id','grade_stars','accommodation_type_id'])
// ->allowedIncludes('gallery')
// ->when($bed_count, function ($q, $bed_count) {
// $q->with([
// 'accommodationRoomsLimited' => function ($q) use ($bed_count) {
// $q->where('bed_count', $bed_count);
// }
// ]);
// })
// ->paginate(10);
// ->get();
// ->orderBy('hits','DESC')->paginate(10);
$data = Accommodation::with(['city','accommodationFacilities', 'gallery', 'accommodationRoomsLimited.discount', 'accommodationRoomsLimited', 'accommodationRoomsLimited.roomPricingHistorySearch' => function ($query) use ($from_date, $to_date) {
$query->whereDate('from_date', '<=', $from_date);
$query->whereDate('to_date', '>=', $to_date);
}])->when($bed_count, function ($q, $bed_count) {
$q->whereHas('accommodationRoomsLimited', function($query) use ($bed_count) {
$query->where('bed_count', $bed_count);
});
})->when($accommodation_name, function ($query, $accommodation_name) {
$query->where('name', 'like', $accommodation_name);
})->when($is_recommended, function ($query,$is_recommended){
$query->where('is_recommended', $is_recommended);
})->when($start_price, function ($query, $start_price) {
$query->with([
'accommodationRoomsLimited.roomPricingHistorySearch' => function ($q) use ($start_price) {
$q->where('sales_price', '<', $start_price);
}
]);
})->when($has_discount, function ($query, $has_discount) {
$query->with([
'accommodationRoomsLimited' => function ($q) use ($has_discount) {
$q->has('discount');
}
]);
})
->whereIn('city_id', $city_id)
->whereIn('grade_stars', $stars)
->orWhere('accommodation_type_id', $type_id);
if ($hits) { // or == 'blabla'
$data = $data->orderBy('hits','DESC');
} elseif ($price) { // == A-Z or Z-A for order asc,desc
$f = $data->get();
foreach ($f as $datas) {
foreach ($datas->accommodationRoomsLimited as $g) {
dd($data);
$data = $data->accommodationRoomsLimited()->orderBy($g->roomPricingHistorySearch->sales_price);
}
}
}
$data = $data->paginate(10);
return $data;
}
So if you read code I added the sales_price that I want to sort my $data by it if the $price exists in the request. So in a short term question, I want to sort $data by sales_price in this query above.
NOTE
: this filters may get more complicated so any other best practice or better way for that like spatie Query builder or local scopes would be appreciated although i tried both and yet they have their own limitation
I've faced that problem before. And it seems I need to explain a little about eager loading first.
You can't order by eager loading, you can order it after you fetch the data. Because
eager load will split join query for better performance. For example you querying accomodation and has relation with city. The accomodation table has 1000 records and the city table has 10.000 records. let's say the maximum id for eager loading is 250, the unique city_id from accomodation table is 780. There will be 5 query generated.
$data = Accomodation::with('city')->get();
select * from accomodation
select * from city where id in [unique_id_1 - unique_id_250]
select * from city where id in [unique_id_251 - unique_id_500]
select * from city where id in [unique_id_501 - unique_id_750]
select * from city where id in [unique_id_751 - unique_id_780]
Then laravel will do the job to create the relation by city query results. By this method you will fix N+1 problem from join query, thus it's should be faster.
Then imagine you want to order accomodation bycity.name with with method in query builder. let's take the 3rd query for example.
$data = Accomodation::with([
'city' => function($q) { return $q->orderBy('name'); },
])->get();
the query will be:
select * from city where id in [unique_id_251 - unique_id_500] order by name
The city results will be ordered, but laravel will read it the same way. It'll create accomodation first, then relate it with city queries. So the order from city won't affected accomodation order.
Then how to order it? I found out couple ways to achieve that.
Join query. this is the easiest way, but will make query slower. if your data isn't really big and the join query won't hurt your performance. Maybe 0.003 seconds better performance isn't really worth your 8 hours.
sortBy in collection function. You can sort it with a method from collection.
for example if you want to order the accomodation based on country.name from city relation, this script will help you.
$data = Accomodation::with('city.country')->get();
$data->sortBy(function($item) {
return $item->city->country->name;
});
Flatten the collection. This method will try to flatten the collection so the results will be like join query then sorting it. You can use map method from collection. I do believe all the filters and searchable strings is should be included in data.
$data->map(function($item) {
return [
'city_name' => $city->name,
...
all_searchable_data,
all_shareable_data,
...
];
})->sortBy('key1);
Change eager loading direction if possible. You can order it with changing base models. For example you use city instead accomodation to order it by city.name
$data = City::with('accomodation')->orderBy('name')->get();
And last, If your data rarely changes (example every 2 hours), You might thinking to use cache. You only need to invalidate the cache every 2 hours and create the new one. From my experiences, cache always faster than querying database if the data is big. You just need to know the interval or event to invalidate the cache.
Anything you choose is up to you. But please remember this, when you processing bulk data with the collection from laravel, It could be slower than querying from the database. Maybe it's because PHP performance.
For me the best way is using eager loading then ->map() it then cache it. Why do I need to map it first before cache it? The reason is, by selecting some attribute will reduce the cache size. Then you'll be gain more performance by. And I can say it will produce more readable and beatiful code.
Bonus
this is how I doing this.
$data = Cache::remember("accomodation", 10, function() {
$data = Accommodation::with([
'city',
...
])
->get();
return $data->map(function($item) {
return [
'city_name' => $item->city->name,
...
all_additional_data,
all_searchable_data,
all_shareable_data,
...
];
});
}
return $data->search(function($item) use ($searchKey, $searchAnnotiation, $searchValue) {
switch ($searchAnnotiation) {
case '>':
return $item[$searchKey] > $searchValue;
break;
case '<':
return $item[$searchKey] < $searchValue;
break;
}
})->sortBy($sortKey)->paginate();
The cache will save the processed data. thus the execution time needed is fetch data from cache, filter it, and sorting it. then transform it into paginate. you can set any additional cache in those flow for faster results.
$data->paginate() by create macro paginate for Collection.
There are many Categories(Category.php) on my index site, which has many Boards(Board.php).
A Board can have many Topics(Topic.php), and a Topic can have many Replies (Reply.php).
I want to fetch the number of replies of the current board, possibly with eager loading.
I have a way, but it executes too many queries. I create a post_count attribute, then I iterate over every topic and gather number of replies. Is there a way to select all replies from the current board?
public function getPostCountAttribute()
{
$count = 0;
foreach ($this->topics()->withCount('replies')->get() as $topic) {
$count += $topic->replies_count;
}
return $count;
}
Found a nice way. Even though, I'll leave this question open, if anyone finds a better way to do it.
I declared a hasManyThrough relationship, and the called the count():
Board.php:
public function replies()
{
return $this->hasManyThrough(Reply::class, Topic::class);
}
And then called:
$board->replies->count()
30 queries executed (45 before).
Still, I would like to find an eager way to load the data, getting the query number down to 2 or 3 queries.
In addition to your reply you can do something like this.
public function repliesCount() {
return $this->replies()->selectRaw('board_id, count(*) as aggregate')
->groupBy('board_id');
}
and to use this use as below
$board->repliesCount()
Please note that you have to change query according to you.
Try it
Change
$count = 0;
foreach ($this->topics()->withCount('replies')->get() as $topic) {
$count += $topic->replies_count;
}
To
$topic_ids = $this -> topics -> pluck('id');
$reply_count = Reply::whereIn('id',$topic_ids) -> count();
I wondered if anyone could offer me some advice on refactoring the below function to reduce the number of database queries? Alternatively, there may be a completely different way of achieving this using Laravel.
I am trying to calculate the P&L for a Job, which is made up of Products, which are made up of Components:
public function jobProfitAndLoss($id)
{
$products_in_job = DB::table('job_product')
->where('job_id', $id)
->get();
$total_price = 0.0000;
$total_cost = 0.0000;
foreach ($products_in_job as $row) {
$total_price = $total_price + ($row->quantity)*($row->price);
$product_id = $row->product_id;
$components_in_product = DB::table('components')
->where('product_id', $product_id)
->get();
foreach ($components_in_product as $component) {
$total_cost = $total_cost + ($component->cost)*($row->quantity);
}
}
return $total_price-$total_cost;
}
Products have Components -
https://www.dropbox.com/s/ncnij8dnh99sb9v/Screenshot%202016-04-09%2015.22.26.png?dl=0
Components belong to Products -
https://www.dropbox.com/s/3dx6u30gbod2rv4/Screenshot%202016-04-09%2015.23.43.png?dl=0
Jobs have many Products -
https://www.dropbox.com/s/q179t0knd7y8z4k/Screenshot%202016-04-09%2015.24.11.png?dl=0
You will see here that there are some of the same queries being executed multiple times, which I am not sure how to avoid in this situation -
https://www.dropbox.com/s/xonbtx9cdqvq1wd/Screenshot%202016-04-09%2015.33.07.png?dl=0
Any help is much appreciated.
Edit: It seems that you aren't using models. If you have not done so, create models for your database entries. You will need to use the protected $table attribute for job_product as Eloquent may not be able to automatically convert your class name into the correct table name.
First of all, set up relations if you have not done so. For example, under Job.php, include the Products relation:
public function products() {
return $this->hasMany(App\Products::class); // Assuming App is the namespace
}
Now, instead of using a Fluent query for $components_in_product, you are able to directly do $components_in_product = $products_in_job->products;. However, this still leads to N+1 queries.
As a result, take a look at this: https://laravel.com/docs/5.2/eloquent-relationships#eager-loading
$books = App\Book::with('author')->get();
foreach ($books as $book) {
echo $book->author->name;
}
For this operation, only two queries will be executed:
select * from books
select * from authors where id in (1, 2, 3, 4, 5, ...)
Therefore, change $products_in_job to an Eloquent query and add ->with('products') to the query.
I'm running this code on Laravel. I'm adding filters/ordering if I receive them and I'm altering the query before running it and then paginate the results.
$aDatas = DB::table('datas');
if (!empty($aLocations)) {
foreach ($aLocations as $oLocation) {
$aDatas->orWhere('pc', '=', $oLocation->pc);
}
}
if (!empty($oFilters->note)) {
$aDatas->where('note', '>=', $oFilters->note);
}
if (!empty($oFilters->nb_comments)) {
$aDatas->where('nb_comments', '>=', $oFilters->nb_comments);
}
if (!empty($oOrder->type)) {
$aDatas->orderBy($oOrder->type, $oOrder->sort);
}
// echo $aDatas->where('note', '>=', 5)->count() ????
It's working fine.
But I'd like to use these results to count several parts of it.
The last line shows what I tried to do, counting how many rows in these filtered results have a note >= 5. But doing this will actually filter my original data.
I thought about assigning $aDatas to another variable and then count on this, but I'll have many counts and that seems dirty.
Is there a sweet way to do this ?
Just save your datas an replace the last line with this:
$datas =$aDatas->where('note', '>=', 5)->get();
echo $datas->count();
//use datas here for more action.
For all of your requirement, you might want to resolve in making several queries because a single query will not be able to do that(based from what I know)
//this is to get your total of note greater than 5
$query = DB::table('datas');
$query->where('note', '>=', 5);
$data = $query->paginate(10);
$count = $data->getTotal();
to get your other data
If you are working with pagination, use getTotal() instead
$query = DB::table('datas');
$query->select(
DB::raw('COUNT(stars) AS count'),
DB::raw('starts'),
);
$query->where('notes', '>=', 5);
$query->groupBy('stars');
$data = $query->get();
In this scenario I have a Ticket model and a TicketReply model.
I can grab all replies to a ticket with $ticket->replies.
Considering status 1 or 3 of a ticket or reply means its state is open/unresolved, this is how I currently find the total of tickets open the Eloquent way.
$tickets = Ticket::all();
$tickets_open = 0;
foreach($tickets as $t)
{
$tickets_open++;
if(($t->replies()->first()->status == 2) || ($t->status == 2))
{
$tickets_open--;
}
}
return $tickets_open;
Is there a more efficient way of doing this with Eloquent?
Since as it is a query will run for each iteration.
If there is not, I can convert $tickets to an array and iterate it.
Update:
$t->replies()->first()->status was causing the 1+N with or without eager loading.
I changed to $t->replies->first()->status and 1+N is gone. A rookie mistake I believe.
You can use Eager Loading to reduce the number of queries.
$tickets = Ticket::with('replies')->get();
Doing this, instead of running 1+N queries, you will end up running two SQL queries:
select * from Tickets
select * from TicketReplies where ticket_id in (1, 2, 3, 4, 5, ...)
Though it's a bit confusing to me but I believe you may try something like this:
$openTickets = Reply::where('status', 2)->count();
Also maybe something like this:
$tickets = Ticket::whereHas('replies', function($query) {
$query->where('status', 2);
})->with('replies')->get();
Then count:
$t = 0;
$tickets->fetch('replies')->each(function($i) use (&$t){
$t += count($i);
});
dd($t); // total
I could be wrong but I guessed it could be done like this.
P/S:Please let me know the update.