I have a homework table in laravel-5.7 where 5000 records in the table also have some Relational records that are coming through HasMany() or HasOne() Relation. I tried many types of Eloquent Queries to get fast results. but Postman result time becomes 10200ms to 10700ms but when i direct dispaly this into postman then i am getting this into 500ms to 1100ms. i want to get it in near about 800ms after binding form Laravel Resource or Normal Array.
problem is, when i try to show the Eloquent result direct then it coming around 600ms to 1000ms. but when i bind into an Array and display in postman then its taking 6200ms why? i do not know?
$page = $req->page ?$req->page:1; // set starting value for look query limit.
$user = Auth::user()->student()->first();
$studentProfile = Auth::user()->student()->first();
// collecting all homework id that have assigned to student.
$studentHWList = StudentHomeWork::where("student_id",$studentProfile->id)
->select('home_work_id')
->get()->pluck('home_work_id');
// collecting page by page of homework id.
$hwLimitList = Homework::where('session_code', dnc($req->require('sid')))
->whereIn('id',$studentHWList )
->where('approved', '1')
->select('id')
->orderBy('updated_at','desc')
->get();
$hwIndexes = $hwLimitList->pluck('id')->forPage($page,$this->recordLimit);
$paginated = Homework::whereIn('id', $hwIndexes)
->with( "user:id,username,name",
'subject:id,subject_name,subject_code',
'approveByUser','publishBy')
->with(["likes"=>function($erw){
$erw->select('id','home_work_id','complete_status','likes')
->where("student_id", $studentProfile->id);
}])
->with(['comment'=>function($qur){
$qur->where('parent_id',0)
->where('user_id',$user->id);
}])
->orderBy('id','desc')
->get( );
if( count($paginated))
{
$paginationData = customPagination('getAllHW',$hwLimitList , $page , $this->recordLimit , $user, $studentProfile );
return response()->json(["error"=>0,"errmsg"=>"","paginationData"=>$paginationData ,
"response"=>['homework_list'=>$this->customResourceHWBinding($paginated , $req )],'auth'=>userType()]);
private function customResourceHWBinding($queryData , $request, $user, $studentProfile )
{
$document_list =[]; $is_seen=0; $resultData =[];
foreach ( $queryData as $query )
{
if( count($query->document) )
{
foreach($query->document as $document){
if( $document->changed_filename )
{
$file=""; $fileName ="";
$path =env('AWS_URL')."/uploads/".dnc($request->header('dbauth'))."/".$query->session_code."/homeWorks/";
if(is_s3FileExist( $path.$document->changed_filename ) )
{
$fileName =$document->changed_filename;
}
$document_list[] = [
'oname'=> $document->changed_filename,
'ext'=>$fileName?explode('.', $document->changed_filename):"",
'url'=>$file,
'file_url'=>$document->changed_filename?$path.$document->changed_filename:""
];
}
}
}
$resultData[] = [
'id'=>enc($query->id),
'ids'=>$query->id,
'pin_user_id'=>"",
'pin_enabled'=>0,
'created_by'=>$query->user->name,
'created_by_image'=>getUserImage($query->user,$query->user->privilege,$request),
'assignment_date'=>getDateFormat($query->assignment_date,0),
'assigment_date_edit'=>"",
'submission_date'=>getDateFormat($query->submission_date,1),
'submission_date_edit'=>"",
'class_code'=>$query->class_code,
'subject'=>$query->subject?$query->subject->subject_name:"",
'topic'=>$query->topic,
'is_student_seen'=> $this->studentHWSeen($query, $user, $studentProfile),
'updated_at'=> date('d-m-Y H:i:s' , strtotime($query->updated_at)),
'approved'=>$query->approved,
'approve_by'=> '',
'can_approve'=>0,
'comment_count'=>0,
'total_like'=>0,
'documents_count'=>count($document_list)?count($document_list):0,
'is_draft'=> $query->draft?$query->draft:0,
];
}
return $resultData;
}
private function studentHWSeen( $query , $user, $studentProfile)
{
if(count($query->studentSeen))
{
foreach($query->studentSeen as $seen){
if( user->privilege == 1 )
{
if($seen->student_id == $studentProfile->id )
return 1;
}
}
}
return 0;
}
I try to use Resource but it's also taking 3+seconds. I try too many others optimize a solution but not work in my case. someone told Use query builder instead of Eloquent to optimize queries. find here Optimising Laravel query . is it a good answer for me? I am not sure. please help me.
please check my image.
Eloquent Query Result
First of all, try to optimize this:
$paginated = Homework::whereIn('id', $hwIndexes)
->with( "user:id,username,name",'subject:id,subject_name,subject_code',
'approveByUser','publishBy')
->with(["likes"=>function($erw){
$erw->select('id','home_work_id','complete_status','likes')
->where("student_id",Auth::user()->student()->first()->id);
}])
->with(['comment'=>function($qur){
$qur->where('parent_id',0)
->where('user_id',Auth::id());
}])
->orderBy('id','desc')
->get( );
You run same code in nested query: Auth::user()->student()->first()->id.
Optimized version:
$studentId = Auth::user()->student()->first()->id;
$paginated = Homework::whereIn('id', $hwIndexes)
->with("user:id,username,name", 'subject:id,subject_name,subject_code', 'approveByUser', 'publishBy')
->with(["likes"=>function($erw) use ($studentId) {
$erw->select('id','home_work_id','complete_status','likes')
->where("student_id", $studentId);
}])
->with(['comment'=>function($qur) {
$qur->where('parent_id',0)
->where('user_id',Auth::id());
}])
->orderBy('id', 'desc')
->get();
Remember to add indexes to fields that you're using in where conditions.
Related
I have a filter like this
I have a working date filter with laravel. The problem is that if I don't filter my records when no records are shown.
public function index(Request $request)
{
$stats = [
'done' => UserTransaction::where('status', UserTransaction::STATUS['done'])->count(),
'canceled' => UserTransaction::where('status', UserTransaction::STATUS['canceled'])->count(),
'all' => UserTransaction::count(),
];
if ($request->start_date && $request->end_date){
$dateS = new Carbon($request->start_date);
$dateE = new Carbon($request->end_date);
$result = UserTransaction::whereBetween('created_at', [$dateS->format('Y-m-d'), $dateE->format('Y-m-d')])->get();
}
if ($request->search) {
$transactions = UserTransaction::search($request->search)->paginate(5);
} else {
$transactions = UserTransaction::paginate(5);
}
if ($transactions->count() == 0 && $request->search ) {
msg()->warning('No records found');
}
return view('transaction::admin.index', compact('transactions', 'stats', 'result'));
}
I would consider changing your logic a little to allow for people to specify either just a start date or just an end date which could be entirely possible from a usability perspective.
Create a query builder, then add where clauses for the start_date and end_date if they appear in your $request object. If they do not appear then you can use the query as you have been doing so already.
$transactions = Transaction::query();
if ($request->start_date) {
$query->whereDate('start_date', '>=', new Carbon($request->start_date));
}
if ($request->end_date) {
$query->whereDate('end_date', '<=', new Carbon($request->end_date));
}
if ($requer->search) {
$query->search($request->search);
}
$query->paginate(5);
You can use if condition inside query and remove outer if condition
$result = UserTransaction::where(function($query)use($request){
if(!empty($request->start_date)&&!empty($request->end_date)){
$dateS = new Carbon($request->start_date);
$dateE = new Carbon($request->end_date);
$query->whereBetween('created_at', [$dateS->format('Y-m-d'), $dateE->format('Y-m-d')]);
}
})->get();
I am working with laravel project which uses a mysql database.It has some tables which has over 5 million data.it takes too much time to get these data to frontend. following is the function i use to get data. i have some filters used such as date range (from,to) search by name($seach) pagination amount(records_number) etc. but when i try to get all records it takes too much time. is there any solutions/optimizations for this matter?
Thanks.
public function transactionListBetween($from, $to, $sort, $search, $records_number, $filter, $previous_sort) {
$query = $this->accountTransactions
->with('transactionType', 'giver', 'recipient')
->leftJoin('tbdb_users as recipient', 'recipient.id', '=', 'tbdb_account_transaction.recipient_id')
->leftJoin('tbdb_users as giver', 'giver.id', '=', 'tbdb_account_transaction.giver_id')
->leftJoin('tbdb_account_transaction_type', 'tbdb_account_transaction.account_transaction_type_id', '=', 'tbdb_account_transaction_type.id')
->where('recipient.name', 'like', "%$search%")
->select('tbdb_account_transaction.*');
if($filter) {
if($filter == -1) {
$transactionTypeModel = \App::make('App\Models\AccountTransactionTypeModel');
$depositTypeIds = $transactionTypeModel->whereIn('name', ['PayPal Deposit', 'Eway Deposit', 'Bank Deposit', 'BPay Deposit', 'Poli Deposit'])->lists('id');
$query = $query->whereIn('tbdb_account_transaction.account_transaction_type_id', $depositTypeIds);
} else {
$query = $query->where('tbdb_account_transaction.account_transaction_type_id', $filter);
}
}
if($from) {
$query = $query->where('tbdb_account_transaction.created_date', '>=', $from);
}
if($to) {
$query = $query->where('tbdb_account_transaction.created_date', '<=', $to->endOfDay());
}
if($records_number == 'no_paginate') {
return $query->orderBy($sort, $previous_sort)
->get();
} else {
$totalAmount = $query->sum('amount');
$collection = $query->orderBy($sort, $previous_sort)->paginate($records_number);
$collection->totalAmount = number_format($totalAmount / 100, 2);
return $collection;
}
}
You seem to be doing 3 left joins and getting the same data as you did with the with.
You can drop them:
public function transactionListBetween($from, $to, $sort, $search, $records_number, $filter, $previous_sort) {
$query = $this->accountTransactions->->with('transactionType', 'giver', 'recipient')
->whereHas('recipient', function ($query) use ($search) { //Query the related model
$query->where('name', 'LIKE', "%$search%"); //Note this will not use an index
});
if ($filter) {
if ($filter == -1) {
$query->whereHas('transactionType', function ($query) {
$query->whereIn('id', \DB::raw("(SELECT id FROM tbdb_account_transaction_type WHERE name IN ('PayPal Deposit', 'Eway Deposit', 'Bank Deposit', 'BPay Deposit', 'Poli Deposit')");
});
} else {
$query->whereHas('transactionType', function ($query) use ($filter) {
$query->where('id', $filter);
});
}
}
if($from) {
$query = $query->where('created_date', '>=', $from);
}
if($to) {
$query = $query->where('created_date', '<=', $to->endOfDay());
}
if($records_number == 'no_paginate') {
return $query->orderBy($sort, $previous_sort)
->get();
} else {
$totalAmount = $query->sum('amount');
$collection = $query->orderBy($sort, $previous_sort)->paginate($records_number);
$collection->totalAmount = number_format($totalAmount / 100, 2);
return $collection;
}
}
This will reduce the main result set greatly making it easier to work with.
In my case i have also having this problem when using mysql database. These much records in table with relationship takes too much time. You have to use raw query to get result. You have to use pagination to show the data. because fetching all the data will slow down.
But i highly recommended you to normalize your table structure. In my case i did normalize of my tables by split it with yearly.
Found out why this is happening. it is because of pagination. it takes lot of time to paginate. there are more than 50000 pages so it takes time. i used simplePaginate() instead of paginate(). then it loads faster than usual
Hello Again everyone,
can anyone help me with this code.
public function submitranking(Request $req){
$dataCandidate = Candidate::all();
foreach($dataCandidate as $Candidate){
$judgeRate = Score::where('canId',$Candidate->id )
->where('catId',$req->catId )
->where('judgeId',$req->judgeId)
->sum('score');
dd($judgeRate);
}
}
It is supposed to display all the scores of the candidates when I dd($judgeRate) it only display one record? How can I make it loop and display all the total sum scores of the candidates... Please help
You should try this:
public function submitranking(Request $req){
$dataCandidate = Candidate::get();
foreach($dataCandidate as $Candidate){
$judgeRate = Score::where('canId',$Candidate->id )
->where('catId',$req->catId )
->where('judgeId',$req->judgeId)
->sum('score');
print('<pre style="color:red;">');
print_r($judgeRate);
print('</pre>');
}
exit;
}
OR You should try this:
public function submitranking(Request $req){
$dataCandidate = Candidate::get();
foreach($dataCandidate as $Candidate){
$judgeRate = Score::select('score_id',DB::raw("SUM(score) as score"))
->where('canId',$Candidate->id )
->where('catId',$req->catId )
->where('judgeId',$req->judgeId)
->get();
print('<pre style="color:red;">');
print_r($judgeRate);
print('</pre>');
}
exit;
}
You can use pluck function to get all ids of candidates and use whereIn.
So, there is no need to use loop, use group by and simple raw query.
Just change your function like,
public function submitranking(Request $req){
$dataCandidate = Candidate::all()->pluck('canId');
$judgeRate = Score::whereIn('canId',$dataCandidate)
->where('catId',$req->catId )
->where('judgeId',$req->judgeId)
->select('catId',DB::raw('COUNT(score) as total_score'))
->groupBy('catId')
->get();
dd($judgeRate);
}
You can see docs here: https://laravel.com/docs/5.5/queries#where-clauses
Hope you understand.
Here's my function to load submissions created by a user.
public function viewSubs()
{
$user = User::find(Input::get('id'));
$submissions = Submission::find($user)->sortByDesc('created_at');
$submissions->load('user')->load('votes')->load('suggestions.votes');
return view('submissions.index' , compact('submissions'));
}
This returns with an error
Call to a member function load() on null
when there are no records on the submission.
How to handle if there are no submission on the DB?
Just check if its null first using an if statement:
public function viewSubs()
{
$user = User::find(Input::get('id'));
if ($submissions = Submission::find($user)->sortByDesc('created_at')) {
$submissions->load('user')->load('votes')->load('suggestions.votes');
}
return view('submissions.index' , compact('submissions'));
}
Also, depending on your DB structure I'm pretty sure you can cut out a lot of the code by utilising your models' relationships by doing something like this:
$user = User::find(Input::get('id'))
->with(['submissions' => function($query) {
$query->orderBy('created_at', 'asc');
}, 'submissions.votes', 'submissions.suggestions.votes']);
Then pass the $user variable to the view, or:
$submissions = Submission::with('user', 'votes', 'suggestions.votes')
->where('user_id', Input::get('id'))
->sortByDesc('created_at')
->first();
Not entirely sure the code will work perfectly, but I'm sure you can tweak it. The point is your code can be a lot shorter and still/or more readable by using relationships you've already set up.
I have created the following for a product catelog/lister:
public function index($type_id = null) {
$filters = $sort = array();
if (isset($type_id)) {
$filters['type'] = $type_id;
} else {
$filters['type'] = Input::get('type');
}
$filters['search'] = Input::get('search');
$filters['brand'] = Input::get('brand');
$sort['sort'] = Input::get('sort');
$sort['sortdir'] = Input::get('dir');
$productsPaginated = $this->fetchProducts($filters, $sort);
return View::make('products.products', array(
'productsList' => $productsPaginated
)
);
}
public function fetchProducts($filters, $sorts, $perpage = 2) {
print_r($filters);
$Product = Product::query();
if (!empty($filters['search']))
$Product->where('name', 'LIKE', '%' . $filters['search'] . '%');
if (isset($filters['type']))
$Product->where('type_id', $filters['type']);
if (isset($filters['brand']))
$Product->where('brand_id', $filters['brand']);
if (isset($sorts['sort']))
$Product->orderBy($sorts['sort'], $sorts['sortdir']);
$Product = $Product->paginate($perpage);
return $Product;
}
Which works well so far.
I am now trying to create some filters so a user can further filter the results.
How can I access and determine distinct rows based on a column in:
$productsPaginated = $this->fetchProducts($filters, $sort);
?
The groupBy method not only exists on the query builder but also on the collection class. (which will be returned when calling paginate)
Take a look at the source on github
So add an argument to your function and use groupBy
public function fetchProducts($filters, $sorts, $perpage = 2, $groupBy = null) {
// code omitted for brevity
$Product = $Product->paginate($perpage);
if($groupBy){
$Product = $Product->groupBy($groupBy);
}
return $Product;
}
Update
Then there's the lists function that works on collections as well as on query builders...
$Product->lists('column-name');
Update 2
I was curious so I did some testing and a found something very weird and I have no idea if its a bug or a feature I don't understand
When calling groupBy the collection returned has actually only one item (index "") and this item contains an array of the "original" items. So to make lists work. I found this workaround
$Product = $Product->groupBy($groupBy);
$Product = new Collection($Product[""]); // \Illuminate\Support\Collection
$Product = $Product->lists('column-name');