I am building an online tool for soccerclubs in php. Part of the system is a match module where you can make a video analysis, give grades to players, and more.
Everything is working fine, only problem being the performance of the initial pageload. The reason behind the problem is the database and the way im getting and presenting the data from the database.
My current database looks like this(I removed all unnecessary fields):
http://i.stack.imgur.com/VZnpC.png
When I'm done getting my data from my database I have an object like this:
$match->formations[]->positiongroups[]->positions[]->users[]->statgroups[]->stats[]
The way I'm getting the data takes way to much time( about 12 seconds ) and I'm probably doing it completely wrong. You can see the code below. I use laravel 4 as framework so most code isn't plain php but I think when you read the code you will understand what every line of code does. I do want to notice that an non-laravel solution is fine!
/*
* Get fullmatch info
* Returned value consists of
* $match->formation->positiongroups[]->positions[]->users[]->statgroups[]->stats[]
*/
public static function fullMatch($id){
//Get match of given id with formation and team
$match = Match::where('id', '=', $id)->with('formation', 'team.formation.positions', 'team.category')->first();
if($match->formation){
//Set all positiongroups in $match->formation
$match->formation->positiongroups = Positiongroup::all();
//Get possible positions
foreach(Formation::find($match->formation->id)->positions as $position){
$positions[] = $position->id;
}
//Loop through all positiongroups in $match->formation
foreach($match->formation->positiongroups as $positiongroup){
//Set all positions in positiongroups
$positiongroup->positions = Position::where('positiongroup_id', '=', $positiongroup->id)->whereIn('id', $positions)->get();
foreach($positiongroup->positions as $position){
$position->users = DB::table('formation_position_match_user')
->leftJoin('users', 'user_id', '=', 'users.id')
->where('formation_id', '=', $match->formation->id)
->where('position_id', '=', $position->id)
->where('match_id', '=', $match->id)
->get();
foreach($position->users as $user){
$user->statgroups = Statgroup::where('video', '=', 1)->with('Stats')->get();
$user->stats = DB::table('stat_statdate_user')
->leftJoin('statdates', 'statdate_id', '=', 'statdates.id')
->where('stat_statdate_user.user_id', '=', $user->id)
->groupBy('stat_statdate_user.stat_id')
->orderBy('stat_statdate_user.stat_id')
->get();
}
}
}
}
return $match;
}
If there is more information needed I'm happy to add it to the post.
I haven't seen the queries it produces, but I think you have too many nested foreach loops and you are sending too many queries to database. You should minimize number of queries. You can do it manually or use some library. For example NotORM
edit: Here is example of what you could easily do and improve the performance:
You should focus on getting more data at once, than doing it row by row. For example replace some = in WHERE conditions with IN ();
so instead of sending a lot of queries like
SELECT * FROM positions WHERE position_group_id = x;
SELECT * FROM positions WHERE position_group_id = y;
SELECT * FROM positions WHERE position_group_id = z;
you send only one SELECT to server:
SELECT * FROM positions WHERE position_group_id IN (x, y, z);
You will have to modify your code, but it won't be that difficult... I don't know how your where method works, and if it supports IN statement, but if it does, do something like this:
$position_group_ids = array();
foreach ($match->formation->positiongroups as $positiongroup) {
$position_group_ids[] = $positiongroup->id;
}
$all_positions = Position::where('positiongroup_id', 'IN', $position_group_ids);
Related
I have laravel project where I need to periodically check if one of 3.000 application users is in Anti Money Laundering database(5.000.000 rows big Microsoft SQL table).
In frontend I did it async with ajax, so when I click on "Check Users" button I will wait for response.
$users = User::all(); // 3.000 rows
foreach($users as $user){
$aml = DB::table('anti_money_laundering') // 5.000.000 rows
->select('ID')
->whereRaw("LOWER([FULL_NAME]) = ?", [$user->full_name])
->first();
if($aml){
//Bingo, do stuff
//Continue
}
}
But I get maximum execution time of 30 seconds error and I think that increasing request time in php.ini configuration is not solution to my problem.
How should I do it? What is the best practice for big query/long request?
You are currently running a query within a loop - and a loop with 3.000 iterations will therefor make 3.000 queries - that takes time!
Instead, you can just run one query where you join the two tables together and see if any results were returned.
SELECT aml.id
FROM anti_money_laundering AS aml
JOIN users AS u
ON aml.FULL_NAME = u.full_name
In Eloquent, you can do it like this
$query = DB::table('anti_money_laundering ')
->join('users', 'users.full_name', '=', 'anti_money_laundering.full_name')
->select('anti_money_laundering.id')
->get();
If there are any results, there is a match.
Build up an array with usernames and check whether any anti_money_laundering record is in your set:
$users = User::all(); // 3.000 rows
$fullNames = [];
foreach($users as $user){
$fullNames[]=$user->full_name;
}
$badUsers = DB::table('anti_money_laundering') // 5.000.000 rows
->select('ID')
->whereIn("LOWER([FULL_NAME]) = ?", $fullNames)
->get();
foreach($badUsers as $badUser){
//Bingo, do stuff
//Continue
}
Note, that I'm not fluent in Laravel or Eloquent, so if there are typos in my answer, then that's the cause. However, the idea is superior to your initial idea of sending 3000 requests (very slow) or the other answer which does an unnecessary join.
Sorry if my title is confusing, not sure how to explain this within a line. Let's say I have a table with some columns and I have this
$model = Document::where('systemName', '=', $systemName)->where('ticketNumber', '=', ($nextTicketNumber))->get(); ticketNumber is unique where as there are quite a few systemNames
The above will get exactly what I want but I want more. I want an another array which will store all the rows under the same systemName. I know I can do this by doing
$allSystemNameModel = Document::where('systemName', '=', $systemName)
But is there a possible way to not having two variables and be easier?
No, you can't get both collections into one variable with one statement, however, you can create an array and store your results there:
$both = [];
$both['model'] = ...
$both['all'] = ...
UPDATE:
To avoid querying the database twice, you can use a first method that laravel provides us with.
$allSystemNameModel = Document::where('systemName', '=', $systemName);
$model = $allSystemNameModel->first(function ($doc) use ($nextTicketNumber) {
return $doc->ticketNumber == $nextTicketNumber;
});
$both['model'] = $model;
$both['all'] = $allSystemNameModel->all();
Note: Be sure to use use when working with php closures since $nextTicketNumber will be undefined otherwise.
I want to query my Laravel model using eloquent for results that may need to match some where clauses, then take and skip predefined numbers.
This isn't a problem in itself, but I also need to know the number of rows that were found in the query before reducing the result set with take and skip - so the original number of matches which could be every row in the table if no where clauses are used or a few if either is used.
What I want to do could be accomplished by making the query twice, with the first omitting "->take($iDisplayLength)->skip($iDisplayStart)" at the end and counting that, but that just seems messy.
Any thoughts?
$contacts = Contact::where(function($query) use ($request)
{
if (!empty($request['firstname'])) {
$query->where(function($query) use ($request)
{
$query->where('firstname', 'LIKE', "%{$request['firstname']}%");
});
}
if (!empty($request['lastname'])) {
$query->where(function($query) use ($request)
{
$query->where('lastname', 'LIKE', "%{$request['lastname']}%");
});
}
})
->take($iDisplayLength)->skip($iDisplayStart)->get();
$iTotalRecords = count($contacts);
You can use count then get on the same query.
And by the way, your whole query is a bit over complicated. It results in something like this:
select * from `contacts` where ((`firstname` like ?) and (`lastname` like ?)) limit X, Y
Closure in where is used to make a query like this for example:
select * from table where (X or Y) and (A or B);
So to sum up you need this:
$query = Contact::query();
if (!empty($request['firstname'])) {
$query->where('firstname', 'like', "%{$request['firstname']}%");
}
if (!empty($request['lastname'])) {
$query->where('lastname', 'like', "%{$request['lastname']}%");
}
$count = $query->count();
$contacts = $query->take($iDisplayLength)->skip(iDisplayStart)->get();
The Collection class offers a splice and a count method which you can take advantage of.
First you would want to get the collection..
$collection = $query->get();
Then you can get the count and splice it.
$count = $collection->count();
$records = $collection->splice($iDisplayStart, $iDisplayLength);
This might be hard on performance because you are querying for the entire collection each time rather than putting a limit on the query, so it might be beneficial to cache the collection if this page is going to be hit often. At the same time though, this will hit the database only once, so it's a bit of a trade off.
I have tried various methods to resolve this issue, but none worked for me.
1st method:
$title = Character::find($selected_char->id)->title()->where('title', '=', 'Castle');
$title = $title->where('title', '=', 'City');
$title = $title->get();
2nd method:
$title = Character::find($selected_char->id)->title()->where('title', '=', 'Castle')->where('title', '=', 'City')->get();
3rd method:
$title = DB::select(DB::raw("select * from titles where titles.char_id = 5 and title = 'Castle' and title = 'City'"));
None of the above methods work. If I take only one where clause it works perfectly. Example:
$title = Character::find($selected_char->id)->title()->where('title', '=', 'City')->get();
$title = Character::find($selected_char->id)->title()->where('title', '=', 'Castle')->get();
I even tried to take another column than title, but it doesn't work with a second where function. I want to retreive the rows from titles table where the title is City AND Castle I have used multiple where clauses before in a single select statement and it worked. Not now. Any suggestions? Thanks in advance.
You said:
I want to retreive the rows from titles table where the title is City AND Castle
You may try this:
$rowCOllection = DB::table('titles')
->whereIn('title', array('City', 'Castle'))->get();
Using multiple where:
$rowCOllection = DB::table('titles')
->where('title', 'City')
->where('title', 'Castle')->get();
If you want to add another where clause for titles.char_id then you may use it like:
$rowCOllection = DB::table('titles')
->where('title', 'City')
->where('title', 'Castle')
->where('char_id', 5)->get();
You may chain as much where as you need before you call get() method. You can add the where('char_id', 5) after the whereIn like whereIn(...)->where('char_id', 5) and then call get().
If you have a Title model then you may do the same thing using:
Title::where(...)->where(...)->get();
Same as using DB, only replace the DB::table('titles') with Title, for example:
$rowCOllection = Title::where('title', 'City')
->where('title', 'Castle')
->where('char_id', 5)->get();
What about Character here ?
I don't really know how work your double ->where( in php, but in sql here is the mistake :
When you say where title = 'a' and title = 'b', it's like you say : ok give me something where 0=1 it returns nothing.
You can do :
select * from titles where titles.char_id = 5 and (title = 'Castle' or title = 'City')
Retrieve all data where title equals castle or city
Or
select * from titles where titles.char_id = 5 and title IN ('Castle','City')
Retrieve all data where title equals castle or city using IN
I'm pretty sure you will find a way to do that in PHP too.
Assuming you are using Laravel 4
And Character is your model extended from Eloquent
don't mix FIND and WHERE.
Find is for single usage find AND sorting afterward (so order by, and etc)
So if you want to chain up your query
Character::where()->where()->where()-get() (don't forget the get or else you wont get a result)
this way you respect eloquent's features.
Note your first method with ->title() is flawed because your calling a function that you custom created inside your model - thats why it wouldn't have worked.
Note: WereWolf Alpha's method will also work IF you don't want to use Eloquent because the code that he presented will work but thats Fluent notation...so take your pick.
Of course I can use order_by with columns in my first table but not with columns on second table because results are partial.
If I use 'join' everything works perfect but I need to achieve this in eloquent. Am I doing something wrong?
This is an example:
//with join
$data = DB::table('odt')
->join('hdt', 'odt.id', '=', 'hdt.odt_id')
->order_by('hdt.servicio')
->get(array('odt.odt as odt','hdt.servicio as servicio'));
foreach($data as $v){
echo $v->odt.' - '.$v->servicio.'<br>';
}
echo '<br><br>';
//with eloquent
$data = Odt::get();
foreach($data as $odt){
foreach($odt->hdt()->order_by('servicio')->get() as $hdt){
echo $odt->odt.' - '.$hdt->servicio.'<br>';
}
}
In your model you will need to explicitly tell the relation to sort by that field.
So in your odt model add this:
public function hdt() {
return $this->has_many('hdt')->order_by('servicio', 'ASC');
}
This will allow the second table to be sorted when using this relation, and you wont need the order_by line in your Fluent join statement.
I would advise against including the order by in the relational method as codivist suggested. The method you had laid is functionally identical to codivist suggestion.
The difference between the two solutions is that in the first, you are ordering odt ( all results ) by hdt.servicio. In the second you are retrieving odt in it's natural order, then ordering each odt's contained hdt by servico.
The second solution is also much less efficient because you are making one query to pull all odt, then an additional query for each odt to pull it's hdts. Check the profiler. Considering your initial query and that you are only retrieving one column, would something like this work?
HDT::where( 'odt_id', '>', 0 )->order_by( 'servico' )->get('servico');
Now I see it was something simple! I have to do the query on the second table and get contents of the first table using the function odt() witch establish the relation "belongs_to"
//solution
$data = Hdt::order_by('servicio')->get();
foreach($data as $hdt){
echo $hdt->odt->odt.' - '.$hdt->servicio.'<br>';
}
The simple answer is:
$data = Odt::join('hdt', 'odt.id', '=', 'hdt.odt_id')
->order_by('hdt.servicio')
->get(array('odt.odt as odt','hdt.servicio as servicio'));
Anything you can do with Fluent you can also do with Eloquent. If your goal is to retrieve hdts with their odts tho, I would recommend the inverse query for improved readability:
$data = Hdt::join('odt', 'odt.id', '=', 'hdt.odt_id')
->order_by('hdt.servicio')
->get(array('hdt.servicio as servicio', 'odt.odt as odt'));
Both of these do exactly the same.
To explain why this works:
Whenever you call static methods like Posts::where(...), Eloquent will return a Fluent query for you, exactly the same as DB::table('posts')->where(...). This gives you flexibility to build whichever queries you like. Here's an example:
// Retrieves last 10 posts by Johnny within Laravel category
$posts = Posts::join('authors', 'authors.id', '=', 'posts.author_id')
->join('categories', 'categories.id', '=', 'posts.category_id')
->where('authors.username', '=', 'johnny')
->where('categories.name', '=', 'laravel')
->order_by('posts.created_at', 'DESC')
->take(10)
->get('posts.*');