So I'm just about at my whit's end here. I've been tasked with optimizing a system because pages are loading too slowly. The system is built on Laravel 5.5 (yeah I know not the latest version, but that's a problem for another day). I've installed Clockwork to get a better feel for what could be causing things to be so slow and literally fell off my seat. The load of a single page generated 10412 queries in the database (you read that right, 10000 queries to load a single page)!!!!!
Anyway, I tweaked relationships so that things could be properly lazy/eager loaded when necessary and that divided things by three... but that's still a lot considering how little information is currently displayed.
So I went on to try and tackle another problem they have: They defined custom attributes inside models that fetch informations from three or four layers of relationships. Here's a "simple" example:
ride-->(departureStop, destinationStop)-->(state, event, city)
This path is to resolve the "getNameAttribute()" defined inside the Ride model. So in essence, Laravel generates 5 queries just to calculate the name of a ride (said name being something like "Ottawa, On to Toronto, On"). Of course, being an attribute, this means that we get those 5 queries FOR EVERY INSTANCE OF THE RIDE MODEL. And that's just one attribute, there are dozens of them just like this.
Oh and just to add to the fun, we're also using the compoship package because some tables have composite primary keys.
I'm not sure what to include in terms of code here since everything is such tangled mess but every suggestion you can throw at me will be more than welcome.
Related
I'm somewhat new to laravel and working on my first full-stack laravel project. I've been trying to follow the best practices of laravel to help with better optimization and performance. In the dashboard area of my app, I noticed that the models on some of my pages have 253 counts and the view in my dashboard layout has 17 counts when I check the Laravel debugger stats [see image below].
In the image above, I have a country model that has 249 rows of all countries in the world (which are used as dropdown select for users to pick their countries), as well as a notification (this count varies per user).
From my knowledge so far, I know data like countries that rarely change can be cached but I'd like to know how many is too many when it comes to Models and Views count. So I can always watch out for those values, in case I find myself in situations where I can't use cache or before data is cached.
Thanks
That is fine, if you need to display 249 records from the table then you will have to pull 249 records (or models).
What is an issue is when you involve relationships and you come across issues such as the n+1 query prblem; which is described here: What is the "N+1 selects problem" in ORM (Object-Relational Mapping)?
Currently working on a new project, where different components can be entered and connected to eachother. The trick is that the amount of components and the data in a component can be anything, making it hard to find some sort of database-logic in it when it comes to table-setup and keys.
For example I have the following components:
cars
name: audi, color: red, horsepower: 500
bicycle
name: mybike, color: blue, lights: front and back
Etcetera. These components have some types in common, like name and color, but some different types, like horsepower on a car and lights on a bike.
These components can really be anything, but I need a way to link these together somehow in a database-structure that still makes sense.
Ive figured I could define a component-table that simply feeds an ID, with a seperate component-type table next to it per component. However that would require a different table everytime someone comes up with a new component (which again could be anything at any time).
I really do need to be able to structure these and link them together somehow, for example link that car to that bike.
Im running into a bit of a wall in my head thinking about this and hope someone can help me figure out how to do this in mysql.
The open ended database model that fits this use case is the Entity Attribute Value (EAV) model http://en.wikipedia.org/wiki/Entity%E2%80%93attribute%E2%80%93value_model. The problems that you encounter with this model are performance and complexity of queries. Now some DBs can assist you with pivot functions that will turn rows into column, but MySQL is not one of them. So if you go with this model then go with postgres as it has crosstab which will pivot the single row attributes of an entity into columns.
I am currently working on a huge refactoring project. We have taken over a classic PHP/MySQL project, where most code is procedural, duplicated, and there is very little hint of an architecture.
I am planning on using Doctrine to handle our Data Access, and have all of my tables mapped to entities. However, our MySQL tables are largely messed up.
The table I am currently working with has over 40 columns, and is not normalized by any means. A quick example of what we have:
Brand
id
name
poNumber
orderConfirmationEmail <---- these should go into a BrandConfirmations entity
shippingConfirmationEmail <-----
bill_address <---- these should go into a BrandAddress entity
bill_address2 <-----
city <------
.
.
.
Ideally, what I would like to have is for Doctrine to pull out the fields that reference different Entities, and actually put them into those Entities. So for instance id, name, and poNumber would get pulled out into a Brand entity. orderConfirmationEmail and shippingConfirmationEmail would get pulled out into a BrandNotification entity. Next, bill_address, and the rest of the address fields would get pulled out into a BrandBillAddress entity. Is there a way to configure Doctrine to split the table into these models for me, or do I have to custom write code myself that would do that?
If I do have to write the code to split this table myself, do you have any resources or advice that tackle a similar issue? I haven't been able to find many yet.
The latest version of Doctrine 2 supports what they call embeddables: http://doctrine-orm.readthedocs.org/en/latest/tutorials/embeddables.html. It may solve some of your problems. However, it requires D2.5+. Currently, S2 uses Doctrine 2.4. You could experiment with using the very latest doctrine.
What you can do is make your domain models (entities) act as though you had value objects. So $brand->getOrderConfirmation() would actually return an order confirmation object. You have to do some messing around to keep everything mapped to one table and you might be limited on some of your queries but it's not that hard. The advantage is that the rest of your new applications deals with proper normalized objects. It's only the internal persistence code that needs to get messy.
There are quite a few links on this approach. Here is one: http://russellscottwalker.blogspot.com/2013/11/entities-vs-value-objects-and-doctrine-2.html
Your best bet of course is to refactor your database schema. I like to do kind of a raw dump of the original database into a yaml file with the desired object nesting. I then load the yaml file into the new schema. If you are really lucky then you might even be able to create new views for your existing application which will allow it to keep working in parallel with your new application.
I moved from an old sever running centOS on a managed hoster to a new one running Ubuntu in AWS.
Post the move I've noticed that the page that loads a list of items is now taking ~10-12 secs to render (sometimes even up to 74secs). This was never noticed on the old server. I used newrelic to look at what was taking so long and found that the sfPHPView->render() was taking 99% of the time. From nerelic there is approximately ~500 calls to the DB to render the page.
The page is a list of ideas, with each idea a row. I use the $idea->getAccounts()->getSlug() ability of Doctrine 1.2. Where accounts is another table linked to the idea as a foreign relation. This is called several times for each idea row. A partial is not currently used to hold the code for each row element.
Is there a performance advantage to using a partial for the row element? (ignoring for now the benefit of code maintability)
What is best practice for referencing data connected via a foreign relation? I'm surprised that a call is made to the DB everytime $idea->getAccounts()->getSlug() is called.
Is there anything obvious in ubuntu that would otherwise be making sfPHPView->render() run slower than centOS?
I'll give you my thought
When using a partial for a row element, it's more easy to put it in cache, because you can affine the caching by partial.
Because you don't explicit define the relation when making the query, Doctrine won't hydrate all elements with the relation. For this point, you can manually define relations you want to hydrate. Then, your call $idea->getAccounts()->getSlug() won't perform a new query every time.
$q = $this->createQuery();
$q->leftJoin('Idea.Account');
No idea for the point 3.
PS: for the point 2, it's very common to have lots of queries in admin gen when you want to display an information from a relation (in the list view). The solution is to define the method to retrieve data:
In your generator.yml:
list:
table_method: retrieveForBackendList
In the IdeaTable:
public function retrieveForBackendList(Doctrine_Query $q)
{
$rootAlias = $q->getRootAlias();
$q->leftJoin($rootAlias . '.Account');
return $q;
}
Though I would add what else I did to improve the speed of page load in addition to jOk's recommendations.
In addition to the explicit joins I did the following:
Switched to returning a DQL Query object which was then passed to Doctrine's paginator
Changed from using include_partial() to PHP's include() which reduced the object creation time of include_partial()
Hydrate the data from the DB as an array instead of an object
Removed some foreach loops by doing more leftJoins in the DB
Used result & query caching to reduce the number of DB calls
Used view caching to reduce PHP template generation time
Interestingly, by doing 1 to 4 it made 5 and 6 more effective and easier to implement. I think there is something to be said for improving your code before jumping in with caching.
In my MVC web app, I'm finding myself doing a lot of actions with ActiveRecords where I'll fetch a specific subset of Products from the database (for a search query, say), and then loop through again to display each one -- but to display each one requires several more trips to the database, to fetch things like price, who supplies them, and various other pieces of metadata. To calculate each of these pieces of metadata isn't very simple; it's not really something that could be achieved with a simple JOIN. However, it WOULD be possible (for most of these cases anyway) to batch the required database calls and do them all at once before the loop, and then within the loop refer to those pre-fetched data to do the various calculations.
Just as an example of the type of thing -- in a search, I might want to know what regions the product is provided by. In the database I have various rows which represent a particular supplier's stock of that item, and I can look up all the different suppliers which supply that item, and then get all the regions supplied by those suppliers. That fits nicely into one query, but it would start getting a bit complex to join into the original product search (wouldn't it?).
I have two questions:
does anyone else do something similar to this, and does it sound like a good way to handle the problem? Or does it sound more like a design problem (perhaps the application has grown out of ActiveRecord's usefulness, or perhaps the models need to be split up and combined in different ways, for instance).
If I do pre-fetch a bunch of different things I think I'll use inside the loop, I'm having a hard time deciding what would be the best way to pass the appropriate data back to the model. At the moment I'm using a static method on the model to fetch all the data I need at the start of the array, like fetchRegionsForProductIds(array $ids) and so forth; these methods return an array keyed by the ID of the product, so when I'm working inside the loop I can get the regions for the current product and pass them as a parameter to the model method that needs them. But that seems kind of hacky and laborious to me. So, can anyone tell me if there is just some really obvious and beautiful design pattern I'm missing which could totally resolve this for me, or if it's just a bit of a complex problem that needs a kind of ugly complex solution?
Small update: I wonder if using a datamapper class would put me on the right track? Is there a common way of implementing a data mapper so that it can be told to do large batch queries up front, store that information in an array, and then drip feed it out to the records as they request it?
I really hope this question makes sense; I've done the best I can to make it clear, and will happily add more detail if someone thinks they can have a go at it!