We're in the middle of an upgrade to Pimcore 10, and we're experiencing severe performance degradation and a possible memory leak. We're working with PHP v8.1.2, Symfony, and and MySQL v8.
Worth noting - we're talking about VERY little data. The source tables all have fewer than 1000 rows.
I've verified that the generated queries are lightning fast when they run directly from workbench, or when I circumvent the ORM (Doctrine) and run the SQL from its connection object.
When I use the ORM as intended, the load() below should bring back all 219 rows from the table almost instantly. Instead it takes close to a minute.
Nothing changed about this code since before the upgrade, and we never had performance or memory issues before.
I'm new to Pimcore, Symfony, and Doctrine and this is my first time using PHP in probably 10 years. I think the problem is somewhere in the hydration of the model but I'm running low on ideas at this point.
Things I tried:
Running queries in MySQL Workbench (no issues)
Circumventing the ORM to run queries through the application (no issues)
Removed all other service and repository calls so I'm just looking at this situation.
Checked dump() of the hydrated model to see if there are recursion issues (no issues)
// The very simple controller
return $sampleService->getEverything();
// The very simple service
public function getEverything()
{
return $this->sampleRepository->findAll()->load();
}
// The very simple Repository
class SampleRepository
{
protected $listingClass;
public function findAll()
{
return new ($this->listingClass)();
}
}
Related
We're currently rewriting our codebase, making this change:
// From
public function __construct(EntityManager $em){} // then use $this->em->getRepository(Example::class)
// To:
public function __construct(ExampleRepository $exampleRepo){}
In order to facilitate that, we update the EntityReposity to a ServiceEntityRepository. That helps significantly with unittest and goes well most of the time.
In some rare cases however we get a 502 Bad Gateway and if we run php bin/console
we get a 'Segmentation fault', which (at least to our understanding) means that C just breaks.
During our research we've created the theory that the ServiceEntityRepository load the EntityManager, the EM the Repos, which in turn load the EM again, etc.
Our packages are currently up to date per everyones suggestions everywhere, I'm hoping someone here will come up with a solution, now or maybe in the future. So, does anyone have any idea?
This issue appears related: https://github.com/symfony/symfony/issues/30091
For now I have a "temporary" solution which I'll post here so that others might have at least a work-around if they're searching for a solution. If I find out myself, I'll also update this question.
The error appears to be as is explained in the topic. The solution is to go that step back. Remove the Repository injection from constructor and use the $this->em->getRepository(Example::class) in the methods. The following will not work, as you're still loading the repo in the construct, resulting in the loop:
function __construct(EntityManager $em){
// THIS WILL NOT WORK!
$this->exampleRepo = $this->em->getRepository(Example::class);
}
The issue appears a timing problem. The EntityManager loads the repositories, and each repository loads the Entitymanager, which in turn loads the repositories, ...
This results in an endless loop, which results in the interpreter running out of resources and returning the C error.
Note: Not all repositories have this problem, thusfar we haven't found any pattern yet.
Laravel's eloquent models are set to lazy load by default. The problem is that it makes a lot of query to the database and especially during high traffic, the laravel application crashes whereas a similar application build on Yii 1 has no issues.
After installing the Laravel's debug bar, the problem is too many queries being made on every page load. The next step is to query optimization. I have been using eager loading as directed in the Laravel's documentation but still too many queries.
I was wondering if there is a way to set Eloquent to only "Eager Load" in dev environment. That way when the page fails to load, identifying issue would be easier.
You could set defaults relations to "eager load" directly on the models :
Class MyModel extends Model {
protected $with = ['relation'];
}
The solution for high database load is Cache.
Caching properly could give you incredible performance during high traffic, because it reduces common database queries to zero, and redirect them to RAM ones, which are faster.
Enabling Route Caching will increase perfomance too:
php artisan route:cache
EDIT:
As Fx32 points, you should make sure that you need Eloquent and wouldn't be better to make the same query directly to the DB, joining the tables you need and making a single query instead of a lot:
Cache is not a good solution as a fix for bad database querying.
Eloquent is great, but often it's better to write proper queries with
some joins. Don't just bash away at your DB and then throw all the
results in a cache, as it will introduce new problems. If your use
case is not a flat CRUD API, ActiveRecord patterns might not be the
best solution anyway. If you carefully select and join results from
the DB, and want to speed up retrieval of such items, caching can
help.
I moved from an old sever running centOS on a managed hoster to a new one running Ubuntu in AWS.
Post the move I've noticed that the page that loads a list of items is now taking ~10-12 secs to render (sometimes even up to 74secs). This was never noticed on the old server. I used newrelic to look at what was taking so long and found that the sfPHPView->render() was taking 99% of the time. From nerelic there is approximately ~500 calls to the DB to render the page.
The page is a list of ideas, with each idea a row. I use the $idea->getAccounts()->getSlug() ability of Doctrine 1.2. Where accounts is another table linked to the idea as a foreign relation. This is called several times for each idea row. A partial is not currently used to hold the code for each row element.
Is there a performance advantage to using a partial for the row element? (ignoring for now the benefit of code maintability)
What is best practice for referencing data connected via a foreign relation? I'm surprised that a call is made to the DB everytime $idea->getAccounts()->getSlug() is called.
Is there anything obvious in ubuntu that would otherwise be making sfPHPView->render() run slower than centOS?
I'll give you my thought
When using a partial for a row element, it's more easy to put it in cache, because you can affine the caching by partial.
Because you don't explicit define the relation when making the query, Doctrine won't hydrate all elements with the relation. For this point, you can manually define relations you want to hydrate. Then, your call $idea->getAccounts()->getSlug() won't perform a new query every time.
$q = $this->createQuery();
$q->leftJoin('Idea.Account');
No idea for the point 3.
PS: for the point 2, it's very common to have lots of queries in admin gen when you want to display an information from a relation (in the list view). The solution is to define the method to retrieve data:
In your generator.yml:
list:
table_method: retrieveForBackendList
In the IdeaTable:
public function retrieveForBackendList(Doctrine_Query $q)
{
$rootAlias = $q->getRootAlias();
$q->leftJoin($rootAlias . '.Account');
return $q;
}
Though I would add what else I did to improve the speed of page load in addition to jOk's recommendations.
In addition to the explicit joins I did the following:
Switched to returning a DQL Query object which was then passed to Doctrine's paginator
Changed from using include_partial() to PHP's include() which reduced the object creation time of include_partial()
Hydrate the data from the DB as an array instead of an object
Removed some foreach loops by doing more leftJoins in the DB
Used result & query caching to reduce the number of DB calls
Used view caching to reduce PHP template generation time
Interestingly, by doing 1 to 4 it made 5 and 6 more effective and easier to implement. I think there is something to be said for improving your code before jumping in with caching.
Recently I integrated Doctrine 2 ORM into CodeIgniter 2. I configured Doctrine 2 as a library and autoloaded it in CodeIgniter. Within a page I instantiate doctrine entity manager in the following way:
private static $em = null;
public function __construct() {
parent::__construct();
$this->em = $this->doctrine->em;
}
And then I start to use the Entity Manager when needed. The issue I have is that in each page request Entity Manager takes some time to initialize (appr. 1 second). This causes the user to wait until the page is loaded. Below you can see some performance results I measured:
BENCHMARKS
Loading Time: Base Classes 0.0166
Doctrine 0.0486
GetArticle 1.0441
Functions 0.0068
Controller Execution Time 1.1770
Total Execution Time 1.1938
The GetArticle function basicly makes an EntityManager->find() call:
$currentart = $this->em->find('Entities\Article', $artid);
I have to wait that 1 second even if I use the EntityManager->createQuery() method.
In every page, I have a time loss of approximately 1 second because of EntityManager's first request.
Is this common?
Does this 1 second come from the fact that EntityManager needs to establish a connection to the DB? The functions/requests after the first request are quite fast though.
The most time consuming thing that Doctrine does is load metadata for your entities, whether it's annotations, XML, or YAML. Doctrine lazy loads the metadata when possible, so you will not see the performance hit until you start using entities. Since the metadata doesn't change unless you make changes in your code, Doctrine allows you to cache the metadata across requests. DQL queries also need to be parsed into SQL, so Doctrine provides another caching configuration for this.
In a production environment you should set these caches up (it sounds like you have already, but for others reading this):
$cache = new \Doctrine\Common\Cache\ApcCache(); // or MemcacheCache
$configuration->setMetadataCachImpl($cache); // caches metadata for entities
$configuration->setQueryCachImpl($cache); // caches SQL from DQL queries
In order to prevent the first page load from taking the full metadata load, you can set up a cache warmer that loads all of the class metadata and save it to the cache.
$em->getMetadataFactory()->getAllMetadata();
Another potential bottleneck is the generation of proxy classes. If this is not configured correctly in a production environment, Doctrine will generate the classes and save them to the file system on every page load. These proxy classes do not change unless the entity's code changes, so it is again unnecessary for this to happen. To speed things up, you should generate the proxies using the command line tool (orm:generate-proxies) and disable auto-generation:
$configuration->setAutoGenerateProxyClasses(false);
Hopefully this helps you out. Some more information can be found at http://www.doctrine-project.org/docs/orm/2.0/en/reference/improving-performance.html#bytecode-cache
I'd like to use a transaction rollback method to isolate my database for unit testing purposes. Ideally, I would use a structure something like this:
public static function setUpBeforeClass(){
Mage_Core_Model_Resource_Transaction::beginTransaction();
}
public function testOne(){...}
public function testTwo(){...}
public static function tearDownAfterClass(){
Mage_Core_Model_Resource_Transaction::rollBack();
}
Unfortunately the Mage_Core_Model_Resource_Transaction class doesn't expose public begin/rollbackTransaction functions. I can't find any public Magento functions to meet the requirement. Is there a Zend equivalent that will work?
I guess I could rewrite Mage_Core_Model_Resource_Transaction and add public wrappers for the protected methods, but I'm hesitant to override such a core class.
I have also tried using
$this->model = Mage::getModel('model_being_tested');
$this->model->getResource()->beginTransaction();
...
$this->model->getResource()->rollBack();
and then use the $this->model in the tests but that can't be used in static functions.
Any suggestions or alternative architectures? Thanks in advance.
Frankly speaking, I am going to create some kind of test suite for Magento to make possible writing test cases in your module without taking care of the initialization and so on. Of course I don't have enough time because of the projects, but I want to share the idea wich I am going to use considering database fixtures. I came up with creating a separate database for unit tests (copied from current by test case), because each test case will load initial data for it via fixture. The database connection credentials will be set up prior to Mage::app()->run(), so it will be possible to protect your development copy from possible changes by unit tests.
You cannot rely on transactions, especially in case of saving product... There is set a commit callback for starting re-indexing process and it may cause unpredictable results if you haven't committed data to product tables. As well mysql server may gone away in this case, especially if you have a large database.
UPDATE:
The extension for it:
http://www.ecomdev.org/2011/02/01/phpunit-and-magento-yes-you-can.html