I have encountered a weird "bug" in PHP and since I'm a novice I'm at the end of my knowledge.
I'm developing a TYPO3 extension that has some major performance issues with data, or so I thought.
It turns out that the first use of the array, which stores all the objects I got from my database query, is taking way to long.
Every use or loop after that is fast again.
The code looks like this:
$productsArr = $this->productRepository->findByDetail($category, $properties);
$newSortArr = array();
$familyProductList = array();
$counter = count($productsArr);
/** #var Product $product */
for($i = 0; $i < $counter; $i++) {
//it takes to long to do this
$product = $productsArr[$i];
if(!empty($productsArr[$i])) {
$newSortArr[$product->getInFamily()->getUid()][] = $product;
}
}
It doesn't matter where I first use the object array. The first use of the array is always taking around 30 sec.
Has anybody encountered something similar?
If you need more information I will gladly provide that.
Thanks in advance!
Your $productsArr is not an array but an Object of Exbase's class QueryResult which you can iterate over with foreach or do index access. This Object execute the query and build it's objects only when needed, so at the moment you do $product = $productsArr[$i];, all Product-objects of $productsArrare built. The major problem is that building objects in PHP has bad performance and consumpts a lot of memory.
So, to avoid the performance issue, consider using a custom query with
$this->productRepository->createQuery()->statement('select * from ...')->execute();
to get exactly what you want instead of loading a huge amount of objects and refine them later in PHP.
As Jay already mentioned, your result isn't an array, but a QueryResult. Just FYI, it's possible to transform it to an array by adding ->toArray() at the end of your query:
$productsArr = $this->productRepository->findByDetail($category, $properties)->toArray();
But this won't improve the situation. There are two possible issues:
Iterating all objects
The advantage of a QueryResult is that it reflects only the result of the query but doesn't resolve all objects already. A QueryResult can be passed e.g. to a Pagination widget and will then only load the results requested (e.h. 1-10, 11-20 etc.).
Since you're applying manual sorting, all your objects (depending on your project this can be a lot...) are loaded.
Apparently you would like to sort the products by their family UID? Why not do that with Extbase functionality in your ProductRepository:
protected $defaultOrderings = array(
'inFamily.uid' => \TYPO3\CMS\Extbase\Persistence\QueryInterface::ORDER_ASCENDING
);
Eager loading of sub objects
Your model Product might have relations to other models (e.h. Product to Category, Product to Options etc.). By default, Extbase resolves all these relations on accessing the objects.
To prevent this, you can use Lazy Loading for relations. This makes sense for sub objects that are not used in all the views. E.g. in your list view you only need to title, image and price of your product, but you don't need all options of the product.
To configure lazy loading for these sub objects, you just need to #lazy annotation in the model:
/**
* #var \TYPO3\CMS\Extbase\Persistence\ObjectStorage<\My\Extension\Domain\Model\ObjectStorageModel>
* #lazy
*/
protected $categories;
/**
* #var \My\Extension\Domain\Model\OtherModel
* #lazy
*/
protected $author;
Lazy loading can have some drawbacks, e.g. in certain situations when checking for an object being an instance of OtherModel, you get an object of type LazyLoadingProxy instead. You can work around most of these issues or maybe don't even stumple upon them in normal scenarios. A common workaround if you really depend an object on not being a LazyLoadingProxy is a check like that:
if ($product->getAuthor() instanceof \TYPO3\CMS\Extbase\Persistence\Generic\LazyLoadingProxy) {
$product->getAuthor()->_loadRealInstance();
}
This makes sure that in any case you have a "real" instance of the object.
Please don't forget to flush system caches when you're doing a change regarding either one of the issues.
I assume the array is populated on the first line. Have you considered populating $product using foreach($productArr as $product) instead o using for?
Related
I have a strange problem with \Doctrine\ORM\UnitOfWork::getScheduledEntityDeletions used inside onFlush event
foreach ($unitOfWork->getScheduledEntityDeletions() as $entity) {
if ($entity instanceof PollVote) {
$arr = $entity->getAnswer()->getVotes()->toArray();
dump($arr);
dump($entity);
dump(in_array($entity, $arr, true));
dump(in_array($entity, $arr));
}
}
And here is the result:
So we see that the object is pointing to a different instance than the original, therefore in_array no longer yields expected results when used with stick comparison (AKA ===). Furthermore, the \DateTime object is pointing to a different instance.
The only possible explanation I found is the following (source):
Whenever you fetch an object from the database Doctrine will keep a copy of all the properties and associations inside the UnitOfWork. Because variables in the PHP language are subject to “copy-on-write” the memory usage of a PHP request that only reads objects from the database is the same as if Doctrine did not keep this variable copy. Only if you start changing variables PHP will create new variables internally that consume new memory.
However, I did not change anything (even the created field is kept as it is). The only operations that were preformed on entity are:
\Doctrine\ORM\EntityRepository::findBy (fetching from DB)
\Doctrine\Common\Persistence\ObjectManager::remove (scheduling for removal)
$em->flush(); (triggering synchronization with DB)
Which leads me to think (I might be wrong) that the Doctrine's change tracking method has nothing to do with the issue that I'm experiencing. Which leads me to following questions:
What causes this?
How to reliably check if an entity scheduled for deletion is inside a collection (\Doctrine\Common\Collections\Collection::contains uses in_array with strict comparison) or which items in a collection are scheduled for deletion?
The problem is that when you tell doctrine to remove entity, it is removed from identity map (here):
<?php
public function scheduleForDelete($entity)
{
$oid = spl_object_hash($entity);
// ....
$this->removeFromIdentityMap($entity);
// ...
if ( ! isset($this->entityDeletions[$oid])) {
$this->entityDeletions[$oid] = $entity;
$this->entityStates[$oid] = self::STATE_REMOVED;
}
}
And when you do $entity->getAnswer()->getVotes(), it does the following:
Load all votes from database
For every vote, checks if it is in identity map, use old one
If it is not in identity map, create new object
Try to call $entity->getAnswer()->getVotes() before you delete entity. If the problem disappears, then I am right. Of cause, I would not suggest this hack as a solution, just to make sure we understand what is going on under the hood.
UPD instead of $entity->getAnswer()->getVotes() you should probably do foreach for all votes, because of lazy loading. If you just call $entity->getAnswer()->getVotes(), Doctrine probably wouldn't do anytning, and will load them only when you start to iterate through them.
From the doc:
If you call the EntityManager and ask for an entity with a specific ID twice, it will return the same instance
So calling twice findOneBy(['id' => 12]) should result in two exact same instances.
So it all depends on how both instances are retrieved by Doctrine.
In my opinion, the one you get in $arr is from a One-to-Many association on $votes in the Answer entity, which results in a separate query (maybe a id IN (12)) by the ORM.
Something you could try is to declare this association as EAGER (fetch="EAGER"), it may force the ORM to make a specific query and keep it in cache so that the second time you want to get it, the same instance is returned ?
Could you have a look at the logs and post them here ? It may indicates something interesting or at least relevant to investigate further.
I have an Extbase-based extension for Typo3, which has a hierarchical data model. I had to insert an additional layer to this model, ie the original structure was Project contains multiple items. Now, I have Project contains multiple sub-projects and Sub-project contains multiple items. Everything is modelled using MM-relation tables and works in the backend. I can add, remove, sort the sub-projects and items.
However, the fluid template does not show anything and if I pass eg the sub-Project to t3lib_utilities_debug::Debug, I get
You should never see this warning. If you do, you probably used PHP
array functions like current() on the
Tx_Extbase_Persistence_ObjectStorage. To retrieve the first result,
you can use the rewind() and current() methods.
when printing the ObjectStorage for the items. I assume that the MM-relation I added is somehow broken, but I cannot figure out how. Furthermore, it seems that the __construct method of the domain model is not called (I have added a debug output, which is not printed).
The enumeration works if I pass the result of a call to findAll of the repository, but it does not work for my filtered calls (which worked before I added the additional layer). The filtering method looks like eg for the item
public function findBySubProject(SubProject $p) {
$query = $this->createQuery();
$query->getQuerySettings()->setRespectStoragePage(false);
$query->matching($query->equals('subproject', $p));
return $query->execute();
}
As I said, the query yields results, but they are somehow broken wrt. their relations.
Any ideas how to fix that?
I don't know on which version of Extbase you're developing on.
But on TYPO3 4.6+ you should be aware of the object and reflection caching. During development you can disable this caching by:
$GLOBALS['TYPO3_CONF_VARS']['SYS']['caching']['cacheConfigurations']['extbase_reflection']['backend'] = 't3lib_cache_backend_NullBackend';
$GLOBALS['TYPO3_CONF_VARS']['SYS']['caching']['cacheConfigurations']['extbase_object']['backend'] = 't3lib_cache_backend_NullBackend';
Since your problem has something to do with modifications in your model you should try to truncate the tables cf_extbase_object, cf_extbase_object_tags, cf_extbase_reflection and cf_extbase_reflection_tags after any change.
If this doesn't help you to resolve your problem then you should give us more insight about your configuration (especially the TCA configuration because Extbase rely on it).
How to test a Extbase QueryResult
$items = $this->itemRepository->findAll();
echo count($items);
if ($items) {
echo '<pre>';
foreach ($items as $item) {
print_r($item);
}
echo '</pre>';
}
-- edit --
Did you define the field subproject in your TCA? It should be atleast available as type passtrough:
'subproject' => array(
'config' => array(
'type' => 'passthrough',
),
),
I case someone else encounters the same issue: I accidentally used an object without items as test object. If you try to enumerate/debug/display an empty ObjectStorage, the warning is printed.
If anyone has an idea, I couldn't think of a better way to phase the question.
I'll try to not make this to complicated an explination.
I'm writing a "quotes" class that is the main class. This class has "overall" functions that preform calculations based on "items" stored in its array. Suffice it to say, the end-developer will call it as $q = new apiQuote/quote().
Then, before it's of any use, the first item must be added and it's properties set so it can do it's "own" calculations. Something like $q->createItem(). Once the item is created with this call, an "item" of the class "item" is added to an array in "quotes" named "items".
The currently editable item, $q->item is always the last one added to the array via the createItem method. Which looks like:
public function createNewItem() {
$this->items[] = new item();
$this->item = $this->items[count($this->items)-1];
}
I added setItem method, whereby the parameter would be an integer representing item index in the array and would set $q->item to the item index chosen. This works, but still seems "not as productive" as I'd like".
What I'm curious about, is if anyone has any suggestions on a better way to go about this. I tried looking for a "cards/deck" php example, but all I could find was array shuffles, which is kinda useless here. I know how to do such associations in .NET and thought this would be just as easy, but I don't have the same property abilities in PHP that I have in a .NET language, thus negating what I'm used to in created this kind of "class/subclass[items]" type structure.
Really I would just like to know if anyone has done anything similar and if I'm doing things to the "best of ability" or if there might be a better way, via PHP, to allow an "end-developer" to call on one main class and create a list "items" based on a subclass that can later be used for methods of the main class?
I really hope this sums it all up well and I havn't gone outside the guidelines of "question asking" here, but I can't think of a better place, other than maybe Code Review to pose such a question and get great developer feed back. If y'all feel I need move it to Code Review, let me know. My main reason for choosing this over CR is this site tends to get faster responses.
Perhaps a view of what I have and what I "might" like to see:
Way it works now
$q = new apiQuote\quote(TRUE);
$q->createNewItem();
$q->item->totalHeight = 100;
$q->item->totalWidth = 250;
...
$q->createNewItem();
$q->item->totalHeight = 300;
$q->item->set_coverage('25%');
...
$q->setItem(1);
$q->item->totalHeight = 250;
...
$q->getTotalsCurv(); // an array to create a graph curve of totals from each item
What I "think" I might like:
$q = new apiQuote\quote(TRUE);
$q->items[] = new apiQuote\item();
$q->items[0]->totalHeight = 100;
$q->items[0]->totalWidth = 250;
...
$q->items[] = new apiQuote\item();
$q->items[1]->totalHeight = 300;
$q->items[1]->set_coverage('25%');
...
$q->items[0]->totalHeight = 250;
...
$q->getTotalsCurv();
However, something like the second idea mean leaving the "items" array public, which could lead to a vast amount of "other" problems as I'm trying to set this whole thing up to be "near dummy proof". Thus the usage of protected variables with specific get/set methods as if they where C# properties.
I think your problem is how to identify an "item" outside of the quote instance. And by using its array index you feel you are going to run into the problems. And you will, when you will try to delete an item. It would successfully invalidate any index already known/stored outside. The simplest patch to it is to give every item a unique ID and store them in the map instead of storing it as a vector.
Also, in your solution item by itself cannot provide you with any helpful information how to access this item in your collection of items (a quote).
public function createNewItem() {
static $counter;
$id = $counter++;
return $this->item = $this->items[$id] = new item($id);
}
public function editItem($id) {
return $this->item = $this->items[$id];
}
public function removeItem($id) {
$this->item = null;
unset($this->item[$id]);
}
Alternatively I recommend you not to reinvent the wheel and take a look here:
http://php.net/manual/en/spl.datastructures.php
and here in specific
http://www.php.net/manual/en/class.splobjectstorage.php
Anything that implements Iterator interface can be iterated with foreach
http://www.php.net/manual/en/class.iterator.php
Ah well you could return the new item from the createItem function.
An equivalent of your .NET example would be to simply return the new item
public function createNewItem() {
$this->items[] = new item();
$this->item = $this->items[count($this->items)-1];
return $this->items[count($this->items)-1];
}
then you could do
$item = $q->createNewItem();
$item->totalHeight = 100;
...
And as you are already adding the new item to the array from within createNewItem so no need for something like $q->add($item);
And to get at any other item you could do a
function getItem($index){
return $this->items[count($this->items)-1];
}
$otheritem = $q->getItem(3);
$otheritem->totalHeight = 100;
I am currently a beginner in CakePHP, and have played around with CakePHP 1.3, but recently CakePHP 2.0 has been released.
So far I like it but the only thing is being a pain is the fact that it doesn't return Objects, rather it just returns arrays. I mean, it hardly makes sense to have to do $post['Post']['id']. It is (in my opinion) much more practical to just do $post->id.
Now after Google I stumbled upon this link, however, this kept generating errors about indexes not being defined when using the Form class (guessing this is because it was getting the objectified version rather than the array version).
I am following the Blog tutorial (already have followed it under 1.3 but going over it again for 2.0)
So, anyone know how to achieve this without it interfering with the Form class?
Hosh
Little known fact: Cake DOES return them as objects, or well properties of an object, anyway. The arrays are the syntactical sugar:
// In your View:
debug($this->viewVars);
Shwoing $this is a View object and the viewVars property corresponds with the $this->set('key', $variable) or $this->set(compact('data', 'for', 'view')) from the controller action.
The problem with squashing them into $Post->id for the sake of keystrokes is Cake is why. Cake is designed to be a heavy lifter, so its built-in ORM is ridiculously powerful, unavoidable, and intended for addressing infinity rows of infinity associated tables - auto callbacks, automatic data passing, query generation, etc. Base depth of multidimensional arrays depends on your find method, as soon as you're working with more than one $Post with multiple associated models (for example), you've introduced arrays into the mix and there's just no avoiding that.
Different find methods return arrays of different depths. From the default generated controller code, you can see that index uses $this->set('posts', $this->paginate()); - view uses $this->set('post', $this->Post->read(null, $id)); and edit doesn't use $this->set with a Post find at all - it assigns $this->data = $this->Post->read(null, $id);.
FWIW, Set::map probably throws those undefined index errors because (guessing) you happen to be trying to map an edit action, amirite? By default, edit actions only use $this->set to set associated model finds to the View. The result of $this->read is sent to $this->data instead. That's probably why Set::map is failing. Either way, you're still going to end up aiming at $Post[0]->id or $Post->id (depending on what you find method you used), which isn't much of an improvement.
Here's some generic examples of Set::map() property depth for these actions:
// In posts/index.ctp
$Post = Set::map($posts);
debug($Post);
debug($Post[0]->id);
// In posts/edit/1
debug($this-viewVars);
debug($this->data);
// In posts/view/1
debug($this-viewVars);
$Post = Set::map($post);
debug($Post->id);
http://api13.cakephp.org/class/controller#method-Controllerset
http://api13.cakephp.org/class/model#method-Modelread
http://api13.cakephp.org/class/model#method-ModelsaveAll
HTH.
You could create additional object vars. This way you wouldn't interfere with Cake's automagic but could access data using a format like $modelNameObj->id; format.
Firstly, create an AppController.php in /app/Controller if you don't already have one. Then create a beforeRender() function. This will look for data in Cake's standard naming conventions, and from it create additional object vars.
<?php
App::uses('Controller', 'Controller');
class AppController extends Controller {
public function beforeRender() {
parent::beforeRender();
// camelcase plural of current model
$plural = lcfirst(Inflector::pluralize($this->modelClass));
// create a new object
if (!empty($this->viewVars[$plural])) {
$objects = Set::map($this->viewVars[$plural]);
$this->set($plural . 'Obj', $objects);
}
// camelcase singular of current model
$singular = lcfirst(Inflector::singularize($this->modelClass));
// create new object
if (!empty($this->viewVars[$singular])) {
$object = Set::map($this->viewVars[$singular]);
$this->set($singular . 'Obj', $object);
}
}
}
Then in your views you can access the objects like so:
index.ctp
$productsObj;
view.ctp
$productObj->id;
All we're doing is adding 'Obj' to the variable names that Cake would already provide. Some example mappings:
Products -> $productsObj
ProductType -> $productTypesObj
I know this is not perfect but it would essentially achieve what you wanted and would be available across all of your models.
While I like the idea Moz proposes there are a number of existing solutions to this problem.
The quickest one I found is https://github.com/kanshin/CakeEntity - but it looks like you might need to refactor it for 2.x - there might even already be a 2.x branch or fork but I didn't look.
I also ran this question couple of time in my head. Now a few Cake based apps later, I see the benefit to be able to branch and merge (am, in_array etc.) result sets more conveniently with arrays than using objects.
The $Post->id form would be a sweet syntactic sugar, but not a real benefit over arrays.
You could write a function that iterates over your public propertys (see ReflectionClass::getProperties) and save it in an array (and return the array).
If you have access to the class, you can implement the ArrayAccess Interface and easily access your object as an array.
P.S.: Sorry, i've never used CakePHP but i think object-to-array conversion doesn't have to be a framework specific problem
I'm looking for a way to prevent repeated calls to the database if the item in question has already been loaded previously. The reason is that we have a lot of different areas that show popular items, latest releases, top rated etc. and sometimes it happens that one item appears in multiple lists on the same page.
I wonder if it's possible to save the object instance in a static array associated with the class and then check if the data is actually in there yet, but then how do I point the new instance to the existing one?
Here's a draft of my idea:
$baseball = new Item($idOfTheBaseballItem);
$baseballAgain = new Item($idOfTheBaseballItem);
class Item
{
static $arrItems = array();
function __construct($id) {
if(in_array($id, self::arrItems)){
// Point this instance to the object in self::arrItems[$id]
// But how?
}
else {
// Call the database
self::arrItems[id] = $this;
}
}
}
If you have any other ideas or you just think I'm totally nuts, let me know.
You should know that static variables only exist in the page they were created, meaning 2 users that load the same page and get served the same script still exist as 2 different memory spaces.
You should consider caching results, take a look at code igniter database caching
What you are trying to achieve is similar to a singleton factory
$baseball = getItem($idOfTheBaseballItem);
$baseballAgain =getItem($idOfTheBaseballItem);
function getItem($id){
static $items=array();
if(!isset($items[$id])$items[$id]=new Item($id);
return $items[$id];
}
class Item{
// this stays the same
}
P.S. Also take a look at memcache. A very simple way to remove database load is to create a /cache/ directory and save database results there for a few minutes or until you deem the data old (this can be done in a number of ways, but most approaches are time based)
You can't directly replace "this" in constructor. Instead, prepare a static function like "getById($id)" that returns object from list.
And as stated above: this will work only per page load.