Yii NestedSetBehavior Memory usage - php

I use NestedSetBehavior model extension in my project for using db table as a tree.
I wroute example:
$model = SiteMap()->findAll();
$log .= $this->debug_memory('used')."<br>";
$ancestors = null;
foreach ($model as $item) {
$ancestors = $item->ancestors()->findAll();
}
$log .= $this->debug_memory('used')."<br>";
echo $log;
(debug_memory source just return friendly memory_get_usage(), $model has 50 items)
The result is:
used: 10525440
used: 15892712
After simple calculation - memory usage increased on 5,24 Mb.
But i must use $item->ancestors()->findAll(); many times in cycle, so my memory increased on 138 Mb. And i get #out of memory error".
I try use unset():
$model = SiteMap()->findAll();
$log .= $this->debug_memory('used')."<br>";
$ancestors = null;
foreach ($model as $item) {
$ancestors= $item->ancestors()->findAll();
}
$ancestors = null;
unset($ancestors);
$log .= $this->debug_memory('used')."<br>";
echo $log;
But i steel get result:
used: 10525984
used: 15893320
Behavior ancestors function source is:
public function ancestors($depth=null)
{
$owner=$this->getOwner();
$db=$owner->getDbConnection();
$criteria=$owner->getDbCriteria();
$alias=$db->quoteColumnName($owner->getTableAlias());
$criteria->mergeWith(array(
'condition'=>$alias.'.'.$db->quoteColumnName($this->leftAttribute).'<'.$owner->{$this->leftAttribute}.
' AND '.$alias.'.'.$db->quoteColumnName($this->rightAttribute).'>'.$owner->{$this->rightAttribute},
'order'=>$alias.'.'.$db->quoteColumnName($this->leftAttribute),
));
if($depth!==null)
$criteria->addCondition($alias.'.'.$db->quoteColumnName($this->levelAttribute).'>='.($owner->{$this->levelAttribute}-$depth));
if($this->hasManyRoots)
{
$criteria->addCondition($alias.'.'.$db->quoteColumnName($this->rootAttribute).'='.CDbCriteria::PARAM_PREFIX.CDbCriteria::$paramCount);
$criteria->params[CDbCriteria::PARAM_PREFIX.CDbCriteria::$paramCount++]=$owner->{$this->rootAttribute};
}
return $owner;
}
So, my questions is, why this function use so many memory and why when i unset variable memory is not cleaning?

Comment out the logging in your protected/config/main.php file. (Or wherever you define your configuration settings.)
What you are seeing is likely a result of all the logs being written on each Active Record call, which would explain why unsetting the object doesn't release memory: the memory used isn't in the model, it's in the log.
Try that and report results.

Answer
It's always good to disable the logs (as #willem-renzema already stated) to prevent it from using memory (especially when you're testing for leaks).
However you should call detachBehaviors() on each ancestor in your example code to stop it from leaking memory. It's good conduct to do this for every object instance with attached behaviors before using unset on it to prevent memory leaks from happening. Even when you're not explicitly "unsetting" the object (eg. your variables are moving out of scope).
See https://github.com/yiiext/nested-set-behavior/issues/25 for more info about this issue in NestedSetBehavior.
Also see https://github.com/yiisoft/yii/issues/1329#issuecomment-18729026 for a more general discussion on memory leaks and circular references in PHP and Yii.
Note
You could be tempted to use the a destructor (__destruct()) to handle the detaching of behaviors for you. But in this specific case that won't work because of the destructor rule mentioned here.
"Objects will only free their resources and trigger their
__destruct() method when all references are unset. Even when they
are in the object... sigh!" - (nox at oreigon dot de, 2009).
This rule won't be satisfied by just using unset on an "ancestor" because the NestedSetBehavior still has a reference to the "ancestor" instance stored in a static array called: $_cached. This reference will only be cleared by destroying the NestedSetBehavior instance itself, which will happen of you call detachBehaviors() on the "owner" object of the behavior.

Related

Testing method with no output

I have the following method I want to test:
class SomeObject {
public function actionFromSomeController() {
$obj = new OtherObject();
$obj -> setAttributes();
$obj -> doAction();
}
}
class OtherObject {
private $_attr;
public function setAttributes() {
$this -> _attr = 'something';
Database :: execute('INSERT INTO table VALUES (' . $this -> _attr . ')');
$fileObj = new FileObj();
$content = $fileObj -> getSomeFileContent();
// do something else
}
public function doAction() {
echo $this -> _attr;
}
}
Now I want to test this method, its output depends on database content and one file on the server. It does a lot of things on the way, and the output is just one ID and success => 1.
How should I test it properly?
Some ideas on how to test small code pieces like this:
Generate test-data and pass it to your methods (also, fake database return data or file contents)
Use echo / var_dump() / die() to check property and variable content at different positions in your methods
Also use these commands to check whether execution reaches a certain point (for example to see whether a function got called or not)
If something doesn't work as expected without an error message: Check line by line with the above methods until you find the problem
Consider using interfaces and dependency injection if your code gets bigger - this is a bit over-the-top for this amount of code, but can be a tremendous time-saver when your application becomes big
Testing is never an automatic process and you will always have to think about what makes sense to do and what not. These things to do are never magic but basic PHP.
You should consider letting your scripts throw errors/exceptions if something goes wrong. Writing "silent" applications is almost never good since you can, if you really need a silent execution for production environments, just turn off error reporting and have the same effect. Many PHP functions return something special on failure and/or success and you can check for this. Database handlers do so, too. Do yourself a favor and use these return values!

Memory usage goes wild with Doctrine bulk insert

I’m trying to insert a large amount of data (30 000+ lines) in a MySQL database using Doctrine2 and the Symfony2 fixture bundle.
I looked at the right way to do it. I saw lots of questions about memory leaks and Doctrine, but no satisfying answer for me. It often comes the Doctrine clear() function.
So, I did various shapes of this:
while (($data = getData()) {
$iteration++;
$obj = new EntityObject();
$obj->setName('henry');
// Fill object...
$manager->persist($obj);
if ($iteration % 500 == 0) {
$manager->flush();
$manager->clear();
// Also tried some sort of:
// $manager->clear($obj);
// $manager->detach($obj);
// gc_collect_cycles();
}
}
PHP memory still goes wild, right after the flush() (I’m sure of that). In fact, every time the entities are flushed, memory goes up for a certain amount depending on batch size and the entities, until it reaches the deadly Allowed Memory size exhausted error. With a very very tiny entity, it works but memory consumption increase too much: several MB whereas it should be KB.
clear(), detach() or calling GC doesn’t seem to have an effect at all. It only clears some KB.
Is my approach flawed? Did I miss something, somewhere? Is it a bug?
More info:
Without flush() memory barely moves;
Lowering the batch do not change the outcome;
Data comes from a CSV that need to be sanitized;
EDIT (partial solution):
#qooplmao brought a solution that significantly decrease memory consumption, disable doctrine sql logger: $manager->getConnection()->getConfiguration()->setSQLLogger(null);
However, it is still abnormally high and increasing.
I resolved my problem using this resource, as #Axalix suggested.
This is how I modified the code:
// IMPORTANT - Disable the Doctrine SQL Logger
$manager->getConnection()->getConfiguration()->setSQLLogger(null);
// SUGGESION - make getData as a generator (using yield) to to save more memory.
while ($data = getData()) {
$iteration++;
$obj = new EntityObject();
$obj->setName('henry');
// Fill object...
$manager->persist($obj);
// IMPORTANT - Temporary store entities (of course, must be defined first outside of the loop)
$tempObjets[] = $obj;
if ($iteration % 500 == 0) {
$manager->flush();
// IMPORTANT - clean entities
foreach($tempObjets as $tempObject) {
$manager->detach($tempObject);
}
$tempObjets = null;
gc_enable();
gc_collect_cycles();
}
}
// Do not forget the last flush
$manager->flush();
And, last but not least, as I use this script with Symfony data fixtures, adding the --no-debug parameter in the command is also very important. Then memory consumption is stable.
I found out that Doctrine logs all SQLs during execute. I recommend to disable it with code below, it can really save memory:
use Doctrine\ORM\EntityManagerInterface;
public function __construct(EntityManagerInterface $entity_manager)
{
$em_connection = $entity_manager->getConnection();
$em_connection->getConfiguration()->setSQLLogger(null);
}
My suggestion is to drop the Doctrine approach for bulk inserts. I really like Doctrine but I just hate this kind of stuff on bulk inserts.
MySQL has a great thing called LOAD DATA. I would rather use it or even if I have to sanitize my csv first and do the LOAD after.
If you need to change the values, I would read csv to array $csvData = array_map("str_getcsv", file($csv));. Change whatever you need on the array and save it to the line. After that, use the new .csv to LOAD with MySQL.
To support my claims on why I wouldn't use Doctrine for this here described on the top.

PHP variable loses its value

I have a really serious problem that I have not seen before.
On a website we are using opensource SQC eshop, PHP Version 5.3.3-7+squeeze15 and there is some kind of problem with variable memory I think.
SQC uses notORM and here the problem starts with fatal error "Call to function on non object notORMResult" .
So I dug deeper and found the constructor of NotORM that looks like this:
function __construct(PDO $connection, NotORM_Structure $structure = null,NotORM_Cache $cache = null) {
$this->connection = $connection;
if($_GET['test']){
var_dump($structure);
}
if (!isset($structure)) {
$structure = new NotORM_Structure_Convention;
}
if($_GET['test']){
var_dump($structure);
}
$this->structure = $structure;
if($_GET['test']){
var_dump($this->structure);
exit("1");
}
$this->cache = $cache;
}
And so the output is NULL because the constructor gets no structure param so we create an object. Second output is the object. Then we set the object to attribute and then the THIRD OUTPUT IS NULL
How is this even possible? The site was running for about year and half and no problems till yesterday. I didn't made yet any updates to php and this thing really freaks me out 'cause it's not a constant problem. It just happens sometimes after 2 hours, sometimes after 2 mins and I have really no idea why is this happening.
And btw ... this is just the start it happens across the whole script. Object attributes are set but when you want to read them they give you NULL. There is also second website running on the same server, same php same configuration without problem.
Thanks for any ideas :)

PHP - Releasing resultset

I've been looking online and every article I've come across seems to say something slightly different. If I'm working with a resultset and the program exits before releasing the results what exactly happens?
Is there a better way to ensure that it always happens? The same goes for database connections.
if($statement->fetch()) {
exit("Result!");
}
$statement->free_result();
The free_result tells the database engine it can release the result set.
It only needs to be called if you are concerned about how much memory is being used for queries that return large result sets. All associated result memory is automatically freed at the end of the script's execution.
You can use register_shutdown_function to make sure your results are freed from memory
function shutdown(){
global $statement;
if($statement){
$statement->free_result();
}
}
register_shutdown_function('shutdown');
You can use register_shutdown_function in PHP to make sure that a function / a lot of functions fire on exit. In your example, you could also change the if statement to the following:
$exitResult = null;
if($statement->fetch()) {
$exitResult = "Result!";
}
$statement->free_result();
if ($exitResult !== null) exit($exitResult);
It should be noted though, that most plugins/extensions (that connects to databases etc.) clean up after themself after shutdown of the PHP process (on the end of a script).
If you use this in a class, you can save the reference to $statement and then use a __destruct function in the class.

Php memory leak question

I'm having a terrible amount of problems with an XML parsing script leaking some memory in PHP.
I've made a solution by rewriteing my whole OOP code to non OOP, which was mostly database checks and inserts, and that seemed to plug the hole, but I'm curious as to what caused it? I'm using Zend Framework and once I removed all of the model stuff, there are no leaks.
Just to give you and idea how bad it was:
I'm running through some 30k items on the same number of files. So, one per file. It started out by using 5mb!!! or RAM, when the file itself was only about 20kb big.
Could it be those referencing functions that I've read about? I thought that that bug was fixed?!
EDIT
I found out, that the leak was due to using Zend Framework database classes. Is there a way to call a shutdown function after each iteration, so that it would clear the resources?
Its pretty dificult to answer this as we have no code to work with.
Revert back to the OOP version of your sources and create a small class like so:
abstract class MemoryLeakLogger
{
public static $_logs = array();
public function Start($id,$action)
{
self::$_logs[$id] = array(
'action' => $action,
'start_ts' => microtime(),
'memory_start' => memory_get_usage()
);
}
public function End($id)
{
self::$_logs[$id]['end_ts'] = microtime();
self::$_logs[$id]['memory_end'] = memory_get_usage();
}
public static function GetInformation(){return self::$_logs;}
}
and then within your application do the following:
MemoryLeakLogger::Start(":xml_parse_links_set_2", "parsing set to of links");
/*
* Here you would do the relative code
*/
MemoryLeakLogger::End(":xml_parse_links_set_2");
And so forth throughout your application, you will need to create calculations to gather the offsets for memory usages and time taken per action, once your script is completed just debug the information by printing it in a readable fashion and look for peaks
You can also use xdebug to trace your application.
Hope this helps

Categories