I've been looking online and every article I've come across seems to say something slightly different. If I'm working with a resultset and the program exits before releasing the results what exactly happens?
Is there a better way to ensure that it always happens? The same goes for database connections.
if($statement->fetch()) {
exit("Result!");
}
$statement->free_result();
The free_result tells the database engine it can release the result set.
It only needs to be called if you are concerned about how much memory is being used for queries that return large result sets. All associated result memory is automatically freed at the end of the script's execution.
You can use register_shutdown_function to make sure your results are freed from memory
function shutdown(){
global $statement;
if($statement){
$statement->free_result();
}
}
register_shutdown_function('shutdown');
You can use register_shutdown_function in PHP to make sure that a function / a lot of functions fire on exit. In your example, you could also change the if statement to the following:
$exitResult = null;
if($statement->fetch()) {
$exitResult = "Result!";
}
$statement->free_result();
if ($exitResult !== null) exit($exitResult);
It should be noted though, that most plugins/extensions (that connects to databases etc.) clean up after themself after shutdown of the PHP process (on the end of a script).
If you use this in a class, you can save the reference to $statement and then use a __destruct function in the class.
Related
I have the following method I want to test:
class SomeObject {
public function actionFromSomeController() {
$obj = new OtherObject();
$obj -> setAttributes();
$obj -> doAction();
}
}
class OtherObject {
private $_attr;
public function setAttributes() {
$this -> _attr = 'something';
Database :: execute('INSERT INTO table VALUES (' . $this -> _attr . ')');
$fileObj = new FileObj();
$content = $fileObj -> getSomeFileContent();
// do something else
}
public function doAction() {
echo $this -> _attr;
}
}
Now I want to test this method, its output depends on database content and one file on the server. It does a lot of things on the way, and the output is just one ID and success => 1.
How should I test it properly?
Some ideas on how to test small code pieces like this:
Generate test-data and pass it to your methods (also, fake database return data or file contents)
Use echo / var_dump() / die() to check property and variable content at different positions in your methods
Also use these commands to check whether execution reaches a certain point (for example to see whether a function got called or not)
If something doesn't work as expected without an error message: Check line by line with the above methods until you find the problem
Consider using interfaces and dependency injection if your code gets bigger - this is a bit over-the-top for this amount of code, but can be a tremendous time-saver when your application becomes big
Testing is never an automatic process and you will always have to think about what makes sense to do and what not. These things to do are never magic but basic PHP.
You should consider letting your scripts throw errors/exceptions if something goes wrong. Writing "silent" applications is almost never good since you can, if you really need a silent execution for production environments, just turn off error reporting and have the same effect. Many PHP functions return something special on failure and/or success and you can check for this. Database handlers do so, too. Do yourself a favor and use these return values!
I’m trying to insert a large amount of data (30 000+ lines) in a MySQL database using Doctrine2 and the Symfony2 fixture bundle.
I looked at the right way to do it. I saw lots of questions about memory leaks and Doctrine, but no satisfying answer for me. It often comes the Doctrine clear() function.
So, I did various shapes of this:
while (($data = getData()) {
$iteration++;
$obj = new EntityObject();
$obj->setName('henry');
// Fill object...
$manager->persist($obj);
if ($iteration % 500 == 0) {
$manager->flush();
$manager->clear();
// Also tried some sort of:
// $manager->clear($obj);
// $manager->detach($obj);
// gc_collect_cycles();
}
}
PHP memory still goes wild, right after the flush() (I’m sure of that). In fact, every time the entities are flushed, memory goes up for a certain amount depending on batch size and the entities, until it reaches the deadly Allowed Memory size exhausted error. With a very very tiny entity, it works but memory consumption increase too much: several MB whereas it should be KB.
clear(), detach() or calling GC doesn’t seem to have an effect at all. It only clears some KB.
Is my approach flawed? Did I miss something, somewhere? Is it a bug?
More info:
Without flush() memory barely moves;
Lowering the batch do not change the outcome;
Data comes from a CSV that need to be sanitized;
EDIT (partial solution):
#qooplmao brought a solution that significantly decrease memory consumption, disable doctrine sql logger: $manager->getConnection()->getConfiguration()->setSQLLogger(null);
However, it is still abnormally high and increasing.
I resolved my problem using this resource, as #Axalix suggested.
This is how I modified the code:
// IMPORTANT - Disable the Doctrine SQL Logger
$manager->getConnection()->getConfiguration()->setSQLLogger(null);
// SUGGESION - make getData as a generator (using yield) to to save more memory.
while ($data = getData()) {
$iteration++;
$obj = new EntityObject();
$obj->setName('henry');
// Fill object...
$manager->persist($obj);
// IMPORTANT - Temporary store entities (of course, must be defined first outside of the loop)
$tempObjets[] = $obj;
if ($iteration % 500 == 0) {
$manager->flush();
// IMPORTANT - clean entities
foreach($tempObjets as $tempObject) {
$manager->detach($tempObject);
}
$tempObjets = null;
gc_enable();
gc_collect_cycles();
}
}
// Do not forget the last flush
$manager->flush();
And, last but not least, as I use this script with Symfony data fixtures, adding the --no-debug parameter in the command is also very important. Then memory consumption is stable.
I found out that Doctrine logs all SQLs during execute. I recommend to disable it with code below, it can really save memory:
use Doctrine\ORM\EntityManagerInterface;
public function __construct(EntityManagerInterface $entity_manager)
{
$em_connection = $entity_manager->getConnection();
$em_connection->getConfiguration()->setSQLLogger(null);
}
My suggestion is to drop the Doctrine approach for bulk inserts. I really like Doctrine but I just hate this kind of stuff on bulk inserts.
MySQL has a great thing called LOAD DATA. I would rather use it or even if I have to sanitize my csv first and do the LOAD after.
If you need to change the values, I would read csv to array $csvData = array_map("str_getcsv", file($csv));. Change whatever you need on the array and save it to the line. After that, use the new .csv to LOAD with MySQL.
To support my claims on why I wouldn't use Doctrine for this here described on the top.
I am doing project using mongodb and zendframework 2 so here I create connection in the constructor
private $conn;
public function __construct(){
$this->conn = new \MongoClient('mongodb://example.com:27017', array("connect" => TRUE));
}
It contain several actions to perform database operations like createdb, dropdb, renamedb like wise. so I close that connection within the __distruct() method
public function __destruct(){
$this->conn->close();
}
my code works fine. but I would like to know is this ok?
PHP closes database connections automatically.
You can read more about this in this post
however
read this
quote:
"Open connections (and similar resources) are automatically destroyed at the end of script execution. However, you should still close or free all connections, result sets and statement handles as soon as they are no longer required. This will help return resources to PHP and MySQL faster."
So basically, in case you have tons of things going in and out of the database, then yes, you might want to close right after.
In my opinion it is rather optional and your own choice wether to kill the process or not.
I use NestedSetBehavior model extension in my project for using db table as a tree.
I wroute example:
$model = SiteMap()->findAll();
$log .= $this->debug_memory('used')."<br>";
$ancestors = null;
foreach ($model as $item) {
$ancestors = $item->ancestors()->findAll();
}
$log .= $this->debug_memory('used')."<br>";
echo $log;
(debug_memory source just return friendly memory_get_usage(), $model has 50 items)
The result is:
used: 10525440
used: 15892712
After simple calculation - memory usage increased on 5,24 Mb.
But i must use $item->ancestors()->findAll(); many times in cycle, so my memory increased on 138 Mb. And i get #out of memory error".
I try use unset():
$model = SiteMap()->findAll();
$log .= $this->debug_memory('used')."<br>";
$ancestors = null;
foreach ($model as $item) {
$ancestors= $item->ancestors()->findAll();
}
$ancestors = null;
unset($ancestors);
$log .= $this->debug_memory('used')."<br>";
echo $log;
But i steel get result:
used: 10525984
used: 15893320
Behavior ancestors function source is:
public function ancestors($depth=null)
{
$owner=$this->getOwner();
$db=$owner->getDbConnection();
$criteria=$owner->getDbCriteria();
$alias=$db->quoteColumnName($owner->getTableAlias());
$criteria->mergeWith(array(
'condition'=>$alias.'.'.$db->quoteColumnName($this->leftAttribute).'<'.$owner->{$this->leftAttribute}.
' AND '.$alias.'.'.$db->quoteColumnName($this->rightAttribute).'>'.$owner->{$this->rightAttribute},
'order'=>$alias.'.'.$db->quoteColumnName($this->leftAttribute),
));
if($depth!==null)
$criteria->addCondition($alias.'.'.$db->quoteColumnName($this->levelAttribute).'>='.($owner->{$this->levelAttribute}-$depth));
if($this->hasManyRoots)
{
$criteria->addCondition($alias.'.'.$db->quoteColumnName($this->rootAttribute).'='.CDbCriteria::PARAM_PREFIX.CDbCriteria::$paramCount);
$criteria->params[CDbCriteria::PARAM_PREFIX.CDbCriteria::$paramCount++]=$owner->{$this->rootAttribute};
}
return $owner;
}
So, my questions is, why this function use so many memory and why when i unset variable memory is not cleaning?
Comment out the logging in your protected/config/main.php file. (Or wherever you define your configuration settings.)
What you are seeing is likely a result of all the logs being written on each Active Record call, which would explain why unsetting the object doesn't release memory: the memory used isn't in the model, it's in the log.
Try that and report results.
Answer
It's always good to disable the logs (as #willem-renzema already stated) to prevent it from using memory (especially when you're testing for leaks).
However you should call detachBehaviors() on each ancestor in your example code to stop it from leaking memory. It's good conduct to do this for every object instance with attached behaviors before using unset on it to prevent memory leaks from happening. Even when you're not explicitly "unsetting" the object (eg. your variables are moving out of scope).
See https://github.com/yiiext/nested-set-behavior/issues/25 for more info about this issue in NestedSetBehavior.
Also see https://github.com/yiisoft/yii/issues/1329#issuecomment-18729026 for a more general discussion on memory leaks and circular references in PHP and Yii.
Note
You could be tempted to use the a destructor (__destruct()) to handle the detaching of behaviors for you. But in this specific case that won't work because of the destructor rule mentioned here.
"Objects will only free their resources and trigger their
__destruct() method when all references are unset. Even when they
are in the object... sigh!" - (nox at oreigon dot de, 2009).
This rule won't be satisfied by just using unset on an "ancestor" because the NestedSetBehavior still has a reference to the "ancestor" instance stored in a static array called: $_cached. This reference will only be cleared by destroying the NestedSetBehavior instance itself, which will happen of you call detachBehaviors() on the "owner" object of the behavior.
I'm having problems with a batch insertion of objects into a database using symfony 1.4 and doctrine 1.2.
My model has a certain kind of object called "Sector", each of which has several objects of type "Cupo" (usually ranging from 50 up to 200000). These objects are pretty small; just a short identifier string and one or two integers. Whenever a group of Sectors are created by the user, I need to automatically add all these instances of "Cupo" to the database. In case anything goes wrong, I'm using a doctrine transaction to roll back everything. The problem is that I can only create around 2000 instances before php runs out of memory. It currently has a 128MB limit, which should be more than enough for handling objects that use less than 100 bytes. I've tried increasing the memory limit up to 512MB, but php still crashes and that doesn't solve the problem. Am I doing the batch insertion correctly or is there a better way?
Here's the error:
Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 71 bytes) in /Users/yo/Sites/grifoo/lib/vendor/symfony/lib/log/sfVarLogger.class.php on line 170
And here's the code:
public function save($conn=null){
$conn=$conn?$conn:Doctrine_Manager::connection();
$conn->beginTransaction();
try {
$evento=$this->object;
foreach($evento->getSectores() as $s){
for($j=0;$j<$s->getCapacity();$j++){
$cupo=new Cupo();
$cupo->setActivo($s->getActivo());
$cupo->setEventoId($s->getEventoId());
$cupo->setNombre($j);
$cupo->setSector($s);
$cupo->save();
}
}
$conn->commit();
return;
}
catch (Exception $e) {
$conn->rollback();
throw $e;
}
Once again, this code works fine for less than 1000 objects, but anything bigger than 1500 fails. Thanks for the help.
Tried doing
$cupo->save();
$cupo->free();
$cupo = null;
(But substituting my code) And I'm still getting memory overflows. Any other ideas, SO?
Update:
I created a new environment in my databases.yml, that looks like:
all:
doctrine:
class: sfDoctrineDatabase
param:
dsn: 'mysql:host=localhost;dbname=.......'
username: .....
password: .....
profiler: false
The profiler: false entry disables doctrine's query logging, that normally keeps a copy of every query you make. It didn't stop the memory leakage, but I was able to get about twice as far through my data importing as I was without it.
Update 2
I added
Doctrine_Manager::connection()->setAttribute(Doctrine_Core::ATTR_AUTO_FREE_QUERY_OBJECTS, true );
before running my queries, and changed
$cupo = null;
to
unset($cupo);
And now my script has been churning away happily. I'm pretty sure it will finish without running out of RAM this time.
Update 3
Yup. That's the winning combo.
I have just did "daemonized" script with symfony 1.4 and setting the following stopped the memory hogging:
sfConfig::set('sf_debug', false);
For a symfony task, I also faced to this issue and done following things. It worked for me.
Disable debug mode. Add following before db connection initialize
sfConfig::set('sf_debug', false);
Set auto query object free attribute for db connection
$connection->setAttribute(Doctrine_Core::ATTR_AUTO_FREE_QUERY_OBJECTS, true );
Free all object after use
$object_name->free()
Unset all arrays after use unset($array_name)
Check all doctrine queries used on task. Free all queries after use. $q->free()
(This is a good practice for any time of query using.)
That's all. Hope it may help someone.
Doctrine leaks and there's not much you can do about it. Make sure you use $q->free() whenever applicable to minimize the effect.
Doctrine is not meant for maintenance scripts. The only way to work around this problem is to break you script to parts which will perform part of the task. One way to do that is to add a start parameter to your script and after a certain amount of objects had been processed, the script redirects to itself with a higher start value. This works well for me although it makes writing maintenance scripts more cumbersome.
Try to unset($cupo); after every saving. This should be help. An other thing is to split the script and do some batch processing.
Try to break circular reference which usually cause memory leaks with
$cupo->save();
$cupo->free(); //this call
as described in Doctrine manual.
For me , I've just initialized the task like that:
// initialize the database connection
$databaseManager = new sfDatabaseManager($this->configuration);
$connection = $databaseManager->getDatabase($options['connection'])->getConnection();
$config = ProjectConfiguration::getApplicationConfiguration('frontend', 'prod', true);
sfContext::createInstance($config);
(WITH PROD CONFIG)
and use free() after a save() on doctrine's object
the memory is stable at 25Mo
memory_get_usage=26.884071350098Mo
with php 5.3 on debian squeeze
Periodically close and re-open the connection - not sure why but it seems PDO is retaining references.
What is working for me is calling the free method like this:
$cupo->save();
$cupo->free(true); // free also the related components
unset($cupo);