How to keep memory low when using a hierarchical object structure - php

Question

I have a simple object thing that is able to have children of the same type.
This object has a toHTML method, which does something like:
$html = '<div>' . $this->name . '</div>';
$html .= '<ul>';
foreach($this->children as $child)
$html .= '<li>' . $child->toHTML() . '</li>';
$html .= '</ul>';
return $html;
The problem is that when the object is complex, like lots of children with children with children etc, memory usage skyrockets.
If I simply print_r the multidimensional array that feeds this object I get like 1 MB memory usage, but after I convert the array to my object and do print $root->toHtml() it takes 10 MB !!
How can I fix this?
====================================
Made a simple class that is similar to my real code (but smaller):
class obj{
protected $name;
protected $children = array();
public function __construct($name){
$this->name = $name;
}
public static function build($name, $array = array()){
$obj = new self($name);
if(is_array($array)){
foreach($array as $k => $v)
$obj->addChild(self::build($k, $v));
}
return $obj;
}
public function addChild(self $child){
$this->children[] = $child;
}
public function toHTML(){
$html = '<div>' . $this->name . '</div>';
$html .= '<ul>';
foreach($this->children as $child)
$html .= '<li>' . $child->toHTML() . '</li>';
$html .= '</ul>';
return $html;
}
}
And tests:
$big = array_fill(0, 500, true);
$big[5] = array_fill(0, 200, $big);
print_r($big);
// memory_get_peak_usage() shows 0.61 MB
$root = obj::build('root', $big);
// memory_get_peak_usage() shows 18.5 MB wtf lol
print $root->toHTML();
// memory_get_peak_usage() shows 24.6 MB

Answer 1

The problem is that you're buffering all the data in memory, which you don't actually need to do, as you're just outputting the data, rather than actually processing it.
Rather than buffering everything in memory, if all you want to do is output it you should just output it to wherever it's going to:
public function toHTMLOutput($outputStream){
fwrite($outputStream, '<div>' . $this->name . '</div>';
fwrite($outputStream, '<ul>');
foreach($this->children as $child){
fwrite($outputStream, '<li>');
$child->toHTMLOutput($outputStream);
fwrite($outputStream, '</li>');}
}
fwrite($outputStream, '</ul>');
}
$stdout = fopen('php://stdout', 'w');
print $root->toHTMLOutput($stdout);
or if you want to save the output to a file
$stdout = fopen('htmloutput.html', 'w');
print $root->toHTMLOutput($stdout);
Obviously I've only implemented it for the toHTML() function but the same principle should be done for the build function, which could lead to you skipping a separate toHTML function at all.

Answer 2

Introduction
Since you are sill going to output the HTML there is no need to save it indirectly consuming memory.
Here is a simple class that :
Builds menu from multidimensional array
Memory efficient uses Iterator
Can Write to Socket , Stream , File , array , Iterator etc
Example
$it = new ListBuilder(new RecursiveArrayIterator($big));
// Use Echo
$m = memory_get_peak_usage();
$it->display();
printf("%0.5fMB\n", (memory_get_peak_usage() - $m) / (1024 * 1024));
Output
0.03674MB
Other Output Interfaces
$big = array_fill(0, 500, true);
$big[5] = array_fill(0, 200, $big);
Simple Compare
// Use Echo
$m = memory_get_peak_usage();
$it->display();
$responce['echo'] = sprintf("%0.5fMB\n", (memory_get_peak_usage() - $m) / (1024 * 1024));
// Output to Stream or File eg ( Socket or HTML file)
$m = memory_get_peak_usage();
$it->display(fopen("php://output", "w"));
$responce['stream'] = sprintf("%0.5fMB\n", (memory_get_peak_usage() - $m) / (1024 * 1024));
// Output to ArrayIterator
$m = memory_get_peak_usage();
$it->display($array = new ArrayIterator());
$responce['iterator'] = sprintf("%0.5fMB\n", (memory_get_peak_usage() - $m) / (1024 * 1024));
// Output to Array
$m = memory_get_peak_usage();
$it->display($array = []);
$responce['array'] = sprintf("%0.5fMB\n", (memory_get_peak_usage() - $m) / (1024 * 1024));
echo "\n\nResults \n";
echo json_encode($responce, 128);
Output
Results
{
"echo": "0.03684MB\n",
"stream": "0.00081MB\n",
"iterator": "32.04364MB\n",
"array": "0.00253MB\n"
}
Class Used
class ListBuilder extends RecursiveIteratorIterator {
protected $pad = "\t";
protected $o;
public function beginChildren() {
$this->output("%s<ul>\n", $this->getPad());
}
public function endChildren() {
$this->output("%s</ul>\n", $this->getPad());
}
public function current() {
$this->output("%s<li>%s</li>\n", $this->getPad(1), parent::current());
return parent::current();
}
public function getPad($n = 0) {
return str_repeat($this->pad, $this->getDepth() + $n);
}
function output() {
$args = func_get_args();
$format = array_shift($args);
$var = vsprintf($format, $args);
switch (true) {
case $this->o instanceof ArrayIterator :
$this->o->append($var);
break;
case is_array($this->o) || $this->o instanceof ArrayObject :
$this->o[] = $var;
break;
case is_resource($this->o) && (get_resource_type($this->o) === "file" || get_resource_type($this->o) === "stream") :
fwrite($this->o, $var);
break;
default :
echo $var;
break;
}
}
function display($output = null) {
$this->o = $output;
$this->output("%s<ul>\n", $this->getPad());
foreach($this as $v) {
}
$this->output("%s</ul>\n", $this->getPad());
}
}
Conclusion
As you can see looping with iterator is fast but store values in iterator or object might not be that memory efficient.

Answer 3

Total number of elements in Your array is a little over 100000.
Each element of Your array is just one byte (boolean) so for over 100000 elements it takes 100000bytes ~0.1MB
Each of Your objects is ~100 bytes it is 100*100000 = 100000000 bytes ~ 10MB
But You have ~18MB so where is this 8 from?
If You run this code
<?php
$c = 0; //we use this to count object isntances
class obj{
protected $name;
protected $children = array();
public static $c=0;
public function __construct($name){
global $c;
$c++;
$this->name = $name;
}
public static function build($name, $array = array()){
global $c;
$b = memory_get_usage();
$obj = new self($name);
$diff = memory_get_usage()-$b;
echo $c . ' diff ' . $diff . '<br />'; //display change in allocated size
if(is_array($array)){
foreach($array as $k => $v)
$obj->addChild(self::build($k, $v));
}
return $obj;
}
public function addChild(self $child){
$this->children[] = $child;
}
public function toHTML(){
$html = '<div>' . $this->name . '</div>';
$html .= '<ul>';
foreach($this->children as $child)
$html .= '<li>' . $child->toHTML() . '</li>';
$html .= '</ul>';
return $html;
}
}
$big = array_fill(0, 500, true);
$big[5] = array_fill(0, 200, $big);
$root = obj::build('root', $big);
You will notice a change is constant with exception for objects created as
1024th, 2048th, 4096th...
I don't have link to any article or manual page about it but my guess is that php hold references to each created object in array with initial size of 1024. When You make this array full its size will get doubled to make space for new objects.
If You take difference from for example 2048th object subtract a size of object( the constant value You have in other lines) and divide by 2048 You will always get 32 - standard size of pointer in C.
So for 100000 objects this array grown to size of 131072 elements.
131072*32 = 4194304B = 4MB
This calculation are just approximate but I think it answers Your question what takes so much memory.
To answer how to keep memory low - avoid using objects for large set of data.
Obviously objects are nice and stuff but primitive data types are faster and smaller.
Maybe You can make it work with one object containing array with data. Hard to propose any alternative without more info about this objects and what methods/interface they require.

Answer 4

One thing that might be catching you is that you might be getting close to blowing your stack because of recursion. It might make sense in this case to create a rendering function that deals with the tree as a whole to render instead of relying on recursion to do the rendering for you. For informative topics on this see tail call recursion and tail call optimization.
To stick with your code's current structure and dodge a lot of the resource problems that you are likely facing the simplest solution may be to simply pass in the html string as a reference like:
class obj{
protected $name;
protected $children = array();
public function __construct($name){
$this->name = $name;
}
public static function build($name, $array = array()){
$obj = new self($name);
if(is_array($array)){
foreach($array as $k => $v)
$obj->addChild(self::build($k, $v));
}
return $obj;
}
public function addChild(self $child){
$this->children[] = $child;
}
public function toHTML(&$html = ""){
$html .= '<div>' . $this->name . '</div>';
$html .= '<ul>';
foreach($this->children as $child){
$html .= '<li>';
$html .= $child->toHTML($html);
$html .= '</li>';
}
$html .= '</ul>';
}
}
This will keep you from hauling around a bunch of duplicate partial tree renders while the recursive calls are resolving.
As for the actual build of the tree I think a lot of the memory usage is just the price of playing with data that big, your options there are either render instead of building up a hierarchical model just to render (just render output instead of building a tree) or, to employ some sort of caching strategies to either cache copies of the object tree or copies of the rendered html depending on how the data is used within your site. If you have control of the inbound data invalidating relevant cache keys can be added to that work flow to keep the cache from getting stale.