How to handle nested objects in processing a JSON stream - php

Question

I am working on a program where we need to process very large JSON file, so I would like to use a streaming event oriented reader (like jsonstreamingparser) so that we can avoid loading the entire structure into memory at one time. Something I'm concerned about though is the object structure that seems to be required to make this work.
For example, say I'm writing a program like Evite to send out invitations to an activity, with a JSON structure like:
{
"title": "U2 Concert",
"location": "San Jose",
"attendees": [
{"email": "foo#bar.com"},
{"email": "baz#bar.com"}
],
"date": "July 4, 2015"
}
What I would like to do is have a programming "event" that when the stream encounters a new attendee, sends out an invite email. But, I can't do that because the stream has not yet reached the date of the event.
Of course, given the example, it's fine to just read everything into memory - but my dataset has complex objects where the "attendees" attribute are, and there can be tens of thousands of them.
Another "solution" is to just mandate: you HAVE to put all the required "parent" attributes first, but that is what I'm trying to find a way around.
Any ideas?

Answer 1

This is another 'tree walking' problem. The JSON streaming parser reads the source file and starts to build the 'tree'. It does this by collecting 'elements' and storing them in memory. To enable us to process each entry, it 'emits events' at convenient times. Which means it will call your functions passing useful values as required.
Examples of 'tree events' are:
start_object()
end_object()
start_array()
end_array()
...
The 'example' code provided with the 'Parser' is a program that uses the Parser to build the tree in memory. I just modified that example to call our function whenever it has a 'complete Event' stored.
So, how do we identify a 'complete Event'?
The input file consist of an array where each entry is a JSON 'obbject'. Each object consists of 'sub entries' that make up the data of the 'object'.
Now, as we traverse the 'tree' building it, our code will be called at various points as shown above. Specifically when 'starting' and 'ending' of objects and arrays. We need to collect all the data for the 'outer object'.
How do we identify this? We record where we are in the 'tree' as the processing proceeds. This we do by keeping track of the depth of 'nesting' in the tree. Hence the 'levels'. The 'start' of an object 'nests' down one level, the 'end' of an object 'unnests' one level.
The objects we are interested in are a 'level 1'.
The code provided:
1) keeps track of the 'levels' and calls our function when it reaches the end of an object that is at 'level 1'.
2) Accumulates the data in the appropriate structure from the start of the the object at 'level 1'.
Requirements:
1) Call a 'callable' routine when there is a 'complete Event' that can be processed.
Assumptions:
The input file consists of an array of 'Events'.
Processing:
Parse the file
Whenever the current Event is 'complete'
Execute the 'processEvent' callable with access to the current Event.
Source Code:
Source: class Q31079129Listener at Pastebin.com
Source: index.php file at Pastebin.com
Source: test datafile : Q31079129.json at Pastebin.com
Demonstration using the code
Code: index.php
<?php // https://stackoverflow.com/questions/31079129/how-to-handle-nested-objects-in-processing-a-json-stream
require_once __DIR__ .'/vendor/jsonstreamingparser/src/JsonStreamingParser/Parser.php';
require_once __DIR__ .'/vendor/jsonstreamingparser/src/JsonStreamingParser/Listener/IdleListener.php';
require_once __DIR__ .'/Q31079129Listener.php';
/**
* The input file consists of a JSON array of 'Events'.
*
* The important point is that when the file is being 'parsed' the 'listener' is
* 'walking' the tree.
*
* Therefore
* 1) Each 'Event' is at 'level 1' in the tree.
*
* Event Level Changes:
* Start: level will go from 1 => 2
* End: level will go from 2 => 1 !!!!
*
* Actions:
* The 'processEvent' function will be called when the
* 'Event Level' changes to 2 from 1.
*
*/
define('JSON_FILE', __DIR__. '/Q31079129.json');
/**
* This is called when one 'Event' is complete
*
* #param type $listener
*/
function processEvent($listener) {
echo '<pre>', '+++++++++++++++';
print_r($listener->get_event());
echo '</pre>';
}
// ----------------------------------------------------------------------
// the 'Listener'
$listener = new Q31079129Listener();
// setup the 'Event' Listener that will be called with each complete 'Event'
$listener->whenLevelAction = 'processEvent';
// process the input stream
$stream = fopen(JSON_FILE, 'r');
try {
$parser = new JsonStreamingParser_Parser($stream, $listener);
$parser->parse();
}
catch (Exception $e) {
fclose($stream);
throw $e;
}
fclose($stream);
exit;
Code: Q31079129Listener.php
<?php // // https://stackoverflow.com/questions/31079129/how-to-handle-nested-objects-in-processing-a-json-stream
/**
* This is the supplied example modified:
*
* 1) Record the current 'depth' of 'nesting' in the current object being parsed.
*/
class Q31079129Listener extends JsonStreamingParser\Listener\IdleListener {
public $whenLevelAction = null;
protected $event;
protected $prevLevel;
protected $level;
private $_stack;
private $_keys;
public function get_event() {
return $this->event;
}
public function get_prevLevel() {
return $this->prevLevel;
}
public function get_level() {
return $this->prevLevel;
}
public function start_document() {
$this->prevLevel = 0;
$this->level = 0;
$this->_stack = array();
$this->_keys = array();
// echo '<br />start of document';
}
public function end_document() {
// echo '<br />end of document';
}
public function start_object() {
$this->prevLevel = $this->level;
$this->level++;
$this->_start_complex_value('object');
}
public function end_object() {
$this->prevLevel = $this->level;
$this->level--;
$this->_end_complex_value();
}
public function start_array() {
$this->prevLevel = $this->level;
$this->level++;
$this->_start_complex_value('array');
}
public function end_array() {
$this->prevLevel = $this->level;
$this->level--;
$this->_end_complex_value();
}
public function key($key) {
$this->_keys[] = $key;
}
public function value($value) {
$this->_insert_value($value);
}
private function _start_complex_value($type) {
// We keep a stack of complex values (i.e. arrays and objects) as we build them,
// tagged with the type that they are so we know how to add new values.
$current_item = array('type' => $type, 'value' => array());
$this->_stack[] = $current_item;
}
private function _end_complex_value() {
$obj = array_pop($this->_stack);
// If the value stack is now at level 1 from level 2,
// we're done parsing the current complete event, so we can
// move the result into place so that get_event() can return it. Otherwise, we
// associate the value
// var_dump(__FILE__.__LINE__, $this->prevLevel, $this->level, $obj);
if ($this->prevLevel == 2 && $this->level == 1) {
if (!is_null($this->whenLevelAction)) {
$this->event = $obj['value'];
call_user_func($this->whenLevelAction, $this);
$this->event = null;
}
}
else {
$this->_insert_value($obj['value']);
}
}
// Inserts the given value into the top value on the stack in the appropriate way,
// based on whether that value is an array or an object.
private function _insert_value($value) {
// Grab the top item from the stack that we're currently parsing.
$current_item = array_pop($this->_stack);
// Examine the current item, and then:
// - if it's an object, associate the newly-parsed value with the most recent key
// - if it's an array, push the newly-parsed value to the array
if ($current_item['type'] === 'object') {
$current_item['value'][array_pop($this->_keys)] = $value;
} else {
$current_item['value'][] = $value;
}
// Replace the current item on the stack.
$this->_stack[] = $current_item;
}
}