Read big json file with php - php

Question

I've read somewhere that I should use the library salsify/jsonstreamingparser to open a big json file but it's giving me the same error as with json_decode:
PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 20480 bytes) in /data/www/default/database/vendor/salsify/json-streaming-parser/src/Listener/InMemoryListener.php on line 92
I have to do it in php because I'm using a free hosting which doesn't have python.
Basically what I want to do is download a big json file unzip it and process the content. I don't know why in php I wasn't able to do it in all day but in python I did it in 5 minutes:
import os
import json
import urllib
import zipfile
json_file = 'AllSets-x.json'
zip_file = json_file + '.zip'
urllib.urlretrieve ("https://mtgjson.com/json/" + zip_file, zip_file)
dir_path = os.path.dirname(os.path.realpath(__file__))
zip_ref = zipfile.ZipFile(dir_path + "/" + zip_file, 'r')
zip_ref.extractall(dir_path)
zip_ref.close()
json_data = json.load(open(json_file, 'r'))
print json_data.keys()[0]
This is what I have in php:
<?php
require_once __DIR__ . '/vendor/autoload.php';
include "../credentials.php";
error_reporting(E_ALL); # Reports all errors
ini_set('display_errors','Off'); # Do not display errors for the end-users (security issue)
ini_set('error_log','/tmp/php-errors.log'); # Set a logging file
// Override the default error handler behavior
set_exception_handler(function($exception) {
$logger->error($exception);
echo "Something went wrong!";
});
$logger = new Monolog\Logger('channel-name');
$logger->pushHandler(new Monolog\Handler\StreamHandler('/tmp/php-errors.log', Monolog\Logger::DEBUG));
$logger->info("Parsing json file");
$listener = new \JsonStreamingParser\Listener\InMemoryListener();
$json_file = __DIR__ . "/AllSets-x.json";
$stream = fopen($json_file, 'r');
try {
$parser = new \JsonStreamingParser\Parser($stream, $listener);
$parser->parse();
fclose($json_file);
} catch (Exception $e) {
fclose($json_file);
throw $e;
}
$logger->info("Json file parsed");
$json_data = $listener->getJson();
$logger->info("Displaying json data");
var_dump($json_data);

Answer 1

Using the InMemoryListener certainly defeats the purpose of a streaming parser. That'll just unpack everything into memory (likely worse memory-wise than plain json_decode).
You'll need to catch each JSON object block individually, if you want to work under such constraints.
There's the SimpleObjectQueueListener which could possibly fit the bill. If the specific JSON has a bunch of [{…}, {…}, {…}] objects to be processed:
$listener = new \JsonStreamingParser\Listener\SimpleObjectQueueListener("print_r", 0);
// would just print out each object block from the JSON stream
Obviously you would use a callback like "process_my_json_blobs" instead. (Or a prepared callback like [$pdo, "execute"] perhaps.)
Btw, reading the whole JSON input just works on your local Python, because there's usually no memory_limit as for common PHP setups. (Python at best relies on the systems ulimit.)