php json_encode big array - php

I am trying to use json_encode on a big array, and the result returns nothing (yes, I checked that it is utf-8). When I started to investigate this issue I found that the problem arise when a string becomes bigger than 65536.
So when my array is of size 1245, its string from json_encode has length of string(65493), but when I increase array by just one, the string becomes longer than 65536, json_encode fails to output any result.
I thought that the problem is because of memory limit, but when I checked my php.ini I see that it is -1.
Any idea what can be a problem?
Basically I am doing something like this:
$arr = array();
for($i =0; $i<9000; $i++){
$arr[] = array(
'name' => 'test',
'str' => md5($i)
);
}
echo '<pre>'.json_encode($arr).'</pre>';
P.S. sorry guys. I found the problem, thanks to a person with an unreprintable name :-) (thank your Lawrence).
<pre> is the culprit... for some reason it does not print the string in my browser, but it is there.
Lawrence, if you want, you can just write it and I will accept it as correct. Because you were the reason that I came up with this.

Just to remove confusion about this question. The answer is already found and it is in the question.
There is nothing wrong with json_encode function. It works correctly for every output. There is no limitation there except of your memory and how much of it are you giving to your script.
The problem was with browser's implementation of <pre> tag. If you provide too big string to this tag it does not print anything. So the way out is to output answer without <pre> tag

I had the same problem and the array was so big that increasing the memory limit didn't solve my problem. Had to write my own jsonEncode()-method to overcome this:
/**
* Alternative to json_encode() to handle big arrays
* Regular json_encode would return NULL due to memory issues.
* #param $arr
* #return string
*/
private function jsonEncode($arr) {
$str = '{';
$count = count($arr);
$current = 0;
foreach ($arr as $key => $value) {
$str .= sprintf('"%s":', $this->sanitizeForJSON($key));
if (is_array($value)) {
$str .= '[';
foreach ($value as &$val) {
$val = $this->sanitizeForJSON($val);
}
$str .= '"' . implode('","', $value) . '"';
$str .= ']';
} else {
$str .= sprintf('"%s"', $this->sanitizeForJSON($value));
}
$current ++;
if ($current < $count) {
$str .= ',';
}
}
$str.= '}';
return $str;
}
/**
* #param string $str
* #return string
*/
private function sanitizeForJSON($str)
{
// Strip all slashes:
$str = stripslashes($str);
// Only escape backslashes:
$str = str_replace('"', '\"', $str);
return $str;
}

Please try this,
$arr = array();
for($i =0; $i<3000; $i++){
$arr[] = array(
'name' => 'test',
'str' => md5($i)
);
}
$contentArr = str_split(json_encode($arr), 65536);
foreach ($contentArr as $part) {
echo $part;
}

It also occur if the array exceed memory limit, you can try change memory_limit in php.ini like
memory_limit=256M

in my case I found out that the array (derived from my database) contains strings including special characters so I made sure to convert them to utf-8 before using json_encode() function. more on that:
explained here

Related

How to normalise CSV content in PHP?

Problem:
I'm looking for a PHP function to easily and efficiently normalise CSV content in a string (not in a file). I have made a function for that. I provide it in an answer, because it is a possible solution. Unfortuanately it doesn't work when the separator is included in incomming string values.
Can anyone provide a better solution?
Why not using fputcsv / fgetcsv ?
Because:
it requires at least PHP 5.1.0 (which is sometimes not available)
it can only read from files, but not from a string. even though, sometimes the input is not a file (eg. if you fetch the CSV from an email)
putting the content into a temporary file might be unavailable due to security policies.
Why / what kind of normalisation?
Normalise in a way, that the encloser encloses every field. Because the encloser can be optional and different per line and per field. This can happen if one is implementing unclean/incomplete specifications and/or using CSV content from different sources/programs/developers.
Example function call:
$csvContent = "'a a',\"b\",c,1, 2 ,3 \n a a,'bb',cc, 1, 2, 3 ";
echo "BEFORE:\n$csvContent\n";
normaliseCSV($csvContent);
echo "AFTER:\n$csvContent\n";
Output:
BEFORE:
'a a',"b",c,1, 2 ,3
a a,'bb',cc, 1, 2, 3
AFTER:
"a a","b","c","1","2","3"
"a a","bb","cc","1","2","3"
To specifically address your concern regarding f*csv working only with files:
Since PHP 5.3 there's str_getcsv.
For at least PHP >= 5.1 (and I really hope that's the oldest you'll have to deal with these days), you can use stream wrappers:
$buffer = fopen('php://memory', 'r+');
fwrite($buffer, $string);
rewind($buffer);
fgetcsv($buffer) ..
Or obviously the reverse if you want to use fputcsv.
This is a possible solution. But it doesn't consider the case that the separator (,) might be included in incoming strings.
function normaliseCSV(&$csv,$lineseperator = "\n", $fieldseperator = ',', $encloser = '"')
{
$csvArray = explode ($lineseperator,$csv);
foreach ($csvArray as &$line)
{
$lineArray = explode ($fieldseperator,$line);
foreach ($lineArray as &$field)
{
$field = $encloser.trim($field,"\0\t\n\x0B\r \"'").$encloser;
}
$line = implode ($fieldseperator,$lineArray);
}
$csv = implode ($lineseperator,$csvArray);
}
It is a simple chain of explode -> explode -> trim -> implode -> implode .
Although I agree with #deceze that you could expect atleast 5.1 these days, i'm sure there are some internal company servers somewhere who don't want to update.
I altered your method to be able to use field and line separators between double quotes, or in your case the $encloser value.
<?php
/*
In regards to the specs on http://tools.ietf.org/html/rfc4180 I use the following rules:
- "Fields containing line breaks (CRLF), double quotes, and commas should be enclosed in double-quotes."
- "If double-quotes are used to enclose fields, then a double-quote appearing inside a field must be escaped by preceding it with another double quote."
Exception:
Even though the specs says use double quotes, I 'm using your $encloser variable
*/
echo normaliseCSV('a,b,\'c\',"d,e","f","g""h""i","""j"""' . "\n" . "\"k\nl\nm\"");
function normaliseCSV($csv,$lineseperator = "\n", $fieldseperator = ',', $encloser = '"')
{
//We need 4 temporary replacement values
//line seperator, fieldseperator, double qoutes, triple qoutes
$keys = array();
while (count($keys)<3) {
$tmp = "##".md5(rand().rand().microtime())."##";
if (strpos($csv, $tmp)===false) {
$keys[] = $tmp;
}
}
//first we exchange "" (double $encloser) and """ to make sure its not exploded
$csv = str_replace($encloser.$encloser.$encloser, $keys[0], $csv);
$csv = str_replace($encloser.$encloser, $keys[0], $csv);
//Explode on $encloser
//Every odd index is within quotes
//Exchange line and field seperators for something not used.
$content = explode($encloser,$csv);
$len = count($content);
if ($len>1) {
for ($x=1;$x<$len;$x=$x+2) {
$content[$x] = str_replace($lineseperator,$keys[1], $content[$x]);
$content[$x] = str_replace($fieldseperator,$keys[2], $content[$x]);
}
}
$csv = implode('',$content);
$csvArray = explode ($lineseperator,$csv);
foreach ($csvArray as &$line)
{
$lineArray = explode ($fieldseperator,$line);
foreach ($lineArray as &$field)
{
$val = trim($field,"\0\t\n\x0B\r '");
//put back the exchanged values
$val = str_replace($keys[0],$encloser.$encloser,$val);
$val = str_replace($keys[1],$lineseperator,$val);
$val = str_replace($keys[2],$fieldseperator,$val);
$val = $encloser.$val.$encloser;
$field = $val;
}
$line = implode ($fieldseperator,$lineArray);
}
$csv = implode ($lineseperator,$csvArray);
return $csv;
}
?>
Output would be:
"a","b","c","d,e","f","g""h""i","""j"""
"k
l
m"
Codepad example
when i first read this question wasn´t sure if it should be solved or not, since <5.1 environments should be extinguished a long time ago, dispite of that is a hell of a question how to solve this so we should be thinking wich approach to take... and my guess is it should be char by char examination.
I have separated logic in three main scenarios:
A: CHAR is a separator
B: CHAR is a Fuc$€/& quotation
C: CHAR is a Value
Obtaining as a reulst this weapon class (including log for it) for our arsenal:
<?php
Class CSVParser
{
#basic requirements
public $input;
public $separator;
public $currentQuote;
public $insideQuote;
public $result;
public $field;
public $quotation = array();
public $parsedArray = array();
# for logging purposes only
public $logging = TRUE;
public $log = array();
function __construct($input, $separator, $quotation=array())
{
$this->separator = $separator;
$this->input = $input;
$this->quotation = $quotation;
}
/**
* The main idea is to go through the string to parse char by char to analize
* when a complete field is detected it´ll be quoted according and added to an array
*/
public function parse()
{
for($i = 0; $i < strlen($this->input); $i++){
$this->processStream($i);
}
foreach($this->parsedArray as $value)
{
if(!is_null($value))
$this->result .= '"'.addslashes($value).'",';
}
return rtrim($this->result, ',');
}
private function processStream($i)
{
#A case (its a separator)
if($this->input[$i]===$this->separator){
$this->log("A", $this->input[$i]);
if($this->insideQuote){
$this->field .= $this->input[$i];
}else
{
$this->saveField($this->field);
$this->field = NULL;
}
}
#B case (its a f"·%$% quote)
if(in_array($this->input[$i], $this->quotation)){
$this->log("B", $this->input[$i]);
if(!$this->insideQuote){
$this->insideQuote = TRUE;
$this->currentQuote = $this->input[$i];
}
else{
if($this->currentQuote===$this->input[$i]){
$this->insideQuote = FALSE;
$this->currentQuote ='';
$this->saveField($this->field);
$this->field = NULL;
}else{
$this->field .= $this->input[$i];
}
}
}
#C case (its a value :-) )
if(!in_array($this->input[$i], array_merge(array($this->separator), $this->quotation))){
$this->log("C", $this->input[$i]);
$this->field .= $this->input[$i];
}
}
private function saveField($field)
{
$this->parsedArray[] = $field;
}
private function log($type, $value)
{
if($this->logging){
$this->log[] = "CASE ".$type." WITH ".$value." AS VALUE";
}
}
}
and example of how to use it would be:
$original = 'a,"ab",\'ab\'';
$test = new CSVParser($original, ',', array('"', "'"));
echo "<PRE>ORIGINAL: ".$original."</PRE>";
echo "<PRE>PARSED: ".$test->parse()."</PRE>";
echo "<pre>";
print_r($test->log);
echo "</pre>";
and here are the results:
ORIGINAL: a,"ab",'ab'
PARSED: "a","ab","ab"
Array
(
[0] => CASE C WITH a AS VALUE
[1] => CASE A WITH , AS VALUE
[2] => CASE B WITH " AS VALUE
[3] => CASE C WITH a AS VALUE
[4] => CASE C WITH b AS VALUE
[5] => CASE B WITH " AS VALUE
[6] => CASE A WITH , AS VALUE
[7] => CASE B WITH ' AS VALUE
[8] => CASE C WITH a AS VALUE
[9] => CASE C WITH b AS VALUE
[10] => CASE B WITH ' AS VALUE
)
I might have mistakes since i only dedicated 25 mins to it, so any comment will be appreciated an edited.

Keys and indexes to string - implode with a twist

Snippet:
$temp = array();
foreach ($data as $key => $value) {
// This line varies, from HTML to URLs, to anything in between
array_push($temp, "<b>$key:</b> $value");
}
$request = implode('<br/>', $temp); // Glue also varies depending on needs
unset($temp);
This is a getaway from the usual $string .= 'blah<br/>'. Never mind the rtrim.
Can anyone show me a better way to achieve above without use of a temp array and possibly without a loop? Bonus points for not using above code in a function, lambda acceptable though.
P.S. While writing this question I have found a wonderful http_build_query(). One down, one to go.
Edit:
What result should look like:
<b>Title: </b> Value</br>
<b>Title2: </b> Value2</br>
<b>Title3: </b> Value3
Or with different settings (not required, but shows possibility):
key=value&key2=value2&key3=value3
I am trying to improve my code, I use the above snippet everywhere.
My answer: There are some ways, but:
Loops & arrays are the best friends of any programmer.
They provide pretty good readability, reusability and generally are considered to be a right way for performing pretty much the same actions.
You may also take a look on array_map or array_walk. Less code, but it is a loop anyways.
array_walk($data, function($value, $key){
$temp[] = "<b>{$key}:</b> {$value}" ; //Shorthand for array_push is $arr[] = $value ;
}) ;
I suppose this is one way; yay for "one-liners":
$data = ...;
echo join("\n", array_map(function($key) use ($data) {
return sprintf('<b>%s:</b> %s',
htmlspecialchars($key, ENT_QUOTES, 'UTF-8'),
htmlspecialchars($data[$key], ENT_QUOTES, 'UTF-8')
);
}, array_keys($data)));
You basically map the keys using a closure that binds the original array as well so that you have access to both the key and value in each invocation.
I think you will prefere your solution:
$data = range('a', 'e'); // some data
$request = ''; // empty string
array_map(function($value, $key)use(&$request, $data){
static $i = 1;
if($i == count($data)){
$request .= "<b>$key:</b> $value";
}else{
$request .= "<b>$key:</b> $value<br/>";
}
$i++;
}, $data, array_keys($data));
echo $request;
Online demo
A solution with a loop but without $temp array:
$data = range('a', 'e'); // some data
$request = ""; // empty string
foreach ($data as $key => $value) {
$request .= "<b>$key:</b> $value<br/>"; // add to string
}
$request = substr($request,0,-5); // remove last <br/>
echo $request; //output

PHP foreach overwrite value with array

I'm making a simple PHP Template system but I'm getting an error I cannot solve, the thing is the layout loads excellent but many times, can't figure how to solve, here my code
Class Template {
private $var = array();
public function assign($key, $value) {
$this->vars[$key] = $value;
}
public function render($template_name) {
$path = $template_name.'.tpl';
if (file_exists($path)) {
$content = file_get_contents($path);
foreach($this->vars as $display) {
$newcontent = str_replace(array_keys($this->vars, $display), $display, $content);
echo $newcontent;
}
} else {
exit('<h1>Load error</h1>');
}
}
}
And the output is
Title is : Welcome to my template system
Credits to [credits]
Title is : [title]
Credits to Credits to Alvaritos
As you can see this is wrong, but don't know how to solve it.
You're better off with strtr:
$content = file_get_contents($path);
$new = strtr($content, $this->vars);
print $new;
str_replace() does the replaces in the order the keys are defined. If you have variables like array('a' => 1, 'aa' => 2) and a string like aa, you will get 11 instead of 2. strtr() will order the keys by length before replacing (highest first), so that won't happen.
Use this:
foreach($this->vars as $key => $value)
$content = str_replace($key,$value,$content);
echo $content;

CSV encoding php

Here's the problem i need to post a .csv file from one server to another.
I do this by reading the contents of the .csv file and sending that with curl as post data.
This is working without problems.
But then when i try to parse the data and store it in a table in the database the trouble begins.
I have all the variables in a array, if i print this array it displays correctly.
But if i echo a value from that array i get all kinds of weird characters.
My best guess is it has something to do with the encoding of the csv file but i wouldnt have a clue how to fix that.
here's the function i use to parse the csv data:
public function parseCsv($data)
{
$quote = '"';
$newline = "\n";
$seperator = ';';
$dbQuote = $quote . $quote;
// Clean up file
$data = trim($data);
$data = str_replace("\r\n", $newline, $data);
$data = str_replace($dbQuote,'"', $data);
$data = str_replace(',",', ',,', $data);
$data .= $seperator;
$inquotes = false;
$startPoint = $row = $cellNo = 0;
for($i=0; $i<strlen($data); $i++) {
$char = $data[$i];
if ($char == $quote) {
if ($inquotes) $inquotes = false;
else $inquotes = true;
}
if (($char == $seperator or $char == $newline) and !$inquotes) {
$cell = substr($data,$startPoint,$i-$startPoint);
$cell = str_replace($quote,'',$cell);
$cell = str_replace('"',$quote,$cell);
$result[$row][$this->csvMap[$cellNo]] = $this->_parseValue($cellNo, $cell);
++$cellNo;
$startPoint = $i + 1;
if ($char == $newline) {
$cellNo = 0;
++$row;
}
}
}
return $result;
}
any help is appreciated!
EDIT:
Ok so after some more trial and error i found out its just the very first value of the first row that has some extra characters. If i echo that value everything i output after that gets messed up.
So i tried to change the encoding now if i echo the value its all good but i have a new problem, its a string but i need a int:
echo $val; //output: 7655 but messes up everything outputted after it
$val = mb_convert_encoding($val, "UTF-8");
echo $val // output: 7655
echo intval($val) //output: 0
EDIT:
expected output:
7655Array ( [kenmerk] => ÿþ7655 [status] => 205 [status_date] => 1991-12-30 [dob] => 1936-09-04 ) succes
messed up output
7655牁慲੹ਨ††歛湥敭歲⁝㸽@㟾㘀㔀㔀਀††獛慴畴嵳㴠‾㈀㤀㔀਀††獛慴畴彳慤整⁝㸽 201ⴱ㄀㈀ⴀ30 †嬠潤嵢㴠‾㄀㤀㘀㘀-08〭㐀਀਩畳捣獥
i first echo the element 'kenmerk' after that i print the array
as you can see in the array the element 'kenmerk' has some extra charcters..
converting the data to utf-8 like so:
$data = mb_convert_encoding($data, "UTF-8");
eliminates the problem with messed up output and removes the 'ÿþ' (incorrectly-interpreted BOM?) but i still cant convert the values to a int
EDIT:
ok i sort of found a solution..
but as i have no idea why it works i'd appreciate any info
var_dump((int) $val); // output: 0
var_dump((int) strip_tags($val); // output: 7655
You need to remove ÿþ from 7655. intval() and int ($val = (int)$val;) will always output 0 when the first character is not a number. Ex. 765ÿþ5 will return 765, etc.
Regarding your first problem, I would also recommend you to read this answer. PHP messing with HTML Charset Encoding
I hope that it will give you more clarity about what you struggle with.
I will also build you striping process more stable, so it ex. match 7655 instead of ÿþ7655.

How to recursively create a multidimensional array?

I am trying to create a multi-dimensional array whose parts are determined by a string. I'm using . as the delimiter, and each part (except for the last) should be an array
ex:
config.debug.router.strictMode = true
I want the same results as if I were to type:
$arr = array('config' => array('debug' => array('router' => array('strictMode' => true))));
This problem's really got me going in circles, any help is appreciated. Thanks!
Let’s assume we already have the key and value in $key and $val, then you could do this:
$key = 'config.debug.router.strictMode';
$val = true;
$path = explode('.', $key);
Builing the array from left to right:
$arr = array();
$tmp = &$arr;
foreach ($path as $segment) {
$tmp[$segment] = array();
$tmp = &$tmp[$segment];
}
$tmp = $val;
And from right to left:
$arr = array();
$tmp = $val;
while ($segment = array_pop($path)) {
$tmp = array($segment => $tmp);
}
$arr = $tmp;
I say split everything up, start with the value, and work backwards from there, each time through, wrapping what you have inside another array. Like so:
$s = 'config.debug.router.strictMode = true';
list($parts, $value) = explode(' = ', $s);
$parts = explode('.', $parts);
while($parts) {
$value = array(array_pop($parts) => $value);
}
print_r($parts);
Definitely rewrite it so it has error checking.
Gumbo's answer looks good.
However, it looks like you want to parse a typical .ini file.
Consider using library code instead of rolling your own.
For instance, Zend_Config handles this kind of thing nicely.
I really like JasonWolf answer to this.
As to the possible errors: yes, but he supplied a great idea, now it is up to the reader to make it bullet proof.
My need was a bit more basic: from a delimited list, create a MD array. I slightly modified his code to give me just that. This version will give you an array with or without a define string or even a string without the delimiter.
I hope someone can make this even better.
$parts = "config.debug.router.strictMode";
$parts = explode(".", $parts);
$value = null;
while($parts) {
$value = array(array_pop($parts) => $value);
}
print_r($value);
// The attribute to the right of the equals sign
$rightOfEquals = true;
$leftOfEquals = "config.debug.router.strictMode";
// Array of identifiers
$identifiers = explode(".", $leftOfEquals);
// How many 'identifiers' we have
$numIdentifiers = count($identifiers);
// Iterate through each identifier backwards
// We do this backwards because we want the "innermost" array element
// to be defined first.
for ($i = ($numIdentifiers - 1); $i >=0; $i--)
{
// If we are looking at the "last" identifier, then we know what its
// value is. It is the thing directly to the right of the equals sign.
if ($i == ($numIdentifiers - 1))
{
$a = array($identifiers[$i] => $rightOfEquals);
}
// Otherwise, we recursively append our new attribute to the beginning of the array.
else
{
$a = array($identifiers[$i] => $a);
}
}
print_r($a);

Categories