I'm using FQL to retrieve a list of users from Facebook. For consistency I get the result as JSON. This causes a problem - since the returned JSON encodes the user IDs as numbers, json_decode() converts these numbers to floating point values, because some are too big to fit in an int; of course, I need these IDs as strings.
Since json_decode() does its own thing without accepting any behavior flags, I'm at a loss. Any suggestions on how to resolve this?
json_decode() can convert large integers to strings, if you specify a flag in the function call:
$array = json_decode($json, true, 512, JSON_BIGINT_AS_STRING)
I've resolved the issue by adding &format=json-strings to my the FQL api call, like so:
$myQuery = "SELECT uid2 FROM friend WHERE uid1=me()";
$facebook->api("/fql?q=" . urlencode($myQuery) . "&format=json-strings")
This tells facebook to wrap all the numbers in quotes, which leads json_decode to use neither
int-s not floats.
Because I was afraid this issue is not restricted to FQL but to all graph API calls that choose to represent some of the IDs as BIG-INTs I've went as far as patching facebook's PHP SDK a bit to force Facebook to return all of its numbers as strings.
I've added this one line to the _graph function.
This would be line 738 in facebook_base.php, version 3.1.1
$params['format'] = 'json-strings';
Sure fix
I had a similar problem where json_decode was converting recent twitter/tweet IDs into exponential numbers.
Björn's answer is great if you want your BIGINT to become a string - and have PHP 5.3+. If neither of those things are true, another option is to up PHP's float precision.
This can be done a different few ways...
find the precision value in your php.ini and change it to precision = 20
add ini_set('precision', 20); to your PHP app
add php_value precision 20 to your app's .htaccess or virtual host file
Quick and dirty, seems to work for now :
$sJSON = preg_replace('/:(\d+)/', ':"${1}"', $sJSON);
I use this and it works almost great.
json_decode(preg_replace('/("\w+"):(\d+)/', '\\1:"\\2"', $jsonString), true)
The json breaks when there is geo data included, eg. {"lat":54.2341} results in "lat":"54".2341
Solution:
$json = preg_replace('/("\w+"):(\d+)(.\d+)?/', '\\1:"\\2\\3"', $json);
This (preg_replace('/("\w+"):(\d+)(.\d+)?/', '\\1:"\\2\\3"', $json);) worked for me (for parsing result from facebook api)
A major oversight on PHP's part is that browsers don't choke on integers starting with BigInt (64 bits), but before that (53 bits). The introduction of BigInt to modern browsers doesn't help much, when JSON does not support it. So I am trying to remedy that.
My approach is to handle the decoded array (before potentially re-encoding it to a string):
function fix_large_int(&$value)
{
if (is_int($value) && $value > 9007199254740991)
$value = strval($value);
}
$json_arr = json_decode($json_str, flags: JSON_BIGINT_AS_STRING | JSON_OBJECT_AS_ARRAY);
echo('before: ' . json_encode($json_arr) . '<br />' . PHP_EOL);
array_walk_recursive($json_arr, 'fix_large_int');
echo('after: ' . json_encode($json_arr) . '<br />' . PHP_EOL);
The number 9007199254740991 is what JavaScript has as its Number.MAX_SAFE_INTEGER value. Read about that here: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Number/MAX_SAFE_INTEGER
(By the way, I got here, because Twitter API's stringify_ids parameter for blocks/ids and friends/ids does not work for me. There seems to be no mention of that anywhere I could find, though.)
Related
I use a database that stores data in decimal. I convert it to binary so I can just read it as on and off. I haven't had any issues up until the decimal length gets over 6 characters.
The following works:
$value = 2147614720;
$value = decbin($value);
Output: 10000000000000100000000000000000
However, if I try to pull that value from the database, it doesn't work if it's over 6 characters.
$value = $row['decimalvalue'];
$value = decbin($value);
Output: 1111111111111111111111111111111
Any help would be great. Thank you.
Until very recently, PHP was built with only 32-bit integers. That can explain why the second fails. But it is puzzling why the first example worked.
2147614720 > 2^31-1, so it turned into 2^31-1 = 2147483647.
Upgrade your PHP.
I've read all over about arithmetic regarding floating point numbers, but I'm just trying to accurately store the darn things.
I have a mysql field with the type of DECIMAL (40,20).
I am saving a value in php of 46457.67469999996. After updating the record with this value, the end result is 46457.67470000000000000000. Not sure why it's being rounded at all just being saved to the database.
The value is not being converted to a string or anything beforehand. The field value that is passed into PDO is the value I expected to be saved and it is returned as a float... Perhaps it's because I'm saving a PHP float to a mysql decimal type where the rounding is occurring?
What am I missing here?
EDIT: Added example code that has the issue
// Query placeholder variables. Hard-coded for the example test
$query_vars = array(
":vendor_id" => 33154,
":year" => 2018,
":coop_committed_dollar" => 46457.67469999996,
":coop_committed_dollar_update" => 46457.67469999996
);
$statement = " INSERT INTO vendor_data_yearly
(vendor_id, year, coop_committed_dollar) VALUES
(:vendor_id, :year, :coop_committed_dollar)
ON DUPLICATE KEY UPDATE
coop_committed_dollar = :coop_committed_dollar_update;";
$query = $connection->conn->prepare($statement);
$query->execute($query_vars);
When I run this, the resulting value of coop_committed_dollar is 46457.67470000000000000000. This code is legit all I am doing.
Possible solution
// Note that I am casting the string using the BC Math library.
// I dunno how to just initialize the number (lame documentation), so I'm adding 0 to it.
$number = "46457.674699999967";
$number = bcadd("46457.674699999967", 0, 20);
$query_vars = array(
":vendor_id" => 33154,
":year" => 2018,
":coop_committed_dollar" => $number,
":coop_committed_dollar_update" => $number
);
$statement = " INSERT INTO vendor_data_yearly
(vendor_id, year, coop_committed_dollar) VALUES
(:vendor_id, :year, :coop_committed_dollar)
ON DUPLICATE KEY UPDATE
coop_committed_dollar = :coop_committed_dollar_update;";
$query = $conn->prepare($statement);
$query->execute($query_vars);
This results in the number as expected in the DB.
ONLY SOLUTION I FOUND TO WORK CORRECTLY
The data I am working with is passed in via ajax. I had to take a few steps to get this to work correctly.
Use ini_set('precision', 20);
Manually set the data in question to be a string BEFORE sending it via ajax so PHP would not round it, extended with extra floating point madness, padd it, etc.
I found that PHP would just not let me reliably work with large numbers coming from a variable set outside the script's scope (ajax). Once PHP got it's hands on the number, it would do what it had to do in order to make it make sense as a float.
If anyone has a better solution for this particular scenario I'm all ears and eyes :)
The problem is that PHP's precision is not allowing you to store the exact number you think you are storing.
When you set ":coop_committed_dollar" => 46457.67469999996
PHP is actually storing it as a different value, depending on the precision.
The solution is to store the value in PHP as a string instead of a float.
Since your question is: "what am I missing", I will try to provide an answer.
Basically it comes down to storing floats internally using binary representation. Since 46457.67469999996 cannot be exactly in binary (it ends up with an infinite number, similar to 33% (.3333...) in base-10), the closest rounding is used based on PHP's precision (set in php ini).
I was given a great explanation in this question that I asked a while back.
In your particular case, it also seems that the value that you are sending via AJAX is being stored as a float when parsed by PHP on the server-side. You want it to be stored as a string instead. If you're using json_decode, add this option: JSON_BIGINT_AS_STRING.
I have the following php snippet
$newData = serialize(array('ep' => 50733372961735.4));
echo "New data: " . print_r($newData, 1);
Output:
New data: a:1:{s:2:"ep";d:5.07333729617E+13;}
But I would like the float value as it is and not E+13.
What could I do without having to make drastic changes as this is just an example. In my actual code the 'ep' value could be inside a complex array hierarchy
Firstly, a general note: serialize should never be used on data that could be manipulated in any way. It's useful for things like session data and caches, but should not be relied on for transporting data between applications or data storage. In many cases, you're better off using a standard serialization format like JSON.
You also certainly shouldn't care what the serialized string looks like - the only thing you should do with that string is pass it back to unserialize(). So the fact that there is E+13 is not a problem if the actual value it gives back when you unserialize is the one you wanted.
However, it's clear in your example that you have lost precision - the last digits are ...29617 rather than ...29617354 - so back to the point: there is a PHP setting serialize_precision, described in the manual here. It's default value has varied over the years, but setting it to an explicit value other than -1 will serialize floats with that number of significant figures:
ini_set('serialize_precision', 2);
echo serialize(50733372961735.4), PHP_EOL;
// d:5.1E+13;
ini_set('serialize_precision', 20);
echo serialize(50733372961735.4), PHP_EOL;
// d:50733372961735.398438;
Note that the first example has clearly thrown away information, whereas the second has actually stored more precision than you realised you had - because of the inaccuracy of storing decimals in binary floating point format.
The problem is not the serialize at all!
it's numbers in general.
$num = 50733372961735.4;
print($num);
=> 50733372961735
to "solve" this problem you can use:
ini_set('serialize_precision', 15);
I just executed your code on www.writephponline.com
I got the below value if I did not put your value in string :- $newData = serialize(array('ep' => 50733372961735.4));
New data: a:1:{s:2:"ep";d:50733372961735.398;}
and this after adding it in string :- $newData = serialize(array('ep' => '50733372961735.4'));
New data: a:1:{s:2:"ep";s:16:"50733372961735.4";}
So I'm using php 5.2.6 in a WAMP environment.
I'm trying to use the json_decode function to make a json string into an array. The JSON is coming from a REST API elsewhere so I have no control over formatting of the JSON string. Here is an example of one of the json strings I'm trying to use:
[{
"webinarKey":795855906,
"sessionKey":100000000041808257,
"startTime":"2011-12-16T13:56:15Z",
"endTime":"2011-12-16T14:48:37Z",
"registrantsAttended":2
}]
I'm specifically after the sessionKey value here. PHP is treating the value as a float and I can't seem to do anything to retrieve the original value.
I've tried the following:
json_decode($json, true, 512, JSON_BIGINT_AS_STRING);
# This produces the following error because my php version isn't up to snuff and I
# can't upgrade to the version required
# Warning: json_decode() expects at most 2 parameters, 4 given
I've also tried this:
$json_obj = json_decode($json, true);
number_format($json_obj[0]["sessionKey"], 0, '.', '');
# This results in precision issues where the value was 100000000041808257
# but is number_formated out as 100000000041808256
As I said, upgrading to php 5.4 (where the 4 parameter json_decode call is supported) isn't an option. Please help!
Thanks!
To quality JSON spec use:
// wrap numbers
$json = preg_replace('/:\s*(\-?\d+(\.\d+)?([e|E][\-|\+]\d+)?)/', ': "$1"', $json);
// as object
$object = json_decode($json);
// as array
$array = json_decode($json, true);
Thanks #Scott Gottreu and #pospi.
The answer was in the last comment for the accepted answer on this question.
Use the preg_replace() function to surround all integer values with quotes.
json_decode(preg_replace('/("\w+"):(\d+)/', '\\1:"\\2"', $jsonString), true);
Actually after testing the above line it screws up JSON with floating point numbers in as values so to fix that issue I used the following to just enclose all numbers (integer or floating point numbers) in quotes:
json_decode(preg_replace('/("\w+"):(\d+(\.\d+)?)/', '\\1:"\\2"', $jsonString), true);
Meanwhile, PHP has fixed this problem ... well, somehow. Starting sometime around PHP 5.4 they added an option which does the just what the Regex solutions posted above do:
json_decode($json, false, 512, JSON_BIGINT_AS_STRING);
The 512 refers to the default maximum nesting depth.
(I'm re-posting my answer from another question (Handling big user IDs returned by FQL in PHP) here, because this is more generic; not related to Facebook / FQL.)
A major oversight on PHP's part is that browsers don't choke on integers starting with BigInt (64 bits), but before that (53 bits). The introduction of BigInt to modern browsers doesn't help much, when JSON does not support it. So I am trying to remedy that.
The often cited versions using preg_replace() do not account for large integers in indexed arrays. My approach is to handle the decoded array (before potentially re-encoding it to a string):
function fix_large_int(&$value)
{
if (is_int($value) && $value > 9007199254740991)
$value = strval($value);
}
$json_arr = json_decode($json_str, flags: JSON_BIGINT_AS_STRING | JSON_OBJECT_AS_ARRAY);
echo('before: ' . json_encode($json_arr) . '<br />' . PHP_EOL);
array_walk_recursive($json_arr, 'fix_large_int');
echo('after: ' . json_encode($json_arr) . '<br />' . PHP_EOL);
The number 9007199254740991 is what JavaScript has as its Number.MAX_SAFE_INTEGER value. Read about that here: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Number/MAX_SAFE_INTEGER
I am curious, is there a size limit on serialize in PHP. Would it be possible to serialize an array with 5,000 keys and values so it can be stored into a cache?
I am hoping to cache a users friend list on a social network site, the cache will need to be updated fairly often but it will need to be read almost every page load.
On a single server setup I am assuming APC would be better then memcache for this.
As quite a couple other people answered already, just for fun, here's a very quick benchmark (do I dare calling it that ? ) ; consider the following code :
$num = 1;
$list = array_fill(0, 5000, str_repeat('1234567890', $num));
$before = microtime(true);
for ($i=0 ; $i<10000 ; $i++) {
$str = serialize($list);
}
$after = microtime(true);
var_dump($after-$before);
var_dump(memory_get_peak_usage());
I'm running this on PHP 5.2.6 (the one bundled with Ubuntu jaunty).
And, yes, there are only values ; no keys ; and the values are quite simple : no object, no sub-array, no nothing but string.
For $num = 1, you get :
float(11.8147978783)
int(1702688)
For $num = 10, you get :
float(13.1230671406)
int(2612104)
And, for $num = 100, you get :
float(63.2925770283)
int(11621760)
So, it seems the bigger each element of the array is, the longer it takes (seems fair, actually). But, for elements 100 times bigger, you don't take 100 times much longer...
Now, with an array of 50000 elements, instead of 5000, which means this part of the code is changed :
$list = array_fill(0, 50000, str_repeat('1234567890', $num));
With $num = 1, you get :
float(158.236332178)
int(15750752)
Considering the time it took for 1, I won't be running this for either $num = 10 nor $num = 100...
Yes, of course, in a real situation, you wouldn't be doing this 10000 times ; so let's try with only 10 iterations of the for loop.
For $num = 1 :
float(0.206310987473)
int(15750752)
For $num = 10 :
float(0.272629022598)
int(24849832)
And for $num = 100 :
float(0.895547151566)
int(114949792)
Yeah, that's almost 1 second -- and quite a bit of memory used ^^
(No, this is not a production server : I have a pretty high memory_limit on this development machine ^^ )
So, in the end, to be a bit shorter than those number -- and, yes, you can have numbers say whatever you want them to -- I wouldn't say there is a "limit" as in "hardcoded" in PHP, but you'll end up facing one of those :
max_execution_time (generally, on a webserver, it's never more than 30 seconds)
memory_limit (on a webserver, it's generally not muco more than 32MB)
the load you webserver will have : while 1 of those big serialize-loop was running, it took 1 of my CPU ; if you are having quite a couple of users on the same page at the same time, I let you imagine what it will give ;-)
the patience of your user ^^
But, except if you are really serializing long arrays of big data, I am not sure it will matter that much...
And you must take into consideration the amount of time/CPU-load using that cache might help you gain ;-)
Still, the best way to know would be to test by yourself, with real data ;-)
And you might also want to take a look at what Xdebug can do when it comes to profiling : this kind of situation is one of those it is useful for!
The serialize() function is only limited by available memory.
There's no limit enforced by PHP. Serialize returns a bytestream representation (string) of the serialized structure, so you would just get a large string.
The only practical limit is your available memory, since serialization involves creating a string in memory.
There is no limit, but remember that serialization and unserialization has a cost.
Unserialization is exteremely costly.
A less costly way of caching that data would be via var_export() as such (since PHP 5.1.0, it works on objects):
$largeArray = array(1,2,3,'hello'=>'world',4);
file_put_contents('cache.php', "<?php\nreturn ".
var_export($largeArray, true).
';');
You can then simply retrieve the array by doing the following:
$largeArray = include('cache.php');
Resources are usually not cache-able.
Unfortunately, if you have circular references in your array, you'll need to use serialize().
As suggested by Thinker above:
You could use
$string = json_encode($your_array_here);
and to decode it
$array = json_decode($your_array_here, true);
This returns an array. It works well even if the encoded array was multilevel.
Ok... more numbers! (PHP 5.3.0 OSX, no opcode cache)
#Pascal's code on my machine for n=1 at 10k iters produces:
float(18.884856939316)
int(1075900)
I add unserialize() to the above as so.
$num = 1;
$list = array_fill(0, 5000, str_repeat('1234567890', $num));
$before = microtime(true);
for ($i=0 ; $i<10000 ; $i++) {
$str = serialize($list);
$list = unserialize($str);
}
$after = microtime(true);
var_dump($after-$before);
var_dump(memory_get_peak_usage());
produces
float(50.204112052917)
int(1606768)
I assume the extra 600k or so are the serialized string.
I was curious about var_export and its include/eval partner $str = var_export($list, true); instead of serialize() in the original produces
float(57.064643859863)
int(1066440)
so just a little less memory (at least for this simple example) but way more time already.
adding in eval('$list = '.$str.';'); instead of unserialize in the above produces
float(126.62566018105)
int(2944144)
Indicating theres probably a memory leak somewhere when doing eval :-/.
So again, these aren't great benchmarks (I really should isolate the eval/unserialize by putting the string in a local var or something, but I'm being lazy) but they show the associated trends. var_export seems slow.
Nope, there is no limit and this:
set_time_limit(0);
ini_set('memory_limit ', -1);
unserialize('s:2000000000:"a";');
is why you should have safe.mode = On or a extension like Suhosin installed, otherwise it will eat up all the memory in your system.
I think better than serialize is json_encode function. It got a drawback, that associative arrays and objects are not distinguished, but string result is smaller and easier to read by human, so also to debug and edit.
If you want to cache it (so I assume performance is the issue), use apc_add instead to avoid the performance hit of converting it to a string + gain cache in memory.
As stated above the only size limit is available memory.
A few other gotchas:
serialize'd data is not portable between multi-byte and single-byte character encodings.
PHP5 classes include NUL bytes that can cause havoc with code that doesn't expect them.
Your use case sounds like you're better off using a database to do that rather than relying solely on PHP's available resources. The advantages to using something like MySQL instead is that it's specifically engineered with memory management in mind for such things as storage and lookup.
It's really no fun constantly serializing and unserializing data just to update or change a few pieces of information.
I've just come across an instance where I thought I was hitting an upper limit of serialisation.
I'm persisting serialised objects to a database using a mysql TEXT field.
The limit of the available characters for a single-byte characters is 65,535 so whilst I can serialize much larger objects than that with PHP It's impossible to unserialize them as they are truncated by the limit of the TEXT field.
i have a case in which unserialize throws an exception on a large serialized object, size: 65535 (the magic number: 16bits full bit = 65536)