7 ms·

> JSON is basically perfect Until you realize you can't actually store real integers because every number in js is a float...

by ohazi 3y ago

data-ottawa 3y ago
You can store the first 2^53 integers with either sign, and if you need accurate integer values beyond that size you can stringify them and parse as big ints. It’s not ideal, but 2^64 integers is also finite.
gliptic 3y ago
JSON allows you to store arbitrarily large integers/floats. It's only in JS this is a problem, not if you use JSON in languages that support larger (than 54-bit) integers.
- the_gipsy 3y ago
  That's the freedom of unspecified behavior.
  - marcosdumay 3y ago
    As long as the same person is on both sides of a communication channel, he has total freedom on what to say and will understand it flawlessly! That's what standards are for, isn't it?
    - deleted 3y ago
      [deleted]
- no_wizard 3y ago
  Annoyingly, it also doesn't support BigInt, which would alleviate this problem in JS as well
  - Simran-B 3y ago
    A number in JSON can have an arbitrary number of digits, i.e. it can represent any BigInt value.
    - Supermancho 3y ago
      > A number in JSON can have an arbitrary number of digits, i.e. it can represent any BigInt value. In my experience, violating type constraints causes problems in downstream systems (usually with parsing or trying to operate on invalid values). Number, as defined by the JSON Schema spec. A 32-bit signed integer. It has a minimum value of -2,147,483,648 and a maximum value of 2,147,483,647 BigInt is defined by various (MSFT, MySQL, etc): -9,223,372,036,854,775,808 to 9,223,372,036,854,775,807 Most systems use a JSON String for large numbers, out of necessity, not JSON Number.
      - ghusbands 3y ago
        In context, BigInt refers to arbitrary precision integers [1], rather than any particular size of integer, hence "arbitrary number of digits". [1] https://v8.dev/features/bigint https://v8.dev/features/bigint
josephcsible 3y ago
You can store arbitrary precision numbers in JSON. The spec explicitly doesn't lock you into floats or any other specific number format.
- Spivak 3y ago
  Only if you are both sides of the transmission. If you're sending JSON to code you didn't write you will eventually get bitten by software lossy re-encoding. Lots of places use strings for this reason. It's like API's that mess up the semantics of PUT/GET so implementing idempotency is extra annoying.
  - somewhereoutth 3y ago
    But JSON is strings. So 123.4 is essentially "123.4" but with the indication that it is supposed to be semantically a numerical value.
    - josephg 3y ago
      Right. I want to see an indication of what sort of numerical value it is. Big integers interpreted as floats lose precision. And floats decoded as integers truncate anything after the decimal place. JSON makes it way too easy to get this stuff wrong when decoding.
      - somewhereoutth 3y ago
        If it has a decimal point then it is a decimal. And if it doesn't (or if it only has zeros after the point) then it's an integer. JSON is absolutely unambiguous as to the actual numerical value - how badly that gets translated into the decoding language is entirely on that language.
        josephg 3y ago
        This isn't right. JSON can also store exponential numbers (eg {"google": 1e+100}). You could decode this to an arbitrary-sized BigInt, but I can make you waste an arbitrary number of bytes in RAM if you do that. And even then, "look for a decimal point" doesn't give you enough information to tell whether the number is an integer. Eg, 1.1e+100 is an integer, and 1e-100 is not an integer. One of JSON's biggest benefits is that you don't need to know the shape of the data when you parse. JSON's syntax tells you the type of all of its fields. Unfortunately, that stops being true with numbers as soon as double precision float isn't appropriate. If you use more digits in a JSON number, you can't decode your JSON without knowing what precision you need to decode your data. Even javascript has this problem if you need BigInts, since there's no obvious or easy way to decode a bigint from JSON without losing precision. In the wild, I've seen bigints awkwardly embedded in a JSON string. Gross. Putting responsibility for knowing the number precision into the language you're using to decode JSON misses the point. Everywhere else, JSON tells you the type of your data as you decode, without needing a schema. Requiring a schema for numbers is a bad design.
Kwpolska 3y ago
If both sides are using a language with integer types, this is a non-issue. JSON does not prescribe the number types in use, so the implementations may just say that the field contains 64-bit integers, and just parse them to and from the usual int64 type of their language. It is also legal for JSON parsers to parse numeric literals into an arbitrary-precision decimal type instead of IEEE 574 floats.
__s 3y ago
Except JSON can't even serialize all js numbers when it comes to NaN or infinity
mst 3y ago
53 bit integers should be enough for anyone (in practice for config it usually is but enforcing it is horribly patchy)
Waterluvian 3y ago
JSON’s numbers are not IEEE-754. They’re numbers with an optionally infinite number of decimal places. It’s up to a parser to handle it. Python can parse these into integers if there isn’t a decimal place. It’s in the name, but be careful not to get confused with JSON being JavaScript.
- nly 3y ago
  Doesn't matter. The baseline is anything written in C and C++, which don't have bignum or decimal types and so more or less always parse JSON numbers to either int64 or double, at best.
- dijit 3y ago
  You wrote this as if it’s a defense but honestly I feel even more terrified of JSON numbers now than I was before entering this thread, and before reading your comment. Not following a set standard is undefined behaviour, leaving it up to the implementation is a large problem in other areas of computer science. Such as C compilers.
  - Waterluvian 3y ago
    JSON isn’t intended to narrow all details. That’s up to the producer and consumer. If you use JSON you will specify these details in your API. JSON isn’t an API. I wonder how many times this gets violated though, and how many times this “I dunno… you decide” approach causes problems.
  - crdrost 3y ago
    Yes but this is a necessary limitation for all human readable numbers. The context decides what to deserialize into and different contexts/languages will choose bigint vs i64 vs u64 vs i32 vs double vs quad vs float, whatever is convenient for them. Heck, some of them will even choose different endian-ness and sometimes it will matter. I still remember the first time I dealt with a Java developer who was trying to send us a 64-bit ID and trying to explain to him that JavaScript only has 52-bit integers and how his eyes widened in such earnest disbelief that anybody would ever accept something so ridiculous. (The top bits were not discardable, they redundantly differentiated between environments that the objects lived in... so all of our dev testing had been fine because the top bits were zero for the dev server in Europe but then you put us on this cluster in your Canadian datacenter and now the top bits are not all zero. Something like a shard of the database or so.) We have bigints now but JSON.parse() can't ever ever support 'em! "Please, it's an ID, why are you even sending it as a number anyway, just make it a string." But they had other customers who they didn't want to break. It was an early powerful argument for UUIDs, hah!
throwaway894345 3y ago
> Until you realize you can't actually store real integers because every number in js is a float JSON !== JS

> JSON is basically perfect Until you realize you can't actually store real integers because every number in js is a float...

by ohazi 3y ago

> JSON is basically perfect Until you realize you can't actually store real integers because every number in js is a float...

data-ottawa 3y ago

You can store the first 2^53 integers with either sign, and if you need accurate integer values beyond that size you can stringify them and parse as big ints. It’s not ideal, but 2^64 integers is also finite.

gliptic 3y ago

JSON allows you to store arbitrarily large integers/floats. It's only in JS this is a problem, not if you use JSON in languages that support larger (than 54-bit) integers.

the_gipsy 3y ago
That's the freedom of unspecified behavior.
- marcosdumay 3y ago
  As long as the same person is on both sides of a communication channel, he has total freedom on what to say and will understand it flawlessly! That's what standards are for, isn't it?
  - deleted 3y ago
    [deleted]
no_wizard 3y ago
Annoyingly, it also doesn't support BigInt, which would alleviate this problem in JS as well
- Simran-B 3y ago
  A number in JSON can have an arbitrary number of digits, i.e. it can represent any BigInt value.
  - Supermancho 3y ago
    > A number in JSON can have an arbitrary number of digits, i.e. it can represent any BigInt value. In my experience, violating type constraints causes problems in downstream systems (usually with parsing or trying to operate on invalid values). Number, as defined by the JSON Schema spec. A 32-bit signed integer. It has a minimum value of -2,147,483,648 and a maximum value of 2,147,483,647 BigInt is defined by various (MSFT, MySQL, etc): -9,223,372,036,854,775,808 to 9,223,372,036,854,775,807 Most systems use a JSON String for large numbers, out of necessity, not JSON Number.
    - ghusbands 3y ago
      In context, BigInt refers to arbitrary precision integers [1], rather than any particular size of integer, hence "arbitrary number of digits". [1] https://v8.dev/features/bigint https://v8.dev/features/bigint

josephcsible 3y ago

You can store arbitrary precision numbers in JSON. The spec explicitly doesn't lock you into floats or any other specific number format.

Spivak 3y ago
Only if you are both sides of the transmission. If you're sending JSON to code you didn't write you will eventually get bitten by software lossy re-encoding. Lots of places use strings for this reason. It's like API's that mess up the semantics of PUT/GET so implementing idempotency is extra annoying.
- somewhereoutth 3y ago
  But JSON is strings. So 123.4 is essentially "123.4" but with the indication that it is supposed to be semantically a numerical value.
  - josephg 3y ago
    Right. I want to see an indication of what sort of numerical value it is. Big integers interpreted as floats lose precision. And floats decoded as integers truncate anything after the decimal place. JSON makes it way too easy to get this stuff wrong when decoding.
    - somewhereoutth 3y ago
      If it has a decimal point then it is a decimal. And if it doesn't (or if it only has zeros after the point) then it's an integer. JSON is absolutely unambiguous as to the actual numerical value - how badly that gets translated into the decoding language is entirely on that language.
      - josephg 3y ago
        This isn't right. JSON can also store exponential numbers (eg {"google": 1e+100}). You could decode this to an arbitrary-sized BigInt, but I can make you waste an arbitrary number of bytes in RAM if you do that. And even then, "look for a decimal point" doesn't give you enough information to tell whether the number is an integer. Eg, 1.1e+100 is an integer, and 1e-100 is not an integer. One of JSON's biggest benefits is that you don't need to know the shape of the data when you parse. JSON's syntax tells you the type of all of its fields. Unfortunately, that stops being true with numbers as soon as double precision float isn't appropriate. If you use more digits in a JSON number, you can't decode your JSON without knowing what precision you need to decode your data. Even javascript has this problem if you need BigInts, since there's no obvious or easy way to decode a bigint from JSON without losing precision. In the wild, I've seen bigints awkwardly embedded in a JSON string. Gross. Putting responsibility for knowing the number precision into the language you're using to decode JSON misses the point. Everywhere else, JSON tells you the type of your data as you decode, without needing a schema. Requiring a schema for numbers is a bad design.

Kwpolska 3y ago

If both sides are using a language with integer types, this is a non-issue. JSON does not prescribe the number types in use, so the implementations may just say that the field contains 64-bit integers, and just parse them to and from the usual int64 type of their language. It is also legal for JSON parsers to parse numeric literals into an arbitrary-precision decimal type instead of IEEE 574 floats.

__s 3y ago

Except JSON can't even serialize all js numbers when it comes to NaN or infinity

mst 3y ago

53 bit integers should be enough for anyone (in practice for config it usually is but enforcing it is horribly patchy)

Waterluvian 3y ago

JSON’s numbers are not IEEE-754. They’re numbers with an optionally infinite number of decimal places. It’s up to a parser to handle it. Python can parse these into integers if there isn’t a decimal place. It’s in the name, but be careful not to get confused with JSON being JavaScript.

nly 3y ago
Doesn't matter. The baseline is anything written in C and C++, which don't have bignum or decimal types and so more or less always parse JSON numbers to either int64 or double, at best.
dijit 3y ago
You wrote this as if it’s a defense but honestly I feel even more terrified of JSON numbers now than I was before entering this thread, and before reading your comment. Not following a set standard is undefined behaviour, leaving it up to the implementation is a large problem in other areas of computer science. Such as C compilers.
- Waterluvian 3y ago
  JSON isn’t intended to narrow all details. That’s up to the producer and consumer. If you use JSON you will specify these details in your API. JSON isn’t an API. I wonder how many times this gets violated though, and how many times this “I dunno… you decide” approach causes problems.
- crdrost 3y ago
  Yes but this is a necessary limitation for all human readable numbers. The context decides what to deserialize into and different contexts/languages will choose bigint vs i64 vs u64 vs i32 vs double vs quad vs float, whatever is convenient for them. Heck, some of them will even choose different endian-ness and sometimes it will matter. I still remember the first time I dealt with a Java developer who was trying to send us a 64-bit ID and trying to explain to him that JavaScript only has 52-bit integers and how his eyes widened in such earnest disbelief that anybody would ever accept something so ridiculous. (The top bits were not discardable, they redundantly differentiated between environments that the objects lived in... so all of our dev testing had been fine because the top bits were zero for the dev server in Europe but then you put us on this cluster in your Canadian datacenter and now the top bits are not all zero. Something like a shard of the database or so.) We have bigints now but JSON.parse() can't ever ever support 'em! "Please, it's an ID, why are you even sending it as a number anyway, just make it a string." But they had other customers who they didn't want to break. It was an early powerful argument for UUIDs, hah!

throwaway894345 3y ago

> Until you realize you can't actually store real integers because every number in js is a float JSON !== JS