I also want to chime in that it is great to see this all gone through with motivating examples, good write-ups, and working code.
I’m pretty interested in this, but want to verify a few things with experiments. In particular, which popular languages reliably read and write integers and floats differently from JSON? And maybe also produce some test vectors and worked example code to demonstrate corner-cases.
I’m overall pretty hesitant to rely on JSON text encoding including ‘.0’ or not to distinguish integers from floats throughout the numerical range. At the end of the day the JSON data model only has a single numerical type (the core issue here), and relying on common behaviors doesn’t feel very robust.
I think it is actually relatively common for atproto implementations to need to work with un-typed / no-schema data and want to be able to pass it through multiple contexts and have it come back with the same CID. I think you have broad experience with this sort of thing, but just to lay out some of my thinking, you might want to persist record data in PostgreSQL as JSONB type (supporting rich queries), read it back later, and be able to confirm the CID version. You might get record data back from an HTTP API in JSON format, with a CID along side it, and want to be able to confirm the CID matches the data. The ‘unknown’ Lexicon type allows stuffing arbitrary bytes in.
What i’d be kind of thinking of is to have an unambiguous JSON encoding, and then CBOR can use the obvious/efficient typed encoding. The JSON encoding could look like the existing CID link and bytes special objects:
{
"$float": 123.456
}
(the float value itself could also be in string format, or even base64 or hex encoded)
This isn’t super fun/aesthetic in raw JSON, but for the subset of application code which is schema-aware, this could all be transparently handled, the same way bytestrings or CID links are today. Though maybe a “float” string format in Lexicon would also work this way (get converted to/from floats via codegen).
In addition to the “is it a float or an integer” ambiguity, we’d need to settle on how special float values are handled, and get that tested and rolled out to the entire ecosystem. I think IPLD DAG-CBOR didn’t get too far in the weeds on this, but I’ve seen a couple different deterministic CBOR flavors floating around the IETF with different rules about this (eg, are NaN and +Inf/-Inf allowed, with both signs or not, etc). There is work happening there, and it should probably be possible to settle on a spec, but I think research/consensus would be needed. I personally have lingering anxiety knowing how complex hardware support for IEEE floats is, and would want to do some more research and experimentation that things like FPU rounding mode wouldn’t come in to play. It would really suck if atproto didn’t work reliably or in an interoperable way on obscure CPU instruction sets or old computers or something like that. This might just be FUD, but I’ve been bitten by softfloat, weird libc/newlib ports, things like that.