How does couchbase store numbers?

Couchbase Server
1

Hi all, hoping for some help here. Dug through the couchbase documentation, stackoverflow, etc, still unclear about what are the limitations of storing numbers in couchbase. My main concern is storing decimals (3 decimal precision to be precise :wink: ).
I expected the storage to be some 64bit floating number, so I thought would hit the same limitations of doubles in Java. Basically both Javascript numbers and Java doubles follow (if my googling serves me) IEEE 754. My problem is that Couchbase somehow seems to store 0.99 precisely while Java can’t represent the double well.

Long story short.
POJO:

@ToString
class BD {
public BigDecimal bdString = new BigDecimal(“0.99”);
public BigDecimal bdDouble = new BigDecimal(0.99);
public int i = 123124;
public String boo = “boo”;
}

Code to write and read in Couchbase:

  	BD bdWrite = new BD();
  	String json = mapper.writeValueAsString(bdWrite);
  	BigDecimal bdWriteString = bdWrite.bdString;
  	BigDecimal bdWriteDouble = bdWrite.bdDouble;
  	JsonDocument jsonDoc = JsonDocument.create("bigdecimal::1", JsonObject.fromJson(json));
  	bucket.upsert(jsonDoc);
  	System.out.println("bdWrite: " + bdWrite);
  	
  	String id = "bigdecimal::1";
  	JsonDocument doc = bucket.get(id);
  	BD bdRead = mapper.readValue(doc.content().toString(), BD.class);
  	BigDecimal bdReadString = bdRead.bdString;
  	BigDecimal bdReadDouble = bdRead.bdDouble;
  	System.out.println("bdRead: " + bdRead);
  	System.out.println("bdWrite: " + bdWrite + ", bdRead:" + bdRead + ", equal:" + bdWrite.bdDouble.equals(bdRead.bdDouble));

The doc in couchbase:

{
“i”: 123124,
“boo”: “boo”,
“bdDouble”: 0.99,
“bdString”: 0.99
}

And the console output:

bdWrite: BD(bdString=0.99, bdDouble=0.9899999999999999911182158029987476766109466552734375, i=123124, boo=boo)
bdRead: BD(bdString=0.99, bdDouble=0.99, i=123124, boo=boo)
bdWrite: BD(bdString=0.99, bdDouble=0.9899999999999999911182158029987476766109466552734375, i=123124, boo=boo), bdRead:BD(bdString=0.99, bdDouble=0.99, i=123124, boo=boo), equal:false

Basically both the BigDecimal from double (immediately loss of precision) and the BigDecimal from String (no loss of precision) seem to be stored as 0.99 in Couchbase, and both are read back as 0.99. If Couchbase stores numbers as doubles in Java, should have seen a loss of precision in the storage (console) and reading back in BigDecimal. Instead it looks like some rounding is going on…

Anyone can shed some light on what’s going on? How come from the bdDouble=0.9899999999999999911182158029987476766109466552734375, couchbase stores 0.99 and returns it as such?

thanks.

2

Hi Ranjix,

How does couchbase store numbers?

A Couchbase document is fundamentally just an array of bytes. If that byte array happens to look like JSON then Couchbase Server will parse and index it. But in in terms of storage and retrieval, it’s just a blob.

If you’re storing JSON documents, numbers are stored in their textual representation. The range of numbers you can store is limited only by the libraries you use for generating and parsing JSON. You may want to look at RFC-8259 which describes the limitations of “interoperable” JSON.

How come from the bdDouble=0.9899999999999999911182158029987476766109466552734375, couchbase stores 0.99 and returns it as such?

Short answer: It’s not Couchbase Server, it’s Jackson (the JSON library used by the Java client). The rounding happens during JSON deserialization.

Couchbase Server always stores exactly what the client sends it. Your JSON is being modified before it’s sent to Couchbase. The culprit is JsonObject.fromJson. This method uses Jackson to deserialize the JSON into a Couchbase JsonObject. In the absense of additional type information, Jackson’s default behavior is to parse a floating point value as a double. So when you call JsonObject.fromJson, the value 0.9899999999999999911182158029987476766109466552734375 gets rounded to 0.99 which is why you see 0.99 in Couchbase.

Since you already have a JSON string, you can (and probably should!) avoid the call to JsonObject.fromJson. Instead of creating a JsonDocument, create a RawJsonDocument which passes whatever String you give it directly to Couchbase:

RawJsonDocument document = RawJsonDocument.create("bigdecimal::1", json);

Now when you inspect the document in the Couchbase web UI you should see all those extra digits.

If you read the document back as a JsonDocument, you’ll see the value 0.99 because once again Jackson’s default parsing is happening. If you want to avoid that, read the document as a RawJsonDocument and do your own JSON parsing:

RawJsonDocument document = bucket.get(RawJsonDocument.create("bigdecimal::1"));
String json = document.content();

ObjectMapper mapper = new ObjectMapper();
mapper.enable(DeserializationFeature.USE_BIG_DECIMAL_FOR_FLOATS);

JsonNode parsed = mapper.readTree(json);
BigDecimal result = parsed.path("bdDouble").decimalValue();
System.out.println("After round trip: " + result);

(For best performance, real code should reuse & share a single ObjectMapper instance.)

This story has two morals to choose from:

  1. If you need to store floating point values in JSON with exact precision, it might be safer to use Strings instead of numbers.
  2. If you want fine-grained control over JSON generation and parsing, use RawJsonDocument.

Cheers,
David

1 Like
3

thanks David, appreciate your thorough answer. Several notes:

  • the 0.99 cannot be represented “precisely” as any floating point (on 32 or 64 bits). So still unclear the “binary representation” of whatever couchbase is storing - the “best” way of representing 0.99 as a floating point is that long 0.9899…375 value, so maybe that is the actually binary representation. But, if this is the case, unclear why Jackson would round anything. What was surprising was that I expected to see the long version in Couchbase and then read back. Instead I see the rounded version in Couchbase and same rounded being read back
  • in this particular case I like the 0.99 rounding, I work with prices and this rounding works. What was unclear was the limitations of this “working”, there will be probably situations when I won’t like how it rounded. Same issue every time one needs to store decimals (no loss of precision) instead of floating points (loss of precision)
  • I appreciate the suggestions about the RawJsonDocument and storing as strings, unfortunately that won’t work for the queries we do (ranges). Maybe they could work if all these numbers are formatted the same (so String comparisons would work on numbers), but that’s some extra String <-> number conversions on all operations, which I assume will be slower.

Thanks again…

4

So still unclear the “binary representation” of whatever couchbase is storing

Couchbase is storing the text of your JSON document.

in this particular case I like the 0.99 rounding,

Then you should not use the BigDecimal constructor that takes a double. The Javadoc for that constructor explains why the resulting value is not exactly 0.99.

I work with prices

It’s notoriously difficult to store currency values in JSON. If using strings is not an option, then the next best solution is to use fixed point values (instead of 0.99 you’d store the integer value 99). Watch out though, because you can still lose precision if the number is too large for the JSON library to parse correctly.

1 Like