Crash with attachment after upgrading from CBL 1.4 to 2.6

Mobile Couchbase Lite
1

Android 9
CBL 2.6

App in production used CBL 1.4. Now an update rolled out to beta users with CBL 2.6 and a when parsing a document with attachment a crash occurs.

The document is:

{
  "barcodes": [],
  "category": {
    "id": "feacf1232d19",
    "name": "épicerie"
  },
  "channels": "CacoWfYv9STcbFGuCyfVjr23M1S2",
  "id": "0a6d44633aed",
  "name": "cornichons frais du Val de Loire au vinaigre d'Orleans à l'ancienne",
  "type": "item",
  "unit": {
    "id": "GENERAL",
    "name": ""
  }
}

Meta data:

{
  "meta": {
    "id": "CacoWfYv9STcbFGuCyfVjr23M1S2::item::0a6d44633aed",
    "rev": "54928-15cb6efd705700000000000000000000",
    "expiration": 0,
    "flags": 0,
    "type": "json"
  },
  "xattrs": {
    "_sync": {
      "rev": "5-6b6d1fbe10392a26ddce597e29b00145",
      "sequence": 140113591,
      "recent_sequences": [
        138365203,
        138365264,
        138365287,
        138368527,
        140113591
      ],
      "history": {
        "revs": [
          "3-b77d61ad4108dd7dad1563d413d53a0e",
          "1-e1d63ba738360ff2c1b0b50e1c40428a",
          "2-194ea30678c2fc70733912b8dd989740",
          "4-fd531c1c8f7f838e045b25db016436a1",
          "5-6b6d1fbe10392a26ddce597e29b00145"
        ],
        "parents": [
          2,
          -1,
          1,
          0,
          3
        ],
        "channels": [
          [
            "CacoWfYv9STcbFGuCyfVjr23M1S2"
          ],
          [
            "CacoWfYv9STcbFGuCyfVjr23M1S2"
          ],
          [
            "CacoWfYv9STcbFGuCyfVjr23M1S2"
          ],
          [
            "CacoWfYv9STcbFGuCyfVjr23M1S2"
          ],
          [
            "CacoWfYv9STcbFGuCyfVjr23M1S2"
          ]
        ]
      },
      "channels": {
        "CacoWfYv9STcbFGuCyfVjr23M1S2": null
      },
      "cas": "0x00005770fd6ecb15",
      "value_crc32c": "0x0d51f271",
      "attachments": {
        "1569666393740": {
          "content_type": "image/jpeg",
          "digest": "sha1-EwD/0Bd/vNzukTyKL8k0skwuQYk=",
          "length": 217957,
          "revpos": 2,
          "stub": true
        }
      },
      "time_saved": "2019-10-07T19:55:29.856862484+02:00"
    }
  }
}

Stacktraces:

#0. Crashed: main
       at com.fasterxml.jackson.databind.ObjectMapper._convert + 3750(ObjectMapper.java:3750)
       at com.fasterxml.jackson.databind.ObjectMapper.convertValue + 3668(ObjectMapper.java:3668)
...
Fatal Exception: java.lang.IllegalArgumentException: Failed to obtain content from BlobStore. digest=sha1-EwD/0Bd/vNzukTyKL8k0skwuQYk= (through reference chain: java.util.HashMap["_attachments"]->java.util.HashMap["1569666393740"]->com.couchbase.lite.Blob["content"])
       at com.fasterxml.jackson.databind.ObjectMapper._convert + 3750(ObjectMapper.java:3750)
       at com.fasterxml.jackson.databind.ObjectMapper.convertValue + 3668(ObjectMapper.java:3668)
...
Caused by com.fasterxml.jackson.databind.JsonMappingException: Failed to obtain content from BlobStore. digest=sha1-EwD/0Bd/vNzukTyKL8k0skwuQYk= (through reference chain: java.util.HashMap["_attachments"]->java.util.HashMap["1569666393740"]->com.couchbase.lite.Blob["content"])
       at com.fasterxml.jackson.databind.ser.std.StdSerializer.wrapAndThrow + 316(StdSerializer.java:316)
       at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields + 727(BeanSerializerBase.java:727)
       at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize + 155(BeanSerializer.java:155)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields + 722(MapSerializer.java:722)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize + 643(MapSerializer.java:643)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize + 33(MapSerializer.java:33)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields + 722(MapSerializer.java:722)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize + 643(MapSerializer.java:643)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize + 33(MapSerializer.java:33)
       at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize + 480(DefaultSerializerProvider.java:480)
       at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue + 319(DefaultSerializerProvider.java:319)
       at com.fasterxml.jackson.databind.ObjectMapper._convert + 3728(ObjectMapper.java:3728)
       at com.fasterxml.jackson.databind.ObjectMapper.convertValue + 3668(ObjectMapper.java:3668)
...
Caused by java.lang.IllegalStateException: Failed to obtain content from BlobStore. digest=sha1-EwD/0Bd/vNzukTyKL8k0skwuQYk=
       at com.couchbase.lite.Blob.getBytesFromDatabase + 588(Blob.java:588)
       at com.couchbase.lite.Blob.getContent + 300(Blob.java:300)
       at java.lang.reflect.Method.invoke(Method.java)
       at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField + 688(BeanPropertyWriter.java:688)
       at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields + 719(BeanSerializerBase.java:719)
       at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize + 155(BeanSerializer.java:155)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields + 722(MapSerializer.java:722)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize + 643(MapSerializer.java:643)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize + 33(MapSerializer.java:33)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields + 722(MapSerializer.java:722)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize + 643(MapSerializer.java:643)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize + 33(MapSerializer.java:33)
       at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize + 480(DefaultSerializerProvider.java:480)
       at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue + 319(DefaultSerializerProvider.java:319)
       at com.fasterxml.jackson.databind.ObjectMapper._convert + 3728(ObjectMapper.java:3728)
       at com.fasterxml.jackson.databind.ObjectMapper.convertValue + 3668(ObjectMapper.java:3668)
...
Caused by com.couchbase.lite.LiteCoreException: No such file or directory
       at com.couchbase.lite.internal.core.C4BlobStore.getContents(C4BlobStore.java)
       at com.couchbase.lite.internal.core.C4BlobStore.getContents + 102(C4BlobStore.java:102)
       at com.couchbase.lite.Blob.getBytesFromDatabase + 572(Blob.java:572)
       at com.couchbase.lite.Blob.getContent + 300(Blob.java:300)
       at java.lang.reflect.Method.invoke(Method.java)
       at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField + 688(BeanPropertyWriter.java:688)
       at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields + 719(BeanSerializerBase.java:719)
       at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize + 155(BeanSerializer.java:155)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields + 722(MapSerializer.java:722)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize + 643(MapSerializer.java:643)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize + 33(MapSerializer.java:33)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields + 722(MapSerializer.java:722)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize + 643(MapSerializer.java:643)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize + 33(MapSerializer.java:33)
       at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize + 480(DefaultSerializerProvider.java:480)
       at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue + 319(DefaultSerializerProvider.java:319)
       at com.fasterxml.jackson.databind.ObjectMapper._convert + 3728(ObjectMapper.java:3728)
       at com.fasterxml.jackson.databind.ObjectMapper.convertValue + 3668(ObjectMapper.java:3668)

User reports that this crash occured after the update without the user doing anything. I created a backup of the cluster, imported it on my dev cluster, signed in as the user and I’m able to parse the document and to view the attachment/blob/image.

Does anyone have an idea what might have happened. Is there a possibility that it is a bug in CBL? Is it crashing because the info of the attachment is in the meta data but the binary file of the image cannot be found on the device? What could be an immediate workaround to stop the crash from happening?
So far this happened to only one user. Once it will be rolled out to all users - not just beta users - there might be many, many more crashes.

Thanks!

2

User confirmed that the older release with CBL 1.4 on a different device which uses the same account to sync the user’s data does not have this issue. The image can be viewed.
I published another app update (with CBL 2.6) and will report back if all documents with an attachment would crash, or if this issue is only with a single document.

3

@benjamin_glatzeder: Is this happening consistently? Does it happen with every document that contains a blob, or just some?

CBL stores blobs in files, not in the DB. The DB just contains a reference to the file. Here, it looks as though the file containing the blob cannot be located.

Android 9 has, yet again, changed how file system access works. Have you, by any chance, tried this with any version other than 9?

4

From the first user report as much as I can remember this happened with 1 document. I believe this user had more than 1 picture. The issue only appeared for a tiny, tiny part of the userbase. But then again not everyone uses the picture feature as it’s optional. It appeared on 3 Android versions so far. Here are the last 30 days with this issue:

crash-stats
crash-stats.png1464×549 24.1 KB

I believe this happens after upgrading from CBL 1.4 to CBL 2.6. So either the issue went away on its own or users deleted the app. That’s why there are spikes which bottom out quickly.

6

We’ve filed a bug for this, here: https://issues.couchbase.com/browse/CBL-608

7

I have an update on this issue. It still occurs with CBL 2.7 on Android and SG 2.7. The original post described that it might be a sync issue, e.g. the meta data is pushed down to the client but not the blob itself. I still believe that this is true.

But just now I had a different user report and maybe this issue also appears after database compaction. I run the compaction at least once per day.

Document:

{
  "channels": "uuLcQeNyEif5fVVC8e5ME0dIGYU2",
  "groupMembers": [],
  "id": "uuLcQeNyEif5fVVC8e5ME0dIGYU2",
  "name": "",
  "type": "group"
}

Meta data:


  "meta": {
    "id": "uuLcQeNyEif5fVVC8e5ME0dIGYU2::group::uuLcQeNyEif5fVVC8e5ME0dIGYU2",
    "rev": "37072-15d13ca4bc1900000000000000000000",
    "expiration": 0,
    "flags": 0,
    "type": "json"
  },
  "xattrs": {
    "_sync": {
      "rev": "5-3e994dd7d4c8559452d7d141d162ea18f56bbf48",
      "sequence": 143550103,
      "recent_sequences": [
        83252104,
        83252602,
        83252674,
        87283954,
        143550103
      ],
      "history": {
        "revs": [
          "5-3e994dd7d4c8559452d7d141d162ea18f56bbf48",
          "2-0c539ee772f357e64d2a50f504898604",
          "4-e5a6000055991b29c2422612df1eece4",
          "3-2fa04e61a5a28041a4f88e00aa215b2f",
          "1-a0e81857cfb61f5c793cb2397dc79472"
        ],
        "parents": [
          2,
          4,
          3,
          1,
          -1
        ],
        "channels": [
          [
            "uuLcQeNyEif5fVVC8e5ME0dIGYU2"
          ],
          [
            "uuLcQeNyEif5fVVC8e5ME0dIGYU2"
          ],
          [
            "uuLcQeNyEif5fVVC8e5ME0dIGYU2"
          ],
          [
            "uuLcQeNyEif5fVVC8e5ME0dIGYU2"
          ],
          [
            "uuLcQeNyEif5fVVC8e5ME0dIGYU2"
          ]
        ]
      },
      "channels": {
        "uuLcQeNyEif5fVVC8e5ME0dIGYU2": null
      },
      "cas": "0x000019bca43cd115",
      "value_crc32c": "0x21942e9e",
      "attachments": {
        "12b23ab85b7029007f89e9080d5f5c20::small": {
          "content_type": "image/jpeg",
          "digest": "sha1-w23dMWQzf76lt3BE8AnotSLwdyg=",
          "length": 37348,
          "revpos": 3,
          "stub": true
        }
      },
      "time_saved": "2019-10-26T17:40:23.158231307+02:00"
    }
  }
}

Stacktrace

Fatal Exception: java.lang.IllegalArgumentException: Failed to read content from database for digest: sha1-w23dMWQzf76lt3BE8AnotSLwdyg= (through reference chain: java.util.HashMap["_attachments"]->java.util.HashMap["12b23ab85b7029007f89e9080d5f5c20::small"]->com.couchbase.lite.Blob["content"])
       at com.fasterxml.jackson.databind.ObjectMapper._convert + 3751(ObjectMapper.java:3751)
       at com.fasterxml.jackson.databind.ObjectMapper.convertValue + 3669(ObjectMapper.java:3669)
       at com.controller.group.CtrGroupReceiver.parseDocument + 65(CtrGroupReceiver.java:65)
...
Caused by com.fasterxml.jackson.databind.JsonMappingException: Failed to read content from database for digest: sha1-w23dMWQzf76lt3BE8AnotSLwdyg= (through reference chain: java.util.HashMap["_attachments"]->java.util.HashMap["12b23ab85b7029007f89e9080d5f5c20::small"]->com.couchbase.lite.Blob["content"])
       at com.fasterxml.jackson.databind.ser.std.StdSerializer.wrapAndThrow + 316(StdSerializer.java:316)
       at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields + 727(BeanSerializerBase.java:727)
       at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize + 155(BeanSerializer.java:155)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields + 722(MapSerializer.java:722)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize + 643(MapSerializer.java:643)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize + 33(MapSerializer.java:33)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields + 722(MapSerializer.java:722)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize + 643(MapSerializer.java:643)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize + 33(MapSerializer.java:33)
       at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize + 480(DefaultSerializerProvider.java:480)
       at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue + 319(DefaultSerializerProvider.java:319)
       at com.fasterxml.jackson.databind.ObjectMapper._convert + 3729(ObjectMapper.java:3729)
       at com.fasterxml.jackson.databind.ObjectMapper.convertValue + 3669(ObjectMapper.java:3669)
       at com.controller.group.CtrGroupReceiver.parseDocument + 65(CtrGroupReceiver.java:65)
...
Caused by java.lang.IllegalStateException: Failed to read content from database for digest: sha1-w23dMWQzf76lt3BE8AnotSLwdyg=
       at com.couchbase.lite.Blob.getContentFromDatabase + 516(Blob.java:516)
       at com.couchbase.lite.Blob.getContent + 341(Blob.java:341)
       at java.lang.reflect.Method.invoke(Method.java)
       at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField + 688(BeanPropertyWriter.java:688)
       at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields + 719(BeanSerializerBase.java:719)
       at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize + 155(BeanSerializer.java:155)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields + 722(MapSerializer.java:722)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize + 643(MapSerializer.java:643)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize + 33(MapSerializer.java:33)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields + 722(MapSerializer.java:722)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize + 643(MapSerializer.java:643)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize + 33(MapSerializer.java:33)
       at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize + 480(DefaultSerializerProvider.java:480)
       at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue + 319(DefaultSerializerProvider.java:319)
       at com.fasterxml.jackson.databind.ObjectMapper._convert + 3729(ObjectMapper.java:3729)
       at com.fasterxml.jackson.databind.ObjectMapper.convertValue + 3669(ObjectMapper.java:3669)
       at com.controller.group.CtrGroupReceiver.parseDocument + 65(CtrGroupReceiver.java:65)
...
Caused by com.couchbase.lite.LiteCoreException: No such file or directory
       at com.couchbase.lite.internal.core.C4BlobStore.getContents + 109(C4BlobStore.java:109)
       at com.couchbase.lite.internal.core.C4BlobStore.getContents + 109(C4BlobStore.java:109)
       at com.couchbase.lite.Blob.getContentFromDatabase + 509(Blob.java:509)
       at com.couchbase.lite.Blob.getContent + 341(Blob.java:341)
       at java.lang.reflect.Method.invoke(Method.java)
       at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField + 688(BeanPropertyWriter.java:688)
       at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields + 719(BeanSerializerBase.java:719)
       at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize + 155(BeanSerializer.java:155)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields + 722(MapSerializer.java:722)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize + 643(MapSerializer.java:643)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize + 33(MapSerializer.java:33)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields + 722(MapSerializer.java:722)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize + 643(MapSerializer.java:643)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize + 33(MapSerializer.java:33)
       at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize + 480(DefaultSerializerProvider.java:480)
       at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue + 319(DefaultSerializerProvider.java:319)
       at com.fasterxml.jackson.databind.ObjectMapper._convert + 3729(ObjectMapper.java:3729)
       at com.fasterxml.jackson.databind.ObjectMapper.convertValue + 3669(ObjectMapper.java:3669)
       at com.controller.group.CtrGroupReceiver.parseDocument + 65(CtrGroupReceiver.java:65)

The user who sent in the report is an avid user and uses the app several times per week. The document was last updated a couple month ago (2019-10-26). That’s why I think there might also be a different root cause. Either database compaction or some other housekeeping task. Currently I’m waiting to hear back if the user made any edits to the document recently.

Also is there a way to fail gracefully? I’d need the document but I can live without the blob.

8

At the Couchbase Lite level I don’t think this is fatal, although java.lang.IllegalStateException doesn’t seem like the right exception class to use. The “Fatal Exception” is coming from the Jackson library. Maybe @blake.meike knows how to avoid that.

9

You may be right. One of the things compaction does is garbage-collect attachment files, deleting any that are no longer referenced by any documents. If there were a bug in that code, it could cause a valid attachment file to be deleted, causing this problem. I’ll take a look.

10

The exception that is causing this is:
Caused by com.couchbase.lite.LiteCoreException: No such file or directory

I believe that this exception is being thrown by the checkErr method in FileReadStream::read. There are a couple ways that could happen. The file name or the directory path, in the key/store objects could be corrupt. … or the file could actually be gone…

Unfortunately, the public API for the method Blob.getContent() does not include a checked exception. The only choice I have, when Core fails, is to throw some unchecked exception. The one we typically use is that IllegalStateException. I agree that it is not very helpful but it does chain the actual culprit.

11

Oops. As a workaround (before 3.0) maybe you could add another method that does the same thing but throws a checked exception?

12

Thank you for looking into it!

The user assured me that they made no edits here. Also after reinstalling the app, signing in and starting a one-shot-pull sync all data was downloaded including blobs. No more crashes for this particular user.

13

I’m looking through the code now (Database::collectBlobs, in Database.cc in couchbase-lite-core).

Do you know anything about the particulars of the document that lost its blob? Like where the blob reference is located in the JSON structure, or whether the doc had a conflict?

When you call Database.compact, do you access the database concurrently (through another instance) or just wait till the compaction finishes?

14

I use an AsyncTask to run Database.compact. My utility class to access all database methods is a singleton. Thus it’s the same database instance. There’s only one for the whole application.
While the compaction is running the app displays a progress dialog. Mostly this task only lasts a few hundred milliseconds. Still there is other database housekeeping code running on the main thread which is called right after the AsyncTask is started. And the AsyncTask and compaction might start running before it or not. And very likely might be running while the other housekeeping tasks run on the main thread. Not sure why I went AsyncTask for the compaction and main thread for similar tasks. But that’s what I see in the code.

Here’s how I save blobs:

mutableDocument.setBlob("<blob-id>",
                        new Blob("image/jpeg", org.apache.commons.io.IOUtils.toByteArray(new ByteArrayInputStream(byteArray))));

The location reference should be in the meta data posted above. Here is an excerpt:

"attachments": {
        "12b23ab85b7029007f89e9080d5f5c20::small": {
          "content_type": "image/jpeg",
          "digest": "sha1-w23dMWQzf76lt3BE8AnotSLwdyg=",
          "length": 37348,
          "revpos": 3,
          "stub": true
        }
      }

Here’s another document from a different user with attachment from CBL 1.x. There the location reference is in the document itself:

  {
    "id": "7JFlD2OEV7U2ZBYiBXCC20aEaIL2::group::7JFlD2OEV7U2ZBYiBXCC20aEaIL2",
    "lister_production_bucket": {
      "_attachments": {
        "40394571ed67e486efeb24442358a4dd::small": {
          "content_type": "image/jpeg",
          "digest": "sha1-mTIaPpVTmY3YamtJVG07wQZ9bn0=",
          "length": 60011,
          "revpos": 2,
          "stub": true
        }
      },
      "channels": "7JFlD2OEV7U2ZBYiBXCC20aEaIL2",
      "groupMembers": [],
      "id": "7JFlD2OEV7U2ZBYiBXCC20aEaIL2",
      "name": "",
      "type": "group"
    }
  }

I’d need to check in with the user if they have more than one device and hence there could have been a conflict. As far as I can remember this particular type of document is created once the user signs up to the sync service. It’s only updated if they change their profile picture. This document was last updated on: 2019-10-26T17:40:43.56415477+02:00 (That’s 20 seconds after the other document was updated. I’m guessing the time_saved is the (Couchbase) server time. Otherwise that’s quite a big difference although these documents should be updated one after the other.)

I went through the user emails and the crash reports again. They had trouble with a blob of a different document. The blob was missing. The two documents are totally unrelated. Next time they tried to open the app it crashed with a missing blob from the document shown above (so of type ‘group’). I’d like to make the raw database files available to check if all blobs went missing. But the user reinstalled the app and rooting an Android device to get the database files is a lot to ask for an average person.

15

The JSON key in that excerpt is "attachments", not "_attachments". Was that a copy/paste error? If it were in the database in that form, it wouldn’t be recognized as an attachment by CBL. (And wouldn’t show up as a Blob in the API.)

16

This document is weird too; has it been modified since being fetched from (presumably) SG? The "_attachments: key must be at the top level in the document to be recognized.

Also, both of those attachments look like they were created by SG’s REST API or by CBL 1.x. When CBL 2.x creates a blob, it’s a dictionary anywhere in the document with a special property "@type":"blob" and a mandatory "digest". When uploaded to SG, for compatibility reasons CBL tacks on a top-level "_attachments" property with a shadow entry for each blob; the key for each entry is a JSON path to where the actual blob is. The keys in the attachments you’ve shown don’t look like that.

17

Skip to end for document with the issue

My bad! I must have posted the results of a query! You are correct about the _attchments:key. Here is the full document (doc id = 7JFlD2OEV7U2ZBYiBXCC20aEaIL2::group::7JFlD2OEV7U2ZBYiBXCC20aEaIL2)

{
  "_attachments": {
    "40394571ed67e486efeb24442358a4dd::small": {
      "content_type": "image/jpeg",
      "digest": "sha1-mTIaPpVTmY3YamtJVG07wQZ9bn0=",
      "length": 60011,
      "revpos": 2,
      "stub": true
    }
  },
  "channels": "7JFlD2OEV7U2ZBYiBXCC20aEaIL2",
  "groupMembers": [],
  "id": "7JFlD2OEV7U2ZBYiBXCC20aEaIL2",
  "name": "",
  "type": "group"
}

---

{
  "meta": {
    "id": "7JFlD2OEV7U2ZBYiBXCC20aEaIL2::group::7JFlD2OEV7U2ZBYiBXCC20aEaIL2",
    "rev": "36620-15979b80d97d00000000000000000000",
    "expiration": 0,
    "flags": 0,
    "type": "json"
  },
  "xattrs": {
    "_sync": {
      "rev": "3-63ebe1c7e77b2e5b5808788678eeac3c",
      "sequence": 82468967,
      "recent_sequences": [
        82158911,
        82468934,
        82468967
      ],
      "history": {
        "revs": [
          "2-6cebe022093547235f48a5b89fba94f6",
          "3-63ebe1c7e77b2e5b5808788678eeac3c",
          "1-7a14cfdc02b0c94b042900e8fd2d3774"
        ],
        "parents": [
          2,
          0,
          -1
        ],
        "channels": [
          [
            "7JFlD2OEV7U2ZBYiBXCC20aEaIL2"
          ],
          [
            "7JFlD2OEV7U2ZBYiBXCC20aEaIL2"
          ],
          [
            "7JFlD2OEV7U2ZBYiBXCC20aEaIL2"
          ]
        ]
      },
      "channels": {
        "7JFlD2OEV7U2ZBYiBXCC20aEaIL2": null
      },
      "cas": "0x00007dd9809b9715",
      "value_crc32c": "0xdaa6d9ed",
      "time_saved": "2018-11-20T10:16:26.629403765+01:00"
    }
  }
}

I created a new document with CBL 2.x and see what you described:

{
  "c4db3394d5c8c43e8657089d1d191cf9::small": {
    "@type": "blob",
    "content_type": "image/jpeg",
    "digest": "sha1-AknVFL9mI3E1TL3bU5hUg1atRIA=",
    "length": 39345
  },
  "channels": "tB4juKcz3cUvBOpBvz5Rg0lucGs1",
  "groupMembers": [],
  "id": "tB4juKcz3cUvBOpBvz5Rg0lucGs1",
  "name": "",
  "type": "group"
}

---

{
  "meta": {
    "id": "tB4juKcz3cUvBOpBvz5Rg0lucGs1::group::tB4juKcz3cUvBOpBvz5Rg0lucGs1",
    "rev": "65501-15f445899abb00000000000000000000",
    "expiration": 0,
    "flags": 0,
    "type": "json"
  },
  "xattrs": {
    "_sync": {
      "rev": "2-e0b5687616accf7e75ef36f6d190321c98c6a4f4",
      "sequence": 163430513,
      "recent_sequences": [
        163430507,
        163430513
      ],
      "history": {
        "revs": [
          "1-b183c20be16f8e6c04b1efca8f8d13271280fe4f",
          "2-e0b5687616accf7e75ef36f6d190321c98c6a4f4"
        ],
        "parents": [
          -1,
          0
        ],
        "channels": [
          [
            "tB4juKcz3cUvBOpBvz5Rg0lucGs1"
          ],
          [
            "tB4juKcz3cUvBOpBvz5Rg0lucGs1"
          ]
        ]
      },
      "channels": {
        "tB4juKcz3cUvBOpBvz5Rg0lucGs1": null
      },
      "cas": "0x0000bb9a8945f415",
      "value_crc32c": "0xce405198",
      "attachments": {
        "blob_/c4db3394d5c8c43e8657089d1d191cf9::small": {
          "content_type": "image/jpeg",
          "digest": "sha1-AknVFL9mI3E1TL3bU5hUg1atRIA=",
          "length": 39345,
          "revpos": 2,
          "stub": true
        }
      },
      "time_saved": "2020-02-17T19:57:06.421592433+01:00"
    }
  }
}

Document with issue
As you describe there must be an issue with the document from the user who experienced a crash. Here’s the document and meta data again:

{
  "channels": "uuLcQeNyEif5fVVC8e5ME0dIGYU2",
  "groupMembers": [],
  "id": "uuLcQeNyEif5fVVC8e5ME0dIGYU2",
  "name": "",
  "type": "group"
}

---

{
  "meta": {
    "id": "uuLcQeNyEif5fVVC8e5ME0dIGYU2::group::uuLcQeNyEif5fVVC8e5ME0dIGYU2",
    "rev": "37072-15d13ca4bc1900000000000000000000",
    "expiration": 0,
    "flags": 0,
    "type": "json"
  },
  "xattrs": {
    "_sync": {
      "rev": "5-3e994dd7d4c8559452d7d141d162ea18f56bbf48",
      "sequence": 143550103,
      "recent_sequences": [
        83252104,
        83252602,
        83252674,
        87283954,
        143550103
      ],
      "history": {
        "revs": [
          "5-3e994dd7d4c8559452d7d141d162ea18f56bbf48",
          "2-0c539ee772f357e64d2a50f504898604",
          "4-e5a6000055991b29c2422612df1eece4",
          "3-2fa04e61a5a28041a4f88e00aa215b2f",
          "1-a0e81857cfb61f5c793cb2397dc79472"
        ],
        "parents": [
          2,
          4,
          3,
          1,
          -1
        ],
        "channels": [
          [
            "uuLcQeNyEif5fVVC8e5ME0dIGYU2"
          ],
          [
            "uuLcQeNyEif5fVVC8e5ME0dIGYU2"
          ],
          [
            "uuLcQeNyEif5fVVC8e5ME0dIGYU2"
          ],
          [
            "uuLcQeNyEif5fVVC8e5ME0dIGYU2"
          ],
          [
            "uuLcQeNyEif5fVVC8e5ME0dIGYU2"
          ]
        ]
      },
      "channels": {
        "uuLcQeNyEif5fVVC8e5ME0dIGYU2": null
      },
      "cas": "0x000019bca43cd115",
      "value_crc32c": "0x21942e9e",
      "attachments": {
        "12b23ab85b7029007f89e9080d5f5c20::small": {
          "content_type": "image/jpeg",
          "digest": "sha1-w23dMWQzf76lt3BE8AnotSLwdyg=",
          "length": 37348,
          "revpos": 3,
          "stub": true
        }
      },
      "time_saved": "2019-10-26T17:40:23.158231307+02:00"
    }
  }
}

There is no _attachments key at the top of the JSON and no "@type": "blob". Finally the attachments key in the meta data does not have a valid path, e.g. "blob_/12b23ab85b7029007f89e9080d5f5c20::small. It’s only 12b23ab85b7029007f89e9080d5f5c20::small and it’s missing blob_/.

If I run this query in Couchbase server I see many which have the path missing, i.e. without blob_/

SELECT meta().xattrs._sync.attachments
FROM my_bucket
WHERE type = 'group'
LIMIT 100;

  {
    "attachments": {
      "6b63ca59a6b3145c2290f7c4f5d0a7d0::small": {
        "content_type": "image/jpeg",
        "digest": "sha1-wQZjX8S/V+ghlflW/MW4bRxZH7o=",
        "length": 31091,
        "revpos": 3,
        "stub": true
      }
    }
  },
...

Would I need to find them all and update them? I’d update them via CBLite.

Finally user reported back that they only use it on a single device. So no conflict.

18

I signed in as another user who has the same “_attachments/attachment” problem. There was no issue. No crash. Picture shows up fine.

The document is

{
  "channels": "03HA...,
  "email": "<email>,
  "id": "03HA...",
  "name": "user name",
  "password": "f53d...",
  "type": "userProfile"
}

---

{
  "meta": {
    "id": "03HA...::userProfile",
    "rev": "37014-15c8e001d09500000000000000000000",
    "expiration": 0,
    "flags": 0,
    "type": "json"
  },
  "xattrs": {
    "_sync": {
      "rev": "6-254c9b770d09c779ee53c15c63c80e9a",
      "sequence": 138564133,
      "recent_sequences": [
        85101511,
        85108109,
        88438161,
        102656915,
        124732450,
        138564133
      ],
      "history": {
        "revs": [
          "3-6b028d28f2ebbd5c351fb566d93afc31",
          "6-254c9b770d09c779ee53c15c63c80e9a",
          "5-915af92377826fa7f684a4055f9ea23b",
          "4-3cf17ff6cc9a061ed9c18a748ace1c16",
          "2-20483f20d17c7f28c164ec9a5ce3bd34",
          "1-bf55d270cdcba8f1b204aef3db621887"
        ],
        "parents": [
          4,
          2,
          3,
          0,
          5,
          -1
        ],
        "channels": [
          [
            "03HA..."
          ],
          [
            "03HA..."
          ],
          [
            "03HA..."
          ],
          [
            "03HA..."
          ],
          [
            "03HA..."
          ],
          [
            "03HA..."
          ]
        ]
      },
      "channels": {
        "03HA...": null
      },
      "cas": "0x000095d001e0c815",
      "value_crc32c": "0x672cee70",
      "attachments": {
        "03HA...::large": {
          "content_type": "image/jpeg",
          "digest": "sha1-Xbw8vpz1Fjjm2ALaR7tq6+ecvzo=",
          "length": 40160,
          "revpos": 3,
          "stub": true
        },
        "03HA...::small": {
          "content_type": "image/jpeg",
          "digest": "sha1-SefQD0ClrGBhCjlLwF5vE5bgU3Y=",
          "length": 20037,
          "revpos": 3,
          "stub": true
        }
      },
      "time_saved": "2019-09-29T11:52:48.537052979+02:00"
    }
  }
}

Since the attachments changed between CBL 1 and 2 I retrieve them in two ways:

    public List<Pair<String, Blob>> getAttachments(Document doc) {
        List<Pair<String, Blob>> blobPairs = new ArrayList<>();
        Dictionary attachments = doc.getDictionary("_attachments");
        if (attachments != null)
            for (String s : attachments.getKeys()) {
                blobPairs.add(new Pair<>(s, attachments.getBlob(s)));
            }
        return blobPairs;
    }

    public List<Pair<String, Blob>> getBlobs(Document doc) {
        List<Pair<String, Blob>> blobPairs = new ArrayList<>();
        for (String key : doc.getKeys()) {
            Blob blob = doc.getBlob(key);
            if (blob != null) {
                blobPairs.add(new Pair<>(key, blob));
            }
        }
        return blobPairs;
    }

Getting the attachments with getAttachments(doc) prints this:
[Pair{03HA...::large Blob[image/jpeg; 39 KB]}, Pair{03HA...::small Blob[image/jpeg; 20 KB]}]

I don’t know why Dictionary attachments = doc.getDictionary("_attachments"); works although it’s only attachments without the underscore in the meta data.

As far as I know there was no automatic upgrade from attachments to blobs in any document when upgrading to CBL 2. That’s why I use those two methods. Also I must believe that I don’t retrieve any attachment/blob when parsing a document to a class via Jackson. It’s odd that this exception showed up at all pointing to the invalid path/missing file.

19

To the last document. I found the location of the attachments. The digests from above were

"digest": "sha1-Xbw8vpz1Fjjm2ALaR7tq6+ecvzo="
and
"digest": "sha1-SefQD0ClrGBhCjlLwF5vE5bgU3Y="

The paths are:

/data/data/com.my.app/files/my_database.cblite2/SefQD0ClrGBhCjlLwF5vE5bgU3Y=.blob
/data/data/com.my.app/files/my_database.cblite2/Xbw8vpz1Fjjm2ALaR7tq6+ecvzo=.blob

Downloading them to my computer and replacing the extensions .blob with .jpg shows the pictures.

20

There is no _attachments key at the top of the JSON and no "@type": "blob"

Hm, this is sort of confusing to me because you’re looking at the documents on Sync Gateway, and through something other than the REST API. I’m not familiar with the current internal form of documents in SG now that it uses Couchbase metadata, but it looks as though the attachments do not show up in the document itself, only in the metadata. Meaning that this document is valid, I think.

Finally the attachments key in the meta data does not have a valid path

The attachment was probably created by CBL 1.x, or by the SG REST API. If the document body does not contain a "@type":"blob" property anywhere, it was definitely CBL 1.x or SG.

21

Today I got another user report and can confirm that attachments also get lost when a user does not use the sync feature. The sync feature in my app is optional.

The stacktraces are just the same as above besides the digest name:

Caused by com.couchbase.lite.LiteCoreException: No such file or directory
       at com.couchbase.lite.internal.core.C4BlobStore.getContents + 109(C4BlobStore.java:109)
       at com.couchbase.lite.internal.core.C4BlobStore.getContents + 109(C4BlobStore.java:109)
       at com.couchbase.lite.Blob.getContentFromDatabase + 509(Blob.java:509)
       at com.couchbase.lite.Blob.getContent + 341(Blob.java:341)
       at java.lang.reflect.Method.invoke(Method.java)
       at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField + 688(BeanPropertyWriter.java:688)
       at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields + 719(BeanSerializerBase.java:719)
       at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize + 155(BeanSerializer.java:155)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields + 722(MapSerializer.java:722)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize + 643(MapSerializer.java:643)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize + 33(MapSerializer.java:33)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields + 722(MapSerializer.java:722)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize + 643(MapSerializer.java:643)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize + 33(MapSerializer.java:33)
       at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize + 480(DefaultSerializerProvider.java:480)
       at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue + 319(DefaultSerializerProvider.java:319)
       at com.fasterxml.jackson.databind.ObjectMapper._convert + 3729(ObjectMapper.java:3729)
       at com.fasterxml.jackson.databind.ObjectMapper.convertValue + 3669(ObjectMapper.java:3669)
       at com.controller.overview.CtrOverviewRecipeReceiver.parseDocument + 164(CtrOverviewRecipeReceiver.java:164)
       at com.controller.overview.CtrOverviewRecipeReceiver.parse + 148(CtrOverviewRecipeReceiver.java:148)

Caused by java.lang.IllegalStateException: Failed to read content from database for digest: sha1-ZGT89Gx/6CYqS8wQYXwJ57Y2sdU=
       at com.couchbase.lite.Blob.getContentFromDatabase + 516(Blob.java:516)
       at com.couchbase.lite.Blob.getContent + 341(Blob.java:341)
       at java.lang.reflect.Method.invoke(Method.java)
       at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField + 688(BeanPropertyWriter.java:688)
       at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields + 719(BeanSerializerBase.java:719)
       at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize + 155(BeanSerializer.java:155)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields + 722(MapSerializer.java:722)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize + 643(MapSerializer.java:643)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize + 33(MapSerializer.java:33)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields + 722(MapSerializer.java:722)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize + 643(MapSerializer.java:643)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize + 33(MapSerializer.java:33)
       at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize + 480(DefaultSerializerProvider.java:480)
       at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue + 319(DefaultSerializerProvider.java:319)
       at com.fasterxml.jackson.databind.ObjectMapper._convert + 3729(ObjectMapper.java:3729)
       at com.fasterxml.jackson.databind.ObjectMapper.convertValue + 3669(ObjectMapper.java:3669)
       at com.controller.overview.CtrOverviewRecipeReceiver.parseDocument + 164(CtrOverviewRecipeReceiver.java:164)
       at com.controller.overview.CtrOverviewRecipeReceiver.parse + 148(CtrOverviewRecipeReceiver.java:148)

For non-sync user I’ll try to delete the reference to the attachment. Then at least the document can be parsed. The attachment seems to be gone for good anyways so no need to worry about it.

For sync users I’ll probably show a dialog to re-sync as the attachment is most likely still on the Couchbase server cluster. Is there an “easy” way to re-sync. So far the user would have to reinstall the app and sign in again. Is there any chance to tell Couchbase Lite to redownload a specific document? Could I purge the document and then it would get downloaded on its own?