Crash in JNI from Android SDK: std::bad_alloc

jeff.lockhart · July 30, 2020, 2:46am

While putting together some benchmarks in a fork of Realm’s benchmarks, I ran into a crash in Couchbase on Android. It happens every time I run the benchmarks with ./gradlew clean benchmarkReport on the couchbase branch.

After running for several minutes, it crashes consistently in one of two places, either from Database.getDocument() or Database.save() .

Here are the functions in the CouchbaseBenchmarks code that cause the crash:

private fun addObjects() {
    val list: MutableList<MutableDocument> = mutableListOf()
    for (i in 0 until size) {
        list.add(MutableDocument(i.toString(), mapOf(
                NAME to dataGenerator.getEmployeeName(i),
                AGE to dataGenerator.getEmployeeAge(i),
                HIRED to dataGenerator.getHiredBool(i)
        )))
    }
    db.inBatch {
        list.forEach { db.save(it) }
    }
}

private fun deleteObjects() {
    val query = QueryBuilder.select(SelectResult.expression(Meta.id))
            .from(DataSource.database(db))
    val results = query.execute()
    for (result in results) {
        val id = result.getString(0)
        db.getDocument(id)?.let {
            db.delete(it)
        }
    }
}

These are the two crash logs:

E/libc++abi: terminating with uncaught exception of type St9bad_alloc: std::bad_alloc
A/libc: Fatal signal 6 (SIGABRT), code -1 (SI_QUEUE) in tid 5197 (BenchmarkRunner), pid 5176 (.benchmark.test)
A/DEBUG: *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
A/DEBUG: Build fingerprint: 'google/coral/coral:11/RPB2.200611.012/6677315:user/release-keys'
A/DEBUG: Revision: 'MP1.0'
A/DEBUG: ABI: 'arm64'
A/DEBUG: Timestamp: 2020-07-29 17:35:57-0600
A/DEBUG: pid: 5176, tid: 5197, name: BenchmarkRunner  >>> io.realm.benchmark.test <<<
A/DEBUG: uid: 10560
A/DEBUG: signal 6 (SIGABRT), code -1 (SI_QUEUE), fault addr --------
A/DEBUG: Abort message: 'terminating with uncaught exception of type St9bad_alloc: std::bad_alloc'
A/DEBUG:     x0  0000000000000000  x1  000000000000144d  x2  0000000000000006  x3  0000007722278020
A/DEBUG:     x4  fefefefefefefeff  x5  fefefefefefefeff  x6  fefefefefefefeff  x7  7f7f7f7f7f7f7f7f
A/DEBUG:     x8  00000000000000f0  x9  9d4b7cecfdd9c274  x10 0000000000000000  x11 ffffffc0fffffbdf
A/DEBUG:     x12 0000000000000001  x13 00000a9c01ede89d  x14 0004b854489b1cad  x15 00000000116bc15c
A/DEBUG:     x16 0000007a1d74dc80  x17 0000007a1d72f230  x18 0000007721c36000  x19 0000000000001438
A/DEBUG:     x20 000000000000144d  x21 00000000ffffffff  x22 0000007a1d750f28  x23 0000007722278190
A/DEBUG:     x24 0000007722278150  x25 00000000166353d0  x26 0000000000000000  x27 000000772227f000
A/DEBUG:     x28 0000000000000043  x29 00000077222780a0
A/DEBUG:     lr  0000007a1d6e2da0  sp  0000007722278000  pc  0000007a1d6e2dcc  pst 0000000000000000
A/DEBUG: backtrace:
A/DEBUG:       #00 pc 000000000004ddcc  /apex/com.android.runtime/lib64/bionic/libc.so (abort+164) (BuildId: 59204a187019804c969948db46cb393a)
A/DEBUG:       #01 pc 000000000004b10c  /system/lib64/libc++.so (abort_message+248) (BuildId: c70d7498491a09dc9af7e1abb723497e)
A/DEBUG:       #02 pc 000000000004b2d0  /system/lib64/libc++.so (demangling_terminate_handler()+212) (BuildId: c70d7498491a09dc9af7e1abb723497e)
A/DEBUG:       #03 pc 000000000004bdc4  /system/lib64/libc++.so (std::__terminate(void (*)())+12) (BuildId: c70d7498491a09dc9af7e1abb723497e)
A/DEBUG:       #04 pc 000000000004b4e0  /system/lib64/libc++.so (__cxxabiv1::failed_throw(__cxxabiv1::__cxa_exception*)+28) (BuildId: c70d7498491a09dc9af7e1abb723497e)
A/DEBUG:       #05 pc 000000000004b448  /system/lib64/libc++.so (__cxa_throw+112) (BuildId: c70d7498491a09dc9af7e1abb723497e)
A/DEBUG:       #06 pc 000000000004ebb4  /system/lib64/libc++.so (operator new(unsigned long)+96) (BuildId: c70d7498491a09dc9af7e1abb723497e)
A/DEBUG:       #07 pc 000000000059febc  /apex/com.android.art/lib64/libart.so (_jobject* art::Thread::CreateInternalStackTrace<false>(art::ScopedObjectAccessAlreadyRunnable const&) const+56) (BuildId: 73638a3e2fcf5ba85c88424926d725cb)
A/DEBUG:       #08 pc 00000000004c5c88  /apex/com.android.art/lib64/libart.so (art::Throwable_nativeFillInStackTrace(_JNIEnv*, _jclass*)+48) (BuildId: 73638a3e2fcf5ba85c88424926d725cb)
A/DEBUG:       #09 pc 0000000000088210  /apex/com.android.art/javalib/arm64/boot.oat (art_jni_trampoline+144) (BuildId: 18cb61f1f9e3337a6277b77a4573b0b40873dd06)
A/DEBUG:       #10 pc 000000000015e018  /apex/com.android.art/javalib/arm64/boot.oat (java.lang.Throwable.fillInStackTrace+88) (BuildId: 18cb61f1f9e3337a6277b77a4573b0b40873dd06)
A/DEBUG:       #11 pc 0000000002033c94  /memfd:jit-cache (deleted) (offset 0x2000000) (com.couchbase.lite.LiteCoreException.throwException+180)
A/DEBUG:       #12 pc 00000000001337e8  /apex/com.android.art/lib64/libart.so (art_quick_invoke_static_stub+568) (BuildId: 73638a3e2fcf5ba85c88424926d725cb)
A/DEBUG:       #13 pc 00000000001a8a94  /apex/com.android.art/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+228) (BuildId: 73638a3e2fcf5ba85c88424926d725cb)
A/DEBUG:       #14 pc 0000000000552dd4  /apex/com.android.art/lib64/libart.so (art::JValue art::InvokeWithVarArgs<art::ArtMethod*>(art::ScopedObjectAccessAlreadyRunnable const&, _jobject*, art::ArtMethod*, std::__va_list)+448) (BuildId: 73638a3e2fcf5ba85c88424926d725cb)
A/DEBUG:       #15 pc 0000000000553288  /apex/com.android.art/lib64/libart.so (art::JValue art::InvokeWithVarArgs<_jmethodID*>(art::ScopedObjectAccessAlreadyRunnable const&, _jobject*, _jmethodID*, std::__va_list)+92) (BuildId: 73638a3e2fcf5ba85c88424926d725cb)
A/DEBUG:       #16 pc 00000000003d0374  /apex/com.android.art/lib64/libart.so (art::JNI<false>::CallStaticVoidMethodV(_JNIEnv*, _jclass*, _jmethodID*, std::__va_list)+656) (BuildId: 73638a3e2fcf5ba85c88424926d725cb)
A/DEBUG:       #17 pc 0000000000128720  /data/app/~~SmCj7FXlpKgfDff6SH1tMg==/io.realm.benchmark.test-VfsN0ro4SQIrqNITUiGWag==/lib/arm64/libLiteCoreJNI.so (_JNIEnv::CallStaticVoidMethod(_jclass*, _jmethodID*, ...)+116) (BuildId: d3d24b6aecef3c236dc9f0a93bb3d61ff70c53be)
A/DEBUG:       #18 pc 000000000012a254  /data/app/~~SmCj7FXlpKgfDff6SH1tMg==/io.realm.benchmark.test-VfsN0ro4SQIrqNITUiGWag==/lib/arm64/libLiteCoreJNI.so (Java_com_couchbase_lite_internal_core_C4Document_get+104) (BuildId: d3d24b6aecef3c236dc9f0a93bb3d61ff70c53be)
A/DEBUG:       #19 pc 00000000000160ac  /data/app/~~SmCj7FXlpKgfDff6SH1tMg==/io.realm.benchmark.test-VfsN0ro4SQIrqNITUiGWag==/oat/arm64/base.odex (art_jni_trampoline+188)
A/DEBUG:       #20 pc 0000000002008824  /memfd:jit-cache (deleted) (offset 0x2000000) (com.couchbase.lite.internal.core.C4Database.get+100)
A/DEBUG:       #21 pc 0000000002001eb0  /memfd:jit-cache (deleted) (offset 0x2000000) (com.couchbase.lite.Document.<init>+256)
A/DEBUG:       #22 pc 00000000020080d8  /memfd:jit-cache (deleted) (offset 0x2000000) (com.couchbase.lite.AbstractDatabase.getDocument+184)
A/DEBUG:       #23 pc 000000000201310c  /memfd:jit-cache (deleted) (offset 0x2000000) (io.realm.benchmark.CouchbaseBenchmarks.deleteObjects+1628)
A/DEBUG:       #24 pc 000000000201a5d8  /memfd:jit-cache (deleted) (offset 0x2000000) (io.realm.benchmark.CouchbaseBenchmarks.simpleQuery+3960)
...

E/libc++abi: terminating with uncaught exception of type St9bad_alloc: std::bad_alloc
A/libc: Fatal signal 6 (SIGABRT), code -1 (SI_QUEUE) in tid 28340 (BenchmarkRunner), pid 28313 (.benchmark.test)
A/DEBUG: *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
A/DEBUG: Build fingerprint: 'google/coral/coral:11/RPB2.200611.012/6677315:user/release-keys'
A/DEBUG: Revision: 'MP1.0'
A/DEBUG: ABI: 'arm64'
A/DEBUG: Timestamp: 2020-07-29 05:09:24-0600
A/DEBUG: pid: 28313, tid: 28340, name: BenchmarkRunner  >>> io.realm.benchmark.test <<<
A/DEBUG: uid: 10555
A/DEBUG: signal 6 (SIGABRT), code -1 (SI_QUEUE), fault addr --------
A/DEBUG: Abort message: 'terminating with uncaught exception of type St9bad_alloc: std::bad_alloc'
A/DEBUG:     x0  0000000000000000  x1  0000000000006eb4  x2  0000000000000006  x3  000000772219de70
A/DEBUG:     x4  fefefefefefefeff  x5  fefefefefefefeff  x6  fefefefefefefeff  x7  7f7f7f7f7f7f7f7f
A/DEBUG:     x8  00000000000000f0  x9  9d4b7cecfdd9c274  x10 0000000000000000  x11 ffffffc0fffffbdf
A/DEBUG:     x12 0000000000000001  x13 000009d3c4283c9d  x14 0032f218addce43c  x15 00000000100c8842
A/DEBUG:     x16 0000007a1d74dc80  x17 0000007a1d72f230  x18 0000007721886000  x19 0000000000006e99
A/DEBUG:     x20 0000000000006eb4  x21 00000000ffffffff  x22 0000007a1d750f28  x23 000000772219dfe0
A/DEBUG:     x24 000000772219dfa0  x25 00000077221a5000  x26 000000000000000b  x27 0000000000000002
A/DEBUG:     x28 0000000000000000  x29 000000772219def0
A/DEBUG:     lr  0000007a1d6e2da0  sp  000000772219de50  pc  0000007a1d6e2dcc  pst 0000000000000000
A/DEBUG: backtrace:
A/DEBUG:       #00 pc 000000000004ddcc  /apex/com.android.runtime/lib64/bionic/libc.so (abort+164) (BuildId: 59204a187019804c969948db46cb393a)
A/DEBUG:       #01 pc 000000000004b10c  /system/lib64/libc++.so (abort_message+248) (BuildId: c70d7498491a09dc9af7e1abb723497e)
A/DEBUG:       #02 pc 000000000004b2d0  /system/lib64/libc++.so (demangling_terminate_handler()+212) (BuildId: c70d7498491a09dc9af7e1abb723497e)
A/DEBUG:       #03 pc 000000000004bdc4  /system/lib64/libc++.so (std::__terminate(void (*)())+12) (BuildId: c70d7498491a09dc9af7e1abb723497e)
A/DEBUG:       #04 pc 000000000004b4e0  /system/lib64/libc++.so (__cxxabiv1::failed_throw(__cxxabiv1::__cxa_exception*)+28) (BuildId: c70d7498491a09dc9af7e1abb723497e)
A/DEBUG:       #05 pc 000000000004b448  /system/lib64/libc++.so (__cxa_throw+112) (BuildId: c70d7498491a09dc9af7e1abb723497e)
A/DEBUG:       #06 pc 000000000004ebb4  /system/lib64/libc++.so (operator new(unsigned long)+96) (BuildId: c70d7498491a09dc9af7e1abb723497e)
A/DEBUG:       #07 pc 000000000059febc  /apex/com.android.art/lib64/libart.so (_jobject* art::Thread::CreateInternalStackTrace<false>(art::ScopedObjectAccessAlreadyRunnable const&) const+56) (BuildId: 73638a3e2fcf5ba85c88424926d725cb)
A/DEBUG:       #08 pc 00000000004c5c88  /apex/com.android.art/lib64/libart.so (art::Throwable_nativeFillInStackTrace(_JNIEnv*, _jclass*)+48) (BuildId: 73638a3e2fcf5ba85c88424926d725cb)
A/DEBUG:       #09 pc 0000000000088210  /apex/com.android.art/javalib/arm64/boot.oat (art_jni_trampoline+144) (BuildId: 18cb61f1f9e3337a6277b77a4573b0b40873dd06)
A/DEBUG:       #10 pc 000000000015e018  /apex/com.android.art/javalib/arm64/boot.oat (java.lang.Throwable.fillInStackTrace+88) (BuildId: 18cb61f1f9e3337a6277b77a4573b0b40873dd06)
A/DEBUG:       #11 pc 00000000001f24c0  /apex/com.android.art/javalib/arm64/boot.oat (java.lang.Error.<init> [DEDUPED]+144) (BuildId: 18cb61f1f9e3337a6277b77a4573b0b40873dd06)
A/DEBUG:       #12 pc 0000000000133564  /apex/com.android.art/lib64/libart.so (art_quick_invoke_stub+548) (BuildId: 73638a3e2fcf5ba85c88424926d725cb)
A/DEBUG:       #13 pc 00000000001a8a78  /apex/com.android.art/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+200) (BuildId: 73638a3e2fcf5ba85c88424926d725cb)
A/DEBUG:       #14 pc 0000000000318248  /apex/com.android.art/lib64/libart.so (art::interpreter::ArtInterpreterToCompiledCodeBridge(art::Thread*, art::ArtMethod*, art::ShadowFrame*, unsigned short, art::JValue*)+376) (BuildId: 73638a3e2fcf5ba85c88424926d725cb)
A/DEBUG:       #15 pc 000000000030e574  /apex/com.android.art/lib64/libart.so (bool art::interpreter::DoCall<false, false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*)+996) (BuildId: 73638a3e2fcf5ba85c88424926d725cb)
A/DEBUG:       #16 pc 000000000067d7c4  /apex/com.android.art/lib64/libart.so (MterpInvokeDirect+576) (BuildId: 73638a3e2fcf5ba85c88424926d725cb)
A/DEBUG:       #17 pc 000000000012d914  /apex/com.android.art/lib64/libart.so (mterp_op_invoke_direct+20) (BuildId: 73638a3e2fcf5ba85c88424926d725cb)
A/DEBUG:       #18 pc 00000000000afa48  [anon:dalvik-classes.dex extracted in memory from /data/app/~~_EQhXdMa5MuSMPuKq2cGQw==/io.realm.benchmark.test-74KDmX9A7pSkji-dzsXqoQ==/base.apk] (com.couchbase.lite.LiteCoreException.<init>)
A/DEBUG:       #19 pc 000000000067da64  /apex/com.android.art/lib64/libart.so (MterpInvokeDirect+1248) (BuildId: 73638a3e2fcf5ba85c88424926d725cb)
A/DEBUG:       #20 pc 000000000012d914  /apex/com.android.art/lib64/libart.so (mterp_op_invoke_direct+20) (BuildId: 73638a3e2fcf5ba85c88424926d725cb)
A/DEBUG:       #21 pc 00000000000afa6c  [anon:dalvik-classes.dex extracted in memory from /data/app/~~_EQhXdMa5MuSMPuKq2cGQw==/io.realm.benchmark.test-74KDmX9A7pSkji-dzsXqoQ==/base.apk] (com.couchbase.lite.LiteCoreException.throwException+4)
A/DEBUG:       #22 pc 0000000000305b70  /apex/com.android.art/lib64/libart.so (art::interpreter::Execute(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame&, art::JValue, bool, bool) (.llvm.6412346068224211370)+268) (BuildId: 73638a3e2fcf5ba85c88424926d725cb)
A/DEBUG:       #23 pc 0000000000669e50  /apex/com.android.art/lib64/libart.so (artQuickToInterpreterBridge+780) (BuildId: 73638a3e2fcf5ba85c88424926d725cb)
A/DEBUG:       #24 pc 000000000013cff8  /apex/com.android.art/lib64/libart.so (art_quick_to_interpreter_bridge+88) (BuildId: 73638a3e2fcf5ba85c88424926d725cb)
A/DEBUG:       #25 pc 00000000001337e8  /apex/com.android.art/lib64/libart.so (art_quick_invoke_static_stub+568) (BuildId: 73638a3e2fcf5ba85c88424926d725cb)
A/DEBUG:       #26 pc 00000000001a8a94  /apex/com.android.art/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+228) (BuildId: 73638a3e2fcf5ba85c88424926d725cb)
A/DEBUG:       #27 pc 0000000000552dd4  /apex/com.android.art/lib64/libart.so (art::JValue art::InvokeWithVarArgs<art::ArtMethod*>(art::ScopedObjectAccessAlreadyRunnable const&, _jobject*, art::ArtMethod*, std::__va_list)+448) (BuildId: 73638a3e2fcf5ba85c88424926d725cb)
A/DEBUG:       #28 pc 0000000000553288  /apex/com.android.art/lib64/libart.so (art::JValue art::InvokeWithVarArgs<_jmethodID*>(art::ScopedObjectAccessAlreadyRunnable const&, _jobject*, _jmethodID*, std::__va_list)+92) (BuildId: 73638a3e2fcf5ba85c88424926d725cb)
A/DEBUG:       #29 pc 00000000003d0374  /apex/com.android.art/lib64/libart.so (art::JNI<false>::CallStaticVoidMethodV(_JNIEnv*, _jclass*, _jmethodID*, std::__va_list)+656) (BuildId: 73638a3e2fcf5ba85c88424926d725cb)
A/DEBUG:       #30 pc 0000000000128720  /data/app/~~_EQhXdMa5MuSMPuKq2cGQw==/io.realm.benchmark.test-74KDmX9A7pSkji-dzsXqoQ==/lib/arm64/libLiteCoreJNI.so (_JNIEnv::CallStaticVoidMethod(_jclass*, _jmethodID*, ...)+116) (BuildId: d3d24b6aecef3c236dc9f0a93bb3d61ff70c53be)
A/DEBUG:       #31 pc 000000000012b2d8  /data/app/~~_EQhXdMa5MuSMPuKq2cGQw==/io.realm.benchmark.test-74KDmX9A7pSkji-dzsXqoQ==/lib/arm64/libLiteCoreJNI.so (Java_com_couchbase_lite_internal_core_C4Document_create2+132) (BuildId: d3d24b6aecef3c236dc9f0a93bb3d61ff70c53be)
A/DEBUG:       #32 pc 00000000020255d4  /memfd:jit-cache (deleted) (offset 0x2000000) (art_jni_trampoline+196)
A/DEBUG:       #33 pc 000000000201bb6c  /memfd:jit-cache (deleted) (offset 0x2000000) (com.couchbase.lite.internal.core.C4Database.create+140)
A/DEBUG:       #34 pc 0000000002000574  /memfd:jit-cache (deleted) (offset 0x2000000) (com.couchbase.lite.AbstractDatabase.saveInTransaction+420)
A/DEBUG:       #35 pc 0000000002000a6c  /memfd:jit-cache (deleted) (offset 0x2000000) (com.couchbase.lite.AbstractDatabase.saveInternal+380)
A/DEBUG:       #36 pc 0000000002003ce8  /memfd:jit-cache (deleted) (offset 0x2000000) (com.couchbase.lite.AbstractDatabase.save+72)
A/DEBUG:       #37 pc 0000000002009a4c  /memfd:jit-cache (deleted) (offset 0x2000000) (io.realm.benchmark.CouchbaseBenchmarks$addObjects$1.run+332)
A/DEBUG:       #38 pc 0000000002005bfc  /memfd:jit-cache (deleted) (offset 0x2000000) (com.couchbase.lite.AbstractDatabase.inBatch+188)
A/DEBUG:       #39 pc 000000000200d1f4  /memfd:jit-cache (deleted) (offset 0x2000000) (io.realm.benchmark.CouchbaseBenchmarks.addObjects+1908)
A/DEBUG:       #40 pc 0000000002025248  /memfd:jit-cache (deleted) (offset 0x2000000) (io.realm.benchmark.CouchbaseBenchmarks.access$addObjects+40)
...

I cut off the bottom portion of the logs, but could provide them if helpful. It’s mostly just the JUnit benchmark instrumentation test runner. Tests were run on a Pixel 4 XL phone with couchbase-lite-android-ee v2.7.1.

I also posted as an issue in the GitHub repo, which has more links (only let me post 2 here as a new user).

blake.meike · August 5, 2020, 8:40pm

I think you are out of memory.

It looks to me as if a call to Database.getDocument in one case and Database.save (in a batch) in the other, failed in the native code (probably because it is out of memory). When it failed it tried to throw an exception. Sadly, creating the exception also failed trying to allocate space for the stack trace.

jeff.lockhart · August 8, 2020, 5:24pm

I can get the benchmarks to run by adding android:largeHeap="true" to the manifest. But do you have any feedback on this code, why it would be using so much memory to execute?

I’ve also experienced various bugs with the AndroidX benchmark library, including the latest stable 1.0.0 version. It actually fails to download the final report JSON on all versions except for the original alpha01. But the other database tests are usually able to execute without running out of memory, so I’m curious what optimizations I could make to the Couchbase code.

Topic		Replies	Views
Android query native crash Couchbase Lite query	4	44	November 13, 2024
Error from couchbase lite 2.7.1 (java) CE Mobile	59	2248	October 6, 2021
CouchbaseLiteException {CouchbaseLite,10,'std::bad_alloc} Couchbase Lite query	4	691	September 13, 2022
Android app with cblite2.8.1 crash on first initialization . Native crash null pointer dereference in couchbase library Couchbase Lite android	2	1541	April 12, 2022
Database corrupt on Android Couchbase Lite java	3	1832	October 12, 2016

Crash in JNI from Android SDK: std::bad_alloc

Related topics