Pthread_mutex_lock called on a destroyed mutex

Mobile
, ,
1

We are experience a crash in our android app in the version v.3.1.7

The logs is the following.
FORTIFY: pthread_mutex_lock called on a destroyed mutex (0x)
backtrace:
#00 pc 0x0000000000066b78 /apex/com.android.runtime/lib64/bionic/libc.so (abort+164)
#01 pc 0x00000000000688fc /apex/com.android.runtime/lib64/bionic/libc.so (__fortify_fatal(char const*, …)+124)
#02 pc 0x00000000000ccb78 /apex/com.android.runtime/lib64/bionic/libc.so (HandleUsingDestroyedMutex(pthread_mutex_t*, char const*)+60)
#03 pc 0x00000000000cca08 /apex/com.android.runtime/lib64/bionic/libc.so (pthread_mutex_lock+280)
#04 pc 0x000000000042d3ac /data/app/~~vI1Xa4DyeYK3hE5NuGN18Q==/com.xxx.xxx.xx==/split_config.arm64_v8a.apk!libLiteCore.so (std::__ndk1::recursive_mutex::lock()+12) (BuildId: 1f08ce94845f278776f150d46f93ba4b9c04ad5b)
#05 pc 0x00000000001c6238 /data/app/~~vI1Xa4DyeYK3hE5NuGN18Q==/com.xxx.xxx.xx==/split_config.arm64_v8a.apk!libLiteCore.so (BuildId: 1f08ce94845f278776f150d46f93ba4b9c04ad5b)
#06 pc 0x00000000001af6c8 /data/app/~~vI1Xa4DyeYK3hE5NuGN18Q==/com.xxx.xxx.xx==/split_config.arm64_v8a.apk!libLiteCore.so (BuildId: 1f08ce94845f278776f150d46f93ba4b9c04ad5b)
#07 pc 0x000000000019959c /data/app/~~vI1Xa4DyeYK3hE5NuGN18Q==/com.xxx.xxx.xx==/split_config.arm64_v8a.apk!libLiteCore.so (c4queryobs_setEnabled+20) (BuildId: 1f08ce94845f278776f150d46f93ba4b9c04ad5b)
#08 pc 0x000000000001ae28 /data/app/~~vI1Xa4DyeYK3hE5NuGN18Q==/com.xxx.xxx.xx==/split_config.arm64_v8a.apk!libLiteCoreJNI.so (Java_com_couchbase_lite_internal_core_impl_NativeC4QueryObserver_free+28) (BuildId: e444f712ef889839b51ecf0a00fd0acd7a8bef12)
#09 pc 0x0000000000339adc /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.oat (art_jni_trampoline+108)
#10 pc 0x000000000077ee0c /apex/com.android.art/lib64/libart.so (nterp_helper+1948)
#11 pc 0x0000000000124b68 /data/app/~~vI1Xa4DyeYK3hE5NuGN18Q==/com.xxx.xxx.xx==/base.apk (com.couchbase.lite.internal.core.impl.NativeC4QueryObserver.nFree)
#12 pc 0x00000000007803e4 /apex/com.android.art/lib64/libart.so (nterp_helper+7540)
#13 pc 0x0000000000121820 /data/app/~~vI1Xa4DyeYK3hE5NuGN18Q==/com.xxx.xxx.xx==/base.apk (com.couchbase.lite.internal.core.n1.accept+16)
#14 pc 0x0000000000adc734 /data/app/~~vI1Xa4DyeYK3hE5NuGN18Q==/com.xxx.xxx.xx==/oat/arm64/base.odex (com.couchbase.lite.internal.core.C4NativePeer.releasePeer+244)
#15 pc 0x000000000077f620 /apex/com.android.art/lib64/libart.so (nterp_helper+4016)
#16 pc 0x0000000000121a74 /data/app/~~vI1Xa4DyeYK3hE5NuGN18Q==/com.xxx.xxx.xx==/base.apk (com.couchbase.lite.internal.core.C4QueryObserver.closePeer+20)
#17 pc 0x000000000077f5c4 /apex/com.android.art/lib64/libart.so (nterp_helper+3924)
#18 pc 0x0000000000121a48 /data/app/~~vI1Xa4DyeYK3hE5NuGN18Q==/com.xxx.xxx.xx==/base.apk (com.couchbase.lite.internal.core.C4QueryObserver.close+16)
#19 pc 0x000000000077f5c4 /apex/com.android.art/lib64/libart.so (nterp_helper+3924)
#20 pc 0x0000000000103c1e /data/app/~~vI1Xa4DyeYK3hE5NuGN18Q==/com.xxx.xxx.xx==/base.apk (com.couchbase.lite.AbstractQuery$LiveQueries.remove+26)
#21 pc 0x000000000077f5c4 /apex/com.android.art/lib64/libart.so (nterp_helper+3924)
#22 pc 0x0000000000104258 /data/app/~~vI1Xa4DyeYK3hE5NuGN18Q==/com.xxx.xxx.xx==/base.apk (com.couchbase.lite.AbstractQuery.removeListener+24)
#23 pc 0x000000000077f5c4 /apex/com.android.art/lib64/libart.so (nterp_helper+3924)
#24 pc 0x00000000001040a8 /data/app/~~vI1Xa4DyeYK3hE5NuGN18Q==/com.xxx.xxx.xx==/base.apk (com.couchbase.lite.AbstractQuery.a)
#25 pc 0x000000000077e6a4 /apex/com.android.art/lib64/libart.so (nterp_helper+52)
#26 pc 0x0000000000103924 /data/app/~~vI1Xa4DyeYK3hE5NuGN18Q==/com.xxx.xxx.xx==/base.apk (com.couchbase.lite.f.accept+8)
#27 pc 0x00000000007803e4 /apex/com.android.art/lib64/libart.so (nterp_helper+7540)
#28 pc 0x00000000001111fa /data/app/~~vI1Xa4DyeYK3hE5NuGN18Q==/com.xxx.xxx.xx==/base.apk (com.couchbase.lite.ListenerToken.remove+22)
#29 pc 0x000000000077f5c4 /apex/com.android.art/lib64/libart.so (nterp_helper+3924)

Reviewing this forum we found similar issues such
CBL-5631: CBL-5631: pthread_mutex_lock called on a destroyed mutex by jianminzhao · Pull Request #2013 · couchbase/couchbase-lite-core · GitHub
CBL-5540: CBL-5540: pthread_mutex_lock called on a destroyed mutex by jianminzhao · Pull Request #2025 · couchbase/couchbase-lite-core · GitHub

but seems like we already have these fixes since the version in our codebase ( v.3.1.7)

A per our investigation, we are calling ListenerToken.remove method from android, but in some cases it is generating “pthread_mutex_lock called on a destroyed mutex”. Is it possible that we need to add some logic in order to call ListenerToken.remove in a better way so we can avoid this crash? Our understanding is like we just need to call it when we don’t need to be notified and from thread managing we are seeing some thread managing in the couchbase android library such as the following class: couchbase-lite-java-common/common/main/java/com/couchbase/lite/ListenerToken.java at a5ab529cf95f85620814b28cbd7d2f303d56cb38 · couchbase/couchbase-lite-java-common · GitHub

2

The crash occured within c4queryobs_setEnabled, called from Java method NativeC4QueryObserver.free.

I don’t work on the Java codebase, but IIRC @blake.meike has mentioned an issue regarding memory management of the native QueryObserver. I’ll check with him.

3

This smells to me like a thread-safety issue, so it may help if you remove the listener token on the same thread that you created it, or if you avoid doing anything else with that Database instance on other threads while removing the listener token.

4

This smells to me like a thread-safety issue, so it may help if you remove the listener token on the same thread that you created it, or if you avoid doing anything else with that Database instance on other threads while removing the listener token.

Yes, we are thinking kind of the same. I noticed that there is a thread logic for these situations in couchbase-lite library side ( Using Atomic booleans ). So we just need to call ListenerToken.remove() without taking care of thread safety issues.

Reference code here → couchbase-lite-java-common/common/main/java/com/couchbase/lite/ListenerToken.java at a5ab529cf95f85620814b28cbd7d2f303d56cb38 · couchbase/couchbase-lite-java-common · GitHub

Maybe this management in couchbase-lite is not enough? If that is the case, is there any reference code where I can see what would be the recommended way?

5

This is a fairly old version of CBL, so I had to review it, carefully.
I have seen similar problems but none that manifest in exactly this way.

In fact, if you look here: couchbase-lite-java-common/common/main/java/com/couchbase/lite/internal/core/C4NativePeer.java at a5ab529cf95f85620814b28cbd7d2f303d56cb38 · couchbase/couchbase-lite-java-common · GitHub
You can see that it is not at all likely that, even from multiple threads, the C4QueryObserver could be released twice.
That suggests that there are two different references to that QueryObserver. I do not, yet, see how that could happen.
I’ve opened * CBL-6676 to track this

6

Great.

Yes, seems to be a problems with multiple thread as you mentioned.
From our side we are just calling Listener.remove().

Will keep an eye on CBL-6676 for any updates.

Thanks!

7

Do you have evidence to support the multiple thread claim? I don’t have any, at the moment, and would love to be able to add something to the ticket, if you have it.

8

We are going to make some tests in order to see if we can get more details for this.

9

That’s great! Please keep me posted.

10

I am closing CBL-6676 as “cannot reproduce”.
It is quite difficult to debug such an old version of CBL. Since it is going EoL very soon now, I recommend giving a 3.2 version a try. If you reproduce the issue there, please open a new ticket. I will chase it down.