OutOfMemoryError

Mobile Couchbase Lite
1

Hi

I have seeing a number of these crashes in my logs, I am currently running couchbaselite 3.1.1.

Not sure where is best to start debugging, any help would be much appreciated.

Fatal Exception: java.lang.OutOfMemoryError: pthread_create (1040KB stack) failed: Try again
       at java.lang.Thread.nativeCreate(Thread.java)
       at java.lang.Thread.start(Thread.java:976)
       at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:946)
       at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1352)
       at com.couchbase.lite.internal.exec.CBLExecutor.execute(CBLExecutor.java:75)
       at com.couchbase.lite.internal.exec.ClientTask.execute(ClientTask.java:67)
       at com.couchbase.lite.internal.exec.ClientTask.execute(ClientTask.java:62)
       at com.couchbase.lite.internal.ReplicationCollection.filterCallback(ReplicationCollection.java:135)
2

I have decided to remove my push replication filter in the meantime as it is not so necessary.
Though the filter was not very complicated.
The document it was filtering out though was being written to multiple times a second though, not sure if that is what caused the exception. I have not been able to replicate this locally.

    private var pushFilter = ReplicationFilter { document, _ ->
        document.getType()?.let {
            if (ignoredTypes.contains(it)) {
                return@ReplicationFilter false
            }
        }
        return@ReplicationFilter true
    }
3

Hey @meirrosendorff :
It would be great to see how many threads there are and what they are doing when this happens.

Although we do not support the use of methods that are not part of the CBL API (mostly public methods in the package com.couchbase.lite) there is a method, CBLExecutor.dumpState that I believe is visible to your code. It will log a bunch of possibly useful information at the Warning level. If you can’t immediately see what’s going on, post the log here and maybe I can.

4

Howzit @blake.meike

Firstly just want to say thank you for always responding so quickly, its an absolute pleasure posting questions on this forum.

Here are all the threads at the time of the crash

stacktrace.txt.zip (5.5 KB)

When would you like me to run CBLExecutor.dumpState, just start my replicators, get my application humming and run it ?

5

Ok… This seems to be a bad guess on my part. Here’s the deal.

A thread pool executor will always add a new thread even if there are idle threads, up to the max threads in the pool.

The pool in the thread dump you included has 54 threads in it, all idle, and is failing trying to create a 55th.

Here’s the definition of the pool:

... 
 * I think it is ok to permit this executor pool to get very large: it is used only
 * for Core callbacks that run client code.  There should not, therefore, be
 * more threads here than there are LiteCore threads (the LiteCore thread is suspended
 * until this task completes).  That means that it shouldn't, actually, get all that big.
...
    private static final CBLExecutor EXECUTOR
        = new CBLExecutor("Client worker", 128, 128, new SynchronousQueue<>());

Wrongo!!

The threads die off after 30 seconds of inactivity. It appears, though, that you are totally DOSing the executor with tasks and that it is creating a new thread for each.

I just need to make that pool a lot smaller.

Filed Loading... to fix this

6

Ahh that makes sense @blake.meike!

Yes the document I was trying to stop from syncing was being updated multiple times a second (I was filtering it out to avoid DOSing my server! turns out I just shifted the issue). I guess always count on a user to break things in unexpected ways.

In the mean time I have found a better way of working with that document that doesn’t require so many updates.

appreciate the assistance