Temporary failure received from server. Code 11

Couchbase Server
1

Hello,

While upserting data into couchbase I am often getting:

{
“message”: “Temporary failure received from server. Try again later”,
“code”: 11
}

I am using the NodeJS SDK to connect to Couchbase Server

Any Idea what’s going wrong here?

Thanks

2

Is this a bulk update? The server will return a “TMPFAIL” when doing bulk loading if you overrun memory before it can persist to disk.

See the doc on handling temporary out of memory errors for some more description. It’s from the Server 3.0 documentation, but the concept applies to the 4.x releases too.

p.s.: I just filed a bug because I couldn’t find this in the 4.x documentation.

3

Hi ingenthr

Thanks for replying.

No it wasn’t a bulk load. I was doing a series of upserts through the sdk.

bucket.upsert(key, doc)

which happens to be throwing the Issue.

I am on Couchbase Server 4.0, Also Is ‘Temporary out of memory’ error the same as this error?

I am sort of performing a transaction operation. I have made sure if I get this error the server will be able to handle the situation. But I really wanted to know regarding the error as it is quite frequent…

I am getting about 120 errors like these per day out of something in the thousands. Which is less I agree but I believe couchbase can handle a lot of ops per second and I think there is something wrong with my configuration (but I believe I am using the default config)?

PS: My buckets are also a little low on RAM, But I have about 200-250mb free in that bucket.

4

Hi,

Any update on the situation? :frowning:

5

Does the Couchbase UI show any TMP OOM events? You may need to zoom to minutes/hours/days.

If it does not, then the cause is probably something more complicated. I’d probably have to then recommend looking at logs to see if there are any process crashes or other issues. In fact, does the Web UI’s log show anything?

Of course, if you have an enterprise subscription, you can also send in a collect_info for expert analysis.

6

Hello sorry for a late reply.

I found the Issue. It is related to RAM, basically when the Server shifts the LRU to Disk I get code 11. But it is strange that I have about 100-200MB RAM still left and yet I get that error code. Will Increasing the Nodes benefit me? Also Is there any way to force empty the RAM through any of the SDKs?

My current node has about 8GB RAM and plenty of disk space.

7

I tried go-couchbase-throughput to do some testing.

I fired 500K Docs resulting at a size of about 649MB of RAM. With Full Eviction or Value Eviction both cases consumed 649MB ram (I was hoping for a size reduction in the case of Full Eviction).

Case 1:
When I did 1024MB Ram quota for my Bucket the Docs were inserted in about 10s. No Errors. Which was awesome.

Case 2:
But when I did 512MB (lower than the data’s size) I got Code 11 assuming that the Couchbase tried to move the data to the disk? Which caused this Issue as illustrated below:

cb_debug.png779×717 48.9 KB

Highlighted in the Yellow Region is when I got Code 11. But I see the “drain rate” also going to 0, Shouldn’t Drain Rate be still high since couchbase is flushing my data from RAM to Disk?

Also what would be an approach to the given situation. Is there a method where I can clear the allocation for that bucket or just keep Increasing the Nodes/RAM to accommodate an ever growing database?

EDIT:
Just saw the Ejections/sec graph. Ejections seem to be happening when I get the Code 11