Couchbase GSI Indexes in Warm Up mode

Couchbase Server
1

Hello,

I have an Index node, and i have a set of handful number of indexes ( 5 indexes that consume around 28 GB of disk space and I have 10 GB of RAM on this node.)

I am creating a new Index on this node and this is supposed to take around 12 GB of disk.

This new Index creation is not successful and all my other indexes in this node goes into Warmup mode and comes back and this is happening again n again.

Can you help me understand the behavior.

After building a certain percentage, It is not moving forward and I see this warning in console …
“Warning: Cannot communicate with indexer process. Information on indexes may be stale. Will retry.”

Regards
Shiv

2

Hi Shiv,

Here’s my take: The indexing process is consuming I/O and CPU resources to such an extent that the existing indexes cannot be maintained properly, which is why they go into the Warm Up state and you see those indexer-process warnings. But I would expect that to only happen if the data services are sending data mutations to the index node.

If that’s not the case and the system is quiet, perhaps it’s a configuration issue.

  • What version of Couchbase Server are you using?
  • What’s the CPU utilization when you create the index?
  • What is your “Indexer Threads” count (4.6) under Settings/Index Settings/Advanced Index Settings?
  • If you’re using Linux, did you set swappiness to 0? See Space and Kernel Swappiness
3

Hello kurtz,

I have dedicated Index and Data nodes (3 each), I am assuming this will Isolate the workloads.
there is no mutations happening on data nodes at the time of index creation and building.

version -> 5.0 Beta Build
Indexer Threads -> 0 ( using all available cores)
swappiness -> 0 ( using default setting)

Regards
Shiv[quote=“jkurtz, post:2, topic:13564”]
resources to such an extent that the existing indexes cannot be maintained properly, which is why they go into the Warm Up state and you see those indexer-process warnings. But I would expect that to only happen if the data services are sending data mutations to the index node.
[/quote]

4

@sgudavalli, it looks like the Indexer process is crashing/restarting over and over. Warmup state of indexes indicate the indexer trying to recover these indexes. Can you check in the indexer.log for any panics/errors or you can share the log here and we can take a look.

5

I;m using 5.0.1 and also got index warnup issue after machine poweroff. This is error in log:

2018-01-12T18:31:00.412+07:00 [Info] cpuCollector: cpu percent 0 for pid 8531
2018-01-12T18:31:00.534+07:00 [Info] plasmaSlice::doRecovery SliceId 0 IndexInstId 6403330654398182307 Recovering from recovery point …
2018-01-12T18:31:00.791+07:00 [Info] plasmaSlice::doRecovery SliceId 0 IndexInstId 6403330654398182307 Warmup took 22.644017033s
2018-01-12T18:31:00.792+07:00 [Info] plasmaSlice:NewplasmaSlice Created New Slice Id 0 IndexInstId 6403330654398182307 WriterThreads 16
2018-01-12T18:31:00.792+07:00 [Info] Indexer::initPartnInstance Initialized Slice:
Index: 6403330654398182307 Slice: SliceId: 0 File: /home/data/couchbase/data/@2i/default_ads_monit_count_idx_6403330654398182307_0.index Index: 6403330654398182307
2018-01-12T18:31:00.792+07:00 [Info] Indexer::initPartnInstance Initialized Partition:
Index: 8604717496499624485 Partition: PartitionId: 0 Endpoints: [:9105]
panic: runtime error: index out of range

goroutine 789 [running]:
panic(0xd43460, 0xc4200160c0)
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.3/go/src/runtime/panic.go:500 +0x1a1 fp=0xc42712b690 sp=0xc42712b600
runtime.panicindex()
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.3/go/src/runtime/panic.go:27 +0x6d fp=0xc42712b6c0 sp=0xc42712b690
github.com/couchbase/plasma.(*Plasma).doRecovery.func1(0xae5b230a, 0xc42a962000, 0x0, 0x1d17e, 0x0, 0x0, 0x0)
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:515 +0xd75 fp=0xc42712b7c0 sp=0xc42712b6c0
github.com/couchbase/plasma.(*lsStore).visitor(0xc425061600, 0xad57d63a, 0xae5b43c2, 0xc42b90e700, 0xc42b90e6e0, 0xd89840, 0xc41fa37801)
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.