Hi, I’m looking for any insight anyone might have into an issue we are seeing.
- Periodically our queries will fail using the Java SDK - “Internal Couchbase Server error” and “indexing.error”
- We are running Community Edition 7.6.2 build 3721 in a 3 node cluster, each node has Data, Query, Index, Search roles.
- Each node has RAM Quotas: Data 4.19GiB | Index 3.71GiB | Search 256MiB
- Each node has 24GB RAM
When the above error happens, checking the logs we see which looks like the indexer service has crash and restarted:
indexer.log:
2025-01-28T21:23:53.931+00:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 4786210713412259321 FlushTime 139ns CommitTime 273.182µs TotalFlushTime 0s TotalCommitTime 710.727448ms
2025-01-28T21:23:53.931+00:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 4786210713412259321 Creating New Snapshot SnapshotInfo: seqnos: 887, 887, 35930 committed:true
2025-01-28T21:23:53.931+00:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 4786210713412259321 PartitionId: 0 SliceId: 0 Crc64: 10896911182613105491 (SnapshotInfo: seqnos: 887, 887, 35930 committed:true) SnapType DISK_SNAP SnapAligned true SnapCreateDur 8.126429ms SnapOpenDur 49.519µs
2025-01-28T21:23:55.492+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 -ipv4=required -ipv6=optional -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=dd4f7aa6d56c4aedee3a392877546cab -isEnterprise=false --deploymentModel=default]
2025-01-28T21:23:55.492+00:00 [Info] Indexer DeploymentModel is set to: default
2025-01-28T21:23:55.492+00:00 [Info] Setting ipv6=false
2025-01-28T21:23:55.499+00:00 [Info] Indexer::NewIndexer Status Warmup
2025-01-28T21:23:55.504+00:00 [Info] IndexerSettingsManager: Setting numSliceWriters to 4 cgroup.max_cpu_percent: 400 runtime.NumCPU: 400
2025-01-28T21:23:55.508+00:00 [Info] Setting buffer block size to 16384 bytes
2025-01-28T21:23:55.509+00:00 [Info] settingsManager::setGlobalSettings: Indexer # CPU cores: 4, memory quota: 3984588800 bytes (3891200.000 KB, 3800.000 MB, 3.711 GB)
2025-01-28T21:23:55.509+00:00 [Info] Setting log level to Info
2025-01-28T21:23:55.509+00:00 [Warn] SetCICLMgrTimeDiffToForceFetch: Singleton Manager in ClusterInfoCacheLite is not set
2025-01-28T21:23:55.509+00:00 [Warn] SetCICLMgrSleepTimeOnNotifierRestart: Singleton Manager in ClusterInfoCacheLite is not set
2025-01-28T21:23:55.510+00:00 [Info] CpuThrottle::SetCpuTarget: New cpuTarget: 0.95
2025-01-28T21:23:55.510+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091
2025-01-28T21:23:55.513+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.wal_size":4096,"indexer.settings.rebalance.blob_storage_region":"","indexer.plasma.mainIndex.enableInMemoryCompression":true,"indexer.settings.compaction.plasma.optional.quota":25,"indexer.settings.moi.debug":false,"indexer.settings.moi.recovery_threads":4,"indexer.settings.compaction.check_period":30,"indexer.plasma.backIndex.enableInMemoryCompression":true,"indexer.settings.max_writer_lock_prob":20,"indexer.settings.bufferPoolBlockSize":16384,"indexer.settings.max_array_seckey_size":10240,"indexer.settings.enable_page_bloom_filter":false,"indexer.settings.storage_mode":"forestdb","indexer.settings.allow_large_keys":true,"queryport.client.settings.poolOverflow":30,"indexer.settings.percentage_memory_quota":0,"indexer.settings.moi.recovery.max_rollbacks":2,"indexer.settings.scan_timeout":120000,"indexer.settings.enable_shard_affinity":false,"indexer.settings.compaction.min_frag":30,"indexer.settings.compaction.min_size":524288000,"indexer.settings.max_seckey_size":4608,"indexer.settings.persisted_snapshot_init_build.moi.interval":600000,"indexer.settings.thresholds.mem_high":70,"indexer.settings.send_buffer_size":1024,"indexer.settings.sliceBufSize":800,"indexer.settings.persisted_snapshot_init_build.fdb.interval":5000,"indexer.settings.recovery.max_rollbacks":5,"indexer.settings.num_replica":0,"indexer.settings.thresholds.mem_low":50,"indexer.settings.build.batch_size":5,"indexer.settings.rebalance.blob_storage_prefix":"","queryport.client.settings.relConnBatchSize":100,"indexer.settings.snapshotRequestWorkers":2,"indexer.settings.cpuProfDir":"","indexer.settings.largeSnapshotThreshold":200,"indexer.settings.gc_percent":100,"indexer.settings.memory_quota":3984588800,"indexer.settings.persisted_snapshot.moi.interval":600000,"indexer.settings.maxVbQueueLength":0,"indexer.settings.inmemory_snapshot.moi.interval":10,"indexer.settings.storage_mode.disable_upgrade":false,"indexer.settings.persisted_snapshot.fdb.interval":5000,"indexer.settings.eTagPeriod":240,"indexer.settings.cpuProfile":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.rebalance.blob_storage_bucket":"","indexer.settings.compaction.plasma.optional.decrement":5,"indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.smallSnapshotThreshold":30,"projector.settings.log_level":"info","indexer.settings.thresholds.units_low":40,"indexer.settings.enable_corrupt_index_backup":false,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.scan_getseqnos_retries":30,"indexer.settings.compaction.abort_exceed_interval":true,"indexer.plasma.backIndex.enablePageBloomFilter":false,"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.thresholds.units_high":60,"indexer.settings.memProfile":false,"queryport.client.settings.backfillLimit":5120,"indexer.settings.corrupt_index_num_backups":1,"indexer.settings.snapshotListeners":2,"queryport.client.settings.poolSize":5000,"indexer.settings.compaction.plasma.manual":false,"indexer.settings.fast_flush_mode":true,"queryport.client.settings.minPoolSizeWM":1000,"indexer.settings.compaction.plasma.optional.min_frag":20,"indexer.settings.max_cpu_percent":0,"indexer.settings.compaction.compaction_mode":"circular","indexer.settings.inmemory_snapshot.fdb.interval":200,"indexer.settings.rebalance.blob_storage_scheme":"","indexer.settings.persisted_snapshot_init_build.interval":5000,"indexer.settings.log_level":"info","indexer.settings.minVbQueueLength":30,"indexer.settings.moi.persistence_threads":2,"indexer.settings.memProfDir":"","indexer.settings.compaction.interval":"02:00,04:00","indexer.settings.plasma.recovery.max_rollbacks":2,"indexer.settings.statsLogDumpInterval":60}. Any setting change will get applied once Indexer is ready.
2025-01-28T21:23:55.537+00:00 [Info] InitSystemEventLogger: Started SystemEventLogger
2025-01-28T21:23:55.538+00:00 [Info] Receive security change notification. code 15
2025-01-28T21:23:55.538+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false
2025-01-28T21:23:55.559+00:00 [Info] security context initialized
2025-01-28T21:23:55.559+00:00 [Info] Indexer::NewIndexer Build Mode Set Community
2025-01-28T21:23:55.559+00:00 [Info] Indexer::Cluster Storage Mode Set forestdb
2025-01-28T21:23:55.559+00:00 [Info] memcachedTimeout set to 120
2025-01-28T21:23:55.559+00:00 [Info] Indexer::NewIndexer done initializing from config
2025-01-28T21:23:55.562+00:00 [Info] SetMajorVersionCICL: Setting cluster version to 7
2025-01-28T21:23:55.562+00:00 [Info] newClusterInfoCacheLiteManager: started New clusterInfoCacheManager
2025-01-28T21:23:55.562+00:00 [Info] NewClusterInfoCacheLiteClient started new cicl client for indexer
2025-01-28T21:23:55.563+00:00 [Info] mutationMgr::setEnableAuth: enableAuth set to 1
2025-01-28T21:23:55.563+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ...
2025-01-28T21:23:55.568+00:00 [Warn] PollForDeletedBucketsV2: Falling back to pollForDeletedBuckets() as use_cinfo_lite is false or not present
2025-01-28T21:23:55.573+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available.
2025-01-28T21:23:55.577+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index@127.0.0.1:8091-default
2025-01-28T21:23:55.578+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification
2025-01-28T21:23:55.583+00:00 [Info] Terminating WatchClusterVersionChanges: Cluster version is >= 9
2025-01-28T21:23:55.583+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: rdr-prod
2025-01-28T21:23:55.583+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: rdr-test
2025-01-28T21:23:55.584+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification
2025-01-28T21:23:55.601+00:00 [Info] Timekeeper::setMaxTsQueueLen 1000
2025-01-28T21:23:55.602+00:00 [Info] OperationsMonitor(kvSender) starting monitor ...
2025-01-28T21:23:55.602+00:00 [Info] [Queryport ":9101"] started ...
2025-01-28T21:23:55.604+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap
2025-01-28T21:23:55.606+00:00 [Info] Indexer::NewIndexer bootstrap storage mode forestdb
2025-01-28T21:23:55.606+00:00 [Info] NewClusterInfoCacheLiteClient started new cicl client for IndexMgr
2025-01-28T21:23:55.606+00:00 [Info] NewClusterInfoCacheLiteClient started new cicl client for RequestHandler
2025-01-28T21:23:55.607+00:00 [Info] NewClusterInfoCacheLiteClient started new cicl client for LifecycleMgr
2025-01-28T21:23:55.607+00:00 [Info] Starting metadadta repo: quota 3347054592 sleep duration 900 threshold 30 min file size 0
2025-01-28T21:23:55.664+00:00 [INFO][FDB] Forestdb blockcache size 3347054592 initialized in 55704 us
2025-01-28T21:23:55.665+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.90
2025-01-28T21:23:55.668+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode
2025-01-28T21:23:55.669+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server
2025-01-28T21:23:55.695+00:00 [Info] indexer:: Staring http server : :9102
2025-01-28T21:23:55.695+00:00 [Info] Indexer::indexer version 9 with priority 7.6.0
2025-01-28T21:23:55.695+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId
2025-01-28T21:23:55.696+00:00 [Info] Indexer Id dd4f7aa6d56c4aedee3a392877546cab
2025-01-28T21:23:55.696+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning
2025-01-28T21:23:55.696+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken
2025-01-28T21:23:55.696+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken <nil>
2025-01-28T21:23:55.696+00:00 [Info] ClustMgr:handleGetLocalValuesWithPrefix Key PauseResumeRunning
2025-01-28T21:23:55.696+00:00 [Info] ClustMgr:handleGetLocalValuesWithPrefix Key PauseToken
2025-01-28T21:23:55.696+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap :
2025-01-28T21:23:55.725+00:00 [Info] internalVersionMonitor:ticker starting ...
2025-01-28T21:23:55.725+00:00 [Info] internalVersionMonitor:monitor starting. Term versions (9:7.0.4)
2025-01-28T21:23:55.727+00:00 [Info] internalVersionMonitor:notifier starting ...
2025-01-28T21:23:55.727+00:00 [Info] internalVersionMonitor:monitor terminate. Cluster version reached 9
2025-01-28T21:23:55.727+00:00 [Info] internalVersionMonitor:ticker stopping ...
2025-01-28T21:23:55.862+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap
2025-01-28T21:23:55.884+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID dd4f7aa6d56c4aedee3a392877546cab
2025-01-28T21:23:55.885+00:00 [Info] KVSender::sendShutdownTopic Projector rdr-db01.oc.net.nz:9999 Topic MAINT_STREAM_TOPIC_dd4f7aa6d56c4aedee3a392877546cab
2025-01-28T21:23:55.886+00:00 [Info] NewClusterInfoCacheLiteClient started new cicl client for schedIndexCreator
2025-01-28T21:23:55.886+00:00 [Info] schedIndexCreator: intialized.
2025-01-28T21:23:55.886+00:00 [Info] RebalanceServiceManager::NewRebalanceServiceManager false <nil>
2025-01-28T21:23:55.886+00:00 [Info] DDLServiceMgr::runTokenCleaner: Starting with period 5m0s
2025-01-28T21:23:55.886+00:00 [Info] RebalanceServiceManager::initService Init
2025-01-28T21:23:55.886+00:00 [Info] MasterServiceManager::registerWithServer: *indexer.MasterServiceManager implements service.AutofailoverManager; *indexer.MasterServiceManager implements service.Manager; *indexer.MasterServiceManager implements service.ServerlessManager
2025-01-28T21:23:55.890+00:00 [Info] requestHandlerContext::getCachedIndexerNodeUUIDs: Returning 3 NodeUUIDs: [b68bb6897e4107f83006b87d5d5de2e2 8a0fd5ad9edf493c7dd02cb4d0acd487 dd4f7aa6d56c4aedee3a392877546cab]
2025-01-28T21:23:55.891+00:00 [Info] RebalanceServiceManager::getCachedIndexerNodeUUIDs: Returning 3 NodeUUIDs: [b68bb6897e4107f83006b87d5d5de2e2 8a0fd5ad9edf493c7dd02cb4d0acd487 dd4f7aa6d56c4aedee3a392877546cab]
2025-01-28T21:23:55.891+00:00 [Info] RebalanceServiceManager::updateNodeList: Initialized with 3 NodeUUIDs: [b68bb6897e4107f83006b87d5d5de2e2 8a0fd5ad9edf493c7dd02cb4d0acd487 dd4f7aa6d56c4aedee3a392877546cab]
2025-01-28T21:23:55.891+00:00 [Info] GenericServiceManager::GetTaskList: called with rev: []
2025-01-28T21:23:55.891+00:00 [Info] GenericServiceManager::GetTaskList: return from rev [] call. taskList: &{Rev:[0 0 0 0 0 0 0 1] Tasks:[]}
2025-01-28T21:23:55.894+00:00 [Info] RebalanceServiceManager::GetCurrentTopology []
2025-01-28T21:23:55.894+00:00 [Info] RebalanceServiceManager::GetCurrentTopology returns &{Rev:[0 0 0 0 0 0 0 1] Nodes:[b68bb6897e4107f83006b87d5d5de2e2 8a0fd5ad9edf493c7dd02cb4d0acd487 dd4f7aa6d56c4aedee3a392877546cab] IsBalanced:true Messages:[]}
2025-01-28T21:23:55.895+00:00 [Info] RebalanceServiceManager::GetCurrentTopology [0 0 0 0 0 0 0 1]
2025-01-28T21:23:55.895+00:00 [Info] GenericServiceManager::GetTaskList: called with rev: [0 0 0 0 0 0 0 1]
2025-01-28T21:23:55.922+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector rdr-db01.oc.net.nz:9999 Topic MAINT_STREAM_TOPIC_dd4f7aa6d56c4aedee3a392877546cab. Err genServer.closed
2025-01-28T21:23:55.922+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received genServer.closed from rdr-db01.oc.net.nz:9999
2025-01-28T21:23:55.922+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating genServer.closed As Success
2025-01-28T21:23:55.922+00:00 [Info] KVSender::sendShutdownTopic Projector rdr-db02.oc.net.nz:9999 Topic MAINT_STREAM_TOPIC_dd4f7aa6d56c4aedee3a392877546cab
2025-01-28T21:23:55.946+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector rdr-db02.oc.net.nz:9999 Topic MAINT_STREAM_TOPIC_dd4f7aa6d56c4aedee3a392877546cab. Err genServer.closed
2025-01-28T21:23:55.946+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received genServer.closed from rdr-db02.oc.net.nz:9999
2025-01-28T21:23:55.946+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating genServer.closed As Success
2025-01-28T21:23:55.946+00:00 [Info] KVSender::sendShutdownTopic Projector rdr-db03.oc.net.nz:9999 Topic MAINT_STREAM_TOPIC_dd4f7aa6d56c4aedee3a392877546cab
2025-01-28T21:23:55.987+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector rdr-db03.oc.net.nz:9999 Topic MAINT_STREAM_TOPIC_dd4f7aa6d56c4aedee3a392877546cab. Err genServer.closed
2025-01-28T21:23:55.987+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received genServer.closed from rdr-db03.oc.net.nz:9999
2025-01-28T21:23:55.987+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating genServer.closed As Success
2025-01-28T21:23:55.987+00:00 [Info] KVSender::sendShutdownTopic Projector rdr-db01.oc.net.nz:9999 Topic INIT_STREAM_TOPIC_dd4f7aa6d56c4aedee3a392877546cab
2025-01-28T21:23:55.988+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector rdr-db01.oc.net.nz:9999 Topic INIT_STREAM_TOPIC_dd4f7aa6d56c4aedee3a392877546cab. Err projector.topicMissing
2025-01-28T21:23:55.988+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from rdr-db01.oc.net.nz:9999
2025-01-28T21:23:55.989+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success
2025-01-28T21:23:55.989+00:00 [Info] KVSender::sendShutdownTopic Projector rdr-db02.oc.net.nz:9999 Topic INIT_STREAM_TOPIC_dd4f7aa6d56c4aedee3a392877546cab
2025-01-28T21:23:55.989+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector rdr-db02.oc.net.nz:9999 Topic INIT_STREAM_TOPIC_dd4f7aa6d56c4aedee3a392877546cab. Err projector.topicMissing
2025-01-28T21:23:55.989+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from rdr-db02.oc.net.nz:9999
2025-01-28T21:23:55.989+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success
2025-01-28T21:23:55.989+00:00 [Info] KVSender::sendShutdownTopic Projector rdr-db03.oc.net.nz:9999 Topic INIT_STREAM_TOPIC_dd4f7aa6d56c4aedee3a392877546cab
2025-01-28T21:23:55.994+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector rdr-db03.oc.net.nz:9999 Topic INIT_STREAM_TOPIC_dd4f7aa6d56c4aedee3a392877546cab. Err projector.topicMissing
2025-01-28T21:23:55.994+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from rdr-db03.oc.net.nz:9999
2025-01-28T21:23:55.994+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success
2025-01-28T21:23:55.994+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: <nil>
2025-01-28T21:23:55.995+00:00 [Info] Indexer::local storage mode forestdb
2025-01-28T21:23:56.000+00:00 [Info] Indexer::initPartnInstance Initialized Partition:
2025-01-28T21:23:55.001+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/rdr-prod_idx_customers_name_contains_4963260860397104879_0.index/data.fdb.40
2025-01-28T21:23:56.002+00:00 [Info] NewForestDBSlice(): file version ForestDB v2.x format
2025-01-28T21:23:56.003+00:00 [Info] ForestDBSlice:NewForestDBSlice Created New Slice Id 0 IndexInstId 4963260860397104879 WriterThreads 1
2025-01-28T21:23:56.003+00:00 [Info] Indexer::initPartnInstance Initialized Slice:
2025-01-28T21:23:56.003+00:00 [Info] StorageMgr::updateIndexSnapMapForIndex, New IndexSnapshotContainer is being created for indexInst: 4963260860397104879
2025-01-28T21:23:56.003+00:00 [Info] StorageMgr::updateIndexSnapMapForIndex IndexInst 4963260860397104879 Partitions [0]
2025-01-28T21:23:56.012+00:00 [Info] StorageMgr::openSnapshot IndexInst:4963260860397104879 Partition:0 Attempting to open snapshot (SnapshotInfo: seqnos: 11724, 11583, 35219 committed:true)
2025-01-28T21:23:56.012+00:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 4963260860397104879 Creating New Snapshot SnapshotInfo: seqnos: 11724, 11583, 35219 committed:true
2025-01-28T21:23:56.012+00:00 [Info] ScanCoordinator::handleAddIndexInstance &{0xc000044ba0 ScanCoordinator {4963260860397104879 {3880825516611699905 idx_customers_name_contains forestdb rdr-prod 786bffe66a79fdc18c62d62da50fe831 false [`_class` array lower(`v`) for `v` in [`customerName`] end] N1QL SINGLE (`_class` = "company.rdr.api.document.customers.Customer") [false false] false false [] false false 0 [] false 0 {true 0 0 0} _default customers 0 14 false false false 0 0 0 0 0 0 0 0 [] [] 0 0 0 map[] map[0:[]] [`_class` array lower(`v`) for `v` in [`customerName`] end] `_class`,array lower(`v`) for `v` in [`customerName`] end} 4 0 1 0xc005586c60 [] 0 0 false forestdb 0} map[0:{{0 0 [:9105] []} 0xc000485180}] 0xc004c49908 0xc000413d40}
babysitter.log:
[ns_server:debug,2025-01-28T21:23:55.168Z,babysitter_of_ns_1@cb.local:indexer-goport<0.28952.94>:goport:handle_eof:592]Stream 'stdout' closed
[ns_server:debug,2025-01-28T21:23:55.168Z,babysitter_of_ns_1@cb.local:indexer-goport<0.28952.94>:goport:handle_eof:592]Stream 'stderr' closed
[ns_server:info,2025-01-28T21:23:55.168Z,babysitter_of_ns_1@cb.local:indexer-goport<0.28952.94>:goport:handle_process_exit:573]Port exited with status 134.
[error_logger:error,2025-01-28T21:23:55.169Z,babysitter_of_ns_1@cb.local:<0.28885.94>:ale_error_logger_handler:do_log:101]
=========================ERROR REPORT=========================
** Generic server <0.28885.94> terminating
** Last message in was {<0.28952.94>,{exit_status,134}}
** When Server state == {state,<0.28952.94>,16810,
{index,"/opt/couchbase/bin/indexer",
["-adminPort=9100","-scanPort=9101",
"-httpPort=9102","-streamInitPort=9103",
"-streamCatchupPort=9104","-streamMaintPort=9105",
"-ipv4=required","-ipv6=optional",
"-cluster=127.0.0.1:8091",
"-storageDir=/opt/couchbase/var/lib/couchbase/data/@2i",
"-diagDir=/opt/couchbase/var/lib/couchbase/crash",
"-logDir=/opt/couchbase/var/lib/couchbase/logs",
"-nodeUUID=dd4f7aa6d56c4aedee3a392877546cab",
"-isEnterprise=false","--deploymentModel=default"],
[via_goport,exit_status,stderr_to_stdout,
{env,
[{"GOTRACEBACK","single"},
{"GOMAXPROCS","4"},
{"CBAUTH_REVRPC_URL",
{dynamic,
{ns_ports_setup,
remote_build_cbauth_revrpc_url,
[index]}}}]}]},
{ringbuffer,1300,1024,
{[{<<"Breakpad caught a crash in forestdb. Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/d8a7f0ef-9d34-4f2f-49a34b9e-d6aae049.dmp before te"...>>,
159},
{<<"assertion failed [hbmeta.prefix != NULL] at /home/couchbase/jenkins/workspace/couchbase-server-unix/forestdb/src/hbtrie.cc:1267 ((nil) != 0x7fbe"...>>,
154},
{<<"2025-01-28T21:23:53.931+00:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 4786210713412259321 PartitionId: 0 SliceId: "...>>,
308},
{<<"2025-01-28T21:23:53.931+00:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 4786210713412259321 Creating New Snapshot Snapsho"...>>,
182}],
[{<<"2025-01-28T21:23:53.925+00:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 9802324416138102223 PartitionId: 0 SliceId: 0 Cr"...>>,
311},
{<<"2025-01-28T21:23:53.931+00:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 4786210713412259321 FlushTime 139ns CommitTime 273.182µs To"/utf8...>>,
186}]}},
index,undefined,[],0}
** Reason for termination ==
** {abnormal,134}
[error_logger:error,2025-01-28T21:23:55.170Z,babysitter_of_ns_1@cb.local:<0.28885.94>:ale_error_logger_handler:do_log:101]
=========================CRASH REPORT=========================
crasher:
initial call: ns_port_server:init/1
pid: <0.28885.94>
registered_name: []
exception exit: {abnormal,134}
in function gen_server:handle_common_reply/8 (gen_server.erl, line 1241)
ancestors: [<0.28884.94>,<0.28883.94>,ns_child_ports_sup,
ns_babysitter_sup,<0.115.0>]
message_queue_len: 1
messages: [{'EXIT',<0.28952.94>,normal}]
links: [<0.28884.94>]
dictionary: []
trap_exit: true
status: running
heap_size: 17731
stack_size: 28
reductions: 10431439
neighbours:
[ns_server:info,2025-01-28T21:23:55.170Z,babysitter_of_ns_1@cb.local:<0.28884.94>:supervisor_cushion:handle_info:66]Cushion managed supervisor for index failed: {abnormal,134}
[error_logger:error,2025-01-28T21:23:55.170Z,babysitter_of_ns_1@cb.local:<0.28884.94>:ale_error_logger_handler:do_log:101]
=========================ERROR REPORT=========================
** Generic server <0.28884.94> terminating
** Last message in was {die,{abnormal,134}}
** When Server state == {state,index,5000,-575984698605870436,undefined,
infinity,
#{always_delay => false}}
** Reason for termination ==
** {abnormal,134}
[error_logger:error,2025-01-28T21:23:55.170Z,babysitter_of_ns_1@cb.local:<0.28884.94>:ale_error_logger_handler:do_log:101]
=========================CRASH REPORT=========================
crasher:
initial call: supervisor_cushion:init/1
pid: <0.28884.94>
registered_name: []
exception exit: {abnormal,134}
in function gen_server:handle_common_reply/8 (gen_server.erl, line 1241)
ancestors: [<0.28883.94>,ns_child_ports_sup,ns_babysitter_sup,<0.115.0>]
message_queue_len: 0
messages: []
links: [<0.28883.94>]
dictionary: []
trap_exit: true
status: running
heap_size: 6772
stack_size: 28
reductions: 11200
neighbours:
[error_logger:error,2025-01-28T21:23:55.171Z,babysitter_of_ns_1@cb.local:<0.28883.94>:ale_error_logger_handler:do_log:101]
=========================CRASH REPORT=========================
crasher:
initial call: erlang:apply/2
pid: <0.28883.94>
registered_name: []
exception exit: {abnormal,134}
in function restartable:loop/4 (src/restartable.erl, line 63)
ancestors: [ns_child_ports_sup,ns_babysitter_sup,<0.115.0>]
message_queue_len: 0
messages: []
links: [<0.132.0>]
dictionary: []
trap_exit: true
status: running
heap_size: 4185
stack_size: 28
reductions: 2885
neighbours:
[error_logger:error,2025-01-28T21:23:55.171Z,babysitter_of_ns_1@cb.local:ns_child_ports_sup<0.132.0>:ale_error_logger_handler:do_log:101]
=========================SUPERVISOR REPORT=========================
supervisor: {local,ns_child_ports_sup}
errorContext: child_terminated
reason: {abnormal,134}
offender: [{pid,<0.28883.94>},
{id,{index,"/opt/couchbase/bin/indexer",
["-adminPort=9100","-scanPort=9101","-httpPort=9102",
"-streamInitPort=9103","-streamCatchupPort=9104",
"-streamMaintPort=9105","-ipv4=required",
"-ipv6=optional","-cluster=127.0.0.1:8091",
"-storageDir=/opt/couchbase/var/lib/couchbase/data/@2i",
"-diagDir=/opt/couchbase/var/lib/couchbase/crash",
"-logDir=/opt/couchbase/var/lib/couchbase/logs",
"-nodeUUID=dd4f7aa6d56c4aedee3a392877546cab",
"-isEnterprise=false","--deploymentModel=default"],
[via_goport,exit_status,stderr_to_stdout,
{env,
[{"GOTRACEBACK","single"},
{"GOMAXPROCS","4"},
{"CBAUTH_REVRPC_URL",
{dynamic,
{ns_ports_setup,
remote_build_cbauth_revrpc_url,
[index]}}}]},
{log,"indexer.log"}]}},
{mfargs,
{restartable,start_link,
[{supervisor_cushion,start_link,
[index,5000,infinity,ns_port_server,start_link,
[#Fun<ns_child_ports_sup.2.17249174>]]},
86400000]}},
{restart_type,permanent},
{significant,false},
{shutdown,infinity},
{child_type,worker}]
[ns_server:debug,2025-01-28T21:23:55.172Z,babysitter_of_ns_1@cb.local:<0.8340.98>:supervisor_cushion:init:39]Starting supervisor cushion for index with delay of 5000
[error_logger:info,2025-01-28T21:23:55.182Z,babysitter_of_ns_1@cb.local:ale_dynamic_sup<0.96.0>:ale_error_logger_handler:do_log:101]
=========================PROGRESS REPORT=========================
supervisor: {local,ale_dynamic_sup}
started: [{pid,<0.8342.98>},
{id,'sink-index'},
{mfargs,
{ale_dynamic_sup,delay_death,
[{ale_disk_sink,start_link,
['sink-index',
"/opt/couchbase/var/lib/couchbase/logs/indexer.log",
[{rotation,
[{compress,true},
{size,41943040},
{num_files,10},
{buffer_size_max,52428800}]}]]},
1000]}},
{restart_type,permanent},
{significant,false},
{shutdown,5000},
{child_type,worker}]
[ns_server:debug,2025-01-28T21:23:55.470Z,babysitter_of_ns_1@cb.local:<0.8339.98>:restartable:start_child:92]Started child process <0.8340.98>
MFA: {supervisor_cushion,start_link,
[index,5000,infinity,ns_port_server,start_link,
[#Fun<ns_child_ports_sup.2.17249174>]]}
[error_logger:info,2025-01-28T21:23:55.471Z,babysitter_of_ns_1@cb.local:ns_child_ports_sup<0.132.0>:ale_error_logger_handler:do_log:101]
=========================PROGRESS REPORT=========================
supervisor: {local,ns_child_ports_sup}
started: [{pid,<0.8339.98>},
{id,{index,"/opt/couchbase/bin/indexer",
["-adminPort=9100","-scanPort=9101","-httpPort=9102",
"-streamInitPort=9103","-streamCatchupPort=9104",
"-streamMaintPort=9105","-ipv4=required",
"-ipv6=optional","-cluster=127.0.0.1:8091",
"-storageDir=/opt/couchbase/var/lib/couchbase/data/@2i",
"-diagDir=/opt/couchbase/var/lib/couchbase/crash",
"-logDir=/opt/couchbase/var/lib/couchbase/logs",
"-nodeUUID=dd4f7aa6d56c4aedee3a392877546cab",
"-isEnterprise=false","--deploymentModel=default"],
[via_goport,exit_status,stderr_to_stdout,
{env,
[{"GOTRACEBACK","single"},
{"GOMAXPROCS","4"},
{"CBAUTH_REVRPC_URL",
{dynamic,
{ns_ports_setup,
remote_build_cbauth_revrpc_url,
[index]}}}]},
{log,"indexer.log"}]}},
{mfargs,
{restartable,start_link,
[{supervisor_cushion,start_link,
[index,5000,infinity,ns_port_server,start_link,
[#Fun<ns_child_ports_sup.2.17249174>]]},
86400000]}},
{restart_type,permanent},
{significant,false},
{shutdown,infinity},
{child_type,worker}]
Does anyone know what might cause the indexer to crash like this?
Thanks!