5001 Panic Error during MERGE query

christopherz · September 20, 2024, 2:28pm

Hello,
I’m experiencing a consistent 5001 panic error when attempting to run the following MERGE query:

MERGE INTO `merge-dest` d
USING `merge-source` s
ON (meta(d).id = meta(s).id)
WHEN MATCHED THEN
  UPDATE SET d = s
WHEN NOT MATCHED THEN
  INSERT (meta(s).id, s)

The intention with this query is to simply replace the document in the destination table with that from the source when the document key matches.
The failure occurs every time once there’s ~70000 or more records in the source table. The issue seems to occur based on the number of rows being processed, as varying the size of each record did not seem to have an effect.

The destination and source collection each have only the primary indexes initialized, which Index Advisor claims is sufficient. The error occurs whether the command was issued from the Java SDK or from the web console in the Query UI.

query.log output:

2024-09-20T10:18:08.483-04:00 [SEVERE] panic: runtime error: invalid memory address or nil pointer dereference
2024-09-20T10:18:08.483-04:00 [SEVERE] request text: <ud>MERGE INTO `merge-dest` d
USING `merge-source` s
ON (meta(d).id = meta(s).id)
WHEN MATCHED THEN
  UPDATE SET d = s
WHEN NOT MATCHED THEN
  INSERT (meta(s).id, s)</ud>
2024-09-20T10:18:08.483-04:00 [SEVERE] query context: <ud>default:performance-bucket.performance-tests</ud>
2024-09-20T10:18:08.483-04:00 [SEVERE] stack: goroutine 888 [running]:
github.com/couchbase/query/execution.(*Context).Recover(0xc000983188, 0xc01ffe7188)
	C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/query/execution/context.go:1710 +0x85
panic({0x1f5cf60?, 0x1d11080?})
	C:/Users/Administrator/cbdepscache/exploded/amd64/go-1.22.2/go/src/runtime/panic.go:770 +0x132
github.com/couchbase/query/execution.(*base).close(0xc01ffe7188, 0xc000983188)
	C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/query/execution/base.go:305 +0x1a5
panic({0x1f5cf60?, 0x1d11080?})
	C:/Users/Administrator/cbdepscache/exploded/amd64/go-1.22.2/go/src/runtime/panic.go:770 +0x132
github.com/couchbase/query/execution.(*base).notify(0xc01ffe7188)
	C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/query/execution/base.go:1099 +0x105
panic({0x1f5cf60?, 0x1d11080?})
	C:/Users/Administrator/cbdepscache/exploded/amd64/go-1.22.2/go/src/runtime/panic.go:770 +0x132
io.ReadAtLeast({0x0, 0x0}, {0xc002782080, 0x1, 0x80}, 0x1)
	C:/Users/Administrator/cbdepscache/exploded/amd64/go-1.22.2/go/src/io/io.go:335 +0x67
io.ReadFull(...)
	C:/Users/Administrator/cbdepscache/exploded/amd64/go-1.22.2/go/src/io/io.go:354
github.com/couchbase/query/value.(*spillFile).Read(0x150?, {0xc002782080?, 0x648225?, 0x1ea1400?})
	C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/query/value/annotated_array.go:67 +0x2f
github.com/couchbase/query/value.readSpillValue({0x2cb58e0, 0xc00f212960}, {0x0, 0xc0009c5bb0?, 0x0})
	C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/query/value/spill.go:315 +0xdb
github.com/couchbase/query/value.(*spillFile).nextValue(0xc00f212960)
	C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/query/value/annotated_array.go:73 +0x73
github.com/couchbase/query/value.(*AnnotatedArray).nextUnsorted(0xc0042f2f00)
	C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/query/value/annotated_array.go:377 +0x46
github.com/couchbase/query/value.(*AnnotatedArray).Foreach(0xc0042f2f00, 0xc0009c5df0)
	C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/query/value/annotated_array.go:356 +0xfe
github.com/couchbase/query/execution.(*Merge).RunOnce.func1()
	C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/query/execution/merge.go:290 +0x11da
github.com/couchbase/query/util.(*Once).Do(0xc0056dac60?, 0x2cc3048?)
	C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/query/util/sync.go:136 +0x2d
github.com/couchbase/query/execution.(*Merge).RunOnce(0x3fdbcc0?, 0x3fdbcd0?, {0x0?, 0x0?})
	C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/query/execution/merge.go:145 +0x5c
github.com/couchbase/query/execution.opFork({0x203ac80?, 0xc0147037d0?})
	C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/query/execution/base.go:562 +0x9c
github.com/couchbase/query/util.(*worker).work(0xc003c09950)
	C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/query/util/sync.go:66 +0x3b
created by github.com/couchbase/query/util.Fork in goroutine 703
	C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/query/util/sync.go:98 +0x1e5
goroutine 888 [running]:
github.com/couchbase/query/execution.(*Context).Recover(0xc000983188, 0xc01ffe7188)
	C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/query/execution/context.go:1710 +0x85
panic({0x1f5cf60?, 0x1d11080?})
	C:/Users/Administrator/cbdepscache/exploded/amd64/go-1.22.2/go/src/runtime/panic.go:770 +0x132
github.com/couchbase/query/execution.(*base).close(0xc01ffe7188, 0xc000983188)
	C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/query/execution/base.go:305 +0x1a5
panic({0x1f5cf60?, 0x1d11080?})
	C:/Users/Administrator/cbdepscache/exploded/amd64/go-1.22.2/go/src/runtime/panic.go:770 +0x132
github.com/couchbase/query/execution.(*base).notify(0xc01ffe7188)
	C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/query/execution/base.go:1099 +0x105
panic({0x1f5cf60?, 0x1d11080?})
	C:/Users/Administrator/cbdepscache/exploded/amd64/go-1.22.2/go/src/runtime/panic.go:770 +0x132
io.ReadAtLeast({0x0, 0x0}, {0xc002782080, 0x1, 0x80}, 0x1)
	C:/Users/Administrator/cbdepscache/exploded/amd64/go-1.22.2/go/src/io/io.go:335 +0x67
io.ReadFull(...)
	C:/Users/Administrator/cbdepscache/exploded/amd64/go-1.22.2/go/src/io/io.go:354
github.com/couchbase/query/value.(*spillFile).Read(0x150?, {0xc002782080?, 0x648225?, 0x1ea1400?})
	C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/query/value/annotated_array.go:67 +0x2f
github.com/couchbase/query/value.readSpillValue({0x2cb58e0, 0xc00f212960}, {0x0, 0xc0009c5bb0?, 0x0})
	C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/query/value/spill.go:315 +0xdb
github.com/couchbase/query/value.(*spillFile).nextValue(0xc00f212960)
	C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/query/value/annotated_array.go:73 +0x73
github.com/couchbase/query/value.(*AnnotatedArray).nextUnsorted(0xc0042f2f00)
	C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/query/value/annotated_array.go:377 +0x46
github.com/couchbase/query/value.(*AnnotatedArray).Foreach(0xc0042f2f00, 0xc0009c5df0)
	C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/query/value/annotated_array.go:356 +0xfe
github.com/couchbase/query/execution.(*Merge).RunOnce.func1()
	C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/query/execution/merge.go:290 +0x11da
github.com/couchbase/query/util.(*Once).Do(0xc0056dac60?, 0x2cc3048?)
	C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/query/util/sync.go:136 +0x2d
github.com/couchbase/query/execution.(*Merge).RunOnce(0x3fdbcc0?, 0x3fdbcd0?, {0x0?, 0x0?})
	C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/query/execution/merge.go:145 +0x5c
github.com/couchbase/query/execution.opFork({0x203ac80?, 0xc0147037d0?})
	C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/query/execution/base.go:562 +0x9c
github.com/couchbase/query/util.(*worker).work(0xc003c09950)
	C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/query/util/sync.go:66 +0x3b
created by github.com/couchbase/query/util.Fork in goroutine 703
	C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/query/util/sync.go:98 +0x1e5

I’m running a single-node server on my local machine, using Couchbase Enterprise 7.6.2 build 3721.

I haven’t been able to find much information online in relation to this error. Is this a bug, a potential configuration error on the server/buckets, or some problem with the query? Thanks in advance to anyone who can assist.

mreiche · September 20, 2024, 5:26pm

There is a known issue with spilling to disk. [MB-63270] - Couchbase Cloud

1. Have they restarted all Query nodes since upgrading. **If not, this must be the first step.**
2. They can set the feature control bit indicated by 0x80000 to switch back to legacy-style MERGE statements.
  * Firstly leave spilling disabled so they don't encounter the linked bug. The only way to avoid it is to leave spilling disabled.
  * Set the feature control value to 0x80000|0x8000|0x4c = 0x8804c = **557132**
    * this disables both "new merge" (i.e. reverts to legacy merge) and spilling

here’s the doc for setting features - Query without Indexes | Couchbase Docs

Topic		Replies	Views
5001 Query Error Couchbase Server query , n1ql , dot-net	4	377	February 14, 2024
Intermittent 5001 panic runtime error Couchbase Server query , n1ql	2	752	March 14, 2022
Panic: runtime error: invalid memory address or nil pointer dereference Couchbase Server query , n1ql	10	4434	February 26, 2023
Panic runtime error while running Hello Couchbase application Go SDK	3	1646	May 13, 2020
Error: Panic: runtime error: invalid memory address or nil pointer dereference SQL++ query , n1ql	9	2309	May 12, 2021

5001 Panic Error during MERGE query

Related topics