Upgrading to Couchbase Server 7.6 causes query to fail

I’m currently using Couchbase Server 7.1.5 and Java SDK 3.4.7. I have a document model named diagnosticsReports.

{
    "propCode": "MEMCW",
    "roomNumber": "118",
    "macAddress": "E81863521C47",
    "date": "2020-03-23",
    "lastReportedTime": 1603224693,
    "diagnosticsReport": [
        {
            "created": 1603224693,
            "payload": [
                {
                    "data": {
                        "innCode": "MSYOR",
                        "location": 0,
                        "roomNumber": "1413"
                    },
                    "name": "Provision Info",
                    "isError": 0,
                    "timestamp": 1608087137
                },
                {
                    "data": {
                        "isWiFiConnected": 0,
                        "isEthernetConnected": 1
                    },
                    "name": "Network Status",
                    "isError": 0,
                    "timestamp": 1608087137
                }
            ]
        }
    ]
}

With a query to get the latest date for each group:

SELECT MAX([d.date, d][1]).*
FROM (
    SELECT document.*
    FROM diagnosticReports AS document
    WHERE document.propCode = $propCode
        AND ($roomNumber IS NULL
            OR document.roomNumber = $roomNumber)
        AND ($macAddress IS NULL
            OR document.macAddress = $macAddress) ) AS d
GROUP BY d.propCode,
         d.roomNumber,
         d.macAddress
ORDER BY MAX([d.date, d][1]).lastReportedTime DESC
OFFSET $offset
LIMIT $limit;

Using server version 7.1.5, the above query works. After upgrading to 7.6.1, the below error is observed:

{
    "completed": true,
    "coreId": "0x1a7e82c00000001",
    "errors": [
        {
            "code": 5001,
            "message": "Panic: runtime error: invalid memory address or nil pointer dereference",
            "retry": false
        }
    ],
    "httpStatus": 200,
    "idempotent": true,
    "lastDispatchedFrom": "127.0.0.1:53307",
    "lastDispatchedTo": "localhost:8093",
    "requestId": 343,
    "requestType": "QueryRequest",
    "retried": 0,
    "service": {
        "bucket": "connectedroom",
        "operationId": "null",
        "scope": "ecmp",
        "statement": "SELECT MAX([d.date, d][1]).* FROM (    SELECT document.*    FROM guestDiagnosticReports AS document    WHERE document.propCode = $propCode        AND ($roomNumber IS NULL            OR document.roomNumber = $roomNumber)        AND ($macAddress IS NULL            OR document.macAddress = $macAddress) ) AS d GROUP BY d.propCode,          d.roomNumber,          d.macAddress ORDER BY MAX([d.date, d][1]).lastReportedTime DESC OFFSET $offset LIMIT $limit",
        "type": "query"
    },
    "timeoutMs": 45000,
    "timings": {
        "dispatchMicros": 8446,
        "totalDispatchMicros": 8446,
        "totalMicros": 8990
    }
}

New working query.

SELECT document.*
FROM (
    SELECT MAX([d.date, d][1]).*
    FROM diagnosticReports AS d
    WHERE d.propCode = $propCode
        AND ($roomNumber IS NULL
            OR d.roomNumber = $roomNumber)
        AND ($macAddress IS NULL
            OR d.macAddress = $macAddress)
    GROUP BY d.propCode,
             d.roomNumber,
             d.macAddress
    ORDER BY MAX([d.date, d][1]).lastReportedTime DESC ) AS document
OFFSET $offset
LIMIT $limit;

I’m using the query as a String in the statement variable.

return scope.query(
        statement,
        queryOptions()
            .parameters(JsonObject.create()
                .put("propCode", propCode)
                .put("roomNumber", roomNumber)
                .put("macAddress", macAddress)
                .put("offset", offset)
                .put("limit", limit))
            .scanConsistency(QueryScanConsistency.REQUEST_PLUS)
            .metrics(true)
            .readonly(true)
            .adhoc(false))
    .rowsAs(DiagnosticReportsEntity.class);

Placing the aggregate function in the FROM clause resolves this error. I believe this is because of the first order of execution?

Is this a memory configuration issue, something more related to how the query engine works or poor query construction?

Hi @eric_balawejder - I opened a ticket [MB-63420] - Couchbase Cloud . I think it is a known issue with a planned fix, someone will investigate and verify. I’m glad you found a work-around.

  • Mike

Check query.log and search panic. Please post stack trace here.
The stack gives important information and see if this related existing issue or new issue

Go to UI===>Settings==>Advanced Query Settings

Take the value from N1QL feature control and add 32768 (0x000000000000008000) to existing value, set this value and save and try again

Due to above adhoc=true might perform better depends on index it can use index aggregation

When I add 32768 (2^15) to the N1QL feature control value, it doesn’t resolve the issue.

Here is the stack trace from query.log:

/opt/couchbase/var/lib/couchbase/logs# cat query.log | grep panic
2024-09-05T16:49:34.694+00:00 [SEVERE] panic: runtime error: invalid memory address or nil pointer dereference
panic({0x2c49fc0?, 0x4dc1660?})
	/home/couchbase/.cbdepscache/exploded/aarch64/go-1.21.8/go/src/runtime/panic.go:914 +0x218
panic({0x2c49fc0?, 0x4dc1660?})
	/home/couchbase/.cbdepscache/exploded/aarch64/go-1.21.8/go/src/runtime/panic.go:914 +0x218
panic({0x2c49fc0?, 0x4dc1660?})
	/home/couchbase/.cbdepscache/exploded/aarch64/go-1.21.8/go/src/runtime/panic.go:914 +0x218
panic({0x2c49fc0?, 0x4dc1660?})
	/home/couchbase/.cbdepscache/exploded/aarch64/go-1.21.8/go/src/runtime/panic.go:914 +0x218
2024-09-05T16:49:34.746+00:00 [SEVERE] panic: runtime error: invalid memory address or nil pointer dereference
panic({0x2c49fc0?, 0x4dc1660?})
	/home/couchbase/.cbdepscache/exploded/aarch64/go-1.21.8/go/src/runtime/panic.go:914 +0x218
panic({0x2c49fc0?, 0x4dc1660?})
	/home/couchbase/.cbdepscache/exploded/aarch64/go-1.21.8/go/src/runtime/panic.go:914 +0x218
panic({0x2c49fc0?, 0x4dc1660?})
	/home/couchbase/.cbdepscache/exploded/aarch64/go-1.21.8/go/src/runtime/panic.go:914 +0x218
panic({0x2c49fc0?, 0x4dc1660?})
	/home/couchbase/.cbdepscache/exploded/aarch64/go-1.21.8/go/src/runtime/panic.go:914 +0x218
2024-09-05T16:49:34.781+00:00 [SEVERE] panic: runtime error: invalid memory address or nil pointer dereference
panic({0x2c49fc0?, 0x4dc1660?})
	/home/couchbase/.cbdepscache/exploded/aarch64/go-1.21.8/go/src/runtime/panic.go:914 +0x218
panic({0x2c49fc0?, 0x4dc1660?})
	/home/couchbase/.cbdepscache/exploded/aarch64/go-1.21.8/go/src/runtime/panic.go:914 +0x218
panic({0x2c49fc0?, 0x4dc1660?})
	/home/couchbase/.cbdepscache/exploded/aarch64/go-1.21.8/go/src/runtime/panic.go:914 +0x218
panic({0x2c49fc0?, 0x4dc1660?})
	/home/couchbase/.cbdepscache/exploded/aarch64/go-1.21.8/go/src/runtime/panic.go:914 +0x218
2024-09-05T16:49:35.077+00:00 [SEVERE] panic: runtime error: invalid memory address or nil pointer dereference
panic({0x2c49fc0?, 0x4dc1660?})
	/home/couchbase/.cbdepscache/exploded/aarch64/go-1.21.8/go/src/runtime/panic.go:914 +0x218
panic({0x2c49fc0?, 0x4dc1660?})
	/home/couchbase/.cbdepscache/exploded/aarch64/go-1.21.8/go/src/runtime/panic.go:914 +0x218
panic({0x2c49fc0?, 0x4dc1660?})
	/home/couchbase/.cbdepscache/exploded/aarch64/go-1.21.8/go/src/runtime/panic.go:914 +0x218
panic({0x2c49fc0?, 0x4dc1660?})
	/home/couchbase/.cbdepscache/exploded/aarch64/go-1.21.8/go/src/runtime/panic.go:914 +0x218
2024-09-05T16:49:35.105+00:00 [SEVERE] panic: runtime error: invalid memory address or nil pointer dereference
panic({0x2c49fc0?, 0x4dc1660?})
	/home/couchbase/.cbdepscache/exploded/aarch64/go-1.21.8/go/src/runtime/panic.go:914 +0x218
panic({0x2c49fc0?, 0x4dc1660?})
	/home/couchbase/.cbdepscache/exploded/aarch64/go-1.21.8/go/src/runtime/panic.go:914 +0x218
panic({0x2c49fc0?, 0x4dc1660?})
	/home/couchbase/.cbdepscache/exploded/aarch64/go-1.21.8/go/src/runtime/panic.go:914 +0x218
panic({0x2c49fc0?, 0x4dc1660?})
	/home/couchbase/.cbdepscache/exploded/aarch64/go-1.21.8/go/src/runtime/panic.go:914 +0x218
2024-09-05T17:08:46.868+00:00 [SEVERE] panic: runtime error: invalid memory address or nil pointer dereference
panic({0x2c49fc0?, 0x4dc1660?})
	/home/couchbase/.cbdepscache/exploded/aarch64/go-1.21.8/go/src/runtime/panic.go:914 +0x218
panic({0x2c49fc0?, 0x4dc1660?})
	/home/couchbase/.cbdepscache/exploded/aarch64/go-1.21.8/go/src/runtime/panic.go:914 +0x218
panic({0x2c49fc0?, 0x4dc1660?})
	/home/couchbase/.cbdepscache/exploded/aarch64/go-1.21.8/go/src/runtime/panic.go:914 +0x218
panic({0x2c49fc0?, 0x4dc1660?})
	/home/couchbase/.cbdepscache/exploded/aarch64/go-1.21.8/go/src/runtime/panic.go:914 +0x218
2024-09-05T17:10:01.364+00:00 [SEVERE] panic: runtime error: invalid memory address or nil pointer dereference
panic({0x2c49fc0?, 0x4dc1660?})
	/home/couchbase/.cbdepscache/exploded/aarch64/go-1.21.8/go/src/runtime/panic.go:914 +0x218
panic({0x2c49fc0?, 0x4dc1660?})
	/home/couchbase/.cbdepscache/exploded/aarch64/go-1.21.8/go/src/runtime/panic.go:914 +0x218
panic({0x2c49fc0?, 0x4dc1660?})
	/home/couchbase/.cbdepscache/exploded/aarch64/go-1.21.8/go/src/runtime/panic.go:914 +0x218
panic({0x2c49fc0?, 0x4dc1660?})
	/home/couchbase/.cbdepscache/exploded/aarch64/go-1.21.8/go/src/runtime/panic.go:914 +0x218
2024-09-05T17:21:10.226+00:00 [SEVERE] panic: runtime error: invalid memory address or nil pointer dereference
panic({0x2c49fc0?, 0x4dc1660?})
	/home/couchbase/.cbdepscache/exploded/aarch64/go-1.21.8/go/src/runtime/panic.go:914 +0x218
panic({0x2c49fc0?, 0x4dc1660?})
	/home/couchbase/.cbdepscache/exploded/aarch64/go-1.21.8/go/src/runtime/panic.go:914 +0x218
panic({0x2c49fc0?, 0x4dc1660?})
	/home/couchbase/.cbdepscache/exploded/aarch64/go-1.21.8/go/src/runtime/panic.go:914 +0x218
panic({0x2c49fc0?, 0x4dc1660?})
	/home/couchbase/.cbdepscache/exploded/aarch64/go-1.21.8/go/src/runtime/panic.go:914 +0x218
2024-09-05T17:21:12.302+00:00 [SEVERE] panic: runtime error: invalid memory address or nil pointer dereference
panic({0x2c49fc0?, 0x4dc1660?})
	/home/couchbase/.cbdepscache/exploded/aarch64/go-1.21.8/go/src/runtime/panic.go:914 +0x218
panic({0x2c49fc0?, 0x4dc1660?})
	/home/couchbase/.cbdepscache/exploded/aarch64/go-1.21.8/go/src/runtime/panic.go:914 +0x218
panic({0x2c49fc0?, 0x4dc1660?})
	/home/couchbase/.cbdepscache/exploded/aarch64/go-1.21.8/go/src/runtime/panic.go:914 +0x218
panic({0x2c49fc0?, 0x4dc1660?})
	/home/couchbase/.cbdepscache/exploded/aarch64/go-1.21.8/go/src/runtime/panic.go:914 +0x218

grep only gives like which has panic word. Open file go to panic

need stack it gives routines (caller stack) where panic occurred. paste around one frame or around 40-50lines

Got it. query.log is a very large file. This appears to be one event of the panic failure taken from that file.

2024-09-05T16:49:34.694+00:00 [SEVERE] panic: runtime error: invalid memory address or nil pointer dereference
2024-09-05T16:49:34.694+00:00 [SEVERE] request text: <ud>PREPARE SELECT MAX([d.date, d][1]).* FROM (    SELECT document.*    FROM diagnosticReports AS document    WHERE document.propCode = $propCode        AND ($roomNumber IS NULL            OR document.roomNumber = $roomNumber)        AND ($macAddress IS NULL            OR document.macAddress = $macAddress) ) AS d GROUP BY d.propCode,          d.roomNumber,          d.macAddress ORDER BY MAX([d.date, d][1]).lastReportedTime DESC OFFSET $offset LIMIT $limit</ud>
2024-09-05T16:49:34.694+00:00 [SEVERE] query context: <ud>`default`:`connectedroom`.`ecmp`</ud>
2024-09-05T16:49:34.694+00:00 [SEVERE] stack: goroutine 636 [running]:
github.com/couchbase/query/execution.(*Context).Recover(0x4002f57180, 0x40032f1b00)
	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/context.go:1709 +0x6c
panic({0x2c49fc0?, 0x4dc1660?})
	/home/couchbase/.cbdepscache/exploded/aarch64/go-1.21.8/go/src/runtime/panic.go:914 +0x218
github.com/couchbase/query/execution.(*InitialProject).RunOnce.(*base).runConsumer.func1.2()
	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/base.go:983 +0x13c
panic({0x2c49fc0?, 0x4dc1660?})
	/home/couchbase/.cbdepscache/exploded/aarch64/go-1.21.8/go/src/runtime/panic.go:914 +0x218
github.com/couchbase/query/value.(*annotatedValue).Actual(0x4000187030?)
	<autogenerated>:1 +0x28
github.com/couchbase/query/execution.(*InitialProject).processTerms(0x40032f1b00, {0x399b398?, 0x4001dfd700?}, 0x4002f57180)
	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/project_initial.go:304 +0x5ac
github.com/couchbase/query/execution.(*InitialProject).processItem(0x40032f1b00, {0x399b398, 0x4001dfd700}, 0x4002f57180)
	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/project_initial.go:170 +0x518
github.com/couchbase/query/execution.(*InitialProject).RunOnce.(*base).runConsumer.func1()
	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/base.go:1025 +0x344
github.com/couchbase/query/util.(*Once).Do(0x4000580400?, 0x12e0dd8?)
	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/util/sync.go:136 +0x64
github.com/couchbase/query/execution.(*base).runConsumer(...)
	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/base.go:949
github.com/couchbase/query/execution.(*InitialProject).RunOnce(0x4000e8a0c0?, 0x52942a0?, {0x0?, 0x0?})
	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/project_initial.go:63 +0x70
github.com/couchbase/query/execution.opFork({0x2d41b40?, 0x4002544ae0?})
	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/base.go:560 +0x88
github.com/couchbase/query/util.(*worker).work(0x4001749980)
	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/util/sync.go:66 +0x40
created by github.com/couchbase/query/util.Fork in goroutine 612
	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/util/sync.go:98 +0x268
goroutine 636 [running]:
github.com/couchbase/query/execution.(*Context).Recover(0x4002f57180, 0x40032f1b00)
	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/context.go:1709 +0x6c
panic({0x2c49fc0?, 0x4dc1660?})
	/home/couchbase/.cbdepscache/exploded/aarch64/go-1.21.8/go/src/runtime/panic.go:914 +0x218
github.com/couchbase/query/execution.(*InitialProject).RunOnce.(*base).runConsumer.func1.2()
	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/base.go:983 +0x13c
panic({0x2c49fc0?, 0x4dc1660?})
	/home/couchbase/.cbdepscache/exploded/aarch64/go-1.21.8/go/src/runtime/panic.go:914 +0x218
github.com/couchbase/query/value.(*annotatedValue).Actual(0x4000187030?)
	<autogenerated>:1 +0x28
github.com/couchbase/query/execution.(*InitialProject).processTerms(0x40032f1b00, {0x399b398?, 0x4001dfd700?}, 0x4002f57180)
	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/project_initial.go:304 +0x5ac
github.com/couchbase/query/execution.(*InitialProject).processItem(0x40032f1b00, {0x399b398, 0x4001dfd700}, 0x4002f57180)
	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/project_initial.go:170 +0x518
github.com/couchbase/query/execution.(*InitialProject).RunOnce.(*base).runConsumer.func1()
	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/base.go:1025 +0x344
github.com/couchbase/query/util.(*Once).Do(0x4000580400?, 0x12e0dd8?)
	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/util/sync.go:136 +0x64
github.com/couchbase/query/execution.(*base).runConsumer(...)
	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/base.go:949
github.com/couchbase/query/execution.(*InitialProject).RunOnce(0x4000e8a0c0?, 0x52942a0?, {0x0?, 0x0?})
	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/project_initial.go:63 +0x70
github.com/couchbase/query/execution.opFork({0x2d41b40?, 0x4002544ae0?})
	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/base.go:560 +0x88
github.com/couchbase/query/util.(*worker).work(0x4001749980)
	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/util/sync.go:66 +0x40
created by github.com/couchbase/query/util.Fork in goroutine 612
	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/util/sync.go:98 +0x268
1 Like

Thanks. We need to see what causing this.
In meantime see if this any way help you

CREATE INDEX ix1 ON diagnosticReports(propCode, roomNumber, macAddress, date, lastReportedTime);


WITH docs AS ( SELECT RAW mx[2]
               FROM diagnosticReports AS d
               WHERE d.propCode = $propCode
                     AND ($roomNumber IS NULL OR d.roomNumber = $roomNumber)
                     AND ($macAddress IS NULL OR d.macAddress = $macAddress)
               GROUP BY d.propCode, d.roomNumber, d.macAddress
               LETTING mx = MAX([d.date, d.lastReportedTime, META(d).id])
               ORDER BY mx[1] DESC
               OFFSET $offset
               LIMIT $limit)
SELECT d1.* FROM diagnosticReports AS d1 USE KEYS docs;

When adhoc=true, it will NOT reuse the prepared statement it creates for positional parameters?

In this particular case, this query is not fully covered and requires fetching from the data service. I believe one of the hard requirements for index aggregation is having a fully covered index?

adhoc=true will not use prepare statement. But if you have $roomNumber , $macAddress it will be way faster. If really want use prepare statement based on two parameter if null or not null generate dynamic statement without OR it will perform better
The query in WITH should have covered and use index aggregate and produced document keys you interested check what is missing in index.