Sync gateway database can't go online

Mobile Sync Gateway
1

I recently upgraded from couchbase/sync-gateway:3.0.4-community to couchbase/sync-gateway:3.1.0-community (using these images for my docker container).

I have one database, and after the upgrade it is not possible to get this db online.
When starting sync gateway, I get the following error:

Database has collections that require resync before it can go online: [_default._default] -- rest.(*ServerContext).TakeDbOnline() at server_context.go:1069

I have done a resync of the db. It seems it is not possible to specifically resync a collection.
I can tried to resync with “regenerate_sequences: true”, which will temporarily get the db online, but after stopping and starting the container, I’m back at the beginning.

Anyone knows how to proceed from here?

Thanks!

2

Are there any clues in the log file as to why resync is failing? I wouldn’t expect an upgrade to require a resync in the first place.

The next steps I’d take are to enable and examine debug logs.

3

Thanks for the fast reply!

Found the problem:

I updated the db config /{db}/_config with a non-existent bucket.
This returns http status code 500 with payload

{
    "error": "Internal Server Error",
    "reason": "Internal error: Authentication failure"
}

If you then restart sync gateway, the db isn’t able to start with the mentioned error.

Updating the config back to an existent bucket will bring the db online again.

Thanks for the help!

4

I had this issue as well on two servers.
@fifteen_renditions how did you figure out the solution?
Thank you,

5

@fifteen_renditions in my case, after every restart of couchbase sever and gateway I have to do the same operations:

  • call _offline
  • call _config with a wrong bucket name
  • call again _config with the right bucket name
  • call _online

Here are my full logs: JustPaste.it - Share Text & Images the Easy Way

6

I ran into this issue while testing on a local dev deployment - so in that case I just set up a fresh installation.
Unfortunately, I don’t know how to reliably recover from this. Hope someone from couchbase can help you.

7

I have identified the use-case,
when we upgrade from couchbase/sync-gateway:3.0.4-community to couchbase/sync-gateway:3.1.0-community
using a docker-compose that looks like

  couchbase-server:
    image: couchbase:community-7.2.0
    ports:
      - "8091:8091"
      - "8092:8092"
      - "8093:8093"
      - "8094:8094"
      - "11210:11210"
    volumes:
      - /data/couchbase-data:/opt/couchbase/var
      - ./config:/config
      - ./logs:/opt/couchbase/var/lib/couchbase/logs
    command: ["/config/init-cluster.sh" ]
    healthcheck:
      test: ["CMD", "sh", "/config/health-check.sh" ]
      interval: 30s
      timeout: 10s
      retries: 3
    restart: unless-stopped

  couchbase-sync-gateway:
    image: couchbase/sync-gateway:3.1.0-community
    depends_on:
      - couchbase-server
    ports:
      - "4985:4985"
      - "4984:4984"
      - "14985:14985"
    volumes:
      - ./config:/config
    command: /config/config.json
    healthcheck:
      test: ["CMD", "sh", "/config/sync_gateway_health_check.sh" ]
      interval: 30s
      timeout: 10s
      retries: 3

couchbase gateway will not be put online automatically, it will fail with 503 DB is currently under maintenance -- rest.(*handler).writeError() at handler.go:1306
I have to run _offline, push the config again and then to put it back online.
This will make it work.

Using the docker-compose file by creating a brand new cluster I don’t have this issue, so it seems a backwards compatibility problem.

8

@crarau Can you provide the SG config you’re using, as well as SG logs?