Unable to add import_filter function to SG JSON config file

Following the docs instruction to use backticks for config properties that are javascript, we’ve created the following config file but are getting errors on our ent2.8 server that only go away when we remove the content in the backticks. (empty, same for sync property but showing import_filter for brevity)

sync-gateway-config.json

{
    "interface": ":4984",
    "adminInterface": ":4985",
    "log": ["*"],
    "logging": {
        "log_file_path": "/var/tmp/sglogs",
        "console": {
            "log_level": "debug",
            "log_keys": ["*"]
        },
        "error": {
            "enabled": true,
            "rotation": {
                "max_size": 20,
                "max_age": 180
            }
        },
        "warn": {
            "enabled": true,
            "rotation": {
                "max_size": 20,
                "max_age": 90
            }
        },
        "info": {
            "enabled": false
        },
        "debug": {
            "enabled": false
        }
    },
    "databases": {
        "salesrabbit": {
            "import_docs": true,
            "bucket": "test-bucket",
            "server": "http://ec2-52-37-187-174.us-west-2.compute.amazonaws.com:8091,ec2-18-237-136-91.us-west-2.compute.amazonaws.com:8091,ec2-54-191-150-5.us-west-2.compute.amazonaws.com:8091",
            "enable_shared_bucket_access": true,
            "delta_sync": {
                "enabled": true
            },
            "username": "test-user",
            "password": "password",
            "users": {
                "user1": {"password": "password1", "admin_channels": ["USR-user1"]},
                "user2": {"password": "password2", "admin_channels": ["USR-user2", "ORG-office-RD"]},
                "user3": {"password": "password3", "admin_channels": ["USR-user3", "ORG-office-WT"]}
            },
            "num_index_replicas": 0,
            "import_filter": `function(doc) {
                return true;
            }`
        }
    }
}

The error we are getting points to the import_filter property. This happens when we start the SG as a service on ubuntu. When we run ./sync_gateway /home/sync_gateway/sync_gateway.json directly we don’t get an error.

2021-01-06 17:20:38 [INFO] [run-sync-gateway] Starting configuration of Sync Gateway...
2021-01-06 17:20:38 [INFO] [run-sync-gateway] Setting Sync Gateway systemd unit /lib/systemd/system/sync_gateway.service config path to /home/sync_gateway/sync_gateway.json
2021-01-06 17:20:38 [INFO] [run-sync-gateway] No ASG auto-fill params specified.
2021-01-06 17:20:38 [INFO] [run-sync-gateway] No auto-fill params specified.
parse error: Invalid numeric literal at line 49, column 33

Any guidance with this error would be appreciated. Doing a google search on the error message was not helpful.

Don’t see anything wrong with the config. Backticks will work. In fact, I just copied and pasted the config and connected just fine. Not sure how the config file was edited/created on your end - maybe some hidden character in your config file …or maybe the right one isn’t loaded

2021-01-06T22:29:47.225Z ==== Couchbase Sync Gateway/2.8.0(376;e2e7d42) EE ====

2021-01-06T22:29:47.225Z [INF] Logging: Console to stderr

2021-01-06T22:29:47.225Z [INF] Logging: Files to /var/tmp/sglogs

2021-01-06T22:29:47.225Z [INF] Logging: Console level: debug

2021-01-06T22:29:47.225Z [INF] Logging: Console keys: [* HTTP]

2021-01-06T22:29:47.225Z [INF] Logging: Redaction level: none

2021-01-06T22:29:47.225Z [INF] requestedSoftFDLimit < currentSoftFdLimit (5000 < 1048576) no action needed

2021-01-06T22:29:47.225Z [INF] Logging stats with frequency: 1m0s

2021-01-06T22:29:47.225Z [INF] Opening db /salesrabbit as bucket "test-bucket", pool "default", server <http://ec2-52-37-187-174.us-west-2.compute.amazonaws.com:8091,ec2-18-237-136-91.us-west-2.compute.amazonaws.com:8091,ec2-54-191-150-5.us-west-2.compute.amazonaws.com:8091>

2021-01-06T22:29:47.225Z [INF] GoCBCustomSGTranscoder Opening Couchbase database test-bucket on <http://ec2-52-37-187-174.us-west-2.compute.amazonaws.com:8091,ec2-18-237-136-91.us-west-2.compute.amazonaws.com:8091,ec2-54-191-150-5.us-west-2.compute.amazonaws.com:8091> as user "test-user"

2021-01-06T22:29:47.226Z [INF] Auth: Attempting credential authentication against bucket test-bucket on http://ec2-52-37-187-174.us-west-2.compute.amazonaws.com:8091,ec2-18-237-136-91.us-west-2.compute.amazonaws.com:8091,ec2-54-191-150-5.us-west-2.compute.amazonaws.com:8091

2021-01-06T22:29:47.738Z [INF] Successfully opened bucket test-bucket

2021-01-06T22:29:47.981Z [INF] Set query timeouts for bucket test-bucket to cluster:1m15s, bucket:1m15s

2021-01-06T22:29:47.981Z [INF] Initializing indexes with numReplicas: 0...

2021-01-06T22:29:48.711Z [INF] Verifying index availability for bucket test-bucket...

2021-01-06T22:29:48.711Z [DBG] Query+: Verifying index availability for index sg_channels_x1...

2021-01-06T22:29:48.711Z [DBG] Query+: Verifying index availability for index sg_access_x1...

2021-01-06T22:29:48.711Z [DBG] Query+: Verifying index availability for index sg_roleAccess_x1...

2021-01-06T22:29:48.785Z [DBG] Query+: Index sg_channels_x1 verified as ready

2021-01-06T22:29:48.852Z [DBG] Query+: Index sg_access_x1 verified as ready

2021-01-06T22:29:48.852Z [DBG] Query+: Index sg_roleAccess_x1 verified as ready

2021-01-06T22:29:48.852Z [INF] Indexes ready for bucket test-bucket.

2021-01-06T22:29:48.852Z [DBG] Import+: Creating new ImportFilterFunction

2021-01-06T22:29:48.852Z [INF] delta_sync enabled=true with rev_max_age_seconds=86400 for database salesrabbit

2021-01-06T22:29:48.985Z [INF] Created background task: "CleanAgedItems" with interval 1m0s

2021-01-06T22:29:48.985Z [DBG] Cache+: Initialized channel cache with maxChannels:50000, HWM: 40000, LWM: 30000

2021-01-06T22:29:48.985Z [INF] Cache: Initializing changes cache for database salesrabbit with options {ChannelCacheOptions:{ChannelCacheMinLength:50 ChannelCacheMaxLength:500 ChannelCacheAge:1m0s MaxNumChannels:50000 CompactHighWatermarkPercent:80 CompactLowWatermarkPercent:60 ChannelQueryLimit:5000} CachePendingSeqMaxWait:5s CachePendingSeqMaxNum:10000 CacheSkippedSeqMaxWait:1h0m0s}

2021-01-06T22:29:48.985Z [INF] Created background task: "InsertPendingEntries" with interval 2.5s

2021-01-06T22:29:48.985Z [INF] Created background task: "CleanSkippedSequenceQueue" with interval 30m0s

2021-01-06T22:29:49.124Z [DBG] SGCluster+: Sending node heartbeats at interval: 1s

2021-01-06T22:29:49.124Z [INF] DCP: Attempting to start import DCP feed...

2021-01-06T22:29:49.124Z [INF] DCP: Registering PindexImplType for syncGateway-import-salesrabbit

2021-01-06T22:29:49.124Z [INF] DCP: Starting DCP import feed for bucket: "test-bucket"

2021-01-06T22:29:49.124Z [DBG] SGCluster+: c:test-bucket-cfgSG cfg_sg: Get, key: nodeDefs-known, cas: 0

2021-01-06T22:29:49.187Z [DBG] SGCluster+: c:test-bucket-cfgSG cfg_sg: Get, key: version, cas: 0

2021-01-06T22:29:49.255Z [INF] DCP: manager: setting the nodeDefs implVersion to 5.5.0

2021-01-06T22:29:49.255Z [DBG] SGCluster+: c:test-bucket-cfgSG cfg_sg: Set, key: nodeDefs-known, cas: 1609972174347894784

2021-01-06T22:29:49.320Z [DBG] SGCluster+: c:test-bucket-cfgSG cfg_sg: Get, key: nodeDefs-wanted, cas: 0

2021-01-06T22:29:49.397Z [DBG] SGCluster+: c:test-bucket-cfgSG cfg_sg: Get, key: version, cas: 0

2021-01-06T22:29:49.464Z [INF] DCP: manager: setting the nodeDefs implVersion to 5.5.0

2021-01-06T22:29:49.464Z [DBG] SGCluster+: c:test-bucket-cfgSG cfg_sg: Set, key: nodeDefs-wanted, cas: 1609972174346518528

2021-01-06T22:29:49.533Z [DBG] SGCluster+: c:test-bucket-cfgSG cfg_sg: Subscribe, key: indexDefs

2021-01-06T22:29:49.533Z [DBG] SGCluster+: c:test-bucket-cfgSG cfg_sg: Subscribe, key: nodeDefs-wanted

2021-01-06T22:29:49.533Z [INF] DCP: janitor: awakes, op: kick, msg: start

2021-01-06T22:29:49.533Z [DBG] SGCluster+: c:test-bucket-cfgSG cfg_sg: Get, key: planPIndexes, cas: 0

2021-01-06T22:29:49.533Z [INF] DCP: planner: awakes, op: kick, msg: start

2021-01-06T22:29:49.533Z [DBG] SGCluster+: c:test-bucket-cfgSG cfg_sg: Subscribe, key: planPIndexes

2021-01-06T22:29:49.533Z [INF] DCP: planner: once, reason: start

2021-01-06T22:29:49.533Z [DBG] SGCluster+: c:test-bucket-cfgSG cfg_sg: Subscribe, key: curMetaKvPlanKey

2021-01-06T22:29:49.533Z [DBG] SGCluster+: c:test-bucket-cfgSG cfg_sg: Get, key: version, cas: 0

It’s definitely not the file encoding since it works when we run ./sync_gateway /home/sync_gateway/sync_gateway.json This however is not ideal for a production system.

It’s the same file used by the service definition.

The service uses the run_sync_gateway script that your package installs. When we try to run that directly with the same environment as the service it fails with the same error.

We have the sync gateway installed according to the instructions here:

However, when we apply a custom import filter or sync function to the config located at /home/sync_gateway/sync_gateway.json we experience the parse error mentioned above.

OK- That’s a key bit of info that you missed mentioning in your initial description. I had tested with docker and it worked just fine as I reported above.

Also what OS are you running on ?

Tagging @bbrks for his thoughts

It is on Ubuntu 18lts on aws instances

I can’t find anywhere here that references a script called this.

I found this terraform package that has a file of the same name though:

Where exactly are you getting the Sync Gateway package from?

Just as a bit of a pointer to narrow down what’s going wrong, I’ve found jq will produce the same error message for a config file using backticks (which is obviously intentionally non-standard JSON), which makes me think a package or script is trying to incorrectly parse the config file before it is being sent to Sync Gateway.

{
  "logging": {
    "console": {
      "log_level": "debug",
      "log_keys": ["*"]
    }
  },
  "databases": {
    "db1": {
      "sync": `function(doc) {
          return true;
      }`
    }
  }
}

$ cat sg_config.json | jq
parse error: Invalid numeric literal at line 10, column 29

To be clear, we are using the gruntworks terraform modules to setup our infrastructure.

Is that not advisable for any reason?

I can’t comment on suitability as I’ve never seen it before this forum post, let alone tried to use it., but it does appear you’ve uncovered a bug in their module in that it assumes our config files are pure JSON (which they’re intentionally not).

The docs do detail the “JSON-like” format, and specifically the fact that backticks are used for multiline strings:

Looks like the issue has been raised on their repo recently, so they’re at least aware of the issue:

@bbrks yes, that’s @jmallory that raised the issue and made a PR to fix.