Utility to make building sync functions for Sync Gateway easier

Mobile
1

Apologies if this is not the appropriate place for such announcements. Our primary goal is not self promotion but rather to give back to the community.

After more than a year of working on our own Couchbase Server and Couchbase Mobile projects, we at Kashoo have refined our process for writing sync functions with comprehensive validation to the point that it can now be distributed as a free and open source (MIT licensed) utility called synctos.

What is synctos? It is a commandline tool, distributed via npm, that allows a user to define their Sync Gateway database/bucket’s documents in a declarative JavaScript format. It drastically reduces the amount of (and in many cases eliminates altogether) boilerplate code needed to comprehensively validate a document’s contents and permissions (i.e. channels). And, if a document ever fails validation for some reason, the error messages that are returned are full of details that make it easy to find the problem.

For instance, no longer will you have to write pages and pages of logic that validate each of the properties of each of the documents that might exist in a Sync Gateway database/bucket. Instead, you create a document definitions object like the following real-world example:

{
  notificationTransportProcessingSummary: {
    channels: {
      write: serviceChannel
    },
    typeFilter: function(doc, oldDoc) {
      return doc.type === 'notificationTransportProcessingSummary';
    },
    propertyValidators: {
      processedBy: {
        type: 'string',
        immutable: true
      },
      processedAt: {
        type: 'datetime',
        required: true,
        immutable: true
      },
      sentAt: {
        type: 'datetime'
      },
      type: {
        type: 'string',
        required: true,
        mustNotBeEmpty: true,
        immutable: true
      }
    }
  }
}

Looking for more info?

6 Likes
2

That’s very interesting stuff - thanks for sharing!

3

Very interesting! Will have to take a look, thanks for sharing.

4

thanks for sharing! I will try it.

5

FYI, we just published v1.2.0 of this utility. Highlights include finer grained control over the immutability of documents and their properties/elements, better range validation constraints for floating point numbers, and convenience functions that make it easier to write test cases for document contents that are expected to be invalid. You can find out more about what’s changed in the change log.

Let me know if this an inappropriate forum for these sorts of release announcements and I’ll cut it out. :wink:

4 Likes
6

A new version of synctos (1.3.0) has been published to npm. This release includes a couple of big features that make synctos significantly more useful and easier to use:

  • Added an option to relax document validation so that you are no longer required to define all the fields that are supported by a document or its nested objects
  • Support for dynamically assigning channels to users and/or roles (via the sync function API’s access function) as documents are created, replaced or deleted

Find out more about what’s new in the change log.

1 Like
7

Today sees another new version of synctos (1.4.0) that brings with it some significant improvements to authorization of document write operations. Specifically:

  • Support for authorizing users to create, replace or delete documents based on their role membership (implemented using the sync function API’s requireRole function)
  • Support for authorizing users to create, replace or delete documents based on their username (implemented using the sync function API’s requireUser function)

As always, you can find out more in the project change log.

3 Likes
8

Today’s release of synctos 1.5.0 is a significant milestone in the project’s development. The features that are introduced now make it possible to generate a sync function that is capable of virtually anything that may be implemented in a custom sync function while continuing to abstract away much of the boilerplate for the vast majority of needs. Of note:

  • Assign role access to users on the fly (implemented using the sync function API’s role function)
  • Support for defining completely custom actions to be executed at various stages of the sync function lifecycle (e.g. when authorization is granted, when document contents are validated). A custom action is a function defined in the document definition that receives the new document, old document and an object containing various metadata generated by the sync function and allows direct calls to the built in sync function API (e.g. the requireUser, channel, access, etc. functions) or any other arbitrary operation. This feature affords a great deal of flexibility for those who need it while remaining completely optional for those who don’t.

See the project’s change log and README for more details.

3 Likes
9

A new year, a new improved version of synctos: 1.6.0. This update brings with it a couple small quality of life improvements and one major new feature:

  • A helper function that makes it easy to determine whether a document is missing/nonexistant (i.e. null or undefined) or deleted (i.e. its _deleted property is true): isDocumentMissingOrDeleted
  • A predefined property validator that can be applied to a document’s type identifier property (e.g. the type property): typeIdValidator
  • Support for modular document definitions. This feature can be invaluable in keeping your code/configuration organized and maintainable by allowing you to decompose document definitions into smaller individual fragments that are imported into the main document definitions file using the importDocumentDefinitionFragment macro. The built in samples (see the project’s samples/ directory) have been updated to demonstrate the feature in action.

As always, you’ll find more info in the project’s change log and README.

1 Like
10

Development on synctos keeps on chugging along at a healthy rate. Today sees the release of version 1.7.0, which adds a few notable conveniences:

  • When a document specifies the simple type filter (simpleTypeFilter), it is no longer necessary to explicitly include a type property in that document’s list of property validators. Now, in such cases, the type property will be implicitly allowed and will use the typeIdValidator that was introduced in the last release.
  • An enum property validation type has been added that allows you to limit the values that may be assigned to a property/element by specifying an accompanying list of predefined values. Predefined values may be either strings or integers.
  • The hashtable property validation type can now enforce minimum and maximum size limits on hashtable/object property values. In other words, you can fully control how many elements/key-value pairs a hashtable property can hold.

Also, behind the scenes the sync function template file has been decomposed into three main components/modules: authorization, validation and access assignment. This is not a breaking change; existing document definitions will continue to work as before. But the change will make it easier to maintain the project (i.e. add and update features) in the future.

Take a look at the project’s change log and README for all the details and examples.

11

Today’s release of synctos 1.8.0 is all about providing greater control over your documents’ attachments. Specifically:

  • limit the number of attachments that can be assigned to a single document
  • limit the file size of each individual attachment assigned to a single document
  • limit the total file size of all attachments assigned to a single document
  • restrict the file extensions that are allowed
  • restrict the content/MIME types that are allowed (e.g. “image/png”, “application/xml”, “text/plain”)
  • require each attachment to have a corresponding property of the attachmentReference validation type

The release also includes some minor behind-the-scenes refactoring that helps with documentation discoverability and keeping the project maintainable.

Remember to check out the change log and README for more information, including examples.

12

I’m pleased to announce the release of synctos 1.9.0. It brings with it several significant improvements that mainly help to make document constraints more dynamic. Some highlights:

  • The minimum and maximum value constraints for the date and datetime validation types now accept values defined as JavaScript Date objects
  • The test-helper module now gives you the option to load document definitions directly from the document definitions file, rather than forcing you to generate a sync function from the document definitions and then load the sync function file
  • No longer are you restricted to defining document constraints as static values (e.g. maximumValue: 1.0); now it’s also possible to define them as JavaScript functions (e.g. maximumValue: function(doc, oldDoc, value, oldValue) { ... }). See the dynamic constraint validation section of the README for more info.

As always, the full details, including examples, can be found in the project’s change log and README.

Coinciding with the new release, the post Testing your Sync Gateway functions with synctos on the official Couchbase blog goes into detail on how to set up a test suite for your synctos document definitions. It’s always a good idea to validate the behaviour of the code you write and the custom logic in your document definitions is no different.

1 Like
13

Today’s release of synctos 1.9.1 is a maintenance release that includes no new features, but it does provide a workaround to a bug in the Sync Gateway admin UI web app that prevented sync functions generated by previous versions of synctos from being loaded in the app’s editor view.

The change log includes a link to the issue description if you’d like more details.

14

Today sees a new version of synctos (1.9.3) that fixes a user-reported issue with validation of immutable properties in an object that is nested in an array. It follows a patch (1.9.2) from earlier this month that fixes an issue in which document permissions were applied incorrectly in certain situations.

As always, see the change log for the details.

15

A couple small bugs were addressed in today’s new version of synctos (1.9.4). Please refer to the change log for more details on the release.

1 Like
16

A new version of synctos (1.10.0) has been released today. It brings some welcome new features:

  • Though it may not be widely known, Sync Gateway includes the Underscore.js utility belt library as a convenience for sync function code. Now the test-helper module properly supports Underscore.js so that you can use it in document definitions.
  • Exercise finer grained control over whether null and undefined values are accepted with the introduction of the mustNotBeMissing and mustNotBeNull constraints
  • Exercise finer grained control over whether null and undefined are considered equal with regards to immutability with the immutableStrict and immutableWhenSetStrict constraints
  • Use the uuid validation type for properties that are meant to be random, unique IDs
  • Test for a specific value with the mustEqual and mustEqualStrict constraints. Useful when a property’s value is supposed to be calculated from the values of other properties or the document’s ID. Most useful when specified dynamically, as a function. For example, suppose that the entityId property should match the value of a component of the document ID:
entityId: {
  type: 'uuid',
  mustEqual: function(doc, oldDoc, value, oldValue) {
    var entityIdRegex = /^entity\.(.+)$/;

    return entityIdRegex.exec(doc._id)[1];
  }
}

In addition, the test-helper module has been refactored into a new module and the old module has been deprecated. The API is virtually identical, so most should be able to switch to the new module in existing test cases without issue by including it as follows:

var testHelper = require('synctos').testHelper;

Check out the change log for further information about the release.

1 Like
17

I am particularly pleased to announce a major new release of synctos: 2.0.0. The marquee feature of this release is a brand new command line utility that scans a document definitions file for structural and semantic errors and then reports any problems it finds. Combined with synctos’ pre-existing test-helper module, this utility provides comprehensive schema validation of document definitions that is easy to incorporate into your own sync function build and deployment processes. See the Validating section of the project’s README for instructions.

This release introduces a number of other helpful features:

  • Validation types for time of day (time) and time zone (timezone) to complement the existing date and datetime types
  • The equality (mustEqual) and immutability (immutable and immutableWhenSet) constraints now use a smarter algorithm when comparing values of specialized string types (e.g. date, datetime, time, timezone, uuid). The strict variants (mustEqualStrict, immutableStrict and immutableWhenSetStrict) continue to require strict equality for specialized string types, however.
  • A new constraint (mustBeTrimmed) for the string validation type that prevents candidate values with leading or trailing whitespace
  • The sync function maker utility now automatically creates the output target directory if it does not already exist
  • Modules/fragments of document definitions may now be nested an arbitrary number of levels deep

Since the project adheres to Semantic Versioning and this is a major release, there are a handful of breaking changes as well:

  • Document validation errors have been updated to make them clearer and more consistent
  • The minimum officially supported version of Node.js is now v8.9.0
  • Modules that were deprecated in a previous 1.x release have been removed

In addition, several bugs have been squashed and the project’s code structure has been updated.

As ever, for more details please refer to the new release’s changelog.

2 Likes
18

Earlier today a patch was released for synctos: 2.0.1. The update fixes a bug in the parameters that are passed to a customValidation constraint function. The details can be found in the changelog.

19

Today’s patch release of synctos 2.0.2 fixes a security vulnerability in the handling of channel and role access assignments. If you make use of the accessAssignments constraint in any of your synctos document definitions, I urge you to upgrade to this version ASAP. Details of the fix can be found in the project changelog.

20

Version 2.1.0 was published today. It adds a couple new features and fixes a bug, along with a number of under-the-hood improvements. The highlights:

  • Documents can always be deleted via the admin API now, whether or not the document’s type can be identified
  • Channel and role access assignments can now be defined dynamically (e.g. accessAssignments: function(doc, oldDoc) { ... })
  • The test-helper module now uses null to indicate an old document that is missing to be consistent with the actual behaviour of Sync Gateway

The changelog has the details.

1 Like