How do I download all data to client?

Mobile
1

Hi,

I have a sample app I have built to demo an internal application for our business with very little functionality. The goal is to have a set of master (or static) and transaction data that is accessible from all clients. In a real world scenario the master data would be changing often so we cannot package the master data in a prebuilt db as suggested in the tutorials. I understand how to filter this from the sync gateway, but for now I just want to pull all of the data as its very little (maybe 100 documents).

What I have so far:

  • Working swift4 application that can create a set of master data and can create transactions as needed fine.
  • Working sync gateway and server connected with client iOS devices.

My issue is if I open another device to demo the speed of couchbase in a meeting all of the master data is not downloading. I checked the sync gateway during the application and nothing seems to be syncing unless its a specific change to one of these and in that case my change listeners pick up the data fine. So my question, is how do I sync the data that doesn’t have changes and already exists in the bucket? The following is my sync gateway code and snippets of code setting up replication from the iOS app. When building my database manager class, on initialization I configure the DB, start the live query and try to print any of the data to the console.
Sync Gateway:

{
  "interface":":4984",
  "log": ["*"],
  "databases": {
"unittrack": {
  "import_docs": "continuous",
  "unsupported": {
    "enable_extended_attributes": true,
    "replicator_2":true
  },
  "server": "http://localhost:8091",
  "enable_shared_bucket_access":true,
  "username": "admin",
  "password": "password",
  "users":{
      "UnitAdmin": {"password": "password"},
	  "admin": {"password":"password"}
  }
}
  }
}

Swift:
func startReplicaiton(){
guard let url = URL.init(string: kSyncURL) else {
print("Cannot get url for: ", kSyncURL)
return
}
let dburl = url.appendingPathComponent(kDBName)

        var config = ReplicatorConfiguration(database: _db, targetURL:dburl)
        config.replicatorType = .pushAndPull
        config.continuous = true
        config.authenticator = BasicAuthenticator(username: kUser, password: kPassword)
        _repl = Replicator(config: config)
        _replListener = _repl?.addChangeListener({change in
            let s = change.status
            print("PushPull Replicator: \(s.progress.completed)/\(s.progress.total), error: \(String(describing: s.error)), activity = \(s.activity)")
            // Workarond for BUG :https://github.com/couchbase/couchbase-lite-ios/issues/1816.
            if s.progress.completed == s.progress.total {
                self.postNotificationOnReplicationState(.idle)
            } else {
                self.postNotificationOnReplicationState(s.activity)
            }
        })
        _repl?.start()
    }
    func postNotificationOnReplicationState(_ status:Replicator.ActivityLevel){
        switch status {
        case .offline:
            NotificationCenter.default.post(Notification.offlineNotification())
        case .connecting:
            NotificationCenter.default.post(Notification.connectingNotification())
        case .stopped:
            NotificationCenter.default.post(Notification.stoppedNotification())
        case .idle:
            NotificationCenter.default.post(Notification.idleNotification())
        case .busy:
            NotificationCenter.default.post(Notification.inProgressNotification())
        }
    }
    
    // MARK: Live Query
    func setupLiveQuery() {
        self.liveQuery = Query
            .select(SelectResult.all())
            .from(DataSource.database(_db))
            .where(Expression.property("type").equalTo("UnitDetails")).toLive()
        self.liveQuery?.addChangeListener({change in
            if change.error == nil {
                for (_, row) in (change.rows?.enumerated())!{
                    let data = row.value(forKey: "unittrack") as! DictionaryObject
                    print(data.keys)
                }
            } else {
                print(change.error?.localizedDescription ?? "Problem getting data from live query. Error exists, but description cannot be printed.")
            }
        })
        self.liveQuery?.start()
        
    }
2

Each device and each time you install your application with CBL its creates a UUID internally.

So when CBL connects to SG. It will create a checkpoint ,using the above UUID, and check to see if that checkpoint is in SG.
If not It will ask for all the changes since=0&active_only=true (don’t include deleted documents) when SG was first created.
After its finish pulling all the data the app will update the checkpoint in Sync gateway to EX 1000.

So next time it pulls data from SG it will say since=1,000&active_only=false(include deleted documents) and get the changes and new documents since 1,000

MORE DETAIL on Replication : Replication Algorithm · couchbase/couchbase-lite-ios Wiki · GitHub

DELETES vs PURGE

PURGE - So if you purged a document(Completed Removed) from SG it will not sync(b/c it doesn’t exist).
DELETE - If you _deleted:true a document it will NOT remove the document , but it will mark it as deleted:true(i.e. a tombstone) so that CBL will be aware that document should no longer should be available on the device.

PRO TIP - The other side of the conversation.
Many times people might expriance slowness in downloading large data sets. To understand why you have to see what SG is doing via logs. Here is a forum post that talks about changed SG cache setting and found the performance it needed. Understanding in sync gateway logging - #4 by househippo

2 Likes
3

Thanks this all makes sense. Though I am not sure this is answering completely how I need to resolve my current issue. So I have a second device which from the above establishes a UUID to that DB on that device and the original connection to the sync gateway should be asking for all. However, this is not happening with my current code above.

Would the UUID generated also be relevant to the user account accessing the sync gateway? Currently I have only been testing with one user.

4

@mraby

The UUID in internal per app install because USER 1 can log into Device 1 & Device 2 and get the same data on both devices.

SG LOGS
Could you post your sync gateway log. make sure its logging levels are set to “log”:[“*”]

That way we can see why SG is not sending data to the second device.

The logs you want are called sync_gateway_error.log

USER
also could you run this command on the SG’s admin port to see what the user has access to.

#curl http://.localhost:4985/{db}/_user/{user name}

5

@househippo, the UUID stuff is irrelevant here, or rather Too Much Information. All you need to know, @mraby, is that any CBL database instance tracks its own replication state, as you’d expect.

If I understand correctly, the problem is that you can push documents to the server, but when you pull into an empty database you get no documents, unless a document is later changed on the other db? That’s unexpected. The initial pull should download all of the existing documents.

First off, are you on the latest developer build (20, I believe?)

Try turning on CBL logging in the Replicator domain. This will log a bunch of stuff, among which should be some indication of what docs are being downloaded. Upload that somewhere like a gist, then post a link to it here.

1 Like
6

@mraby : Please confirm that you are on Developer Build 21 …details of release available here

7

Sorry all some production issues took me away from my couchbase projects, thank you so much for the replies.

@jens you are correct in understanding my issue. I thought I updated it to DB 21, but I don’t see the build in any logging in xcode when I run the app with log level set to all. I have added the framework manually and if its helpful here is the plist in the framework package, but I don’t see the build version.

Info.plist.zip (1.2 KB)

Edit, I will go ahead and reinstall the framework to be certain, I will update you soon.

8

I did not have build 21. After installing I have some updates to make to the code as some functions were removed that I was testing with. I will respond later today if the new build resolves my issues and post the logs as requested if not.

9

Sorry for the delay. New years projects have caused some production issues with some of our applications. I just updated the code and the replication is working as expected with no changes to the sync gateway required. Something in DB20 was preventing the changes from being read in app. Thank you all for helping.