Usage of ReplaceMultiOptions with cas

Python SDK
1

hi! I use python couchbase sdk 4.3.0 and I want to replace bunch of documents in cb, but I want to do it in a single transcation. I wondered if I could use collection.get_and_lock_multi and then collection.replace_multi method to do this since ReplaceMultiOptions supports cas parameter. But I can’t find any usage examples and don’t really understand how it works, since replace_multi seems to accept only single Options parameter and ReplaceMultiOptions apparently accepts only single cas. So I wonder can you actually use this replace_multi in this situation, or you have to iterate through each document and replace them 1 by 1 either in cluster.transcations or using single collection.replace with cas method

2

Hi @khan
Well you certainly don’t want to use non-transactional operations like collection.replace_multi inside a transaction.

Many SDKs make it possible to perform the transactional operations (ctx.replace() etc.) in parallel inside the transaction, but I am not too familiar with the Python SDK. Perhaps my colleague @jcasey can help.

1 Like
3

yeah I mean I hoped I could use cas as optimistic lock since the documents are locked with get_and_lock_multi. but yeah I think if some of the documents fail during the replace, failover strategy will be difficult to implement. maybe it is better to go with cluster.transcations after all. the only thing I am concerned about is a number of queries to couchbase. our environment is pretty high-loaded and I don’t want to overload cb by iterative querying it with ctx.replace method. I hoped that maybe with replace_multi I could do this in 1 query instead of n, if that makes sense

4

On that note - replace_multi is completely an SDK-side operation, implemented by sending N replace requests to the server.

With the additional info - I would avoid using a transaction here as it does add a bit of overhead.

Maybe @jcasey can cover how to provide multiple CAS values in the options. Or perhaps you can do the operations in parallel in your own app logic.

1 Like
5

Hi @khan – Sorry for the delayed response, but an example on using per_key_options is below. This would be how you can associate specific options for specific keys w/in the Python SDK’s suite of multi methods. Let me know if there are any questions.

from datetime import timedelta, datetime

from couchbase.auth import PasswordAuthenticator
from couchbase.cluster import Cluster
from couchbase.options import (ClusterOptions,
                               ReplaceMultiOptions,
                               ReplaceOptions)
def run():
    auth = PasswordAuthenticator('Administrator', 'password')
    opts = ClusterOptions(auth)
    connstr = 'couchbase://localhost'
    cluster = Cluster.connect(connstr, opts)
    bucket = cluster.bucket('beer-sample')
    collection = bucket.default_collection()

    keys = [
        "21st_amendment_brewery_cafe",
        "21st_amendment_brewery_cafe-21a_ipa",
        "21st_amendment_brewery_cafe-563_stout",
        "21st_amendment_brewery_cafe-amendment_pale_ale",
        "21st_amendment_brewery_cafe-bitter_american",
        ]

    glock_res = collection.get_and_lock_multi(keys, timedelta(seconds=30))
    if not glock_res.all_ok:
        raise Exception('Oh no!  Unable to get and lock the docs.')

    keys_and_docs = {k:v for k,v in glock_res.results.items()}
    for k in keys_and_docs.keys():
        new_doc = keys_and_docs[k].content_as[dict]
        new_doc['updated'] = datetime.now().strftime('%Y-%m-%d %H:%M:%S')
        new_doc['what'] = 'New field from replace operation'
        keys_and_docs[k] = new_doc

    per_key_opts = {k:ReplaceOptions(cas=v.cas) for k,v in glock_res.results.items()}
    rep_res = collection.replace_multi(keys_and_docs, ReplaceMultiOptions(per_key_options=per_key_opts))
    if not rep_res.all_ok:
        raise Exception('Oh no!  Unable to replace the docs.')

if __name__ == '__main__':
    run()
2 Likes
6

Using get_and_lock_mutli and replace_multi will not exactly give a transactional behavior. As Graham said, they are implemented as repeated single operations of get_and_lock() and replace(). And replace() of a locked document unlocks it. So the documents will become unlocked as they are replaced - some may already be unlocked by the time the replace of another fails. The good news is - that since the document is locked, the replace would never fail from CasMismatch. It could still fail for other reasons - like a timeout. The safe approach would be to do all the ctx.get and ctx.replace in a couchbase transaction.
Jared is going to post showing how to use replace_multi. The caveats I mentioned will apply to it.

1 Like
7

thanks for the answers! after the discussion I understand that not using a transaction is not an option in my case, since I want to rollback all the changes if one key fails. thanks for clarifying on replace_multi function. I guess I will go with transactions for now and will think about optimization later, chances are I will not need any :slight_smile:

@jcasey thank you so much for clarifying on how replace multi with cas works!

9

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.