Azure cloud backup

DenisB · July 12, 2023, 5:48pm

I deployed couchbase in AKS using helm.
Now I’m trying to configure backups.
Everything is fine when I configure backups with k8s pvm.
But something wrong when I try to configure backups with Azure storage account.
I followed the manual:
create secret and couchbasebackup
apiVersion: v1
kind: Secret
metadata:
name: azure-secret
namespace: apps
type: Opaque
data:
access-key-id: <storage_account_name>
secret-access-key: <key_value>

apiVersion: couchbase.com/v2
kind: CouchbaseBackup
metadata:
name: my-backup
spec:
strategy: full_incremental
full:
schedule: “0 3 * * 0”
incremental:
schedule: “0 3 * * 1-6”
successfulJobsHistoryLimit: 1
failedJobsHistoryLimit: 3
backoffLimit: 2
backupRetention: 24h
logRetention: 24h
size: 5Gi
ephemeralVolume: false
objectStore:
useIAM: true
secret: azure-secret2
uri: az://<account_name>

But when I run a job and check logs at couchbase-backup pod I see

2023-07-12T17:34:12 INFO Checking connection to Kubernetes API…
2023-07-12T17:34:12 INFO Creating repository couchbase-release-2023-07-12T17_34_12 in location az://<account_name>/archive
2023-07-12T17:34:12 ERROR Subprocess call exited with non-zero return code 1
2023-07-12T17:34:12 ERROR Arguments: cbbackupmgr config --archive az://<account_name>/archive --repo couchbase-release-2023-07-12T17_34_12 --obj-staging-dir /data/staging --s3-force-path-style --obj-auth-by-instance-metadata
2023-07-12T17:34:12 ERROR Stdout: b’Backup repository creation failed: failed to get service client: failed to get service URL: failed to determine account name\n’
2023-07-12T17:34:12 ERROR Stderr: b’’
2023-07-12T17:34:12 ERROR Failed to configure repository

what is wrong?

I tried to change “uri”, “access-key-id”, “secret-access-key” in many combinations but result was the same. Even error was the same every time(

Also I created "“fresh” azure storage account, opened all access for testing - result and error were the same

perry · July 13, 2023, 9:58am

Hey @DenisB , sorry you’re having issues here. Just to confirm, have you followed the steps here: Configure Automated Backup and Restore | Couchbase Docs.

Specifically:

For Azure Blob Storage the account name and the account key are expected under the keys access-key-id and secret-access-key.

aemery-cb · July 13, 2023, 12:50pm

I believe the uri should be the container name, not the account name.
EDITED:

DenisB · July 13, 2023, 6:43pm

This is screenshot from my test storage account

Can you tell me what is access-key-id and secret-access-key .?? Number 1, Number 2 or Number 3? Or WHAT?

I see only:
“storage account name”, “Key”, “Connection string”

And connection string looks not like URI
It looks like

DefaultEndpointsProtocol=https;AccountName=straccnametest1;AccountKey=<KEY_VALUE>==;EndpointSuffix=core.windows.net

I tried to put all these values in various combinations with the same result -you can see result(error) at my first post.

Ok. I will try again put
“access-key-id” as “storage account name”
“secret-access-key” as “Key”
But what should I use as uri??

DenisB · July 13, 2023, 6:53pm

I’m confused…
the name storage account NOT the account name

Is this the same? Storage account name and the account name??

I created storage account with the name “straccnametest1”
What should I use as URI? Or to be more specific: WHERE I can get this URI?

DenisB · July 13, 2023, 7:00pm

I’m sorry if my comments are too harsh, but I’ve been trying to figure this out for 3 days now. The instructions on the site are extremely inaccurate, only a few lines are devoted to configuring backups in azure

DenisB · July 14, 2023, 8:56am

I found a reason why I get this error

and now question is how can I specify the values of these env variables?
I’m using not standard k8s deployment -I’m using custom resource “CouchbaseBackup” and I can’t specify something like

containers:
env:
- name: AZURE_STORAGE_ACCOUNT
value: “<account_name>”

perry · July 14, 2023, 12:48pm

Thanks for your feedback @DenisB , I’ve brought this to the attention of our PM team so they can make sure the documentation gets updated with appropriate examples here.

aemery-cb · July 14, 2023, 1:10pm

For the contents of the secret, using your image as reference.

apiVersion: v1
kind: Secret
metadata:
  name: azure-secret
type: Opaque
data:
  access-key-id: <the field numbered 1> straccnametest1 
  secret-access-key: <the key number 2>

From there you’ll need to set a container to store that information in. Ihis must exist already exist, Operator will not have permission to create it for you.

apiVersion: couchbase/v2
kind: CouchbaseBackup
spec:
  ...
  objectStore:
    secret: azure-secret
    uri: az://<name of the container>

From the screenshot, containers is under data storage. clicking there will allow you to create the container you wish to store data in.

DenisB · July 14, 2023, 1:11pm

@perry thanks for your answers

I can formulate the problem I faced

As I said I’m using AKS and deploying Couchbase with helm.
Now I tried to configure Cloud backups with Azure Storage.

I found one custom resource named CouchbaseBackup. This custom resource provided by couchbase-operator and by Couchbase team.

This is description
https://docs.couchbase.com/operator/current/resource/couchbasebackup.html

As you can see with this custom resource you CAN’T provide any specific values to flag or environment variables
But I need to do it following this manual

Question: how is it possible to provide additional flag “-obj-access-key-key” OR The AZURE_STORAGE_ACCOUNT environment variable? OR- The AZURE_STORAGE_CONNECTION_STRING environment variable? Using kubernetes manifests

I can do it using CLI if I deployed Couchbase on VMs, but I can’t do it in AKS

DenisB · July 14, 2023, 1:14pm

Thank you for you answer.
I tried to do what you’re saying but faced with another problem -I described this problem in my previous reply.

DenisB · July 14, 2023, 1:19pm

Just for clarification: For example, I created storage account with name
straccnametest1
create container with name
cbbackup
I get this URL for this container from Azure portal
https://straccnametest1.blob.core.windows.net/cbbackup

Should I provide
az://straccnametest1.blob.core.windows.net/cbbackup as URI ?
or az://https://straccnametest1.blob.core.windows.net/cbbackup ?
or something else?

aemery-cb · July 14, 2023, 1:24pm

You should provide az://cbbackup

The values inside the secret are passed to the backup container as env vars so

access-key-id in the azure-secret is equivalent to the --obj-access-key-id flag and CB_OBJSTORE_ACCESS_KEY_ID env var.
secret-access-key in the azure-secret is equivalent to --obj-secret-access-key and CB_OBJSTORE_SECRET_ACCESS_KEY env var.

DenisB · July 14, 2023, 1:26pm

thank you for clarification!

only one problem is remaining: how can I specify additional flag or env variables with current k8s manifest or helm value files?

aemery-cb · July 14, 2023, 1:32pm

currently you can’t pass in arbitrary values to the backup process

DenisB · July 14, 2023, 1:33pm

ok.
then how I can fix this error?

I mean during your manuals to fix this problem

failed to determine account name, checked ‘–obj-access-key-id’, ‘AZURE_STORAGE_ACCOUNT’ and
‘AZURE_STORAGE_CONNECTION_STRING’

I need to provide additional flag or env variable but according to your answer I can’t do it

aemery-cb · July 14, 2023, 2:19pm

–obj-access-key-id is the access-key-id field in the azure-secret

which should be pass like so

apiVersion: couchbase/v2
kind: CouchbaseBackup
spec:
  ...
  objectStore:
    secret: azure-secret
    uri: az://<name of the container>

DenisB · July 14, 2023, 3:51pm

OK, doublechecked
then I returned to my first post:
I specified values exactly as you said.
created new fresh azure storage account, opened public access without any restrictions, created container cbbackup
first create secret, then create Couchbasebackup resource
And received an error- in pod:
2023-07-14T15:45:34 INFO Checking connection to Kubernetes API…
2023-07-14T15:45:35 INFO Creating repository couchbase-release-2023-07-14T15_45_34 in location az://cbbackup/archive
2023-07-14T15:45:35 ERROR Subprocess call exited with non-zero return code 1
2023-07-14T15:45:35 ERROR Arguments: cbbackupmgr config --archive az://cbbackup/archive --repo couchbase-release-2023-07-14T15_45_34 --obj-staging-dir /data/staging --s3-force-path-style --obj-auth-by-instance-metadata
2023-07-14T15:45:35 ERROR Stdout: b’Backup repository creation failed: failed to get service client: failed to get service URL: failed to determine account name\n’
2023-07-14T15:45:35 ERROR Stderr: b’’
2023-07-14T15:45:35 ERROR Failed to configure repository
2023-07-14T15:45:35 ERROR Command ‘[‘cbbackupmgr’, ‘config’, ‘–archive’, ‘az://cbbackup/archive’, ‘–repo’, ‘couchbase-release-2023-07-14T15_45_34’, ‘–obj-staging-dir’, ‘/data/staging’, ‘–s3-force-path-style’, ‘–obj-auth-by-instance-metadata’]’ returned non-zero exit status 1.
/usr/local/lib/python3.8/dist-packages/requests/init.py:109: RequestsDependencyWarning: urllib3 (2.0.2) or chardet (None)/charset_normalizer (3.1.0) doesn’t match a supported version!
warnings.warn(

what can I check else? I’m SURE that I use proper creds cause I use this creds when I connect to storage account with CLI

aemery-cb · July 14, 2023, 4:14pm

if you describe the pod, can you see

    Environment:
      CB_OBJSTORE_ACCESS_KEY_ID:      <set to the key 'access-key-id' in secret 'azure-secret'>      Optional: false
      CB_OBJSTORE_SECRET_ACCESS_KEY:  <set to the key 'secret-access-key' in secret 'azure-secret'>  Optional: false

maybe just share the whole describe if that is possible

DenisB · July 14, 2023, 8:24pm

IT’S WORKING!
I set parameter
useIAM: false

and after this change I can see secrets in pod description and backups were uploaded to storage account successfully

Thank you SO MUCH!

Topic		Replies	Views
Couchbase backup. No space left on device Kubernetes backup	9	625	September 8, 2023
CouchbaseBackup doesn't create backup cronjob Kubernetes	2	842	November 18, 2021
Using Azure blob for backup Couchbase Server	4	924	December 14, 2022
How to cluster backup and restore with couchbase Autonomous Operator with Kubernetes? Kubernetes	1	393	March 22, 2024
Backup failed due to rollback Couchbase Server backup	4	634	June 26, 2023

Azure cloud backup

Related topics