Couchbase autonomous operator backup error with SSL certificate verify failed: unable to get issuer certificate

Couchbase Server Kubernetes
1

Hi there,

I create CouchbaseBackup manifest yaml with Couchbase autonomous operator 2.5 for doing backup. Backup pod was spawned and ran with error. There is Kubernetes API SSL cert error which is known issue with our side.

Thus, I am trying to update /usr/local/bin/backup.py python script with disable verify Kubernetes API SSL cert and rebuild customization image with base image couchbase/operator-backup:1.3.5. The SSL cert error was gone when backup pod is running.

Any other solution suggestion on this error ? Or can I ignore SSL cert verification / configure SSL cert path ?
Many thanks.

Backup pod log:

2023-10-12T02:12:01 INFO couchbase-operator-backup/1.3.5 (commit/3898d8b19e5978a426a200e108e92b43ada7268f)
2023-10-12T02:12:01 INFO Timestamp: 2023-10-12 02:12:01.392035
2023-10-12T02:12:01 INFO Arguments: cluster=dev-couchbase, mode=backup, full=True, incremental=False, backup_ret=720.0, disable_bucket_config=False, repo=None, start=None, end=None, map_data=None, filter_keys=None, filter_values=None, enable_bucket_config=False, force_updates=False, include_data=None, exclude_data=None, disable_views=False, disable_gsi_indexes=False, disable_ft_indexes=False, disable_ft_alias=False, disable_data=False, disable_analytics=False, disable_eventing=False, disable_cluster_analytics=False, disable_bucket_query=False, disable_cluster_query=False, cacert=None, log_ret=168.0, verbosity=INFO, s3_bucket=None, obj_store=s3://dev-couchbase, obj_auth_by_instance_metadata=None, obj_endpoint=https://xxx.com, obj_cacert=None, s3_force_path_style=True, threads=1, default_recovery=none
2023-10-12T02:12:01 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1131)'))': /version/
2023-10-12T02:12:01 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1131)'))': /version/
2023-10-12T02:12:01 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1131)'))': /version/
2023-10-12T02:12:01 INFO Unable to contact Kubernetes API: HTTPSConnectionPool(host='10.200.128.1', port=443): Max retries exceeded with url: /version/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1131)')))
2023-10-12T02:12:06 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1131)'))': /version/
2023-10-12T02:12:06 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1131)'))': /version/
2023-10-12T02:12:06 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1131)'))': /version/
2023-10-12T02:12:06 INFO Unable to contact Kubernetes API: HTTPSConnectionPool(host='10.200.128.1', port=443): Max retries exceeded with url: /version/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1131)')))

CouchbaseBackup manifest yaml:

apiVersion: couchbase.com/v2
kind: CouchbaseBackup
metadata:
  name: dev-couchbase
spec:
  strategy: full_incremental
  full:
    schedule: "0 3 * * 0"
  incremental:
    schedule: "0 3 * * 1-6"
  size: 20Gi
  autoScaling:
    thresholdPercent: 20 
    incrementPercent: 20 
    limit: 100Gi
  s3bucket: s3://dev-couchbase
  objectStore:
    secret: s3-secret
    uri: s3://dev-couchbase
    endpoint:
      url: https://xxx.com