Good day.
I have Couchbase server in AKS.
I configured couchbase backups using these parameters:
backups:
default-backup:
name: couchbase-backup
strategy: full_incremental
full:
schedule: “0 3 * * 0”
incremental:
schedule: “0 3 * * 1-6”
successfulJobsHistoryLimit: 1
failedJobsHistoryLimit: 3
backoffLimit: 2
backupRetention: 720h
logRetention: 168h
size: 5Gi
ephemeralVolume: false
objectStore:
useIAM: false
secret: azure-secret
uri: az://cbbackup
and I configured cloud backup with Azure storage account.
And after few days I realized that my backups is not working properly.
I checked and found very strange behavior. This is log from pod/couchbasebackup
kubectl logs pod/couchbase-backup-incremental-28221300-ljtch -n apps
/usr/local/lib/python3.8/dist-packages/requests/init.py:109: RequestsDependencyWarning: urllib3 (2.0.2) or chardet (None)/charset_normalizer (3.1.0) doesn’t match a supported version!
warnings.warn(
Traceback (most recent call last):
File “/usr/local/bin/backup.py”, line 1243, in
Backup(context).run()
File “/usr/local/bin/backup.py”, line 379, in run
self._setup_logging()
File “/usr/local/bin/backup.py”, line 1156, in _setup_logging
file_handler = logging.FileHandler(filename=self._get_logs_abs_path(),
File “/usr/lib/python3.8/logging/init.py”, line 1147, in init
StreamHandler.init(self, self._open())
File “/usr/lib/python3.8/logging/init.py”, line 1176, in _open
return open(self.baseFilename, self.mode, encoding=self.encoding)
OSError: [Errno 28] No space left on device: ‘/data/scriptlogs/backup/2023-08-29T03_01_03.log’
No space left on device I checked free space and found that only 1.5 gigabytes out of 5 gigabytes of hard drive is used.
And to give a proof this is outptu from kubectl get couchbasebackup couchbase-backup -o yaml
apiVersion: couchbase.com/v2
kind: CouchbaseBackup
metadata:
annotations:
meta.helm.sh/release-name: couchbase-release
meta.helm.sh/release-namespace: apps
creationTimestamp: “2023-08-02T17:31:13Z”
generation: 109
labels:
app.kubernetes.io/managed-by: Helm
cluster: couchbase-release
name: couchbase-backup
namespace: apps
resourceVersion: “115777197”
uid: 27cbfa84-1cd7-40e6-a1df-3579f7c4aacf
spec:
backoffLimit: 2
backupRetention: 720h
ephemeralVolume: false
failedJobsHistoryLimit: 3
full:
schedule: 0 3 * * 0
incremental:
schedule: 0 3 * * 1-6
logRetention: 168h
objectStore:
secret: azure-secret
uri: az://cbbackup
useIAM: false
services:
analytics: true
bucketConfig: true
bucketQuery: true
clusterAnalytics: true
clusterQuery: true
data: true
eventing: true
ftsAliases: true
ftsIndexes: true
gsIndexes: true
views: true
size: 5Gi
strategy: full_incremental
successfulJobsHistoryLimit: 1
threads: 1
status:
archive: az://cbbackup/archive
backups:
- full: 2023-08-15T20_19_18.026423127Z
incrementals:- 2023-08-16T03_00_23.152342302Z
- 2023-08-17T03_00_26.114923499Z
- 2023-08-18T03_00_32.024195992Z
- 2023-08-19T03_00_30.619394995Z
name: couchbase-release-2023-08-15T20_19_15
- full: 2023-08-20T03_00_34.305474025Z
incrementals:- 2023-08-21T03_00_40.506386234Z
- 2023-08-22T03_00_43.216405629Z
- 2023-08-23T03_00_50.281174379Z
- 2023-08-24T03_00_55.846861426Z
- 2023-08-25T03_00_57.820162133Z
- 2023-08-26T03_01_06.659720506Z
name: couchbase-release-2023-08-20T03_00_17
- full: 2023-08-27T03_01_10.868515494Z
incrementals:
name: couchbase-release-2023-08-27T03_00_31
capacityUsed: 1512Mi
cronjob: deprecated
duration: 62s
failed: true
job: deprecated
lastFailure: “2023-08-28T03:03:58Z”
lastRun: “2023-08-28T03:02:55Z”
lastSuccess: “2023-08-27T03:07:23Z”
pod: deprecated
repo: couchbase-release-2023-08-27T03_00_31
running: false
Question: what’s going on???
And I have another question:
if I use cloud storage for backups, then why is the local copy also stored on disk? why is there no deletion or something like that? After all, that’s why I set up cloud storage, so as not to store data locally, not to think about free space