You are looking at the documentation of a prior release. To read the documentation of the latest release, please visit here.
"repository is already locked " issue
Sometimes, the backend repository gets locked and the subsequent backup fails. In this guide, we are going to explain why this can happen and what you can do to solve the issue.
If the repository gets locked, the new backup will fail. If you describe the
BackupSession, you should see an error message indicating that the repository is already locked by another process.
kubectl describe -n <namespace> backupsession <backupsession name>
You will also see the error message in the backup sidecar/job log.
# For backup that uses sidecar (i.e. Deployment, StatefulSet etc.)
kubectl logs -n <namespace> <workload pod name> -c stash
# For backup that uses job (i.e. Database, PVC, etc.)
kubectl logs -n <namespace> <backup job's pod name> --all-containers
A restic process that modifies the repository, creates a lock at the beginning of its operation. When it completes the operation, it removes the lock so that other restic processes can use the repository. Now, if the process is killed unexpectedly, it can not remove the lock. As a result, the repository remains locked and becomes unusable for other processes.
The repository can get locked in the following scenarios.
If the workload pod that has the
stash sidecar or backup job’s pod gets terminated while a backup is running, the repository can get locked. In this case, you have to find out why the pod was terminated.
Stash uses an
emptyDir as a temporary volume where it stores cache for improving backup performance. By default, the
emptyDir does not have any size limit. However, if you set the limit manually using
spec.tempDir section of
BackupConfiguration make sure you have set it to a reasonable size based on your targeted data size. If the
tempDir limit is too low, cache size may cross the limit resulting in the backup pod getting evicted by Kubernetes. This is a tricky case because you may not notice that the backup pod has been evicted. You can describe the respective workload/job to check if it was the case.
In such a scenario, make sure that you have set the
tempDir size to a reasonable amount. You can also disable caching by setting
spec.tempDir.disableCaching: true. However, this might impact the backup performance significantly.
If your repository gets locked, you have to unlock it manually. You can use one of the following methods.
At first, install the Stash
kubectl plugin by following the instruction here.
Then, run the following command to unlock the repository:
kubectl stash unlock <repository name> --namespace=<namespace>
If you are using a cloud bucket that provides a UI to browse the storage, you can go to the repository directory and delete the
Once you have found the issue of why the repository got locked in the first place, take the necessary measure to prevent it from occurring in the future.