Troubleshooting “ciphertext verification failed” issue

Sometimes, the backup starts failing after a few days with an error indicating ciphertext verification failed. In this guide, we are going to explain what might cause the issue and how to solve it.

Identifying the issue

The backup should run successfully for a few days. Suddenly, it will start failing. Any subsequent backups will fail too. If you describe the respective BackupSession or view the log from the respective backup sidecar/job, you should see the following error:

Fatal: ciphertext verification failed

Possible reasons

This can happen if the backed-up data get corrupted for any of these reasons.

• Someone deleted some files/folders from the backend manually.
• The respective bucket has a policy configured to delete the old data automatically.

Solution

At first, check if the bucket has any policy configured to delete the old data automatically. If this is the case, please remove that policy and depend only on the retention policy provided by Stash Repository to cleanup the old data.

For example, if you are using a GCS bucket, you can check for such policy in the LIFECYCLE tab.

Unfortunately, there is no known way to fix the corrupted repository. You have to delete all the corrupted data from the backend. Only then, the subsequent backups will succeed again.