Storage Vault Locks

Lock files are an important part of Magnus Box's safety design. Magnus Box uses lock files to ensure data consistency during concurrent operations.

Problem statement

Magnus Box supports multiple devices backing up into a shared Storage Vault simultaneously. When Magnus Box runs a retention pass to clean up data, it is important that no other backup jobs are running simultaneously.

A retention pass first looks at what data chunks exist in the Vault, then secondly deletes the unused ones.

A backup job first looks at what data chunks exist in the Vault, then secondly uploads new chunks from the local data, and uploads a backup snapshot that relies on both pre-existing and newly-uploaded chunks.

It is perfectly safe for multiple backup jobs to run simultaneously, even from multiple devices.

It is not safe for a retention pass to run at the same time as a backup job, because if the steps are interleaved (retention A > backup A > retention B > backup B) then a backup job might write a backup snapshot that refers to unknown chunks, resulting in data loss.

Magnus Box MUST prevent you from running a backup job and a retention pass simultaneously.

Lock files

In order to check whether a retention pass is currently running, Magnus Box must communicate between all devices that could potentially be using the Storage Vault.

In order to determine whether any other device is actively using a Storage Vault, Magnus Box writes a temporary text file into the Storage Vault, and deletes it when the job is completed. This is the only mechanism supported across all Storage Vault types (i.e. local disk / SFTP / S3 / etc). Then, any other job can look for these files to see what other operations are taking place concurrently.

Jobs in a Storage Vault are classified into two categories:

Exclusive (retention passes)
Non-exclusive (backup/restore jobs)

Multiple non-exclusive jobs may run simultaneously from any device. A non-exclusive job will refuse to start if any exclusive jobs are currently running. An exclusive job will refuse to start if any other jobs are running.

Specifically:

If a backup job is currently running, Magnus Box will refuse to start a retention pass.
If a retention pass is currently running, Magnus Box will refuse to start a backup job.

Downsides of lock file design

If Magnus Box is stopped suddenly (e.g. PC crash), the lock file would not be removed. All other Magnus Box processes would not realize that the job had stopped. This could prevent the proper functioning of backup jobs and/or retention passes.

Magnus Box will alert you to this issue by failing the job. The error message should explain which device and/or job was responsible for originating the now-stale lock file.

You may see error messages of the form:

Locked by user '...' on this device (PID #...) since ... (... days ago)
Locked by user '...' on computer '...' (PID #...) since ... (... days ago)
However, the responsible process might have stopped.
If you investigate this process, and are absolutely certain it won't resume, then it's safe to ignore it and continue.

It is possible to delete lock files to recover from this situation. However, you MUST investigate the issue to ensure that the responsible process really has stopped. Consider that a PC may go to sleep at any time, and wake up days or weeks later, and immediately resume from the middle of a backup or retention operation; if the lock files were removed incorrectly, data loss is highly likely.

If you are sure that the responsible process is stopped, you can delete the lock files.

You can initiate this either

in Magnus Box Backup, on the "Account" pane > right-click Storage Vault > "Advanced" > "Clean up lock files" option
remotely via the Magnus Box Server web interface (when logged in as an administrator).

Backup strategies to avoid lock file issues

If you are experiencing lock file errors where the duration is 5 minutes or less, consider rescheduling the backups, moving a backup ahead or behind by 30 minutes.
If you are experiencing lock file issues of more than 5 minutes (anywhere from several hours to multiple days of the storage vault being locked), consider moving one or more of the devices to a separate storage vault where the backups will not be affected by the other devices.

Automatic unlock

Magnus Box will automatically delete stale lock files when it determines that it is safe to do so.

When Magnus Box is running on the same PC as a potentially-stale lock file, it can check the running processes to see if the originator process is still running.

A future version of Magnus Box may be able to automatically detect and remove stale lock files in more situations.

Recovering from unsafe unlock operations

If you encounter a Packindex '...' for snapshot '...' refers to unknown pack '...', shouldn't happen error, a data file has been erroneously deleted inside the Storage Vault. Data has been lost. This can happen if the "Unlock" feature is used without proper caution as advised above.

In this situation, you can recover the remaining data in the Storage Vault by contacting [email protected]