Duplicacy compatibility

#1

I’ve succeeded in getting some backups running now and things looks quite promising in my trialling of Vertical Backup; the incremental backups are going pretty fast thanks to the de-duplication…

Now I’m planning to use the duplicacy -copy command to sync backups off-site via SFTP, but first I want to ‘seed’ the initial copy by attaching a new storage locally. But then I ran into some issues with the encryption:

F:\dummy>duplicacy -d init -encrypt from-dummy "F:\verticalbackup"
Reading the environment variable DUPLICACY_PASSWORD
Enter storage password for F:\verticalbackup:*********************************
Using a static salt and 16384 iterations for key derivation
Chunk read levels: [2 3], write level: 2
The storage 'F:\verticalbackup' has already been initialized
Compression level: 100
Average chunk size: 1048576
Maximum chunk size: 1048576
Minimum chunk size: 1048576
Chunk seed: 766572746963616c6261636b7570
F:\dummy will be backed up to F:\verticalbackup with id from-dummy

So far so good.

F:\dummy>duplicacy set -no-backup
New options for storage F:\verticalbackup have been saved

F:\dummy>duplicacy list -all
Storage set to F:\verticalbackup
Enter storage password:*********************************
Failed to decrypt the file snapshots/CDS-Server@cds-esxi6/1: cipher: message authentication failed

Hmm?

F:\dummy>duplicacy -d list -id CDSMail@cds-esxi6 -r 4
Storage set to F:\verticalbackup
Reading the environment variable DUPLICACY_PASSWORD
Reading password from keychain/keyring
Using a static salt and 16384 iterations for key derivation
Chunk read levels: [2 3], write level: 2
Compression level: 100
Average chunk size: 1048576
Maximum chunk size: 1048576
Minimum chunk size: 1048576
Chunk seed: 766572746963616c6261636b7570
Reading the environment variable DUPLICACY_PASSWORD
id: CDSMail@cds-esxi6, revisions: [4], tag: , showFiles: false, showChunks: false
Failed to decrypt the file snapshots/CDSMail@cds-esxi6/4: cipher: message authentication failed

F:\verticalbackup is the path to the Windows/cygwin SSHD storage for Vertical Backup. I also tried to initialise the dummy storage with the original sftp://username@://cygdrive/f/verticalbackup URL, but I get the same error.

I’m 99.999% sure the storage isn’t corrupt in any way, nor should there be permissions issues (I ran Command Prompt as Administrator, just to be sure). On ESXi I’m able to ./vertical list --all --files just fine.

It’s my understanding that Vertical Backup and Duplicacy should be directly compatible. Why would Duplicacy not be able to read the encrypted snapshots? I mean, it seems to be able to read the encrypted config file from the storage using the supplied master password.

Running Vertical Backup v1.2.0 (the memory efficient version posted here) and Duplicacy v2.1.0.

Any help would be much appreciated. :slight_smile:

#2

This is a bug caused by a mistake in Vertical Backup using a different hash to derive the encryption key. It has been fixed in the latest build available from https://acrosync.com/esxi/vertical (sha256: ebe74bcfd81287a24764417d95512d2caa89078af0ae8e8a183a026d71504417).

The storage must be initialized with this build for the fix to work. If you need to work with an existing storage, you can build the latest version of Duplicacy from the github master branch, and set the environment variable DUPLICACY_DECRYPT_WITH_HMACSHA256 to 1 before running Duplicacy. This environment variable was added by this commit to fix this compatibility issue.

#3

Thank you for the very swift fix!

I think, since my storage has only 3 VMs and a handful of revisions, and that the new hashing function is the future(?) and the default now, I’ll just reinitialise to a fresh storage. I’ll also test duplicacy -copy after the first VM in case I run into further problems (although I don’t expect to). Cheers.

#4

Continuing on from my experiments, and although I could be doing something wrong, it seems Vertical Backup isn’t able to decrypt a copy-compatible storage that’s been copied from the original with ‘duplicacy copy’.

I get the following error message:

[root@localhost:/vmfs/volumes/Datastore1/verticalbackup] ./vertical -v init --encrypt cert-esxi6 sftp://username@192.168.xxx.xxx//cygdrive/g/verticalbackup
Vertical Backup 1.2.0
Running init command -- chunk size: 1048576, encrypt: True, temporary directory: None
Storage set to sftp://username@192.168.xxx.xxx//cygdrive/g/verticalbackup
Enter the password to encrypt/decrypt the storage:
The downloaded file 'config' is either corrupted or not encrypted

Basically I’m trying to run a test restore on the ‘copied’ storage that I’m now syncing across the internet from the original host. My setup:

host-A —> storage-A —> [duplicacy add -copy -encrypt ++ duplicacy copy] —> storage-B —> host-B

Both storages are copy-compatible but use a different password for encryption (I didn’t use -bit-identical either, which is intentional). I’m currently successfully running the backup on host-A to storage-A and also copying from storage-A and storage-B is now working.

Just to throw a spanner in the works, my passwords are completely random and the one for storage-B has a backslash in it, which I’ve escaped with double-backslash in the json-format ‘password’ file. But I’ve also tried pasting the storage password from my password manager directly into the PuTTY terminal.

What makes me think this is a Vertical Backup issue is I can initialise and restore one of the VMs’ .vmx files (as a simple test) from storage B, using Duplicacy. I can also, with Vertical Backup, initialise/reconnect to storage-A from host-B across the internet. VB just doesn’t like storage-B for some reason.

Once again, any help is much appreciated!

#5

This is due to this change: https://github.com/gilbertchen/duplicacy/pull/244. Duplicacy now uses a new format for the config file which can’t be read by Vertical Backup. I’ll fix this tomorrow.

#6

Can you try this 1.2.1 build at http://acrosync.com/esxi/vertical (sha256: 5cce486badfef863b0e004ea2d386db2cdf13e6905fd650a339ea421e43254be)?

#7

Sorry for the delay, only just got around to testing the new build…

The init command now works - I get the normal message “The storage has already been initialized with a chunk size of 1048576” and “Preference has been saved to /vmfs/volumes/Datastore1/verticalbackup/.verticalbackup/preferences” - no errors there.

When I try to run ./vertical list -a (or specify a VM in the @ format) I get the following:

[root@localhost:/vmfs/volumes/Datastore1/verticalbackup] ./vertical list -a
Vertical Backup 1.2.1
Storage set to sftp://username@192.168.xxx.xxx//cygdrive/g/verticalbackup
Failed to decrypt the downloaded file 'snapshots/CDS-Server@cds-esxi6/1': MAC check failed
#8

Please run the updated 1.2.1 build at http://acrosync.com/esxi/vertical (sha256: c9cf7a408b6be918c2f2be40c9aa980aca67f82fdeae4ffc9063f0a4cfc27e68), and also add a new file named config.vb next to config in the storage directory with the following content:

{
    "decrypt-with-hmac-sha256": false
}

This is still the same bug caused by the difference in key derivation. The previous fix added a new flag in the config file which will force Vertical Backup to use the same key derivation algorithm as Duplicacy. However, when Duplicacy creates a new compatible config file from an existing storage, this flag gets lost. Therefore, a workaround is to add that flag back in the config.vb file.

This is supposed to be a temporary fix. Once Duplicacy starts to preserve the key derivation flag in the config file, this fix (and the config.vb file) won’t be needed any more.

#9

Many thanks for the quick patch, this is now working and I’m currently running a restore…

Strangely, it’s putting the files in a deeper nested structure than planned:

The destination directory '/vmfs/volumes/datastore1/CDSMail' does not exist; use '/vmfs/volumes/Datastore2/CDSMail/vmfs/volumes/datastore1/CDSMail' instead

…even though I created a blank VM on the new host and tried specifying the VM by name and by path i.e.:

./vertical restore /vmfs/volumes/Datastore2/CDSMail --restore-from CDSMail@cds-esxi6 -r 10 --threads 8

But hopefully, once the restore completes, I can manually move those files back to where they should be(?).

I was wondering gchen if you could possibly extend the trial period for my Vertical Backup as I’m rapidly running out of time to test some other scenarios (thread count, storage on external drive vs internal, copy vs direct to cloud). Cheers.

#10

I generated a 30-day license under your account. You should be able to see it on your customer page. To activate, run ./vertical license activate and enter the license code.

#11

GChen many thanks for the trial extension. Today the original trial activation expired and I tried to input the new activation code, but it doesn’t want to accept it:

Activation code has already been assigned to the host id esxi****

Tried renaming license to license.bak and same result.

#12

It should work now. If this happens again, you can click the host id on the customer page and change it to the matching one.

#13

Excellent, thank you. Works a charm.