Snapshot timeout

Lars · January 13, 2018, 2:32am

Hi,

I am testing backup of an average VM (140 GB of virtual disks), but without success.

Shortly after the backup is started, it fails:

./vertical backup WS2008
Vertical Backup 1.1.5
Trial license expires on 2018-02-11
Storage set to s3://s3.dualstack.eu-central-1.amazonaws.com/bucket
Listing all virtual machines
Backing up WS2008, id: 1, vmx path: /vmfs/volumes/datastore1/WS2008/WS2008.vmx, guest os: windows7Server64Guest
Last backup at revision 1 found
Virtual machine WS2008 is powered on
Removing all snapshots of WS2008
Creating a new virtual machine snapshot for WS2008
Command ‘/bin/vim-cmd vmsvc/snapshot.create 1 2018-01-13-02-01-36 ‘Created by Vertical Backup 1.1.5’ 0 1’ times out
Create Snapshot:
Command ‘/bin/vim-cmd vmsvc/snapshot.create 1 2018-01-13-02-01-36 ‘Created by Vertical Backup 1.1.5’ 0 1’ returned -9
Removing all snapshots of WS2008

When I checked the system, I found out that the snapshot task was still running, finished successfuly after minutes later:

vim-cmd vimsvc/task_list
(ManagedObjectReference) [
‘vim.Task:haTask-1-vim.VirtualMachine.removeAllSnapshots-117620809’,
‘vim.Task:haTask-1-vim.VirtualMachine.removeAllSnapshots-117620872’,
‘vim.Task:haTask-1-vim.VirtualMachine.createSnapshot-117620833’
]

Is there something wrong with my configuration (ESXi 6.5) or is vertical simply too impatient?

Thanks,
Lars

gchen · January 13, 2018, 7:20pm

The timeout for removing all snapshots is 20 minutes which I think is enough for most cases. But, if you haven’t done snapshot consolidation recently, the delta disk file can be very large and removing all snapshots may take a long time.

Now that the snapshot removing task has successfully finished, the timeout should not happen when you run the backup command again.

Lars · January 13, 2018, 10:52pm

I have tried it several times. There is no existing snapshot on the VM.

When I run the same command as seen in the log by hand, it lasts 3-5 minutes and finishes just fine. Vertical consistently fails during snapshot creation, even though - as seen in the ESXi logs - the snapshot succeeds. The point is that vertical fails when the snapshot is still running. The execution time until it fails is cca 2 minutes, definitelly not 20 or even more.

gchen · January 14, 2018, 5:04am

Can you try this new build at http://acrosync.com/esxi/vertical (sha256: e42e52c125f5268b50a5c68ce631ff39bda02a423720e89bf13d89efd44aa920)?

The timeout for creating a new snapshot has been increased from 2 minutes to 10 minutes. This is only a temporary fix – ideally there should be an option to set these timeout values.

Lars · January 14, 2018, 10:04am

Hi, it did not help:

time ./vertical\ (2) backup WS2008
Vertical Backup 1.1.5
Trial license expires on 2018-02-11
Storage set to s3://s3.dualstack.eu-central-1.amazonaws.com/bucket
Listing all virtual machines
Backing up WS2008, id: 1, vmx path: /vmfs/volumes/datastore1/WS2008/WS2008.vmx, guest os: windows7Server64Guest
Last backup at revision 1 found
Virtual machine WS2008 is powered on
Removing all snapshots of WS2008
Creating a new virtual machine snapshot for WS2008
Command ‘/bin/vim-cmd vmsvc/snapshot.create 1 2018-01-14-09-56-03 ‘Created by Vertical Backup 1.1.5’ 0 1’ times out
Create Snapshot:
Command ‘/bin/vim-cmd vmsvc/snapshot.create 1 2018-01-14-09-56-03 ‘Created by Vertical Backup 1.1.5’ 0 1’ returned -9
Removing all snapshots of WS2008
Command exited with non-zero status 2
real 2m 23.44s
user 0m 0.09s
sys 0m 0.00s

sha
256sum vertical*
e42e52c125f5268b50a5c68ce631ff39bda02a423720e89bf13d89efd44aa920 vertical (2)

The total execution time was less than 3 minutes, there is still some problem with the timeout limit. If you plan to introduce timeout option, please consider an option to disable timeout completely if the timeout value is zero.

Thanks,
Lars

gchen · January 14, 2018, 7:08pm

Sorry my bad. There are two places where a new snapshot can be created and I only changed the timeout in one of them.

The new build at http://acrosync.com/esxi/vertical (sha256: d0fa9bebca2f7ae40902c6e10ef6a4b6427a0ffddfab08ac83db6e6a1dbffb0c) should fix this.

Lars · January 15, 2018, 12:29am

I am sorry, not working again.

This time it does create the snapshot, but does not detect the successful creation, then removes it again (visible in ESXi log files) and fails.

time ./vertical\ (3) backup WS2008
Vertical Backup 1.1.5
Trial license expires on 2018-02-11
Storage set to s3://s3.dualstack.eu-central-1.amazonaws.com/jtk-vertical/
Listing all virtual machines
Backing up WS2008, id: 1, vmx path: /vmfs/volumes/datastore1/WS2008/WS2008.vmx, guest os: windows7Server64Guest
Last backup at revision 1 found
Virtual machine WS2008 is powered on
Removing all snapshots of WS2008
Creating a new virtual machine snapshot for WS2008
Backup can’t proceed when there is no snapshot and an attempt to create a new one failed
Removing all snapshots of WS2008
Command exited with non-zero status 2
real 2m 51.69s
user 0m 0.09s
sys 0m 0.00s

gchen · January 15, 2018, 10:49pm

Can you manually run the snapshot creation command, and then check under the vm directory if there is a delta disk file for each disk?

Lars · January 16, 2018, 2:36pm

Confirmed:

vim-cmd vmsvc/snapshot.create 1 20180116 test 0 1

command succeeded, delta file created:

-rw------- 1 root root 16945152 Jan 16 14:34 WS2008-000001-delta.vmdk

gchen · January 18, 2018, 1:04am

I added a 10 second delay after the snapshot is created. Hope that is enough for Vertical Backup to detect the new snapshot.

The latest build can be downloaded from the same link with a sha256 of 11a9459a2a93047d16fe5e9399186c33f383e8843ac19b601527cc543eb897bf.

Lars · January 18, 2018, 7:12am

Hi,

I am sorry to say that your last change did not help again. The same behaviour, vertical creates a snapshot, then fails to detect it and at the end it removes it again.:

Vertical Backup 1.1.5
Trial license expires on 2018-02-11
Storage set to s3://s3.dualstack.eu-central-1.amazonaws.com/bucket
Listing all virtual machines
Backing up WS2008, id: 1, vmx path: /vmfs/volumes/datastore1/WS2008/WS2008.vmx, guest os: windows7Server64Guest
Last backup at revision 1 found
Virtual machine WS2008 is powered on
Removing all snapshots of WS2008
Creating a new virtual machine snapshot for WS2008
Backup can’t proceed when there is no snapshot and an attempt to create a new one failed
Removing all snapshots of WS2008

Just an idea: the VM includes two disks, one regular and one independent-persistent, which is unaffected by snapshots. Could this be the reason?

Thanks,
Lars

gchen · January 19, 2018, 2:20am

Vertical Backup can’t back up independent-persistent disks, so you will have to exclude that disk from the backup:

./vertical backup WS2008 --exclude-disk "*disk_name*"

The exclude-disk argument is the full path to the disk file but you can use wildcards instead to avoid specifying the entire path.

Lars · January 22, 2018, 9:27am

Yup, now it works, thanks.

Can we expect some improvements in the documentation and error detection / messages?

Thanks again for your support, testing resumed
Lars