I tried backing up a VM server (we use as a database) that is quite large (around 800 GBs). The backup process took at least a day and half. At the end I saw this error message:
Creating a new virtual machine snapshot for MYDB
Uploaded file /vmfs/volumes/VM2-SSD/MYDB/MYDB.vmdk
Uploaded file /vmfs/volumes/VM2-SSD/MYDB/MYDB_3.vmdk
Uploading file MYDB_3-flat.vmdk
Uploaded file MYDB_3-flat.vmdk 6.62MB/s 1 day 06:05:43
Uploading file MYDB-flat.vmdk
Uploaded file MYDB-flat.vmdk 13.52MB/s 01:40:57
Uploaded file MYDB.vmx
Uploaded file MYDB.vmxf
Removing all snapshots of MYDB
Command ‘/bin/vim-cmd vmsvc/snapshot.removeall 6’ times out
Exception in thread Thread-12:
Traceback (most recent call last):
File “threading.py”, line 801, in __bootstrap_inner
File “threading.py”, line 1073, in run
File “vertical_backup.py”, line 31, in runCommandTimeout
File “vertical_utils.py”, line 46, in LOG_ERROR
File “vertical_utils.py”, line 80, in log
VerticalBackupError: COMMAND_RUN Command ‘/bin/vim-cmd vmsvc/snapshot.removeall 6’ times out
Remove All Snapshots:
Command ‘/bin/vim-cmd vmsvc/snapshot.removeall 6’ returned -9
/opt/verticalbackup # df -h
The default timeout for removing the snapshots is 1200 seconds. Obviously that was not enough in your case.
So the backup file didn’t get a chance to be uploaded, while all chunks that had been uploaded would remain in the storage.
The real issue here is that the backup file should be uploaded first before removing the snapshots. It doesn’t matter if the final snapshot removal takes too long. It will likely be done when the next backup is to be started.
So I made a change to move the uploading of the backup file to before the snapshot removal and uploaded a new version 1.0.5. You can download it from http://verticalbackup.com/esxi/vertical.
Ok thanks will download! What should I do after downloading the latest version? Just run another backup and it will remove the previous backup attempt?
Currently I have space on the backup drive that is allocated for the backup for this backup but not sure how to free up space?
Right, just run another backup which will be much faster since many chunks have been uploaded by the previous backup attempt. However, some chunks uploaded by the previous backup attempt will become unreferenced due to changes in the disk files, so you’ll need to clean up these chunks after the backup is finished.
This command will print out all unreferenced chunks:
vertical prune -d --exclusive --exhaustive
Without the -d option it will remove all unreferenced chunks (assuming no backup to the same storage is in progress):
You can run pgrep vertical to see if the backup process is running, pkill vertical to kill it.
If pgrep vertical doesn’t report any process while you still get “Another backup job is in progress” then you can manually remove the file .vertical/plock.
No, it doesn’t log to a file unless you redirected stdout to a file when you started the program. You can run strace -p pid – if the backup is running it will print out tons of messages.
Thanks for your response… Now whenever I execute a vertical command I get the following error (was trying to do a vertical list):
File “vertical_main.py”, line 8, in
File “/tmp/pip-build-VI6Ne5/pyinstaller/PyInstaller/loader/pyimod03_importers.py”, line 389, in load_module
File “vertical_backup.py”, line 23, in
File “/tmp/pip-build-VI6Ne5/pyinstaller/PyInstaller/loader/pyimod03_importers.py”, line 389, in load_module
File “vertical_snapshot.py”, line 18, in
OSError: [Errno 28] No space left on device
Failed to execute script vertical_main
But I have space available on the vmware server and the data backup server:
Your /tmp dir is full. There might be some folders similar to /tmp/pip-build-VI6Ne5 which were left by previous Vertical Backup processes not exiting cleanly. You can remove these folders to free up space.
Storage set to sftp://user@172.16.8.55//home/vmwarebackup
Listing all virtual machines
Backing up MYDB, id: 6, vmx path: /vmfs/volumes/VM2-SSD/MYDB/MYDB.vmx, guest os: centos64Guest
No previous backup found
Virtual machine MYDB is powered on
Removing all snapshots of MYDB
Remove All Snapshots:
Remove all snapshots failed
Command ‘/bin/vim-cmd vmsvc/snapshot.removeall 6’ returned 1
The backup is not running. I killed the previous vertical process as I had no idea on what it was doing.
You can do a manual snapshot consolidation in vSphere Client (Right client the VM -> Snapshot -> Consolidate) and wait until it’s completed and then start a backup again.
