No Space left on Device error


#1

Our scheduled cron backup as failed 2 consecutive times since we’ve made a change to ESX on a VM. We’ve moved the drive from a smaller SSD to a spinning disk and assigned this disk 1TB which is thin provisioned (only 200 odd GB in use).

Failure 1:
2019-04-16 19:35:15.063526 INFO COMMAND_FAILURE Failed to run command ‘/bin/vim-cmd vmsvc/snapshot.removeall 1’: No space left on device

2019-04-16 19:35:15.063734 ERROR COMMAND_FAILURE Failed to run command '/bin/vim-cmd vmsvc/snapshot.removeall 1': No space left on device

Failure 2:
2019-04-17 19:32:14.623791 INFO COMMAND_FAILURE Failed to run command ‘/bin/vim-cmd vmsvc/power.getstate 2’: No space left on device

2019-04-17 19:32:14.624127 ERROR COMMAND_FAILURE Failed to run command '/bin/vim-cmd vmsvc/power.getstate 2': No space left on device

Vertical runs from the spinning disk which according to ESX has a capacity of 1.82TB, 216.97GB provisioned and 1.61TB free. This then backs up to a Synology box (SYN-01) which has 1.48TB free as per screenshot below.

I manually run the backup yesterday which worked fine so I don’t see why the cron backup would be failing. Any ideas or suggestions?!

P.s I’ve also run an Inodes check on both Syn-01 and /
[icfadmin@hv:/vmfs/volumes/4c2e0dc0-a2afa1cc-16eb-0025903c61d8/verticalbackup] stat -f /
File: “/”
ID: 100000000 Namelen: 127 Type: visorfs
Block size: 4096
Blocks: Total: 342606 Free: 185852 Available: 185852
Inodes: Total: 655360 Free: 649045
[icfadmin@hv:/vmfs/volumes/4c2e0dc0-a2afa1cc-16eb-0025903c61d8/verticalbackup] stat -f /vmfs/volumes/syn-01/
File: “/vmfs/volumes/syn-01/”
ID: 4280421435b97038 Namelen: 127 Type: nfs
Block size: 4096
Blocks: Total: 719884784 Free: 410421987 Available: 410396387
Inodes: Total: 182845440 Free: 180655669


#2

This is actually an out-of-memory issue. I guess the memory usage was at the border line so sometimes it failed but not when you manually ran it.

The latest version 1.3.1 at http://verticalbackup.com/esxi/vertical should fix this issue. In version 1.3.0 we use a on-disk hash table to store the list of existing chunk hashes which significantly reduced the memory consumption.


#3

Thanks @gchen - upgrading to the latest version worked a treat!