no space left on device

#1

I went to one of my machines where the cron backups are not happening, to find out why.

I executed
./vertical backup

I got
Vertical Backup 1.1.2
Licensed to
Storage set to /vmfs/volumes/datastore2/vbackup
Listing all virtual machines
Backing up CentOS7Minimal, id: 3, vmx path: /vmfs/volumes/datastore1/CentOS7Minimal/CentOS7Minimal.vmx, guest os: centos7_64Guest
Last backup at revision 7 found
Virtual machine CentOS7Minimal is powered on
Removing all snapshots of CentOS7Minimal
Creating a new virtual machine snapshot for CentOS7Minimal
Failed to save ‘/vmfs/volumes/datastore2/vbackup/chunks/35/c5/ffaf3464ed0f8b3da3b2537431aca0cfa0c97a0f0a8b7d473ab5699f5ce7.e602af7b’: No space left on device
Removing all snapshots of CentOS7Minimal

So then I did a df to check space on the datastore2:

VMFS-5     930.2G 151.4G    778.9G  16% /vmfs/volumes/datastore2

It looks like it has plenty of space.

What else could be wrong?

Thanks

#2

It could be that your esxi was out of inodes. Please follow the instructions in this article to see if this is the case: https://kb.vmware.com/kb/1007638

#3

It does not appear out of inodes on root:

> stat -f /
File: "/"
    ID: 100000000 Namelen: 127     Type: visorfs
Block size: 4096      
Blocks: Total: 423400     Free: 234579     Available: 234579
Inodes: Total: 655360     Free: 649056

However, the numbers reported on the datastores are not useful:

> stat -f /vmfs/volumes/datastore2
  File: "/vmfs/volumes/datastore2"
    ID: 365500583c8eaf8 Namelen: 127     Type: vmfs
Block size: 1048576   
Blocks: Total: 952576     Free: 797020     Available: 797020
Inodes: Total: 2147483647 Free: 2147483647

> stat -f /vmfs/volumes/datastore1
  File: "/vmfs/volumes/datastore1"
    ID: 12e1f6e41ec0bc60 Namelen: 127     Type: vmfs
Block size: 1048576   
Blocks: Total: 221184     Free: 112952     Available: 112952
Inodes: Total: 2147483647 Free: 2147483647

I believe it may be out of inodes due to the vertical backups. I can do some pruning, but this is a concern if my storage is already out of inodes. So far, I only have less than 10 days of backups for four VM’s in the storage on datastore2. Is it normal to run out of inodes so quickly?

#4

Allright, I did a file count on the affected device (datastore 2) and it came out to exactly 130000 files.

I’ve seen references that vmfs5 is limited to 130k.

So apparently it’s reached the limit.

I have to figure out what to do next. So far in my searches, I can’t find whether migrating to vmfs6 increases the number of inodes available on a datastore or not.

If so, that’s probably the easiest route long term.

For now, I’m doing a massive pruning to get this machine back working.

#5

Allright - I decided that this is too much of a hassle doing backups directly to VMFS partitions.

So now I’ve set it up to backup to a linux partition of one of my virtual machines, which will then be copied offsite using duplicacy.

So far, this seems to be working much more smoothly (and faster!!!).

It would be very helpful to have a more extensive guide covering backup scenarios/configurations, so that problems like this coming from trial-and-error (which are time wasting), are reduced. Not all of us are experienced esxi sysadmins, and I for one can’t really afford the time that it’s taken me to get everything working.

Though my time is limited, I may be able to write something up about my configuration, especially if e.g. there were some reward for doing so.

Thanks
Morgan

#6

Sure, if you can write such a guide to share your experiences using Vertical Backup, I can extend all your licenses for one year for free!

#7

okay - thanks! I will see if I can find some time soon - this week is very busy!