Problems with ESXI 7

#1

Hi,
for testing i updated a server from VMWare ESXi 6.7U3 to VMWare ESXI 7
But now there is a problem backing up the vmdk files.

Here’s the verbose log of a backup call

[root@esxi:/vmfs/volumes/57e2d23c-90f51f2b-57a5-408d5c4a53e8/verticalbackup] ./vertical -v backup -t mtest Rohrpostix
Vertical Backup 1.4.0
Licensed to ; expires on 2020-11-13
Storage set to sftp://
Listing all virtual machines
Running command: /bin/vim-cmd vmsvc/getallvm, timeout: 30
Vmid Name File Guest OS Version Annotation
…
49 Rohrpostix [PCIE-Slot4] Rohrpostix/Rohrpostix.vmx ubuntu64Guest vmx-11
…
Backing up Rohrpostix, id: 49, vmx path: /vmfs/volumes/PCIE-Slot4/Rohrpostix/Rohrpostix.vmx, guest os: ubuntu64Guest
Last backup at revision 325 found
Running command: /bin/vim-cmd vmsvc/power.getstate 49, timeout: 30
Retrieved runtime info
Powered on
Virtual machine Rohrpostix is powered on
Removing all snapshots of Rohrpostix
Running command: /bin/vim-cmd vmsvc/snapshot.removeall 49, timeout: 600
Remove All Snapshots:
Creating a new virtual machine snapshot for Rohrpostix
Running command: /bin/vim-cmd vmsvc/snapshot.create 49 2020-04-27-08-42-24 ‘Created by Vertical Backup 1.4.0’ 0 1, timeout: 600
Create Snapshot:
Uploaded chunk 7df1eb59107d165f2690b546fd53c70f12049c7475ee572c77b8b8027ac22466 size 504
Uploaded file /vmfs/volumes/PCIE-Slot4/Rohrpostix/Rohrpostix.vmdk
Uploading file Rohrpostix-flat.vmdk
Failed to open file ‘/vmfs/volumes/PCIE-Slot4/Rohrpostix/Rohrpostix-flat.vmdk’: Device or resource busy
Removing all snapshots of Rohrpostix
Running command: /bin/vim-cmd vmsvc/snapshot.removeall 49, timeout: 600
Remove All Snapshots:

So I tried this manually and got the same error even though the file is in Read-Only mode according to vmfsfilelockinfo (I tried to access the file with head, which should be read-only for sure)

#>/bin/vim-cmd vmsvc/snapshot.create 49 2020-04-27-08-42-24 ‘Created by Vertical Backup 1.4.0’ 0 1
Create Snapshot:
#>vmfsfilelockinfo -p /vmfs/volumes/PCIE-Slot4/Rohrpostix/Rohrpostix-flat.vmdk
vmfsfilelockinfo Version 2.0
Looking for lock owners on “Rohrpostix-flat.vmdk”
“Rohrpostix-flat.vmdk” is locked in Read-Only mode by host having mac address [‘xx:xx:xx:xx:xx:xx’]
#>head -n 10 ‘/vmfs/volumes/PCIE-Slot4/Rohrpostix/Rohrpostix-flat.vmdk’
head: /vmfs/volumes/PCIE-Slot4/Rohrpostix/Rohrpostix-flat.vmdk: Device or resource busy

By googeling the error I found this interesting thread in the vmware forums: https://communities.vmware.com/thread/632492

so this seems to be either a undocumented change in vmware 7 or just a bug…

Is there a known workaround? Or maybe gchen can check with vmware if this will be changed again in a future update.

But for now I would be very careful with updating to ESXi 7 if you planning on using verticalbackup…

#2

I too have this same issue with ESXi 7.

...
2020-04-30 14:57:56.855424 INFO BACKUP_UPLOAD Uploaded file /vmfs/volumes/Samsung 860 EVO 500GB/UNIFI1/UNIFI1.vmdk
2020-04-30 14:57:56.856530 INFO BACKUP_UPLOAD Uploading file UNIFI1-flat.vmdk
2020-04-30 14:57:56.856829 ERROR FILE_READ Failed to open file '/vmfs/volumes/Samsung 860 EVO 500GB/UNIFI1/UNIFI1-flat.vmdk': Device or resource busy
...

verticalbackup currently doesn’t work with ESXi 7

#3

Unfortunately this does look like a VMware bug and there isn’t a workaround (other than stopping the vm first). Let’s hope that this is an unintentional bug that VMware will fix soon, otherwise we may be forced to switch to using the VADP api.

#4

It doesn’t look like Dell/VMware are treating this as a bug (feature?). I suspect it’s intentional as it does boost their licensing.
Have you begun any work to produce an API integrated vertical backup?
I believe the API is restricted/locked down when ESXi is running unlicensed. For those of us with licensed ESXi instances this shouldn’t be a problem but I suspect we are the minority of vertical backup users…

#5

@gchen, Are you working on a API version/extension of verticalbackup?

#6

At this time I’m not. I’m working on something else and will come back to think about the next step.

#7

It’s been several months. Any update on this yet?

#8

Sorry I haven’t made the plan to rewrite Vertical Backup to use the VADP api. That would take a few months at least.

#9

Not sure, but by reading the thread https://communities.vmware.com/thread/632492 I got the impression that there is a workaround.
Would you consider implementing it?

1 Like
#10

Not sure if you’ve seen it, but XSIBackup is able to workaround this issue on ESXi 7. Could you maybe do what they’re doing?

https://33hops.com/xsibackup-vmware-esxi-backup.html