failed to upload

#1

Any ideas?

[root@hv:/vmfs/volumes/4c2e0dc0-a2afa1cc-16eb-0025903c61d8/verticalbackup] ./vertical backup ts1
Vertical Backup 1.1.1
Licensed to Tickett Enterprises Limited; expires on 2018-08-30
Storage set to sftp://backup@172.17.0.100/vmfs/volumes/HDD/backup
Listing all virtual machines
Backing up ts1, id: 2, vmx path: /vmfs/volumes/SSD/ts1/ts1.vmx, guest os: windows9_64Guest
Last backup at revision 4 found
Virtual machine ts1 is powered on
Removing all snapshots of ts1
Creating a new virtual machine snapshot for ts1
Failed to upload 'vmfs/volumes/HDD/backup/chunks/4b/4e/04f8729463bf910bc5e9a7802f964638d98ca503cc9cbf10b4d9e7f23b22': Failure
Removing all snapshots of ts1
[root@hv:/vmfs/volumes/4c2e0dc0-a2afa1cc-16eb-0025903c61d8/verticalbackup]
#2

Running out of space? Failure is a very generic SFTP error that can caused by several different reasons. The most likely I can think of is that the server runs of the storage space to save the upload chunk.

#3

That was my first idea. Unfortunately loads of space;

[root@localhost:~] df -h
Filesystem   Size    Used Available Use% Mounted on
VMFS-5     465.5G 1019.0M    464.5G   0% /vmfs/volumes/SSD
VMFS-5       1.8T  107.2G      1.7T   6% /vmfs/volumes/HDD
vfat       285.8M  205.9M     80.0M  72% /vmfs/volumes/59786810-48e9eb79-b2b0-00                                     25900c529e
vfat       249.7M  143.8M    105.9M  58% /vmfs/volumes/fe0638e5-1b6dd0ba-1a9c-dc                                     094911ea19
vfat       249.7M  147.8M    102.0M  59% /vmfs/volumes/d27798e9-7daa00be-dcee-5d                                     9fd8598c30

I just tried backing up another VM which also failed so I don’t think it’s an issue with this specific VM.

What’s next?

#4

Hmmm…

[root@hv:/vmfs/volumes/4c2e0dc0-a2afa1cc-16eb-0025903c61d8/verticalbackup] ./vertical benchmark
Vertical Backup 1.1.1
Test directory: /vmfs/volumes/4c2e0db0-c42d9b1e-ed3c-0025903c61d8
Creating a 1,024M test file with random data
Write time: 12.808, speed: 79.950MB/s
Read time: 6.734, speed: 152.065MB/s
Creating another 1,024M test file with random data
Write time: 12.857, speed: 79.648MB/s
Read and hash time: 9.128, speed: 112.179MB/s
Storage set to sftp://backup@172.17.0.100/vmfs/volumes/HDD/backup
Failed to upload 'vmfs/volumes/HDD/backup/bench/fbdbdedd7ac8e2e8f3f9aaf5c1616ca0015fc5715fbeb2909094dba5893ef9cf': Failure
#5

Very confusing…

[root@hv:/vmfs/volumes/4c2e0dc0-a2afa1cc-16eb-0025903c61d8/verticalbackup] scp ./vertical backup@172.17.0.100:/vmfs/volumes/HDD/backup
Password:
scp: /vmfs/volumes/HDD/backup/vertical: No space left on device
#6

Also tried the scp as root and I get the same error.

Trying to upload through the web interface it appears to’ve hung on 0%. Something very strange is going on.

#7

Is /vmfs/volumes/HDD/backup a mount point that is actually a ram disk?

#8

Nope. It’s one of my datastores (a 2TB spinning disk).

#9

I fear I may need to request a refund. I’m not convinced the product is mature/stable enough. I’ve now got a different error;

[root@hv:/vmfs/volumes/4c2e0dc0-a2afa1cc-16eb-0025903c61d8/verticalbackup] ./vertical backup dc1 ts1 --threads 4
Vertical Backup 1.1.1
Licensed to Tickett Enterprises Limited; expires on 2018-08-30
Storage set to sftp://backup@172.17.0.100/vmfs/volumes/HDD/backup
Listing all virtual machines
Backing up dc1, id: 1, vmx path: /vmfs/volumes/SSD/dc1/dc1.vmx, guest os: windows9Server64Guest
No previous backup found
Virtual machine dc1 is powered on
Removing all snapshots of dc1
Creating a new virtual machine snapshot for dc1
Uploaded file /vmfs/volumes/SSD/dc1/dc1.vmdk
Uploading file dc1-flat.vmdk
Using 4 uploading threads
Uploaded file dc1-flat.vmdk 5.70MB/s 02:59:45
Uploaded file dc1.vmx
Uploaded file dc1.vmxf
Backup dc1@hv1 at revision 1 has been successfully completed
Total 61449 chunks, 61444.52M bytes; 36895 new, 36890.52M bytes, 21027.88M uploaded
Total backup time: 02:59:51
Removing all snapshots of dc1
Backing up ts1, id: 2, vmx path: /vmfs/volumes/SSD/ts1/ts1.vmx, guest os: windows9_64Guest
No previous backup found
Virtual machine ts1 is powered on
Removing all snapshots of ts1
Creating a new virtual machine snapshot for ts1
Uploaded file /vmfs/volumes/SSD/ts1/ts1.vmdk
Uploading file ts1-flat.vmdk
Using 4 uploading threads
Uploaded file ts1-flat.vmdk 6.35MB/s 02:41:13
Uploaded file ts1.vmx
Uploaded file ts1.vmxf
Backup ts1@hv1 at revision 1 has been successfully completed
Total 61449 chunks, 61444.52M bytes; 26472 new, 26468.52M bytes, 17750.72M uploaded
Total backup time: 02:41:20
Removing all snapshots of ts1
Exception in thread Thread-7 (most likely raised during interpreter shutdown):Exception in thread Thread-3 (most likely raised during interpreter shutdown):
#10

No problem with the refund but this is apparently a bug in the shutdown process. I believe the backup had been successfully completed before the error occurred. I’ll fix the bug anyway.

How did you fix the out-of-space error?

Another note, in my own experience ESXi hosts are generally slow when used as SFTP servers. It may be much faster to run a dedicated linux box, physical or virtual, as SFTP servers.

#11

Great, thanks I will persevere for a little longer then (especially if you are adding the “exclude disks” option too).

I am curious why ESXis implementation of sftp might be slow. I will probably avoid using a dedicated linux box (vm or otherwise) as my bottleneck should be bandwidth going forward (at present the source and destination are both in my lab, but they will be deployed to a site with a 10Mbit uplink).

Can we sign up anywhere to receive notifications of new releases?

#12

Oh, and unfortunately i couldn’t fix the space issue. I had to destroy the datastore and recreate (format).

I may consider moving to S3 depending on speed and cost.

#13

I would suggest wasabi.com, a new S3-compatible provider that is faster and cheaper than Amazon S3.