Slow backup to mounted drive

#1

Hi,

For some reason upload to mounted drive is limited to 5Mb/s.

My setup:

ls /vmfs/volumes/

311e72e2-4cae7b69-be37-66b745ef2970 59940e55-4c6c944c-9a3a-38d547ca552a ARCHIVE1 S-SSD2
59940661-89cae3f8-c5a9-38d547ca552a 5a078c6c-374971af-9907-38d547ca552a P-SSD1 b330efe4-9ff20e61-02e6-65cfcc9491b9
59940d62-f9eabb98-83fe-38d547ca552a 5a178628-1e3337e4-a519-38d547ca552a S-SSD1

As you can see from here I have 2 physical SSD and 1 HDD drives mounted to ESXI as storages.
ESXI itself installed to a USB stick.
ESXI version: 6.5.0 Update 1 (Build 6765664) - free license

I installed vertical backup to storage ARCHIVE1:

mkdir /vmfs/volumes/ARCHIVE1/verticalbackup
cd /vmfs/volumes/ARCHIVE1/verticalbackup
wget http://verticalbackup.com/esxi/vertical
chmod u+x vertical

initialized it:

./vertical init esxi /vmfs/volumes/ARCHIVE1/backups/

and started backup of vm-1 with thin provisioned 100GB HDD:

./vertical backup vm-1

The files of vm-1 are placed on SSD1, the backups folder is on ARCHIVE1. Physically these storages on the same machine. I’d expect quite fast speed but instead I see 5MB/s tops.

Uploading *****************-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 4.16MB/s 06:25:00 6.Uploading *****************-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 4.16MB/s 06:25:02 6.Uploading *****************-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 4.16MB/s
…and so on…

Here’s the result of the benchmark run:

./vertical benchmark

Vertical Backup 1.1.5
Test directory: /vmfs/volumes/59940d62-f9eabb98-83fe-38d547ca552a
Creating a 1,024M test file with random data
Write time: 10.439, speed: 98.097MB/s
Read time: 3.422, speed: 299.231MB/s
Creating another 1,024M test file with random data
Write time: 11.264, speed: 90.908MB/s
Read and hash time: 6.559, speed: 156.124MB/s
Storage set to /vmfs/volumes/ARCHIVE1/backups/
Upload time: 189.294, speed: 5.410MB/s
Download time: 56.614, speed: 18.087MB/s

ARCHIVE1 is not used by anything else except the backup tool. What’s wrong with it?

#2 
Storage set to /vmfs/volumes/ARCHIVE1/backups/
Upload time: 189.294, speed: 5.410MB/s
Download time: 56.614, speed: 18.087MB/s

In the upload test, 1024 random files with a size of 1M bytes are uploaded. This reflects the default size of chunks uploaded when backing up a virtual machine. So I think there is something wrong with the writing speed of that disk.

#3

Upgraded ESXI from ESXi-6.5.0-20171004001-standard (Build 6765664) to ESXi-6.5.0-20171201001s-standard (Build 7273056).

./vertical benchmark

Vertical Backup 1.1.5
Test directory: /vmfs/volumes/59940d62-f9eabb98-83fe-38d547ca552a
Creating a 1,024M test file with random data
Write time: 5.281, speed: 193.888MB/s
Read time: 3.256, speed: 314.493MB/s
Creating another 1,024M test file with random data
Write time: 5.335, speed: 191.927MB/s
Read and hash time: 5.896, speed: 173.688MB/s
Storage set to /vmfs/volumes/ARCHIVE1/backups/
Deleting old remote test files

Upload time: 92.026, speed: 11.127MB/s
Download time: 14.169, speed: 72.270MB/s

I tried to move around 30-50Gb vmdk files from SSDs to HDD back and forth, the speed seems to be normal.
According to the benchmark results write/read speeds are good: 193.888MB/s / 314.493MB/s, upload/download speeds are crap. Any thoughts?

#4

Can you run ./vertical benchmark --datastore /vmfs/volumes/ARCHIVE1/backups/ to test the write/read speeds of the HDD?

#5

@gchen
./vertical benchmark /vmfs/volumes/ARCHIVE1/backups/

Vertical Backup 1.1.5

Test directory: /vmfs/volumes/ARCHIVE1/backups/

Creating a 1,024M test file with random data

Write time: 20.812, speed: 49.203MB/s

Read time: 6.064, speed: 168.860MB/s

Creating another 1,024M test file with random data

Write time: 22.423, speed: 45.668MB/s

Read and hash time: 6.311, speed: 162.245MB/s

Storage set to /vmfs/volumes/ARCHIVE1/backups/

Upload time: 90.062, speed: 11.370MB/s

Download time: 14.081, speed: 72.725MB/s

#6

So the speed of writing a 1G file is 49MB/s, while the speed of writing 1024 1M files is 11MB/s. There may be some issue with your HDD? What format is that disk? Can you try copying many small files to it to see what the speed is?

#7

@gchen,
It’s brand new Western Digital 4TB Intellipower SATA 6Gb/s 64 MB Cache 3.5-Inch NAS Desktop Hard Disk Drive - Red (WD40EFRX). I converted it into ESXi’s storage drive.
May be my usb flash is dying and it causes such side effects, will try to replace it.