Out of memory during backup


#1

Vertical backup 1.1.5 on ESXI 6.5, vertical runs out of memory running backup. Tried “–no-chunk-listing”, did not help.

|----Total Memory…5.00 MB
|----Touched…1.12 MB
|----Total Minimum (base)1.22 MB
|----Total Maximum (base)1.22 MB
|----Effective Minimum.1.74 MB
|----Target Allocation.1.22 MB
|----Total Memory…777.00 MB
|----Reserved Memory…773.00 MB
|----COW…28.04 MB
|----Mapped…749.25 MB
|----Touched…68.80 MB
|----Total Minimum (base)773.62 MB
|----Total Maximum (base)773.62 MB
|----Effective Minimum.773.86 MB
|----Target Allocation.773.62 MB

./vertical backup --no-chunk-listing fs
Vertical Backup 1.1.5
Trial license expires on 2018-05-01
Storage set to sftp://esxibackup@192.168.24.6/esxi
Listing all virtual machines
Backing up fs, id: 10, vmx path: /vmfs/volumes/datastore1/fs/fs.vmx, guest os: debian8_64Guest
No previous backup found
Virtual machine fs is powered on
Removing all snapshots of fs
Creating a new virtual machine snapshot for fs
Uploaded file /vmfs/volumes/datastore1/fs/fs.vmdk
Uploaded file /vmfs/volumes/datastore1/fs/data.vmdk
Uploaded file /vmfs/volumes/datastore1/fs/space1.vmdk
Uploaded file /vmfs/volumes/datastore1/fs/mm2.vmdk
Uploaded file /vmfs/volumes/datastore1/fs/data2-000002.vmdk
Uploaded file /vmfs/volumes/datastore1/fs/data2-000001.vmdk
Uploaded file /vmfs/volumes/datastore1/fs/data2.vmdk
Uploaded file /vmfs/volumes/59600a57-09287b8c-d7b0-0025904656a9/fs/mm.vmdk
Uploaded file /vmfs/volumes/datastore1/fs/fs_1.vmdk
Uploading file data2-flat.vmdk
Uploaded file data2-flat.vmdk 197.91MB/s 00:25:52
Uploading file fs-flat.vmdk
Uploaded file fs-flat.vmdk 183.46MB/s 00:01:51
Uploading file data2-000001-delta.vmdk
Uploaded file data2-000001-delta.vmdk 12.92MB/s 00:00:01
Uploading file data-flat.vmdk
Uploaded file data-flat.vmdk 172.00MB/s 01:51:45
Uploading file mm2-flat.vmdk
Uploaded file mm2-flat.vmdk 215.09MB/s 00:25:23
Uploading file data2-000002-delta.vmdk
Uploaded file data2-000002-delta.vmdk 87.93MB/s 00:00:00
Uploading file mm-flat.vmdk
Uploading ***************************************************--------------------------- 101.54MB/s 01:27:00 66.3%
Failed to read file ‘/vmfs/volumes/59600a57-09287b8c-d7b0-0025904656a9/fs/mm-flat.vmdk’: MemoryError()


#2

How big is the mm-flat disk?


#3

mm-flat.vmdk is approx 1.65TB


#4

I’ll release a new version next week that should use much less memory. I’ll let you know when it is ready.


#5

Sorry I hasn’t been able to work on the new version this week. Should be able to get it done early next week. I’ll send you a new 30-day trial license when the new version is out.


#6

I uploaded a new version 1.2.0 at http://acrosync.com/esxi/vertical.

This version should be much more memory efficient than previous versions.

If you log into your customer page you should also see a 30-day license there. You can activate it by running vertical license activate and then entering the license code.


#7

Hi…
Thank you for the updated version. Unfortunately, it also seems to fail with a memory error:

                                    |----Total Memory......24.00 MB
                                         |----Reserved Memory...2.00 MB
                                         |----Touched...........1.12 MB
                                         |----Total Minimum (base)1.22 MB
                                         |----Total Maximum (base)1.22 MB
                                         |----Effective Minimum.2.20 MB
                                         |----Target Allocation.1.22 MB
                                         |----Total Memory......773.00 MB
                                         |----Reserved Memory...751.00 MB
                                         |----COW...............27.33 MB
                                         |----Mapped............720.88 MB
                                         |----Touched...........68.03 MB
                                         |----Total Minimum (base)750.18 MB
                                         |----Total Maximum (base)750.18 MB
                                         |----Effective Minimum.751.62 MB
                                         |----Target Allocation.750.18 MB

[root@localhost:/vmfs/volumes/59600a57-09287b8c-d7b0-0025904656a9/verticalbackup] ./vertical backup fs
Vertical Backup 1.2.0
Trial license expires on 2018-05-01
Storage set to sftp://esxibackup@192.168.24.6/esxi
Listing all virtual machines
Backing up fs, id: 10, vmx path: /vmfs/volumes/datastore1/fs/fs.vmx, guest os: debian8_64Guest
No previous backup found
Virtual machine fs is powered on
Removing all snapshots of fs
Creating a new virtual machine snapshot for fs
Uploaded file /vmfs/volumes/datastore1/fs/fs.vmdk
Uploaded file /vmfs/volumes/datastore1/fs/data.vmdk
Uploaded file /vmfs/volumes/datastore1/fs/space1.vmdk
Uploaded file /vmfs/volumes/datastore1/fs/mm2.vmdk
Uploaded file /vmfs/volumes/datastore1/fs/data2-000003.vmdk
Uploaded file /vmfs/volumes/datastore1/fs/data2-000002.vmdk
Uploaded file /vmfs/volumes/datastore1/fs/data2-000001.vmdk
Uploaded file /vmfs/volumes/datastore1/fs/data2.vmdk
Uploaded file /vmfs/volumes/59600a57-09287b8c-d7b0-0025904656a9/fs/mm-000001.vmdk
Uploaded file /vmfs/volumes/59600a57-09287b8c-d7b0-0025904656a9/fs/mm.vmdk
Uploaded file /vmfs/volumes/datastore1/fs/fs_1-000001.vmdk
Uploaded file /vmfs/volumes/datastore1/fs/fs_1.vmdk
Uploading file data2-flat.vmdk
Uploaded file data2-flat.vmdk 196.56MB/s 00:26:02
Uploading file data2-000003-delta.vmdk
Uploaded file data2-000003-delta.vmdk 35.16MB/s 00:00:03
Uploading file fs-flat.vmdk
Uploaded file fs-flat.vmdk 157.69MB/s 00:02:09
Uploading file data2-000001-delta.vmdk
Uploaded file data2-000001-delta.vmdk 95.27MB/s 00:00:00
Uploading file data-flat.vmdk
Uploaded file data-flat.vmdk 164.25MB/s 01:57:02
Uploading file fs_1-flat.vmdk
Uploaded file fs_1-flat.vmdk 204.75MB/s 01:25:21
Uploading file fs_1-000001-delta.vmdk
Uploaded file fs_1-000001-delta.vmdk 91.37MB/s 00:00:00
Uploading file mm2-flat.vmdk
Uploaded file mm2-flat.vmdk 206.25MB/s 00:26:28
Uploading file data2-000002-delta.vmdk
Uploaded file data2-000002-delta.vmdk 77.13MB/s 00:00:00
Uploading file mm-flat.vmdk
Uploading ******************************************************************************- 74.84MB/s 00:02:44 99.2%Exception in thread Thread-26:
Traceback (most recent call last):
File “threading.py”, line 801, in __bootstrap_inner
File “threading.py”, line 754, in run
File “vertical_snapshot.py”, line 529, in uploadChunkThread
File “vertical_utils.py”, line 391, in encryptAESGCM
MemoryError


#8

You can try to increase the chunk size by initializing the storage this way:

./vertical init -c 4M hostid sftp://esxibackup@192.168.24.6/esxi

This should reduce the size of the chunk list in memory. The downside is that the deduplication will become less efficient with a larger chunk size. However, I think the 4MB chunk size is reasonable for disks larger than 1TB (this is the default average chunk size in Duplicacy).


#9

Seems to be working better with the larger chunksize.
The downside is that while I have a couple of disks in the terabyte range, most of the disks are more in the 20GB to 100GB range.
Tnx