Out of memory during backup

Vertical backup 1.1.5 on ESXI 6.5, vertical runs out of memory running backup. Tried “–no-chunk-listing”, did not help.

|----Total Memory…5.00 MB
|----Touched…1.12 MB
|----Total Minimum (base)1.22 MB
|----Total Maximum (base)1.22 MB
|----Effective Minimum.1.74 MB
|----Target Allocation.1.22 MB
|----Total Memory…777.00 MB
|----Reserved Memory…773.00 MB
|----COW…28.04 MB
|----Mapped…749.25 MB
|----Touched…68.80 MB
|----Total Minimum (base)773.62 MB
|----Total Maximum (base)773.62 MB
|----Effective Minimum.773.86 MB
|----Target Allocation.773.62 MB

./vertical backup --no-chunk-listing fs
Vertical Backup 1.1.5
Trial license expires on 2018-05-01
Storage set to sftp://esxibackup@192.168.24.6/esxi
Listing all virtual machines
Backing up fs, id: 10, vmx path: /vmfs/volumes/datastore1/fs/fs.vmx, guest os: debian8_64Guest
No previous backup found
Virtual machine fs is powered on
Removing all snapshots of fs
Creating a new virtual machine snapshot for fs
Uploaded file /vmfs/volumes/datastore1/fs/fs.vmdk
Uploaded file /vmfs/volumes/datastore1/fs/data.vmdk
Uploaded file /vmfs/volumes/datastore1/fs/space1.vmdk
Uploaded file /vmfs/volumes/datastore1/fs/mm2.vmdk
Uploaded file /vmfs/volumes/datastore1/fs/data2-000002.vmdk
Uploaded file /vmfs/volumes/datastore1/fs/data2-000001.vmdk
Uploaded file /vmfs/volumes/datastore1/fs/data2.vmdk
Uploaded file /vmfs/volumes/59600a57-09287b8c-d7b0-0025904656a9/fs/mm.vmdk
Uploaded file /vmfs/volumes/datastore1/fs/fs_1.vmdk
Uploading file data2-flat.vmdk
Uploaded file data2-flat.vmdk 197.91MB/s 00:25:52
Uploading file fs-flat.vmdk
Uploaded file fs-flat.vmdk 183.46MB/s 00:01:51
Uploading file data2-000001-delta.vmdk
Uploaded file data2-000001-delta.vmdk 12.92MB/s 00:00:01
Uploading file data-flat.vmdk
Uploaded file data-flat.vmdk 172.00MB/s 01:51:45
Uploading file mm2-flat.vmdk
Uploaded file mm2-flat.vmdk 215.09MB/s 00:25:23
Uploading file data2-000002-delta.vmdk
Uploaded file data2-000002-delta.vmdk 87.93MB/s 00:00:00
Uploading file mm-flat.vmdk
Uploading ***************************************************--------------------------- 101.54MB/s 01:27:00 66.3%
Failed to read file ‘/vmfs/volumes/59600a57-09287b8c-d7b0-0025904656a9/fs/mm-flat.vmdk’: MemoryError()

How big is the mm-flat disk?

mm-flat.vmdk is approx 1.65TB

I’ll release a new version next week that should use much less memory. I’ll let you know when it is ready.

Sorry I hasn’t been able to work on the new version this week. Should be able to get it done early next week. I’ll send you a new 30-day trial license when the new version is out.

I uploaded a new version 1.2.0 at http://acrosync.com/esxi/vertical.

This version should be much more memory efficient than previous versions.

If you log into your customer page you should also see a 30-day license there. You can activate it by running vertical license activate and then entering the license code.

Hi…
Thank you for the updated version. Unfortunately, it also seems to fail with a memory error:

                                    |----Total Memory......24.00 MB
                                         |----Reserved Memory...2.00 MB
                                         |----Touched...........1.12 MB
                                         |----Total Minimum (base)1.22 MB
                                         |----Total Maximum (base)1.22 MB
                                         |----Effective Minimum.2.20 MB
                                         |----Target Allocation.1.22 MB
                                         |----Total Memory......773.00 MB
                                         |----Reserved Memory...751.00 MB
                                         |----COW...............27.33 MB
                                         |----Mapped............720.88 MB
                                         |----Touched...........68.03 MB
                                         |----Total Minimum (base)750.18 MB
                                         |----Total Maximum (base)750.18 MB
                                         |----Effective Minimum.751.62 MB
                                         |----Target Allocation.750.18 MB

[root@localhost:/vmfs/volumes/59600a57-09287b8c-d7b0-0025904656a9/verticalbackup] ./vertical backup fs
Vertical Backup 1.2.0
Trial license expires on 2018-05-01
Storage set to sftp://esxibackup@192.168.24.6/esxi
Listing all virtual machines
Backing up fs, id: 10, vmx path: /vmfs/volumes/datastore1/fs/fs.vmx, guest os: debian8_64Guest
No previous backup found
Virtual machine fs is powered on
Removing all snapshots of fs
Creating a new virtual machine snapshot for fs
Uploaded file /vmfs/volumes/datastore1/fs/fs.vmdk
Uploaded file /vmfs/volumes/datastore1/fs/data.vmdk
Uploaded file /vmfs/volumes/datastore1/fs/space1.vmdk
Uploaded file /vmfs/volumes/datastore1/fs/mm2.vmdk
Uploaded file /vmfs/volumes/datastore1/fs/data2-000003.vmdk
Uploaded file /vmfs/volumes/datastore1/fs/data2-000002.vmdk
Uploaded file /vmfs/volumes/datastore1/fs/data2-000001.vmdk
Uploaded file /vmfs/volumes/datastore1/fs/data2.vmdk
Uploaded file /vmfs/volumes/59600a57-09287b8c-d7b0-0025904656a9/fs/mm-000001.vmdk
Uploaded file /vmfs/volumes/59600a57-09287b8c-d7b0-0025904656a9/fs/mm.vmdk
Uploaded file /vmfs/volumes/datastore1/fs/fs_1-000001.vmdk
Uploaded file /vmfs/volumes/datastore1/fs/fs_1.vmdk
Uploading file data2-flat.vmdk
Uploaded file data2-flat.vmdk 196.56MB/s 00:26:02
Uploading file data2-000003-delta.vmdk
Uploaded file data2-000003-delta.vmdk 35.16MB/s 00:00:03
Uploading file fs-flat.vmdk
Uploaded file fs-flat.vmdk 157.69MB/s 00:02:09
Uploading file data2-000001-delta.vmdk
Uploaded file data2-000001-delta.vmdk 95.27MB/s 00:00:00
Uploading file data-flat.vmdk
Uploaded file data-flat.vmdk 164.25MB/s 01:57:02
Uploading file fs_1-flat.vmdk
Uploaded file fs_1-flat.vmdk 204.75MB/s 01:25:21
Uploading file fs_1-000001-delta.vmdk
Uploaded file fs_1-000001-delta.vmdk 91.37MB/s 00:00:00
Uploading file mm2-flat.vmdk
Uploaded file mm2-flat.vmdk 206.25MB/s 00:26:28
Uploading file data2-000002-delta.vmdk
Uploaded file data2-000002-delta.vmdk 77.13MB/s 00:00:00
Uploading file mm-flat.vmdk
Uploading ******************************************************************************- 74.84MB/s 00:02:44 99.2%Exception in thread Thread-26:
Traceback (most recent call last):
File “threading.py”, line 801, in __bootstrap_inner
File “threading.py”, line 754, in run
File “vertical_snapshot.py”, line 529, in uploadChunkThread
File “vertical_utils.py”, line 391, in encryptAESGCM
MemoryError

You can try to increase the chunk size by initializing the storage this way:

./vertical init -c 4M hostid sftp://esxibackup@192.168.24.6/esxi

This should reduce the size of the chunk list in memory. The downside is that the deduplication will become less efficient with a larger chunk size. However, I think the 4MB chunk size is reasonable for disks larger than 1TB (this is the default average chunk size in Duplicacy).

Seems to be working better with the larger chunksize.
The downside is that while I have a couple of disks in the terabyte range, most of the disks are more in the 20GB to 100GB range.
Tnx