misterbean Apr 4 6:46PM 2018
Vertical backup 1.1.5 on ESXI 6.5, vertical runs out of memory running backup. Tried "--no-chunk-listing", did not help.
|----Total Memory......5.00 MB
|----Touched...........1.12 MB
|----Total Minimum (base)1.22 MB
|----Total Maximum (base)1.22 MB
|----Effective Minimum.1.74 MB
|----Target Allocation.1.22 MB
|----Total Memory......777.00 MB
|----Reserved Memory...773.00 MB
|----COW...............28.04 MB
|----Mapped............749.25 MB
|----Touched...........68.80 MB
|----Total Minimum (base)773.62 MB
|----Total Maximum (base)773.62 MB
|----Effective Minimum.773.86 MB
|----Target Allocation.773.62 MB
./vertical backup --no-chunk-listing fs
Vertical Backup 1.1.5
Trial license expires on 2018-05-01
Storage set to sftp://esxibackup@192.168.24.6/esxi
Listing all virtual machines
Backing up fs, id: 10, vmx path: /vmfs/volumes/datastore1/fs/fs.vmx, guest os: debian8_64Guest
No previous backup found
Virtual machine fs is powered on
Removing all snapshots of fs
Creating a new virtual machine snapshot for fs
Uploaded file /vmfs/volumes/datastore1/fs/fs.vmdk
Uploaded file /vmfs/volumes/datastore1/fs/data.vmdk
Uploaded file /vmfs/volumes/datastore1/fs/space1.vmdk
Uploaded file /vmfs/volumes/datastore1/fs/mm2.vmdk
Uploaded file /vmfs/volumes/datastore1/fs/data2-000002.vmdk
Uploaded file /vmfs/volumes/datastore1/fs/data2-000001.vmdk
Uploaded file /vmfs/volumes/datastore1/fs/data2.vmdk
Uploaded file /vmfs/volumes/59600a57-09287b8c-d7b0-0025904656a9/fs/mm.vmdk
Uploaded file /vmfs/volumes/datastore1/fs/fs_1.vmdk
Uploading file data2-flat.vmdk
Uploaded file data2-flat.vmdk 197.91MB/s 00:25:52
Uploading file fs-flat.vmdk
Uploaded file fs-flat.vmdk 183.46MB/s 00:01:51
Uploading file data2-000001-delta.vmdk
Uploaded file data2-000001-delta.vmdk 12.92MB/s 00:00:01
Uploading file data-flat.vmdk
Uploaded file data-flat.vmdk 172.00MB/s 01:51:45
Uploading file mm2-flat.vmdk
Uploaded file mm2-flat.vmdk 215.09MB/s 00:25:23
Uploading file data2-000002-delta.vmdk
Uploaded file data2-000002-delta.vmdk 87.93MB/s 00:00:00
Uploading file mm-flat.vmdk
Uploading ***************************************************--------------------------- 101.54MB/s 01:27:00 66.3%
Failed to read file '/vmfs/volumes/59600a57-09287b8c-d7b0-0025904656a9/fs/mm-flat.vmdk': MemoryError()
gchen Apr 4 9:25PM 2018
How big is the mm-flat disk?
misterbean Apr 6 2:21PM 2018
mm-flat.vmdk is approx 1.65TB
gchen Apr 7 8:37PM 2018
I'll release a new version next week that should use much less memory. I'll let you know when it is ready.
gchen Apr 13 9:44PM 2018
Sorry I hasn't been able to work on the new version this week. Should be able to get it done early next week. I'll send you a new 30-day trial license when the new version is out.
gchen Apr 17 10:11PM 2018
I uploaded a new version 1.2.0 at http://acrosync.com/esxi/vertical.
This version should be much more memory efficient than previous versions.
If you log into your customer page you should also see a 30-day license there. You can activate it by running vertical license activate and then entering the license code.
misterbean Apr 19 1:31PM 2018
Hi.. Thank you for the updated version. Unfortunately, it also seems to fail with a memory error:
|----Total Memory......24.00 MB
|----Reserved Memory...2.00 MB
|----Touched...........1.12 MB
|----Total Minimum (base)1.22 MB
|----Total Maximum (base)1.22 MB
|----Effective Minimum.2.20 MB
|----Target Allocation.1.22 MB
|----Total Memory......773.00 MB
|----Reserved Memory...751.00 MB
|----COW...............27.33 MB
|----Mapped............720.88 MB
|----Touched...........68.03 MB
|----Total Minimum (base)750.18 MB
|----Total Maximum (base)750.18 MB
|----Effective Minimum.751.62 MB
|----Target Allocation.750.18 MB
[root@localhost:/vmfs/volumes/59600a57-09287b8c-d7b0-0025904656a9/verticalbackup] ./vertical backup fs
Vertical Backup 1.2.0
Trial license expires on 2018-05-01
Storage set to sftp://esxibackup@192.168.24.6/esxi
Listing all virtual machines
Backing up fs, id: 10, vmx path: /vmfs/volumes/datastore1/fs/fs.vmx, guest os: debian8_64Guest
No previous backup found
Virtual machine fs is powered on
Removing all snapshots of fs
Creating a new virtual machine snapshot for fs
Uploaded file /vmfs/volumes/datastore1/fs/fs.vmdk
Uploaded file /vmfs/volumes/datastore1/fs/data.vmdk
Uploaded file /vmfs/volumes/datastore1/fs/space1.vmdk
Uploaded file /vmfs/volumes/datastore1/fs/mm2.vmdk
Uploaded file /vmfs/volumes/datastore1/fs/data2-000003.vmdk
Uploaded file /vmfs/volumes/datastore1/fs/data2-000002.vmdk
Uploaded file /vmfs/volumes/datastore1/fs/data2-000001.vmdk
Uploaded file /vmfs/volumes/datastore1/fs/data2.vmdk
Uploaded file /vmfs/volumes/59600a57-09287b8c-d7b0-0025904656a9/fs/mm-000001.vmdk
Uploaded file /vmfs/volumes/59600a57-09287b8c-d7b0-0025904656a9/fs/mm.vmdk
Uploaded file /vmfs/volumes/datastore1/fs/fs_1-000001.vmdk
Uploaded file /vmfs/volumes/datastore1/fs/fs_1.vmdk
Uploading file data2-flat.vmdk
Uploaded file data2-flat.vmdk 196.56MB/s 00:26:02
Uploading file data2-000003-delta.vmdk
Uploaded file data2-000003-delta.vmdk 35.16MB/s 00:00:03
Uploading file fs-flat.vmdk
Uploaded file fs-flat.vmdk 157.69MB/s 00:02:09
Uploading file data2-000001-delta.vmdk
Uploaded file data2-000001-delta.vmdk 95.27MB/s 00:00:00
Uploading file data-flat.vmdk
Uploaded file data-flat.vmdk 164.25MB/s 01:57:02
Uploading file fs_1-flat.vmdk
Uploaded file fs_1-flat.vmdk 204.75MB/s 01:25:21
Uploading file fs_1-000001-delta.vmdk
Uploaded file fs_1-000001-delta.vmdk 91.37MB/s 00:00:00
Uploading file mm2-flat.vmdk
Uploaded file mm2-flat.vmdk 206.25MB/s 00:26:28
Uploading file data2-000002-delta.vmdk
Uploaded file data2-000002-delta.vmdk 77.13MB/s 00:00:00
Uploading file mm-flat.vmdk
Uploading ******************************************************************************- 74.84MB/s 00:02:44 99.2%Exception in thread Thread-26:
Traceback (most recent call last):
File "threading.py", line 801, in __bootstrap_inner
File "threading.py", line 754, in run
File "vertical_snapshot.py", line 529, in uploadChunkThread
File "vertical_utils.py", line 391, in encryptAESGCM
MemoryError
gchen Apr 19 9:17PM 2018
You can try to increase the chunk size by initializing the storage this way:
./vertical init -c 4M hostid sftp://esxibackup@192.168.24.6/esxi
This should reduce the size of the chunk list in memory. The downside is that the deduplication will become less efficient with a larger chunk size. However, I think the 4MB chunk size is reasonable for disks larger than 1TB (this is the default average chunk size in Duplicacy).
misterbean Apr 25 2:08PM 2018
Seems to be working better with the larger chunksize. The downside is that while I have a couple of disks in the terabyte range, most of the disks are more in the 20GB to 100GB range. Tnx