It is the prune command in the pre-backup script that was out of memory. A process running directly inside the ESXi host can only have a limited amount of memory (less then 1GB) so you will run into this memory problem if you have lots of backups to prune. I would suggest using Duplicacy on another computer to prune the backups which is free for this purpose.