If you’re using AWS Backup Gateway to back up a standalone VMware VM on an ESXi 6.5 host and run into issues where backups start failing, you’re not alone. Here’s a straightforward approach to troubleshoot and resolve the problem.
First, it’s important to understand your environment. In this case, the AWS Backup Gateway is running as a virtual machine on the same VMware ESXi host, which is standalone without vCenter server. The production VM is quite large, about 2.4 TB, with an independent disk, and the gateway stays online constantly.
The problem began around June 26, 2026. After that date, backup jobs stopped working, and the error message indicated that the gateway went offline during backup. When analyzing the environment, it’s clear that the gateway’s connectivity to VMware and the network seems solid. Connections, authentication, SSL sessions, and VDDK sessions are all working properly.
To confirm the gateway’s basic functions, a simple test was performed by creating a small, 10 GB test VM on the same host. Backups and restores on that VM went smoothly, indicating that the gateway and its connection to ESXi are functioning correctly, and the backup and restore processes are operational in general.
The next step is to look deeper into the logs. By enabling CloudWatch logs and examining the exported log groups, some key observations were made. The VDDK logs show successful connection, authentication, SSL negotiation, and disk access initialization. The only unusual message appears as an error 20 during certain operations, stating that the command isn’t supported on remote objects. Interestingly, this message appears during both successful and failed backups, suggesting it isn’t the root cause.
Other logs show no authentication failures or network errors, and there are no disconnections or communication issues recorded. The Hypervisor error logs mostly contain messages about managing temporary VMs, which aren’t directly relevant to the ongoing backup failures.
Looking at the timeline, backups operated normally until on or before June 26, when the first failure occurred. After that, every subsequent attempt failed, and the snapshot delta files on the datastore accumulated, consuming all available space. Though cleaning up these snapshots restored the VM’s health, the backup process kept failing.
One suspected trigger is an automatic update to the Backup Gateway software around the time the issues began. Unfortunately, the current version of the gateway installed isn’t directly identifiable from the appliance or AWS console, making it difficult to confirm if a bug was introduced. A possible regression after this update is suspected.
So, what are some questions and potential steps you can take?
First, understand why the gateway reports itself as offline while VDDK communication still works. Is there an internal log or status check not visible in CloudWatch? Are there known issues or bugs with the latest Gateway version, especially for standalone ESXi 6.5 hosts? And, are there problems troubleshooting large VMs or ones with independent disks?
One approach might be to roll back to an earlier Gateway version if possible, to see if the backups succeed again. Additionally, monitoring for updates or patches that address known issues could help. Ensuring enough datastore space and removing orphaned snapshots regularly can also prevent space issues from halting backups.
In short, consistent checks of logs, connectivity, and software versions are key. If similar problems are seen elsewhere, reaching out to AWS support or your vendor’s community forum could provide insights. Large VMs or complex configurations sometimes uncover limitations or bugs, so focusing on these details can guide your troubleshooting.
By methodically examining your environment and logs, and considering software updates, you can better pinpoint the cause and restore reliable backups.