An error occurred while taking a snapshot failed to quiesce the virtual machine

Try reinstalling or updating your VMware tools to fix the issue

Symptoms

1. You are using one of the Acronis Cyber Backup products on an ESX Host to back up virtual machines;
2. The backup of one or several of the virtual machines fails with an error message similar to one of the following:

An error occurred while saving the snapshot: Failed to quiesce the virtual machine. An error occurred while taking a snapshot: Failed to quiesce the virtual machine.

Creating a quiesced snapshot failed

An error occurred while quiescing the virtual machine. See the virtual machine’s event log for details.

VMware failed to create a quiesced snapshot of the selected virtual machine. To back up the machine, power off the machine before the process or uninstall the Volume Shadow Copy Services Support feature of VMware Tools from the VM.

Protocol error from VMX

An error occurred while saving the snapshot: A required file was not found.

Cause

Please see the solution section of this article for specific causes and resolution.

Solution

1.1 VM snapshot should work correctly

Acronis software uses snapshots executed by the hypervisor while taking the virtual machine backup. It is necessary that these snapshots work correctly. Create a virtual machine snapshot to check the consistency :

1. Open the vSphere connection to the ESX host.
2. Right click on the machine > take snapshot > check the «Quiesce guest file system» 
3. Uncheck the «Snapshot the virtual machine’s memory» and click OK.

If taking quiesced snapshot fails, investigate the issue with the virtual machine (contact VMware support if needed)

In most cases, reinstalling/updating VMware tools and rebooting the virtual machine will resolve the issue with quiesced snapshot. Please see the following articles:

• VMware Knowledge Base Article 1009073
• VMware Knowledge Base Article 5962168
• VMware Knowledge Base Article 1007696
• VMware Knowledge Base Article 1007346
• VMware Knowledge Base Article 80246

If quiesced snapshot is created successfully but backup with Acronis still fails with the same error message, contact Acronis Support with a reference to this article

1.2 Verify VSS status

If you backup a virtual machine with transactional applications installed (SQL, Exchange evt.), then:

• VMware Tools should be installed inside that guest OS.
• VSS service should not be disabled inside the guest OS.

In case the guest OS event logs contain errors or warnings for VSS operations during creation of the snapshot, troubleshoot the VSS issues inside the guest OS.

If a virtual machine snapshot fails due to incorrect guest file system quiescing (snapshot without option Quiesce quest file system is successful), for virtual machines with no transcational applications installed you can workaround the issue by uninstalling the VMware Tools for the guest OS (see More information below) or by disabling Volume Shadow Copy (VSS) for Virtual Machines backup option (for Acronis Cyber Protect Cloud or Acronis Cyber Backup 12 and above).

Keep in mind the VMware ESX(i) virtualization platform limitations:

• Agent for VMware backs up a fault tolerant machine only if fault tolerance was enabled in VMware vSphere 6.0 and later,
• VMware vSphere does not support snapshots of independent disks and Raw Device Mapping (RDM) disks in physical compatibility mode. To overcome these limitations install an agent into the guest OS.

1.3 If the error message is:

Creating a quiesced snapshot failed because the create snapshot operation exceeded the time limit for holding off I/O in the frozen virtual machine

Then see the following article for a solution: VMware Knowledge Base Article 1038963 

If ESX(i) version is 6.0, install this patch.

 The error may be caused by pre-backup command failure. A pre-backup command (pre-freeze script located in /usr/sbin/pre-freeze-script on a Linux machine or in a specific directory on a Windows machine) is a user-supplied script that should be executed before backup. Usually, these scripts are used to quiesce an application that does not have its own VSS provider (in Windows) to ensure a consistent state before backup. When such a script exits with a non-zero return code, this indicates that the script failed to perform its quiescing task and quiesced snapshot operation has to be cancelled.

 See also the following solutions:

• If VM freezes during snapshot creation, see A virtual machine can freeze under load when you take quiesced snapshots or use custom quiescing scripts;
• If VM runs a database, try giving it more processors/RAM. Allow VM to use more of the host/cluster resources.

More information

Acronis Cyber Backup 12.5: If the backed up VM does not run a database, you can disable quiescing of VMs via VSS by turning off Volume Shadow Copy for virtual machines (applies to Acronis Cyber Backup 12.5 only — there is no such option in 11.7 or lower). With quiescing disabled, the VM will be backed up in crash-consistent state.

For Acronis Cyber Backup 11.7 and older products, try disabling the VSS component of VMware Tools to solve the issue, but only if the backed up VM does not run a database:

Click to expand instructions for Acronis Cyber Backup 11.7 and older products

Here is the hostd.log of the ESXi:

2014-02-24T18:41:20.030Z [3D8C2B70 verbose ‘Statssvc.vim.PerformanceManager’] HostCtl Exception in stats collection: Sysinfo error on operation returned status : Not initialized. Please see the VMkernel log for detailed error information

2014-02-24T18:41:20.030Z [3D8C2B70 verbose ‘Statssvc.vim.PerformanceManager’] HostCtl Exception in stats collection.  Turn on ‘trivia’ log for details

2014-02-24T18:41:22.812Z [3D881B70 verbose ‘SoapAdapter’] Responded to service state request

2014-02-24T18:41:24.615Z [3D881B70 verbose ‘Cimsvc’] Ticket issued for CIMOM version 1.0, user root

2014-02-24T18:41:25.710Z [3C240B70 verbose ‘Hostsvc.DvsManager’] PersistAllDvsInfo called

2014-02-24T18:41:25.939Z [3D881B70 info ‘Hostsvc’ opID=hostd-9f65] VsanSystemVmkProvider : GetRuntimeInfo: Start

2014-02-24T18:41:25.939Z [3D881B70 info ‘Hostsvc’ opID=hostd-9f65] VsanSystemVmkProvider : GetRuntimeInfo: Complete, runtime info: (vim.vsan.host.VsanRuntimeInfo) {

—>    dynamicType = <unset>,

—>    accessGenNo = 0,

—> }

2014-02-24T18:41:31.114Z [3BEC2B70 verbose ‘Vmsvc.vm:/vmfs/volumes/d9b2943f-86ca1719/slu003/slu003.vmx’] Updating current heartbeatStatus: yellow

2014-02-24T18:41:32.031Z [3D8C2B70 verbose ‘SoapAdapter’] Responded to service state request

2014-02-24T18:41:35.396Z [FFDE4B70 verbose ‘Default’ opID=HB-host-516@6836-77a33b2-eb user=vpxuser] AdapterServer: target=’vim.PerformanceManager:ha-perfmgr’, method=’GetPerfCounter’

2014-02-24T18:41:35.404Z [3D8C2B70 verbose ‘Locale’ opID=HB-host-516@6836-77a33b2-eb user=vpxuser] Default resource used for ‘counter.vsanDomObj.writeThroughput.summary’ expected in module ‘perf’.

2014-02-24T18:41:35.423Z [FFDE4B70 verbose ‘Default’ opID=HB-host-516@6836-77a33b2-eb user=vpxuser] AdapterServer: target=’vim.HostSystem:ha-host’, method=’retrieveInternalCapability’

2014-02-24T18:41:35.425Z [FFDE4B70 verbose ‘Default’ opID=HB-host-516@6836-77a33b2-eb user=vpxuser] AdapterServer: target=’vim.PerformanceManager:ha-perfmgr’, method=’queryPerfCounterInt’

2014-02-24T18:41:35.437Z [FFDE4B70 verbose ‘Default’ opID=HB-host-516@6836-77a33b2-eb user=vpxuser] AdapterServer: target=’vim.LicenseManager:ha-license-manager’, method=’GetLicenses’

2014-02-24T18:41:35.437Z [FFDE4B70 verbose ‘Vimsvc.ha-license-manager’ opID=HB-host-516@6836-77a33b2-eb user=vpxuser] Load: Loading existing file: /etc/vmware/license.cfg

2014-02-24T18:41:35.452Z [FFDE4B70 verbose ‘Default’ opID=HB-host-516@6836-77a33b2-eb user=vpxuser] ha-license-manager:Validate -> Valid license detected for «VMware ESX Server 5.0» (lastError=0, desc.IsValid:Yes)

2014-02-24T18:41:40.029Z [3D8C2B70 verbose ‘Statssvc.vim.PerformanceManager’] HostCtl Exception in stats collection: Sysinfo error on operation returned status : Not initialized. Please see the VMkernel log for detailed error information

2014-02-24T18:41:40.030Z [3D8C2B70 verbose ‘Statssvc.vim.PerformanceManager’] HostCtl Exception in stats collection.  Turn on ‘trivia’ log for details

2014-02-24T18:41:45.422Z [3D881B70 verbose ‘Default’ opID=HB-host-516@6837-6cc68d7b-c6 user=vpxuser] AdapterServer: target=’vim.PerformanceManager:ha-perfmgr’, method=’GetPerfCounter’

2014-02-24T18:41:45.430Z [3D881B70 verbose ‘Locale’ opID=HB-host-516@6837-6cc68d7b-c6 user=vpxuser] Default resource used for ‘counter.vsanDomObj.writeThroughput.summary’ expected in module ‘perf’.

2014-02-24T18:41:45.452Z [3D881B70 verbose ‘Default’ opID=HB-host-516@6837-6cc68d7b-c6 user=vpxuser] AdapterServer: target=’vim.HostSystem:ha-host’, method=’retrieveInternalCapability’

2014-02-24T18:41:45.454Z [3C240B70 verbose ‘Default’ opID=HB-host-516@6837-6cc68d7b-c6 user=vpxuser] AdapterServer: target=’vim.PerformanceManager:ha-perfmgr’, method=’queryPerfCounterInt’

2014-02-24T18:41:45.466Z [FFDE4B70 verbose ‘Default’ opID=HB-host-516@6837-6cc68d7b-c6 user=vpxuser] AdapterServer: target=’vim.LicenseManager:ha-license-manager’, method=’GetLicenses’

2014-02-24T18:41:45.466Z [3D881B70 verbose ‘Vimsvc.ha-license-manager’ opID=HB-host-516@6837-6cc68d7b-c6 user=vpxuser] Load: Loading existing file: /etc/vmware/license.cfg

2014-02-24T18:41:45.481Z [3D881B70 verbose ‘Default’ opID=HB-host-516@6837-6cc68d7b-c6 user=vpxuser] ha-license-manager:Validate -> Valid license detected for «VMware ESX Server 5.0» (lastError=0, desc.IsValid:Yes)

2014-02-24T18:41:51.393Z [3C240B70 verbose ‘Default’ opID=1381cd91-97 user=vpxuser] AdapterServer: target=’vim.VirtualMachine:8′, method=’createSnapshot’

2014-02-24T18:41:51.394Z [3C240B70 info ‘Vimsvc.TaskManager’ opID=1381cd91-97 user=vpxuser] Task Created : haTask-8-vim.VirtualMachine.createSnapshot-228252373

2014-02-24T18:41:51.394Z [3D8C2B70 verbose ‘Vmsvc.vm:/vmfs/volumes/d9b2943f-86ca1719/slu003/slu003.vmx’ opID=1381cd91-97 user=vpxuser] Create Snapshot: Avamar-1393267311a6d576b91e72d43fdb15e1a7fc3dea10e9a47796, memory=false, quiescent=true state=4

2014-02-24T18:41:51.394Z [3D8C2B70 info ‘Vmsvc.vm:/vmfs/volumes/d9b2943f-86ca1719/slu003/slu003.vmx’ opID=1381cd91-97 user=vpxuser] State Transition (VM_STATE_ON -> VM_STATE_CREATE_SNAPSHOT)

2014-02-24T18:41:52.641Z [3BEC2B70 verbose ‘Vmsvc.vm:/vmfs/volumes/d9b2943f-86ca1719/slu003/slu003.vmx’] Handling message _vmx3: The CPU has been disabled by the guest operating system. Power off or reset the virtual machine.

—>

2014-02-24T18:41:52.641Z [3BEC2B70 info ‘Vimsvc.ha-eventmgr’] Event 243 : Message on slu003 on esxi14. in ha-datacenter: The CPU has been disabled by the guest operating system. Power off or reset the virtual machine.

—>

2014-02-24T18:41:52.642Z [3BEC2B70 info ‘Vmsvc.vm:/vmfs/volumes/d9b2943f-86ca1719/slu003/slu003.vmx’] Answered question _vmx3

2014-02-24T18:41:52.813Z [3D8C2B70 verbose ‘SoapAdapter’] Responded to service state request

2014-02-24T18:42:00.029Z [3D581B70 verbose ‘Statssvc.vim.PerformanceManager’] HostCtl Exception in stats collection: Sysinfo error on operation returned status : Not initialized. Please see the VMkernel log for detailed error information

2014-02-24T18:42:00.029Z [3D581B70 verbose ‘Statssvc.vim.PerformanceManager’] HostCtl Exception in stats collection.  Turn on ‘trivia’ log for details

2014-02-24T18:42:01.116Z [3D8C2B70 verbose ‘Vmsvc.vm:/vmfs/volumes/d9b2943f-86ca1719/slu003/slu003.vmx’] Updating current heartbeatStatus: green

2014-02-24T18:42:02.033Z [3D581B70 verbose ‘SoapAdapter’] Responded to service state request

2014-02-24T18:42:03.555Z [FFD815B0 verbose ‘Vmsvc.vm:/vmfs/volumes/d9b2943f-86ca1719/slu003/slu003.vmx’] Version status of tools changed to: 3

2014-02-24T18:42:05.401Z [3D581B70 verbose ‘Default’ opID=HB-host-516@6840-7c1e5c65-1e user=vpxuser] AdapterServer: target=’vim.PerformanceManager:ha-perfmgr’, method=’GetPerfCounter’

2014-02-24T18:42:05.409Z [3D581B70 verbose ‘Locale’ opID=HB-host-516@6840-7c1e5c65-1e user=vpxuser] Default resource used for ‘counter.vsanDomObj.writeThroughput.summary’ expected in module ‘perf’.

2014-02-24T18:42:05.428Z [3D581B70 verbose ‘Default’ opID=HB-host-516@6840-7c1e5c65-1e user=vpxuser] AdapterServer: target=’vim.HostSystem:ha-host’, method=’retrieveInternalCapability’

2014-02-24T18:42:05.430Z [3D581B70 verbose ‘Default’ opID=HB-host-516@6840-7c1e5c65-1e user=vpxuser] AdapterServer: target=’vim.PerformanceManager:ha-perfmgr’, method=’queryPerfCounterInt’

2014-02-24T18:42:05.441Z [3D581B70 verbose ‘Default’ opID=HB-host-516@6840-7c1e5c65-1e user=vpxuser] AdapterServer: target=’vim.LicenseManager:ha-license-manager’, method=’GetLicenses’

2014-02-24T18:42:05.441Z [3D581B70 verbose ‘Vimsvc.ha-license-manager’ opID=HB-host-516@6840-7c1e5c65-1e user=vpxuser] Load: Loading existing file: /etc/vmware/license.cfg

2014-02-24T18:42:05.456Z [3D581B70 verbose ‘Default’ opID=HB-host-516@6840-7c1e5c65-1e user=vpxuser] ha-license-manager:Validate -> Valid license detected for «VMware ESX Server 5.0» (lastError=0, desc.IsValid:Yes)

2014-02-24T18:42:08.332Z [3C240B70 verbose ‘Hostsvc.ResourcePool ha-root-pool’] Root pool capacity changed from 30597MHz/60241MB to 30597MHz/60239MB

2014-02-24T18:42:08.335Z [3D581B70 verbose ‘Default’ opID=SWI-dda157cf user=vpxuser] AdapterServer: target=’vim.ResourcePool:ha-root-pool’, method=’GetConfig’

2014-02-24T18:42:08.336Z [3BEC2B70 verbose ‘Default’ opID=SWI-dda157cf user=vpxuser] AdapterServer: target=’vim.ResourcePool:ha-root-pool’, method=’GetName’

2014-02-24T18:42:15.402Z [3C240B70 verbose ‘Default’ opID=HB-host-516@6842-41f5ef53-39 user=vpxuser] AdapterServer: target=’vim.PerformanceManager:ha-perfmgr’, method=’GetPerfCounter’

2014-02-24T18:42:15.410Z [3C240B70 verbose ‘Locale’ opID=HB-host-516@6842-41f5ef53-39 user=vpxuser] Default resource used for ‘counter.vsanDomObj.writeThroughput.summary’ expected in module ‘perf’.

2014-02-24T18:42:15.429Z [3C240B70 verbose ‘Default’ opID=HB-host-516@6842-41f5ef53-39 user=vpxuser] AdapterServer: target=’vim.HostSystem:ha-host’, method=’retrieveInternalCapability’

2014-02-24T18:42:15.430Z [3C240B70 verbose ‘Default’ opID=HB-host-516@6842-41f5ef53-39 user=vpxuser] AdapterServer: target=’vim.PerformanceManager:ha-perfmgr’, method=’queryPerfCounterInt’

2014-02-24T18:42:15.442Z [3C240B70 verbose ‘Default’ opID=HB-host-516@6842-41f5ef53-39 user=vpxuser] AdapterServer: target=’vim.LicenseManager:ha-license-manager’, method=’GetLicenses’

2014-02-24T18:42:15.442Z [3C240B70 verbose ‘Vimsvc.ha-license-manager’ opID=HB-host-516@6842-41f5ef53-39 user=vpxuser] Load: Loading existing file: /etc/vmware/license.cfg

2014-02-24T18:42:15.457Z [3C240B70 verbose ‘Default’ opID=HB-host-516@6842-41f5ef53-39 user=vpxuser] ha-license-manager:Validate -> Valid license detected for «VMware ESX Server 5.0» (lastError=0, desc.IsValid:Yes)

2014-02-24T18:42:15.711Z [3C240B70 verbose ‘Vmsvc.vm:/vmfs/volumes/d9b2943f-86ca1719/slu003/slu003.vmx’] Version status of tools changed to: 3

2014-02-24T18:42:18.569Z [3C5D2B70 verbose ‘Vmsvc.vm:/vmfs/volumes/d9b2943f-86ca1719/slu003/slu003.vmx’] Create Snapshot translated error to vim.fault.GenericVmConfigFault

2014-02-24T18:42:18.569Z [3C5D2B70 info ‘Vmsvc.vm:/vmfs/volumes/d9b2943f-86ca1719/slu003/slu003.vmx’] Create Snapshot failed: vim.fault.GenericVmConfigFault

2014-02-24T18:42:18.569Z [3C5D2B70 verbose ‘Vmsvc.vm:/vmfs/volumes/d9b2943f-86ca1719/slu003/slu003.vmx’] Create Snapshot message: An error occurred while saving the snapshot: Failed to quiesce the virtual machine.

—> An error occurred while taking a snapshot: Failed to quiesce the virtual machine.

—>

2014-02-24T18:42:18.569Z [3C240B70 verbose ‘Vmsvc.vm:/vmfs/volumes/d9b2943f-86ca1719/slu003/slu003.vmx’] guest operations are not ready

2014-02-24T18:42:18.569Z [3C240B70 verbose ‘Vmsvc.vm:/vmfs/volumes/d9b2943f-86ca1719/slu003/slu003.vmx’] Version status of tools changed to: 3

2014-02-24T18:42:18.571Z [FFD815B0 verbose ‘Vmsvc.vm:/vmfs/volumes/d9b2943f-86ca1719/slu003/slu003.vmx’] Tools are not auto-upgrade capable

2014-02-24T18:42:18.571Z [FFD815B0 verbose ‘Vmsvc.vm:/vmfs/volumes/d9b2943f-86ca1719/slu003/slu003.vmx’] Skipped tools ManifestInfo update.

2014-02-24T18:42:18.574Z [3D8C2B70 verbose ‘Vmsvc.vm:/vmfs/volumes/d9b2943f-86ca1719/slu003/slu003.vmx’] Time to gather Snapshot information ( read from disk, build tree): 2 msecs. needConsolidate is false.

2014-02-24T18:42:18.597Z [3D8C2B70 verbose ‘Vmsvc.vm:/vmfs/volumes/d9b2943f-86ca1719/slu003/slu003.vmx’] Time to gather config: 22 (msecs)

2014-02-24T18:42:18.597Z [3D581B70 verbose ‘Hbrsvc’] Replicator: ReconfigListener triggered for config VM 8

2014-02-24T18:42:18.599Z [3D8C2B70 verbose ‘Vmsvc.vm:/vmfs/volumes/d9b2943f-86ca1719/slu003/slu003.vmx’] Time to gather snapshot file layout: 0 (msecs)

2014-02-24T18:42:18.617Z [3D8C2B70 warning ‘Vmsvc.vm:/vmfs/volumes/d9b2943f-86ca1719/slu003/slu003.vmx’] Failed operation

2014-02-24T18:42:18.617Z [3D8C2B70 info ‘Vmsvc.vm:/vmfs/volumes/d9b2943f-86ca1719/slu003/slu003.vmx’] State Transition (VM_STATE_CREATE_SNAPSHOT -> VM_STATE_ON)

2014-02-24T18:42:18.618Z [3D881B70 verbose ‘Hostsvc’] Received state change for VM ‘8’

2014-02-24T18:42:18.618Z [3D881B70 info ‘Guestsvc.GuestFileTransferImpl’] Entered VmPowerStateListener

2014-02-24T18:42:18.618Z [3D8C2B70 info ‘Vimsvc.TaskManager’] Task Completed : haTask-8-vim.VirtualMachine.createSnapshot-228252373 Status error

2014-02-24T18:42:18.618Z [3D881B70 info ‘Guestsvc.GuestFileTransferImpl’] VmPowerStateListener succeeded

2014-02-24T18:42:18.618Z [3D581B70 verbose ‘Hbrsvc’] Replicator: VmReconfig ignoring VM 8 not configured for replication

2014-02-24T18:42:18.618Z [3D881B70 info ‘Hbrsvc’] Replicator: powerstate change VM: 8 Old: 1 New: 1

2014-02-24T18:42:18.619Z [3D881B70 verbose ‘Hbrsvc’] Replicator: Remove group no matching entry for VM (id=8)

2014-02-24T18:42:20.029Z [3C240B70 verbose ‘Statssvc.vim.PerformanceManager’] HostCtl Exception in stats collection: Sysinfo error on operation returned status : Not initialized. Please see the VMkernel log for detailed error information

2014-02-24T18:42:20.030Z [3C240B70 verbose ‘Statssvc.vim.PerformanceManager’] HostCtl Exception in stats collection.  Turn on ‘trivia’ log for details

2014-02-24T18:42:22.815Z [3BEC2B70 verbose ‘SoapAdapter’] Responded to service state request

2014-02-24T18:42:25.405Z [3C240B70 verbose ‘Default’ opID=HB-host-516@6847-48fcca09-b user=vpxuser] AdapterServer: target=’vim.PerformanceManager:ha-perfmgr’, method=’GetPerfCounter’

2014-02-24T18:42:25.413Z [3C240B70 verbose ‘Locale’ opID=HB-host-516@6847-48fcca09-b user=vpxuser] Default resource used for ‘counter.vsanDomObj.writeThroughput.summary’ expected in module ‘perf’.

2014-02-24T18:42:25.433Z [3C240B70 verbose ‘Default’ opID=HB-host-516@6847-48fcca09-b user=vpxuser] AdapterServer: target=’vim.HostSystem:ha-host’, method=’retrieveInternalCapability’

2014-02-24T18:42:25.434Z [3C240B70 verbose ‘Default’ opID=HB-host-516@6847-48fcca09-b user=vpxuser] AdapterServer: target=’vim.PerformanceManager:ha-perfmgr’, method=’queryPerfCounterInt’

2014-02-24T18:42:25.446Z [3C240B70 verbose ‘Default’ opID=HB-host-516@6847-48fcca09-b user=vpxuser] AdapterServer: target=’vim.LicenseManager:ha-license-manager’, method=’GetLicenses’

2014-02-24T18:42:25.446Z [3C240B70 verbose ‘Vimsvc.ha-license-manager’ opID=HB-host-516@6847-48fcca09-b user=vpxuser] Load: Loading existing file: /etc/vmware/license.cfg

2014-02-24T18:42:25.461Z [3C240B70 verbose ‘Default’ opID=HB-host-516@6847-48fcca09-b user=vpxuser] ha-license-manager:Validate -> Valid license detected for «VMware ESX Server 5.0» (lastError=0, desc.IsValid:Yes)

A Backup or Replication job for a VMware Virtual Machine fails with either of the following errors:

Failed to create VM snapshot. Error: CreateSnapshot failed, vmRef "vm-###", timeout "1800000", snName "VEEAM BACKUP TEMPORARY SNAPSHOT", snDescription "Please do not delete this snapshot. It is being used by Veeam Backup.", memory "False", quiesce "True" Cannot create a quiesced snapshot because the create snapshot operation exceeded the time limit for holding off I/O in the frozen virtual machine.

Failed to create VM snapshot. Error: CreateSnapshot failed, vmRef vm-###, timeout 1800000, snName VEEAM BACKUP TEMPORARY SNAPSHOT, snDescription Please do not delete this snapshot. It is being used by Veeam Backup., memory False, quiesce True

Within the vSphere Environment the following error can be seen in the task list for the VM:

Cannot create a quiesced snapshot because the snapshot operation exceeded the time limit for holding off I/O in the frozen virtual machine

An error occurred while quiescing the virtual machine. See the virtual machine's event log for details. An error occurred while taking a snapshot: Failed to quiesce the virtual machine. An error occurred while saving the snapshot: Failed to quiesce the virtual machine.

On VMware vSphere ESXi 6.5, 6.7, and 7.0, a condition exists where one is unable to take a quiesced snapshot. This is an issue that effects quite a few people and numerous forum threads can be found on the internet by those searching for the solution.

This issues can occur both when taking manual snapshots of virtual machines when one chooses “Quiesce guest filesystem”, or when using snapshot based backup applications such as vSphere Data Protection (vSphere vDP), Veeam, or other applications that utilize quiesced snapshots.

The Issue

I experienced this problem on one of my test VMs (Windows Server 2012 R2), however I believe it can occur on newer versions of Windows Server as well, including Windows Server 2016 and Windows Server 2019.

When this issue occurs, the snapshot will fail and the following errors will be present:

An error occurred while taking a snapshot: Failed to quiesce the virtual machine.
An error occurred while saving the snapshot: Failed to quiesce the virtual machine.

Performing standard troubleshooting, I restarted the VM, checked for VSS provider errors, and confirmed that the Windows Services involved with snapshots were in their correct state and configuration. Unfortunately this had no effect, and everything was configured the way it should be.

I also tried to re-install VMWare tools, which had no effect.

PLEASE NOTE: If you experience this issue, you should confirm the services are in their correct state and configuration, as outlined in VMware KB: 1007696. Source: https://kb.vmware.com/s/article/1007696

The Fix

In the days leading up to the failure when things were running properly, I did notice that the quiesced snapshots for that VM were taking a long time process, but were still functioning correctly before the failure.

This morning during troubleshooting, I went ahead and deleted all the Windows Volume Shadow Copies (VSS Snapshots) which are internal and inside of the Virtual Machine itself. These are the shadow copies that the Windows guest operating system takes on it’s own filesystem (completely unrelated to VMware).

To my surprise after doing this, not only was I able to create a quiesced snapshot, but the snapshot processed almost instantly (200x faster than previously when it was functioning).

If you’re comfortable deleting all your snapshots, it may also be a good idea to fully disable and then re-enable the VSS Snapshots on the volume to make sure they are completely deleted and reset.

I’m assuming this was causing a high load for the VMware snapshot to process and a timeout was being hit on snapshot creation which caused the issue. While Windows volume shadow copies are unrelated to VMware snapshots, they both utilize the same VSS (Volume Shadow Copy Service) system inside of windows to function and process. One must also keep in mind that the Windows volume shadow copies will of course be part of a VMware snapshot since they are stored inside of the VMDK (the virtual disk) file.

PLEASE NOTE: Deleting your Windows Volume Shadow copies will delete your Windows volume snapshots inside of the virtual machine. You will lose the ability to restore files and folders from previous volume shadow copy snapshots. Be aware of what this means and what you are doing before attempting this fix.

Virtual environment has made business systems easily created while the environment also needs to be continuously maintained. Backup and snapshot are often the ways that IT administrators use to protect VM data.

Snapshot is more convenient than backup. Users will get both the VM data and status saved. In fact, snapshot can also help backup VM.

There are generally 2 kinds of snapshot in VMware, standard and quiesced snapshots. Quiesced snapshot ensures that a snapshot disk represents a consistent state of the guest file systems. Quiesced guest file system will be suitable for backup so that’s why backup supplier would like to create quiesced snapshot before backup.

You can create quiesced snapshot in ESXi client or vCenter. Just check Quiesce guest file system (requires VMware Tools) when taking snapshot to create quiesced snapshot.

What if you can’t create quiesced snapshot or your backup software reports that quiecsing VM failed? You will know the causes in the next section.

Why you failed to quiecse the virtual machine?

Creating quiesced snapshot requires VMware Tools installed on the guest so that VMware Tools issue might be the cause.

I/O in the virtual machine and space on VM or datastore will also influence taking quiecsed snapshot.

Some services like VMware Snapshot Provider and Volume Shadow Copy are influencing the process.

If your backup system report this issue, you should also check whether quiesced snapshot can be taken in ESXi or vCenter.

How to fix quiescing virtual machine failed issue?

There are some solutions from the communities that users have used to solve the problem.

Solution 1. Reboot the guest

Rebooting the guest machine is often the easiest way to fix most of the issues in IT environment. Some users reported that this problem appears randomly so you can simply restart the guest machine to see whether quiescing snapshot can be created later.

Solution 2. Update VMware Tools

VMware Tools is a must for quiescing the virtual machine so you should check whether it is on the guest machine or accidentally removed.

Sometimes the out-of-date VMware Tools will cause this problem. Update VMware Tools might solve the problem. You can also read the description of the update to see whether this is a proven issue.

Some users reported that this issue might be related to the installation order of VMware Tools and backup agent. VMware Tools needs to be installed before backup agent and when VMware Tools needs to be updated, you need to uninstall the backup agent first and reinstall it after VMware Tools update, so if you have installed a backup agent, you can remove both VMware Tools and backup agent and reinstall them in order.

If you are going to reinstall VMware Tools, you can try selecting Custom and uncheck VSS option this time.

Solution 3. Reduce the amount of ongoing I/O to the virtual machine

You might have seen the alert that Cannot create a quiesced snapshot because the snapshot operation exceeded the time limit for holding off I/O in the frozen virtual machine.

It means that VM I/O is too high and data transfer can’t be completed so don’t make VM too busy when taking quiesced snapshot.

Solution 4. Release space on VM or datastore

If the space on VM is used up, you can’t create quiesced snapshot. This issue will also occur if the VM has free space but datastore doesn’t.

VMware document says that you cannot quiesce virtual machines that have large capacity disks but haven’t define what is large capacity.

Therefore, check the space on the VM and datastore. Make sure there is enough storage for quiesced snapshot.

Solution 5. Stop «VMware Snapshot Provider» service

Users reported that they fixed this issue by disabling “VMware Snapshot Provider” service on the guest.

Press Windows Key + R to open cmd.exe > type services.msc to open Service > locate VMware Snapshot Provider > right-click it and select Stop

If it doesn’t fix the issue, try stopping Volume Shadow Copy at the same time.

Quiesced snapshot is supported by Vinchin Backup & Recovery

Vinchin is VMware official partner and Vinchin Backup & Recovery is the professional VMware vSphere backup solution which has helped thousands of companies protect virtual environments.

You don’t need to install backup agent on every guest but just add the host into Vinchin backup server and then you can make backup for any guest on the host agentlessly.

Vinchin Backup & Recovery provides many backup strategies to let you backup VM in every way you wish. By enabling Quiesced Snapshot, you can keep file systems or applications on the VM in a consistent state via application-awareness processing before snapshot is taken.

60-day full-featured free trial is available now. You can experience all the features of Vinchin Backup & Recovery including Instant Recovery (recover VM of any size from its backup in 15 seconds), Cross-Platform Recovery (if you have other platforms like XenServer, Red Hat Virtualization, etc., you can migrate VM between different platforms), Data Compression and Deduplication (saving 60% backup storage), etc.

Sum Up

Quiecsed snapshot can ensure that a snapshot disk represents a consistent state of the guest file systems and make a suitable environment for backup.

If you failed to quiesce the virtual machine or received related alert from your backup system, you can read the possible reasons in this post and use the 5 solutions to fix the issue.

Vinchin Backup & Recovery also supports Quiesced Snapshot during VM backup. You can try it for free.

Обновлено 12.12.2019

Добрый день уважаемые читатели блога pyatilistnik.org, сегодня хочу рассказать,что столкнулся с проблемой при попытке сделать полный бэкап машинки через veeam вылезло сообщение.

an error occurred while quiescing the virtual machine. The error code was: 2 The error message was: Custom quiesce script failed.

На Vcenter тоже появляется ошибка такого характера. Которая сообщает что не смогла сделать snapshot.

Если нажать детали

Погуглив, на официальном сайте vmware было предложено сделать ряд шагов.

1. Переустановка или обновление VMware Tools. После процедуры предлагалось перезагрузиться. В некоторых случаях это помогало.

Щелкаем правым кликом по необходимой виртуалке и переходим в меню Guest-Install/Upgrade VMware Tools

Начнется задание

2. Дописывание параметра для HDD

Щелкаем правым кликом по необходимой виртуалке Edit settings

Options-General Configuration Parameters

Добавляем и пишем параметр disk.enableUUID ключ FALSE.

3. Полностью удалить VMware Tools

4. Так же еще vmware советует проверить VSS службы. (ТУТ)

Открываем cmd и пишем vssadmin list providers проверяем запущена ли служба

Проверяем ошибки

vssadmin list writers

5. Так же проблема может быть в косяке прежнего vss провайдера, у меня раньше был symantec. Удалил агента и службы. Материал сайта Pyatilistnik.org