Error an error occurred while saving the snapshot failed to quiesce the virtual machine

Try reinstalling or updating your VMware tools to fix the issue

Symptoms

1. You are using one of the Acronis Cyber Backup products on an ESX Host to back up virtual machines;
2. The backup of one or several of the virtual machines fails with an error message similar to one of the following:

An error occurred while saving the snapshot: Failed to quiesce the virtual machine. An error occurred while taking a snapshot: Failed to quiesce the virtual machine.

Creating a quiesced snapshot failed

An error occurred while quiescing the virtual machine. See the virtual machine’s event log for details.

VMware failed to create a quiesced snapshot of the selected virtual machine. To back up the machine, power off the machine before the process or uninstall the Volume Shadow Copy Services Support feature of VMware Tools from the VM.

Protocol error from VMX

An error occurred while saving the snapshot: A required file was not found.

Cause

Please see the solution section of this article for specific causes and resolution.

Solution

1.1 VM snapshot should work correctly

Acronis software uses snapshots executed by the hypervisor while taking the virtual machine backup. It is necessary that these snapshots work correctly. Create a virtual machine snapshot to check the consistency :

1. Open the vSphere connection to the ESX host.
2. Right click on the machine > take snapshot > check the «Quiesce guest file system» 
3. Uncheck the «Snapshot the virtual machine’s memory» and click OK.

If taking quiesced snapshot fails, investigate the issue with the virtual machine (contact VMware support if needed)

In most cases, reinstalling/updating VMware tools and rebooting the virtual machine will resolve the issue with quiesced snapshot. Please see the following articles:

• VMware Knowledge Base Article 1009073
• VMware Knowledge Base Article 5962168
• VMware Knowledge Base Article 1007696
• VMware Knowledge Base Article 1007346
• VMware Knowledge Base Article 80246

If quiesced snapshot is created successfully but backup with Acronis still fails with the same error message, contact Acronis Support with a reference to this article

1.2 Verify VSS status

If you backup a virtual machine with transactional applications installed (SQL, Exchange evt.), then:

• VMware Tools should be installed inside that guest OS.
• VSS service should not be disabled inside the guest OS.

In case the guest OS event logs contain errors or warnings for VSS operations during creation of the snapshot, troubleshoot the VSS issues inside the guest OS.

If a virtual machine snapshot fails due to incorrect guest file system quiescing (snapshot without option Quiesce quest file system is successful), for virtual machines with no transcational applications installed you can workaround the issue by uninstalling the VMware Tools for the guest OS (see More information below) or by disabling Volume Shadow Copy (VSS) for Virtual Machines backup option (for Acronis Cyber Protect Cloud or Acronis Cyber Backup 12 and above).

Keep in mind the VMware ESX(i) virtualization platform limitations:

• Agent for VMware backs up a fault tolerant machine only if fault tolerance was enabled in VMware vSphere 6.0 and later,
• VMware vSphere does not support snapshots of independent disks and Raw Device Mapping (RDM) disks in physical compatibility mode. To overcome these limitations install an agent into the guest OS.

1.3 If the error message is:

Creating a quiesced snapshot failed because the create snapshot operation exceeded the time limit for holding off I/O in the frozen virtual machine

Then see the following article for a solution: VMware Knowledge Base Article 1038963 

If ESX(i) version is 6.0, install this patch.

 The error may be caused by pre-backup command failure. A pre-backup command (pre-freeze script located in /usr/sbin/pre-freeze-script on a Linux machine or in a specific directory on a Windows machine) is a user-supplied script that should be executed before backup. Usually, these scripts are used to quiesce an application that does not have its own VSS provider (in Windows) to ensure a consistent state before backup. When such a script exits with a non-zero return code, this indicates that the script failed to perform its quiescing task and quiesced snapshot operation has to be cancelled.

 See also the following solutions:

• If VM freezes during snapshot creation, see A virtual machine can freeze under load when you take quiesced snapshots or use custom quiescing scripts;
• If VM runs a database, try giving it more processors/RAM. Allow VM to use more of the host/cluster resources.

More information

Acronis Cyber Backup 12.5: If the backed up VM does not run a database, you can disable quiescing of VMs via VSS by turning off Volume Shadow Copy for virtual machines (applies to Acronis Cyber Backup 12.5 only — there is no such option in 11.7 or lower). With quiescing disabled, the VM will be backed up in crash-consistent state.

For Acronis Cyber Backup 11.7 and older products, try disabling the VSS component of VMware Tools to solve the issue, but only if the backed up VM does not run a database:

Click to expand instructions for Acronis Cyber Backup 11.7 and older products

A Backup or Replication job for a VMware Virtual Machine fails with either of the following errors:

Failed to create VM snapshot. Error: CreateSnapshot failed, vmRef "vm-###", timeout "1800000", snName "VEEAM BACKUP TEMPORARY SNAPSHOT", snDescription "Please do not delete this snapshot. It is being used by Veeam Backup.", memory "False", quiesce "True" Cannot create a quiesced snapshot because the create snapshot operation exceeded the time limit for holding off I/O in the frozen virtual machine.

Failed to create VM snapshot. Error: CreateSnapshot failed, vmRef vm-###, timeout 1800000, snName VEEAM BACKUP TEMPORARY SNAPSHOT, snDescription Please do not delete this snapshot. It is being used by Veeam Backup., memory False, quiesce True

Within the vSphere Environment the following error can be seen in the task list for the VM:

Cannot create a quiesced snapshot because the snapshot operation exceeded the time limit for holding off I/O in the frozen virtual machine

An error occurred while quiescing the virtual machine. See the virtual machine's event log for details. An error occurred while taking a snapshot: Failed to quiesce the virtual machine. An error occurred while saving the snapshot: Failed to quiesce the virtual machine.

Job Fails Due to Quiesced Snapshot Creation Failure

Cheers for trusting us with the spot in your mailbox!

Now you’re less likely to miss what’s been brewing in our knowledge base with this weekly digest

Oops! Something went wrong.

Challenge

A Backup or Replication job for a VMware Virtual Machine fails with either of the following errors:

Within the vSphere Environment the following error can be seen in the task list for the VM:

Cause

This error occurs when the VMware Environment fails to create the quiesced snapshot that was requested by Veeam Backup & Replication.

Veeam Backup & Replication will send the request for a quiesced snapshot only when the option «Enable VMware Tools quiescence» is enabled within the Backup/Replication job settings, and the VM that was being snapshoted was not processed using Application-Aware Processing.

Solution

VMware Provides the following troubleshooting article regarding issues with quiesced snapshot creation:

• You can reproduce the quiesced snapshot creation failure by manually creating a snapshot on the VM that fails to be processed, deselecting the option «Include virtual machine’s memory,» and enabling the «Quiesce guest file system» option.
• The option «Enable VMware tools quiescence» within Veeam backup/replication jobs is disabled by default (unchecked).
• For Windows-based systems, it is recommended to use «Enable application-aware processing» instead of VMware Tools Quiescence.

Источник

vSphere ESXi 6.5, 6.7, 7.0 – Unable to take quiesced snapshot

On VMware vSphere ESXi 6.5, 6.7, and 7.0, a condition exists where one is unable to take a quiesced snapshot. This is an issue that effects quite a few people and numerous forum threads can be found on the internet by those searching for the solution.

This issues can occur both when taking manual snapshots of virtual machines when one chooses “Quiesce guest filesystem”, or when using snapshot based backup applications such as vSphere Data Protection (vSphere vDP), Veeam, or other applications that utilize quiesced snapshots.

The Issue

I experienced this problem on one of my test VMs (Windows Server 2012 R2), however I believe it can occur on newer versions of Windows Server as well, including Windows Server 2016 and Windows Server 2019.

When this issue occurs, the snapshot will fail and the following errors will be present:

Performing standard troubleshooting, I restarted the VM, checked for VSS provider errors, and confirmed that the Windows Services involved with snapshots were in their correct state and configuration. Unfortunately this had no effect, and everything was configured the way it should be.

I also tried to re-install VMWare tools, which had no effect.

PLEASE NOTE: If you experience this issue, you should confirm the services are in their correct state and configuration, as outlined in VMware KB: 1007696. Source: https://kb.vmware.com/s/article/1007696

The Fix

In the days leading up to the failure when things were running properly, I did notice that the quiesced snapshots for that VM were taking a long time process, but were still functioning correctly before the failure.

This morning during troubleshooting, I went ahead and deleted all the Windows Volume Shadow Copies (VSS Snapshots) which are internal and inside of the Virtual Machine itself. These are the shadow copies that the Windows guest operating system takes on it’s own filesystem (completely unrelated to VMware).

To my surprise after doing this, not only was I able to create a quiesced snapshot, but the snapshot processed almost instantly (200x faster than previously when it was functioning).

If you’re comfortable deleting all your snapshots, it may also be a good idea to fully disable and then re-enable the VSS Snapshots on the volume to make sure they are completely deleted and reset.

I’m assuming this was causing a high load for the VMware snapshot to process and a timeout was being hit on snapshot creation which caused the issue. While Windows volume shadow copies are unrelated to VMware snapshots, they both utilize the same VSS (Volume Shadow Copy Service) system inside of windows to function and process. One must also keep in mind that the Windows volume shadow copies will of course be part of a VMware snapshot since they are stored inside of the VMDK (the virtual disk) file.

PLEASE NOTE: Deleting your Windows Volume Shadow copies will delete your Windows volume snapshots inside of the virtual machine. You will lose the ability to restore files and folders from previous volume shadow copy snapshots. Be aware of what this means and what you are doing before attempting this fix.

7 Responses to “vSphere ESXi 6.5, 6.7, 7.0 – Unable to take quiesced snapshot”

I am noticing very odd behaivor on 6.5. Basically I have a zfs zvol presented to esxi as an iscsi target via scst. For whatever reason when deleting a snapshot it causes a thin vmdk file to grow to its provisioned size.

for example i had a server using 700gb of data. The snopshot was 10gb. When i told vcenter to delete the 10gb snapshot it took hours and the vmdk grew to 4tb (the provisioned size). I was just curious if you have seen anything like this.

That’s actually crazy! I haven’t heard of, nor seen that happen.

I’m just curious, is your ESXi host running 6.5 fully patched (I think there’s been a few updates released that I’ve recently seen/deployed via vSphere Update Manager)?

Also, I’m just curious: Does SCST use/provide VAAI? Usually I see Lio-Target being used for iSCSI targets on Linux based storage hosts (since it provides multiple host access and VAAI)?

I’m actually not that familiar with SCST as far as VMware compatibility, VAAI, and clustered file system access on it…

Great catch on how Microsoft shadow copies impact VMWare snapshots. Helped to solve my problem. I think it is a “bug” on VMWare side and nobody reported it to VMWare (on not enough reports for them to spend time on fixing this problem). Technology….;)

I had the same issue but mine was an old backup agent on the machine.

Anything that interfaces, uses, or messes with the Windows VDS providers (such as your old backup agent), can cause this as well!

I had same issue . My Windows Shadow Copy does not activated so i tried another workaround
I powered off my Vm and select Proper ESXi (with Enough Resource for DSR check) and then Start Clone job and it works.

My issue was similar, but I worked out that my shadow copies were being taken exactly the same time as the snapshot was being created by the storage! So I altered the VSS times and its been fine ever since. But your article pointed me in the right directions, many thanks.

Источник

Veeam R&D Forums

Technical discussions about Veeam products and related data center technologies

Backup Fails on VMs with VMware Tools Quiescence Enabled

Post by jbennett » Dec 16, 2015 10:31 pm 1 person likes this post

We’ve been backing up these VMs for months successfully with VMware Tools Quiescence disabled. They’re all running on a single ESXi 6.0 host, all VMs running Windows Server 2012 R2 and VMware Tools build 9536.

The other day, I enabled «VMware Tools Quiescence», the backup job kicked off a failure to create snapshot «an error occurred while saving the snapshot: failed to quiesce the virtual machine». Other 2012 and 2012 R2 VMs with the same VMware Tools build had no issue backing up from the same host.

Found four related articles — none resolved the issue.

What DID fix the issue. I migrated the VMs to a newer ESXi host and upgraded VMware Tools to build 9541. This » appears » to have resolved / fixed issue. I’m am seeing Event Log Errors for 147, 150, 12289, but the backup is now completing when creating the snapshot. I’m going to migrate the VMs back during an off hour and test backup again — (going from virtual standard switch (VSS) to virtual distributed switch (VDS) is ok, but its failing to migrate from VDS to VSS).

Источник

Veeam R&D Forums

Technical discussions about Veeam products and related data center technologies

Veeam B&R 8.0 snapshot/quiescing question?

Veeam B&R 8.0 snapshot/quiescing question?

Post by danswartz » Jul 13, 2015 4:13 am this post

Re: Veeam B&R 8.0 snapshot/quiescing question?

Post by danswartz » Jul 13, 2015 1:40 pm this post

Re: Veeam B&R 8.0 snapshot/quiescing question?

Post by PTide » Jul 13, 2015 2:31 pm this post

If this or this or this does not address the issue, then, I believe, it’s a sign that you need to open a case either with Veeam or VMware support. The best option would be to open two support cases =)

Re: Veeam B&R 8.0 snapshot/quiescing question?

Post by danswartz » Jul 13, 2015 2:42 pm this post

Re: Veeam B&R 8.0 snapshot/quiescing question?

Post by PTide » Jul 13, 2015 2:45 pm this post

Please let us know if it did the trick!

Re: Veeam B&R 8.0 snapshot/quiescing question?

Post by danswartz » Jul 13, 2015 4:03 pm this post

Re: Veeam B&R 8.0 snapshot/quiescing question?

Post by PTide » Jul 13, 2015 5:29 pm this post

Anyway I’d give it a try. If the issue does not match any of VMware’s KB, then I think they might get interested.

Источник

Knowledge Base

4559: Acronis product fails to back up a VMware virtual machine with «Creating a quiesced snapshot failed» or «An error occurred while quiescing the virtual machine»

Also read in:

use Google Translate

Applies to:

Operating Systems:

Last update: 11-08-2022

Try reinstalling or updating your VMware tools to fix the issue

Symptoms

1. You are using one of the Acronis Cyber Backup products on an ESX Host to back up virtual machines;
2. The backup of one or several of the virtual machines fails with an error message similar to one of the following:

An error occurred while saving the snapshot: Failed to quiesce the virtual machine. An error occurred while taking a snapshot: Failed to quiesce the virtual machine.

Creating a quiesced snapshot failed

An error occurred while quiescing the virtual machine. See the virtual machine’s event log for details.

VMware failed to create a quiesced snapshot of the selected virtual machine. To back up the machine, power off the machine before the process or uninstall the Volume Shadow Copy Services Support feature of VMware Tools from the VM.

Protocol error from VMX

An error occurred while saving the snapshot: A required file was not found.

Cause

Please see the solution section of this article for specific causes and resolution.

Solution

1.1 VM snapshot should work correctly

Acronis software uses snapshots executed by the hypervisor while taking the virtual machine backup. It is necessary that these snapshots work correctly. Create a virtual machine snapshot to check the consistency :

1. Open the vSphere connection to the ESX host.
2. Right click on the machine > take snapshot > check the «Quiesce guest file system»
3. Uncheck the «Snapshot the virtual machine’s memory» and click OK.

If taking quiesced snapshot fails, investigate the issue with the virtual machine (contact VMware support if needed)

In most cases, reinstalling/updating VMware tools and rebooting the virtual machine will resolve the issue with quiesced snapshot. Please see the following articles:

• VMware Knowledge Base Article 1009073
• VMware Knowledge Base Article 5962168
• VMware Knowledge Base Article 1007696
• VMware Knowledge Base Article 1007346
• VMware Knowledge Base Article 80246

If quiesced snapshot is created successfully but backup with Acronis still fails with the same error message, contact Acronis Support with a reference to this article

1.2 Verify VSS status

If you backup a virtual machine with transactional applications installed (SQL, Exchange evt.), then:

• VMware Tools should be installed inside that guest OS.
• VSS service should not be disabled inside the guest OS.

In case the guest OS event logs contain errors or warnings for VSS operations during creation of the snapshot, troubleshoot the VSS issues inside the guest OS.

If a virtual machine snapshot fails due to incorrect guest file system quiescing (snapshot without option Quiesce quest file system is successful), for virtual machines with no transcational applications installed you can workaround the issue by uninstalling the VMware Tools for the guest OS (see More information below) or by disabling Volume Shadow Copy (VSS) for Virtual Machines backup option (for Acronis Cyber Protect Cloud or Acronis Cyber Backup 12 and above) .

Keep in mind the VMware ESX(i) virtualization platform limitations:

• Agent for VMware backs up a fault tolerant machine only if fault tolerance was enabled in VMware vSphere 6.0 and later,
• VMware vSphere does not support snapshots of independent disks and Raw Device Mapping (RDM) disks in physical compatibility mode. To overcome these limitations install an agent into the guest OS.

1.3 If the error message is:

Creating a quiesced snapshot failed because the create snapshot operation exceeded the time limit for holding off I/O in the frozen virtual machine

Then see the following article for a solution: VMware Knowledge Base Article 1038963

If ESX(i) version is 6.0, install this patch .

The error may be caused by pre-backup command failure. A pre-backup command (pre-freeze script located in /usr/sbin/pre-freeze-script on a Linux machine or in a specific directory on a Windows machine) is a user-supplied script that should be executed before backup. Usually, these scripts are used to quiesce an application that does not have its own VSS provider (in Windows) to ensure a consistent state before backup. When such a script exits with a non-zero return code, this indicates that the script failed to perform its quiescing task and quiesced snapshot operation has to be cancelled.

See also the following solutions:

• If VM freezes during snapshot creation, see A virtual machine can freeze under load when you take quiesced snapshots or use custom quiescing scripts;
• If VM runs a database, try giving it more processors/RAM. Allow VM to use more of the host/cluster resources.

More information

Acronis Cyber Backup 12.5: If the backed up VM does not run a database, you can disable quiescing of VMs via VSS by turning off Volume Shadow Copy for virtual machines (applies to Acronis Cyber Backup 12.5 only — there is no such option in 11.7 or lower). With quiescing disabled, the VM will be backed up in crash-consistent state.

For Acronis Cyber Backup 11.7 and older products, try disabling the VSS component of VMware Tools to solve the issue, but only if the backed up VM does not run a database:

Click to expand instructions for Acronis Cyber Backup 11.7 and older products

Uninstall the Volume Shadow Copy Services Support feature of VMware Tools from the virtual machine:

1. Select the virtual machine in question on ESX server and go to Inventory -> Virtual Machine -> Install/Upgrade VMware Tools:
2. 
3. This will launch the installation of VMware Tools on the virtual machine:

Uncheck the Volume Shadow Copy Services Support:

Complete the modification:

Источник

On VMware vSphere ESXi 6.5, 6.7, and 7.0, a condition exists where one is unable to take a quiesced snapshot. This is an issue that effects quite a few people and numerous forum threads can be found on the internet by those searching for the solution.

This issues can occur both when taking manual snapshots of virtual machines when one chooses “Quiesce guest filesystem”, or when using snapshot based backup applications such as vSphere Data Protection (vSphere vDP), Veeam, or other applications that utilize quiesced snapshots.

The Issue

I experienced this problem on one of my test VMs (Windows Server 2012 R2), however I believe it can occur on newer versions of Windows Server as well, including Windows Server 2016 and Windows Server 2019.

When this issue occurs, the snapshot will fail and the following errors will be present:

An error occurred while taking a snapshot: Failed to quiesce the virtual machine.
An error occurred while saving the snapshot: Failed to quiesce the virtual machine.

Performing standard troubleshooting, I restarted the VM, checked for VSS provider errors, and confirmed that the Windows Services involved with snapshots were in their correct state and configuration. Unfortunately this had no effect, and everything was configured the way it should be.

I also tried to re-install VMWare tools, which had no effect.

PLEASE NOTE: If you experience this issue, you should confirm the services are in their correct state and configuration, as outlined in VMware KB: 1007696. Source: https://kb.vmware.com/s/article/1007696

The Fix

In the days leading up to the failure when things were running properly, I did notice that the quiesced snapshots for that VM were taking a long time process, but were still functioning correctly before the failure.

This morning during troubleshooting, I went ahead and deleted all the Windows Volume Shadow Copies (VSS Snapshots) which are internal and inside of the Virtual Machine itself. These are the shadow copies that the Windows guest operating system takes on it’s own filesystem (completely unrelated to VMware).

To my surprise after doing this, not only was I able to create a quiesced snapshot, but the snapshot processed almost instantly (200x faster than previously when it was functioning).

If you’re comfortable deleting all your snapshots, it may also be a good idea to fully disable and then re-enable the VSS Snapshots on the volume to make sure they are completely deleted and reset.

I’m assuming this was causing a high load for the VMware snapshot to process and a timeout was being hit on snapshot creation which caused the issue. While Windows volume shadow copies are unrelated to VMware snapshots, they both utilize the same VSS (Volume Shadow Copy Service) system inside of windows to function and process. One must also keep in mind that the Windows volume shadow copies will of course be part of a VMware snapshot since they are stored inside of the VMDK (the virtual disk) file.

PLEASE NOTE: Deleting your Windows Volume Shadow copies will delete your Windows volume snapshots inside of the virtual machine. You will lose the ability to restore files and folders from previous volume shadow copy snapshots. Be aware of what this means and what you are doing before attempting this fix.

[{«Product»:{«code»:»SS8TDQ»,»label»:»Tivoli Storage Manager for Virtual Environments»},»Business Unit»:{«code»:»BU058″,»label»:»IBM Infrastructure w/TPS»},»Component»:»Data Protection for VMware»,»Platform»:[{«code»:»PF016″,»label»:»Linux»},{«code»:»PF033″,»label»:»Windows»}],»Version»:»7.1.2″,»Edition»:»»,»Line of Business»:{«code»:»LOB26″,»label»:»Storage»}}]

Обновлено 12.12.2019

Добрый день уважаемые читатели блога pyatilistnik.org, сегодня хочу рассказать,что столкнулся с проблемой при попытке сделать полный бэкап машинки через veeam вылезло сообщение.

an error occurred while quiescing the virtual machine. The error code was: 2 The error message was: Custom quiesce script failed.

На Vcenter тоже появляется ошибка такого характера. Которая сообщает что не смогла сделать snapshot.

Если нажать детали

Погуглив, на официальном сайте vmware было предложено сделать ряд шагов.

1. Переустановка или обновление VMware Tools. После процедуры предлагалось перезагрузиться. В некоторых случаях это помогало.

Щелкаем правым кликом по необходимой виртуалке и переходим в меню Guest-Install/Upgrade VMware Tools

Начнется задание

2. Дописывание параметра для HDD

Щелкаем правым кликом по необходимой виртуалке Edit settings

Options-General Configuration Parameters

Добавляем и пишем параметр disk.enableUUID ключ FALSE.

3. Полностью удалить VMware Tools

4. Так же еще vmware советует проверить VSS службы. (ТУТ)

Открываем cmd и пишем vssadmin list providers проверяем запущена ли служба

Проверяем ошибки

vssadmin list writers

5. Так же проблема может быть в косяке прежнего vss провайдера, у меня раньше был symantec. Удалил агента и службы. Материал сайта Pyatilistnik.org