Restore job failed error cannot complete file creation operation

Veeam Community discussions and solutions for: Error: Cannot complete file creation operation. of VMware vSphere

unsichtbarre

Service Provider
Posts: 213
Liked: 37 times
Joined: Mar 08, 2010 4:05 pm
Full Name: John Borhek
Contact:

Error: Cannot complete file creation operation.

Case # 04776374

Versions: vSphere 7 / vCenter 7 / ESXi 7 /vSAN 7 / Veeam 11.0.0.837
vSAN: 4 Node 150TB RAW, vSAN Default Storage Policy (unaltered) RAID 1 / 75TB Available

When I try to seed a replication job to the vSAN, it fails with the error: Error: Cannot complete file creation operation. The 2 VMs I am trying to seed are 50GB each

VMware validates storage policies and vSAN configuration as correct, Veeam support (at least my representative) is not willing to consider the issue.

One other thing, https://kb.vmware.com/s/article/2105091 is not the issue due to my version (7.X) and HardwareAcceleratedLocking is already enabled.

Thanks for consideration,
-JB

John Borhek, Solutions Architect
https://vmsources.com


HannesK

Veeam Software
Posts: 12494
Liked: 2400 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: Error: Cannot complete file creation operation.

Post

by HannesK » Apr 27, 2021 5:24 am

Hello,

Thanks for consideration,

please consider to continue working with support, as guessing issues like this makes little sense on a forum.

As far as I understand the case, the logs say that the issue comes from the VMware side (which might be based on similar issues, where it was a VMware issue). I also see a note, that permissions were an issue in the first place.

I also assume that you run vSphere 7.0 or 7.0U1 and not U2 (U2 is not supported yet)

Best regards,
Hannes


unsichtbarre

Service Provider
Posts: 213
Liked: 37 times
Joined: Mar 08, 2010 4:05 pm
Full Name: John Borhek
Contact:

Re: Error: Cannot complete file creation operation.

Post

by unsichtbarre » Apr 28, 2021 5:45 pm

I will continue to work with support, however the agent indicated that Veeam support considered this to be a VMware problem. VMware considers vSAN to be correct, and so (therefore) not their problem. Also, no errors appear in VMware or VMware logs, only Veeam logs. Yes vSphere/vSAN 7.0.

I consider the permission issue to be related to full admin, the error changes but does not disappear. Interestingly, if the job is seeded to a normal SAN datastore, then I migrate the VM to vSAN, it will continue to replicate without issue. It is only when I seed to vSAN that the issue happens.

I can seed to normal SAN with the default permissions.

THX,
-John

John Borhek, Solutions Architect
https://vmsources.com


PetrM

Veeam Software
Posts: 2561
Liked: 409 times
Joined: Aug 28, 2013 8:23 am
Full Name: Petr Makarov
Location: Prague, Czech Republic
Contact:

Re: Error: Cannot complete file creation operation.

Post

by PetrM » Apr 28, 2021 8:08 pm

Hi John,

I’m not sure that you really should reach out VMware support at this stage of investigation, the issue must be confirmed by precise analysis of VMware logs and our logs. One more idea would be to define the exact operation which is failed and to try to reproduce the same problem without Veeam, for instance by invoking the API call manually. I’ve just asked our support team to review the case and to try to help you to isolate the issue at vSphere level and to communicate with VMware support. Also, you may request an escalation of this support case.

Nevertheless, I’m pretty sure that this issue does not come from our code, perhaps this article and that blog might give a better idea, you’ll find different examples of failed attempts to create vSAN objects.

Thanks!


unsichtbarre

Service Provider
Posts: 213
Liked: 37 times
Joined: Mar 08, 2010 4:05 pm
Full Name: John Borhek
Contact:

Re: Error: Cannot complete file creation operation.

Post

by unsichtbarre » Apr 29, 2021 9:25 pm

Thanks for the pointers: verified MTU, verified CLOMOD Liveliness

Still no folder creation. It is a VMware issue, but no resolution so far

John Borhek, Solutions Architect
https://vmsources.com


unsichtbarre

Service Provider
Posts: 213
Liked: 37 times
Joined: Mar 08, 2010 4:05 pm
Full Name: John Borhek
Contact:

Re: Error: Cannot complete file creation operation.

Post

by unsichtbarre » Apr 29, 2021 10:26 pm

The problem seems to be that Veeam is not selecting a storage policy. As I set the target Datastore, I see the vSAN datastore listed under both «Default Policy» (where the vSAN is listed alongside other IP SAN datastores) and in a separate category «vSAN Default Storage Policy». I have tried selecting both, to no effect, it always fails with: Error: Cannot complete file creation operation.

VMware says Veeam is simply not choosing a Storage Policy (even though it looks like I pick one) and, therefore, the files cannot be created.

John Borhek, Solutions Architect
https://vmsources.com


PetrM

Veeam Software
Posts: 2561
Liked: 409 times
Joined: Aug 28, 2013 8:23 am
Full Name: Petr Makarov
Location: Prague, Czech Republic
Contact:

Re: Error: Cannot complete file creation operation.

Post

by PetrM » Apr 30, 2021 10:51 am

Hi John,

Ok, thanks for information. I have 2 questions:
1. How did VMware engineer come to the conclusion that the issue is related to storage policy selection?
2. Have you tried administrator account as it was suggested by our engineer?

Also, please share this information with our support as well, it’s a very important note.

Thanks!



PetrM

Veeam Software
Posts: 2561
Liked: 409 times
Joined: Aug 28, 2013 8:23 am
Full Name: Petr Makarov
Location: Prague, Czech Republic
Contact:

Re: Error: Cannot complete file creation operation.

Post

by PetrM » May 05, 2021 12:18 pm

Hi John,

Glad to hear that the issue is resolved and thanks for sharing the result with the Community!

Thanks!


Who is online

Users browsing this forum: Bing [Bot] and 17 guests

We use Zerto to replicate our production VMs to our 3 Node vSAN 6 cluster in DR.  I can create new VMs, delete them, create folders, copy ISO files, etc…  However, when Zerto tries to do anything; test failover, create new protection group, etc…, I get «Cannot complete file creation operation».  And I see the following in vpxa.log on the host trying to do the operation:

—> Result:

—> (vim.fault.CannotCreateFile) {

—>    faultCause = (vmodl.MethodFault) null,

—>    faultMessage = (vmodl.LocalizableMessage) [

—>       (vmodl.LocalizableMessage) {

—>          key = «com.vmware.esx.hostctl.default»,

—>          arg = (vmodl.KeyAnyValue) [

—>             (vmodl.KeyAnyValue) {

—>                key = «reason»,

—>                value = «Hostsvc::osfs::CreateDirectory : Failed to create directory vm-27637 (Cannot Create File)»

—>             }

—>          ],

—>          message = «Operation failed, diagnostics report: Hostsvc::osfs::CreateDirectory : Failed to create directory vm-27637 (Cannot Create File)»

—>       }

—>    ],

—>    file = «Hostsvc::osfs::CreateDirectory : Failed to create directory vm-27637 (Cannot Create File)»

—>    msg = «Received SOAP response fault from [<cs p:1f31ce78, TCP:localhost:8307>]: CreateDirectory

—> Cannot complete file creation operation.»

—> }

—> Args:

—>

—> Arg spec:

—> (vpxapi.VmLayoutSpec) {

—>    vmLocation = (vpxapi.VmLayoutSpec.Location) null,

—>    multipleConfigs = <unset>,

—>    basename = «Z-VRAH-gvvsan1.rpionline.com-586248»,

—>    baseStorageProfile = <unset>,

—>    disk = (vpxapi.VmLayoutSpec.Location) [

—>       (vpxapi.VmLayoutSpec.Location) {

—>          url = «ds:///vmfs/volumes/vsan:fe0afcdab0b14afe-b39f91c71d1f1a76/1d00fa54-b4a3-9158-b876-ecf4bbcfd398/f79828c5-af94-4c14-8117-712276546bdd/vm-27637/Temo.vmdk»,

—>          key = 16020,

—>          sourceUrl = <unset>,

—>          urlType = «exactFilePath»,

—>          storageProfile = <unset>

—>       }

—>    ],

—>    reserveDirOnly = <unset>

Everything I look into leads me back to an MTU issue and this KB:  VMware KB: Creating new objects on a VMware Virtual SAN Datastore fails and reports the error: Faile…

I enabled the vDS health check and sure enough, it reports and MTU issue.  There is also this:

[root@gvvsan1:~] vmkping -d -s 1472 10.7.7.121

PING 10.7.7.121 (10.7.7.121): 1472 data bytes

1480 bytes from 10.7.7.121: icmp_seq=0 ttl=64 time=0.476 ms

1480 bytes from 10.7.7.121: icmp_seq=1 ttl=64 time=0.401 ms

1480 bytes from 10.7.7.121: icmp_seq=2 ttl=64 time=0.298 ms

— 10.7.7.121 ping statistics —

3 packets transmitted, 3 packets received, 0% packet loss

round-trip min/avg/max = 0.298/0.392/0.476 ms

[root@gvvsan1:~] vmkping -d -s 1473 10.7.7.121

PING 10.7.7.121 (10.7.7.121): 1473 data bytes

sendto() failed (Message too long)

sendto() failed (Message too long)

sendto() failed (Message too long)

— 10.7.7.121 ping statistics —

3 packets transmitted, 0 packets received, 100% packet loss

MTU size of 1472 results in a success and a size of 1473 results in a failure, the ping come back with a size of 1480.

My physical Dell 4032 switches have an MTU of 1518 and the vDS have an MTU of 1500.  My question is, what should they be and is there anywhere else I need to verify/set the MTU?

I have cases opened with Zerto, VMware, and my local networking consultant.  As always… thank you, Zach.

Здравствуйте!

Пытаюсь сделать архивную копию некоторых папок программой «Veeam Endpoint Backup» архивную копию не делает выдает вот такую ошибку

29.08.2016 15:09:28 :: Creating VSS snapshot Error: Failed to create snapshot: Backup job failed. Cannot create a shadow copy of the volumes containing writer’s data. A VSS critical writer has failed. Writer name: [SqlServerWriter]. Class ID: [{a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}].
Instance ID: [{f089a4e2-a605-4f85-8bcd-f26c21eac837}]. Writer’s state: [VSS_WS_FAILED_AT_PREPARE_SNAPSHOT]. Error code: [0x800423f4]. 

«Система архивации данных Windows Server» архивную копию тоже не создает

В журналах windows «приложение» есть такие ошибки

Событие отклонено модулем записи VSS с ошибкой 0x800423f4, Повторяющаяся ошибка средства записи.  Эта ошибка вероятно будет
возникать снова при повторении попытки архивации.
. Изменения компонентов модуля записи, выполненные модулем при обработке события, будут недоступны запрашивающей стороне. Связанные события, поступающие от приложения, в котором размещен модуль записи VSS,
см. в журнале событий. 

Операция:
   Событие PrepareForSnapshot

Контекст:
   Контекст выполнения: Writer
   Код класса модуля записи: {a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}
   Имя модуля записи: SqlServerWriter
   Имя экземпляра модуля записи: Microsoft SQL Server 2012:SQLWriter
   Код экземпляра модуля записи: {f089a4e2-a605-4f85-8bcd-f26c21eac837}
   Командная строка: «C:Program FilesMicrosoft SQL Server90Sharedsqlwriter.exe»
   Идентификатор процесса: 1924

Вывод команды sfc /scannow

Начато сканирование системы.  Этот процесс может занять некоторое время.

Начало стадии проверки при сканировании системы.
Проверка 100% завершена.

Защита ресурсов Windows не обнаружила нарушений целостности.

Вывод команды vssadmin list writers
VSSADMIN 1.1 — Утилита теневого копирования тома
(C) Корпорация Майкрософт, 2001-2005.

Имя компонента записи: «Task Scheduler Writer»
   Id компонента записи: {d61d61c8-d73a-4eee-8cdd-f6f9786b7124}
   Id экземпляра компонента записи: {1bddd48e-5052-49db-9b07-b96f96727e6b}
   Состояние: [1] Стабильный
   Последняя ошибка: Нет ошибок

Имя компонента записи: «VSS Metadata Store Writer»
   Id компонента записи: {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06}
   Id экземпляра компонента записи: {088e7a7d-09a8-4cc6-a609-ad90e75ddc93}
   Состояние: [1] Стабильный
   Последняя ошибка: Нет ошибок

Имя компонента записи: «Performance Counters Writer»
   Id компонента записи: {0bada1de-01a9-4625-8278-69e735f39dd2}
   Id экземпляра компонента записи: {f0086dda-9efc-47c5-8eb6-a944c3d09381}
   Состояние: [1] Стабильный
   Последняя ошибка: Нет ошибок

Имя компонента записи: «SqlServerWriter»
   Id компонента записи: {a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}
   Id экземпляра компонента записи: {f089a4e2-a605-4f85-8bcd-f26c21eac837}
   Состояние: [8] Неисправен
   Последняя ошибка: Неповторяемая ошибка

Имя компонента записи: «System Writer»
   Id компонента записи: {e8132975-6f93-4464-a53e-1050253ae220}
   Id экземпляра компонента записи: {93d46651-9f1c-44ac-ba08-4a0c23c5fa4d}
   Состояние: [1] Стабильный
   Последняя ошибка: Нет ошибок

Имя компонента записи: «ASR Writer»
   Id компонента записи: {be000cbe-11fe-4426-9c58-531aa6355fc4}
   Id экземпляра компонента записи: {44caef9a-4f8c-40f2-a0a4-3bb8b87027ad}
   Состояние: [1] Стабильный
   Последняя ошибка: Нет ошибок

Имя компонента записи: «BITS Writer»
   Id компонента записи: {4969d978-be47-48b0-b100-f328f07ac1e0}
   Id экземпляра компонента записи: {dbe41f83-2443-4ef6-a913-5772313d2f07}
   Состояние: [1] Стабильный
   Последняя ошибка: Нет ошибок

Имя компонента записи: «Registry Writer»
   Id компонента записи: {afbab4a2-367d-4d15-a586-71dbb18f8485}
   Id экземпляра компонента записи: {20d7710c-cff0-4691-a4ff-30d4e7be94c8}
   Состояние: [1] Стабильный
   Последняя ошибка: Нет ошибок

Имя компонента записи: «COM+ REGDB Writer»
   Id компонента записи: {542da469-d3e1-473c-9f4f-7847f01fc64f}
   Id экземпляра компонента записи: {3fb2f123-7432-477d-b07e-0bb063cc9469}
   Состояние: [1] Стабильный
   Последняя ошибка: Нет ошибок

Имя компонента записи: «Shadow Copy Optimization Writer»
   Id компонента записи: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
   Id экземпляра компонента записи: {97c6f8a0-62ea-4885-ba5f-48f67a14aa30}
   Состояние: [5] Ожидание завершения
   Последняя ошибка: Нет ошибок

Имя компонента записи: «WMI Writer»
   Id компонента записи: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
   Id экземпляра компонента записи: {585b97ad-5f52-4b57-88b3-1f764ec40996}
   Состояние: [1] Стабильный
   Последняя ошибка: Нет ошибок

Вывод команды vssadmin List Providers
VSSADMIN 1.1 — Утилита теневого копирования тома
(C) Корпорация Майкрософт, 2001-2005.

Имя поставщика: «Hyper-V IC Software Shadow Copy Provider»
   Тип поставщика: Программное обеспечение
   Id поставщика: {74600e39-7dc5-4567-a03b-f091d6c7b092}
   Версия: 1.0.0.0

Имя поставщика: «Microsoft Software Shadow Copy provider 1.0»
   Тип поставщика: Системный
   Id поставщика: {b5946137-7b9f-4925-af80-51abd60b20d5}
   Версия: 1.0.0.7

Вывод команды vssadmin List ShadowStorage
VSSADMIN 1.1 — Утилита теневого копирования тома
(C) Корпорация Майкрософт, 2001-2005.

Сопоставление хранилища теневой копии
   Для тома: (D:)\?Volume{d639b442-8b73-43a5-a5ac-8c6e8e53af7e}
   Том хранилища теневой копии: (D:)\?Volume{d639b442-8b73-43a5-a5ac-8c6e8e53af7e}
   Использованный объем хранилища теневой копии: 0 B (0%)
   Выделенный объем хранилища теневой копии: 0 B (0%)
   Максимальный объем хранилища теневой копии: 75 GB (15%)

Из того что уже наковырял:

Сделал Resize ShadowStorage увеличил до 15%

Удалил компонент «Система архивации данных Windows Server»

Регистрировал dll 

@echo off
:: Shadow Copy and System Recovery Fix

:: Temporary refreshing pagefile and hyberfil may also help
:: look: http://support.microsoft.com/kb/2506576/ru

:: To generate the problem:
:: regsvr32 /u swprv.dll

:: ======= Stop services: «Volume Shadow Copy» and «MS Software Shadow Copy Provider» ========

cd /d %windir%system32
Net stop vss
Net stop swprv

:: ======= Unregister Block =======

regsvr32 /s /u ole32.dll
regsvr32 /s /u oleaut32.dll
regsvr32 /s /u vss_ps.dll
regsvr32 /s /u swprv.dll
regsvr32 /s /u eventcls.dll
regsvr32 /s /u es.dll
regsvr32 /s /u stdprov.dll
regsvr32 /s /u vssui.dll
regsvr32 /s /u msxml.dll
regsvr32 /s /u msxml3.dll
regsvr32 /s /u msxml4.dll

:: ======= Register Block =========
:: look http://support.microsoft.com/kb/940032/ru

regsvr32 /s ole32.dll
regsvr32 /s oleaut32.dll
regsvr32 /s vss_ps.dll
vssvc /register
regsvr32 /s /i swprv.dll
regsvr32 /s /i eventcls.dll
regsvr32 /s es.dll
regsvr32 /s stdprov.dll
regsvr32 /s vssui.dll
regsvr32 /s msxml.dll
regsvr32 /s msxml3.dll
regsvr32 /s msxml4.dll

:: ======= Trying to run services ====
Net start swprv
Net start vss

echo OK. Please, reboot Windows. & pause

но нет возможности перезагрузить сервер, в работе.

Что еще посмотреть?

Архивная копия папок на диске С: создается без проблем

Понравилась статья? Поделить с друзьями:
  • Restore fail please contact your vendor error code 175 перевод
  • Restore fail please contact your vendor error code 121
  • Restore fail please contact your vendor error code 1167 что значит
  • Restore error general checkra1n
  • Restore dco error api parameter error