Here is the results
[root@ESXi01:~] vmkping -s 8972 -d 192.168.1.201
PING 192.168.1.201 (192.168.1.201): 8972 data bytes
8980 bytes from 192.168.1.201: icmp_seq=0 ttl=64 time=0.693 ms
8980 bytes from 192.168.1.201: icmp_seq=1 ttl=64 time=0.698 ms
8980 bytes from 192.168.1.201: icmp_seq=2 ttl=64 time=0.680 ms
— 192.168.1.201 ping statistics —
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.680/0.690/0.698 ms
[root@ESXi01:~] vmkping -I vmk2 -S vmotion 192.168.1.201
PING 192.168.1.201 (192.168.1.201): 56 data bytes
64 bytes from 192.168.1.201: icmp_seq=0 ttl=64 time=0.166 ms
64 bytes from 192.168.1.201: icmp_seq=1 ttl=64 time=0.186 ms
64 bytes from 192.168.1.201: icmp_seq=2 ttl=64 time=0.178 ms
— 192.168.1.201 ping statistics —
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.166/0.177/0.186 ms
———————————————
[root@ESXi02:~] vmkping -s 8972 -d 192.168.1.200
PING 192.168.1.200 (192.168.1.200): 8972 data bytes
8980 bytes from 192.168.1.200: icmp_seq=0 ttl=64 time=0.644 ms
8980 bytes from 192.168.1.200: icmp_seq=1 ttl=64 time=0.674 ms
8980 bytes from 192.168.1.200: icmp_seq=2 ttl=64 time=0.682 ms
— 192.168.1.200 ping statistics —
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.644/0.667/0.682 ms
[root@ESXi02:~] vmkping -I vmk2 -S vmotion 192.168.1.200
PING 192.168.1.200 (192.168.1.200): 56 data bytes
64 bytes from 192.168.1.200: icmp_seq=0 ttl=64 time=0.182 ms
64 bytes from 192.168.1.200: icmp_seq=1 ttl=64 time=0.185 ms
64 bytes from 192.168.1.200: icmp_seq=2 ttl=64 time=0.185 ms
— 192.168.1.200 ping statistics —
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.182/0.184/0.185 ms
—————————————————
I have tried to vMotion from the NFS datastore to the 2 hosts on VMFS6 datastore works perfectly fine no problems, tried to vMotion the same machine back to the NFS datastore failed :S
We have an ESXi host in our VMware Cluster which we are not able to migrate VMs off of. All VMs on this host are affected by the issue.
vMotion fails at 21% with the following error message: «Failed waiting for data. Error 195887137. Timeout.»
vCenter Logs
What’s interesting is that we are able to migrate VMs TO the problem host. It is only when the problem host is the source host that we see the error message. All other migrations between all other hosts in the cluster work fine.
Logs:
/var/log/vmkernel.log:
2020-01-31T3:26:33.987Z cpu3:8322451)Migrate: vm 8322452: 3885: Setting VMOTION info: Source ts = 1426483424790667834, src ip = <x.x.x.16> dest ip = <x.x.x.45> Dest wid = 4218927 using SHARED swap, encrypted
2020-01-31T13:26:33.989Z cpu3:8322451)Hbr: 3561: Migration start received (worldID=8322452) (migrateType=1) (event=0) (isSource=1) (sharedConfig=1)
2020-01-31T13:26:33.989Z cpu14:8943031)MigrateNet: 1751: 1426483424790667834 S: Successfully bound connection to vmknic vmk2 - 'x.x.x.16'
2020-01-31T13:26:33.991Z cpu20:8902233)MigrateNet: vm 8902233: 3263: Accepted connection from <::ffff:x.x.x.45>
2020-01-31T13:26:33.991Z cpu20:8902233)MigrateNet: vm 8902233: 3351: dataSocket 0x430efb2fce30 receive buffer size is 563272
2020-01-31T13:26:33.991Z cpu14:8943031)MigrateNet: 1751: 1426483424790667834 S: Successfully bound connection to vmknic vmk2 - 'x.x.x.16'
2020-01-31T13:26:33.991Z cpu14:8943031)VMotionUtil: 5199: 1426483424790667834 S: Stream connection 1 added.
2020-01-31T13:26:53.994Z cpu1:8943028)WARNING: VMotionUtil: 862: 1426483424790667834 S: failed to read stream keepalive: Connection closed by remote host, possibly due to timeout
2020-01-31T13:26:53.994Z cpu1:8943028)WARNING: Migrate: 282: 1426483424790667834 S: Failed: Connection closed by remote host, possibly due to timeout (0xbad003f) u/0x41802bf0e273
2020-01-31T13:26:54.012Z cpu35:8322484)WARNING: Migrate: 6189: 1426483424790667834 S: Migration considered a failure by the VMX. It is most likely a timeout, but check the VMX log for the true error.
2020-01-31T13:26:54.013Z cpu35:8322484)Hbr: 3655: Migration end received (worldID=8322452) (migrateType=1) (event=1) (isSource=1) (sharedConfig=1)
2020-01-31T13:26:54.014Z cpu0:8943028)VMotionUtil: 7560: 1426483424790667834 S: Socket 0x430efb2fce30 rcvMigFree pending: 33200/33304 snd 0 rcv
VMX log:
2020-01-31T13:26:33.965Z| vmx| I125: MigrateSetState: Transitioning from state 8 to 9.
2020-01-31T13:26:33.995Z| vmx| I125: MigrateRPC_RetrieveMessages: Informed of a new user message, but can't handle messages in state 4. Leaving the message queued.
2020-01-31T13:26:35.131Z| vmx| I125: Vigor_ClientRequestCb: failed to do op=3 on unregistered device 'Tools' (cmd=queryFields)
2020-01-31T13:26:35.131Z| vmx| I125: Vigor_ClientRequestCb: failed to do op=3 on unregistered device 'GuestInfo' (cmd=queryFields)
2020-01-31T13:26:35.131Z| vmx| I125: Vigor_ClientRequestCb: failed to do op=3 on unregistered device 'GuestPeriodic' (cmd=queryFields)
2020-01-31T13:26:35.131Z| vmx| I125: Vigor_ClientRequestCb: failed to do op=3 on unregistered device 'GuestAppMonitor' (cmd=queryFields)
2020-01-31T13:26:35.131Z| vmx| I125: Vigor_ClientRequestCb: failed to do op=3 on unregistered device 'CrashDetector' (cmd=queryFields)
2020-01-31T13:26:38.833Z| vmx| I125: Vigor_ClientRequestCb: failed to do op=3 on unregistered device 'Tools' (cmd=queryFields)
2020-01-31T13:26:38.833Z| vmx| I125: Vigor_ClientRequestCb: failed to do op=3 on unregistered device 'GuestInfo' (cmd=queryFields)
2020-01-31T13:26:38.833Z| vmx| I125: Vigor_ClientRequestCb: failed to do op=3 on unregistered device 'GuestPeriodic' (cmd=queryFields)
2020-01-31T13:26:38.833Z| vmx| I125: Vigor_ClientRequestCb: failed to do op=3 on unregistered device 'GuestAppMonitor' (cmd=queryFields)
2020-01-31T13:26:38.833Z| vmx| I125: Vigor_ClientRequestCb: failed to do op=3 on unregistered device 'CrashDetector' (cmd=queryFields)
2020-01-31T13:26:53.996Z| vmx| I125: MigrateSetStateFinished: type=2 new state=12
2020-01-31T13:26:53.996Z| vmx| I125: MigrateSetState: Transitioning from state 9 to 12.
2020-01-31T13:26:53.996Z| vmx| I125: Migrate: Caching migration error message list:
2020-01-31T13:26:53.996Z| vmx| I125: [msg.migrate.waitdata.platform] Failed waiting for data. Error bad0021. Timeout.
2020-01-31T13:26:53.996Z| vmx| I125: [vob.vmotion.net.send.start.failed] vMotion migration [a011610:1426483424790667834] failed to send init message to the remote host <x.x.x.16>
2020-01-31T13:26:53.996Z| vmx| I125: [vob.vmotion.net.sbwait.timeout] vMotion migration [a011610:1426483424790667834] timed out waiting 20001 ms to transmit data.
2020-01-31T13:26:53.996Z| vmx| I125: Migrate: cleaning up migration state.
2020-01-31T13:26:53.997Z| vmx| I125: Migrate: Final status reported through Vigor.
2020-01-31T13:26:53.997Z| vmx| I125: MigrateSetState: Transitioning from state 12 to 0.
2020-01-31T13:26:53.997Z| vmx| I125: Module 'Migrate' power on failed.
Things I’ve checked:
-
EVC is configured at the cluster level so there are no CPU compatibility issues.
-
vmkping is successful between the source and destination hosts using the vmotion tp/ip stack:
[root@:~] vmkping ++netstack=vmotion -I vmk2 x.x.x.16 -s 1472
1480 bytes from x.x.x.16: icmp_seq=0 ttl=64 time=0.263 ms
1480 bytes from x.x.x.16: icmp_seq=1 ttl=64 time=0.490 ms
1480 bytes from x.x.x.16: icmp_seq=2 ttl=64 time=0.548 ms
[root@:~] vmkping ++netstack=vmotion -I vmk1 -d x.x.x.45 -s 1472
1480 bytes from x.x.x.45: icmp_seq=0 ttl=64 time=0.351 ms
1480 bytes from x.x.x.45: icmp_seq=1 ttl=64 time=0.299 ms
1480 bytes from x.x.x.45: icmp_seq=2 ttl=64 time=0.393 ms
-
MTU is set to 1500 on the vmotion vmk interfaces, and is higher on the dvswitch and physical switch, so no issue there.
-
VLAN configuration on the physical switches is correct.
-
Tried moving the vmotion vmk to a VSS instead of the VDS.
-
Verified that we have only a single interface configured for vmotion on the host, even though multi vmks is supported.
-
Restarted the management agents on the ESXi host didn’t help.
-
Disabling vmotion encryption for a VM resulted in the same error.
-
Reconfigured the host for HA, but this didn’t help.
Unfortunately the server in question is no longer supported on the HCL, so VMware support would not help with the troubleshooting. However, had we been running Broadwell generation CPUs (v4) instead of Haswell (v3), then it would be supported:
SYS-2048U-RTR4 Intel Xeon E5-4600-v3 Series 6.0 U3
SYS-2048U-RTR4 Intel Xeon E5-4600-v4 Series 6.7 U3
I struggle to imagine this error we’re seeing is caused by running a Haswell CPU instead of Broadwell, but I guess anything is possible…
We’ve made no changes recently to the environment, and vMotion was working previously a few days ago on this host.
Anyway, I just wanted to reach out to see if anyone else is experiencing a similar issue, or if there are some VMware enginners lurking on Reddit who would be interested in troubleshooting this problem to confirm whether it is hardware compatibility related or not.
Update:
I never found the root cause of the issue. As part of my troubleshooting, I configured the ESXi management vmk for vmotion traffic on two of the hosts in my cluster. I was successfully able to migrate a VM off the host with this configuration. I then rolled back the change (disabled vmotion on the management vmk, and re-enabled it on my vmotion vmk) and voilà, I could now vmotion all the VMs off the host.
Problem
Eine VM möchte sich nicht via StorageMotion nicht auf ein VVOL verschieben lassen. Im Aktivitätsprotokoll sieht man den Fehler:
Fehler beim Warten auf Daten. Fehler 195887250. Migration determined a failure by the VMX.
und/oder
Migration determined a failure by the VMX. Storage vMotion konnte die Zielfestplatte /vmfs/volumes/vvol:0000000200004004-825d720126911b58/rfc4122.16630ae1-3756-4e47-b932-cdebf5862fd7/SERVERNAME.vmdk nicht erstellen.
und/oder
Erstellen einer oder mehrerer Zielfestplatten fehlgeschlagen. Es ist ein schwerwiegender interner Fehler aufgetreten. Weitere Details finden Sie im Protokoll der virtuellen Maschine.
Spannend zu suchen ist der Fehler 195887250. Der VMKernel meint damit „VMK_MIGRATE_VMX_FAILURE“, oder in ausgeschrieben „Migration determined a failure by the VMX“.
Lösung
Der weitaus häufigste Grund ist relativ einfach: Eine der Festplatten der betroffenen Maschine hat eine größe, die nicht durch 1Mbyte teilbar ist. VVOLs können nur vielfaches von 1Mbyte allokieren, daher schlägt das anlegen der Platte fehl.
Das sieht man auch im vmware.log der betroffenen Maschine:
2018-08-06T11:50:01.264Z| worker-2161938| I125: DISKLIB-LIB_CREATE : CREATE: Creating disk backed by 'vvol' 2018-08-06T11:50:01.265Z| worker-2161938| W115: OBJLIB-VVOLOBJ : VVolObjCheckSize: Requested size (32217052160) is not an MB multiple. 2018-08-06T11:50:01.265Z| worker-2161938| W115: OBJLIB-VVOLOBJ : VVolObjDetermineSizeInMB: Requested size (32217052160) is not a MB multiple. 2018-08-06T11:50:01.265Z| worker-2161938| W115: Mirror: scsi0:0: SVMotionLocalDiskCreate: Failed to create destination disk: The requested size is not a multiple of 1MB 2018-08-06T11:50:01.265Z| worker-2161938| W115: Mirror: scsi0:0: Failed to create disk from /vmfs/volumes/.../NAME.vmdk to /vmfs/volumes/.../NAME.vmdk.
Ärgerlicherweise behauptet die VCSA stattdessen: „No space left on device“
Der zweite Fall der ich einmal untersuchen durfte, war eine „kaputte“ Netzwerkkarte. Alles lief einwandfrei, nur storage motion auf dieser Karte nicht. NIC ausgetauscht, alles wieder fertig.
vmkerrcode utility displays a table of all VMkernel errors, a brief description, and the corresponding error codes in both decimal and hexadecimal format. This utility was added in ESX 3.0.2
How to use it:-
1. Connect with ESXi Shell (Local or through SSH)
2. Run this command:-
vmkerrcode -l
3. You will get the output like this:-
VMK_ReturnStatus
table for VMware ESX, Build #4564106
Name Dec Hex errno equiv Description
————————————————————————————————————————————————————-
VMK_OK
0 0 OK Success
VMK_FAILURE
195887105 0xbad0001 EINVAL Failure
VMK_WOULD_BLOCK
195887106 0xbad0002 EAGAIN Would block
VMK_NOT_FOUND
195887107 0xbad0003 ENOENT Not found
VMK_BUSY
195887108 0xbad0004 EBUSY
Busy
VMK_EXISTS
195887109 0xbad0005 EEXIST Already exists
VMK_LIMIT_EXCEEDED
195887110 0xbad0006 EFBIG Limit exceeded
VMK_BAD_PARAM 195887111 0xbad0007 EINVAL Bad parameter
VMK_METADATA_READ_ERROR
195887112 0xbad0008 EIO Metadata read error
VMK_METADATA_WRITE_ERROR
195887113 0xbad0009 EIO
Metadata write
error
VMK_IO_ERROR
195887114 0xbad000a EIO I/O error
VMK_READ_ERROR
195887115 0xbad000b EIO Read error
VMK_WRITE_ERROR 195887116 0xbad000c EIO Write error
VMK_INVALID_NAME
195887117 0xbad000d ENAMETOOLONG Invalid name
VMK_INVALID_HANDLE
195887118 0xbad000e EBADF Invalid handle
VMK_INVALID_ADAPTER
195887119 0xbad000f ENODEV No such SCSI adapter
VMK_INVALID_TARGET
195887120 0xbad0010 ENODEV No such target on adapter
VMK_INVALID_PARTITION
195887121 0xbad0011 ENXIO No such partition on target
VMK_INVALID_FS
195887122 0xbad0012 ENXIO No filesystem on the device
VMK_INVALID_MEMMAP
195887123 0xbad0013 EFAULT Memory map mismatch
VMK_NO_MEMORY
195887124 0xbad0014 ENOMEM Out of memory
VMK_NO_MEMORY_RETRY
195887125 0xbad0015 ENOMEM Out of memory (ok to retry)
VMK_NO_LPAGE_MEMORY
195887126 0xbad0016 ENOMEM Out of large pages
VMK_NO_RESOURCES
195887127 0xbad0017 ENOMEM Out of resources
VMK_NO_FREE_HANDLES 195887128 0xbad0018 EMFILE No free handles
VMK_NUM_HANDLES_EXCEEDED
195887129 0xbad0019 ENFILE Exceeded maximum number of allowed
handles
VMK_DEPRECATED_NO_FREE_PTR_BLOCKS
195887130 0xbad001a ENOSPC No free pointer blocks
(deprecated)
VMK_DEPRECATED_NO_FREE_DATA_BLOCKS
195887131 0xbad001b ENOSPC No free data blocks
(deprecated)
VMK_CORRUPT_REDOLOG
195887132 0xbad001c EBADF Corrupt RedoLog
VMK_STATUS_PENDING
195887133 0xbad001d EAGAIN Status pending
VMK_STATUS_FREE
195887134 0xbad001e EAGAIN Status free
VMK_UNSUPPORTED_CPU
195887135 0xbad001f ENODEV Unsupported CPU
VMK_NOT_SUPPORTED
195887136 0xbad0020 ENOSYS Not supported
VMK_TIMEOUT
195887137 0xbad0021 ETIMEDOUT Timeout
VMK_READ_ONLY
195887138 0xbad0022 EROFS Read only
VMK_RESERVATION_CONFLICT
195887139 0xbad0023 EAGAIN SCSI reservation conflict
VMK_FS_LOCKED 195887140 0xbad0024 EADDRINUSE File system locked
VMK_NOT_ENOUGH_SLOTS
195887141 0xbad0025 ENFILE Out of slots
VMK_INVALID_ADDRESS
195887142 0xbad0026 EFAULT Invalid address
VMK_NOT_SHARED
195887143 0xbad0027 ENOMEM Not shared
VMK_SHARED
195887144 0xbad0028 ENOMEM Page is shared
VMK_KSEG_PAIR_FLUSHED
195887145 0xbad0029 ENOMEM Kseg pair flushed
VMK_MAX_ASYNCIO_PENDING
195887146 0xbad002a ENOMEM Max async I/O requests pending
VMK_VERSION_MISMATCH_MINOR
195887147 0xbad002b ENOSYS Minor version mismatch
VMK_VERSION_MISMATCH_MAJOR
195887148 0xbad002c ENOSYS Major version mismatch
VMK_IS_CONNECTED
195887149 0xbad002d EINVAL Already connected
VMK_IS_DISCONNECTED
195887150 0xbad002e ENOTCONN Already disconnected
VMK_IS_ENABLED
195887151 0xbad002f EINVAL Already enabled
VMK_IS_DISABLED 195887152 0xbad0030 EINVAL Already disabled
VMK_NOT_INITIALIZED
195887153 0xbad0031 EINVAL Not initialized
VMK_WAIT_INTERRUPTED
195887154 0xbad0032 EINTR
Wait interrupted
VMK_NAME_TOO_LONG
195887155 0xbad0033 ENAMETOOLONG Name too long
VMK_MISSING_FS_PES
195887156 0xbad0034 ENODEV VMFS volume missing physical
extents
VMK_NICTEAMING_VALID_MASTER
195887157 0xbad0035 EINVAL NIC teaming master valid
VMK_NICTEAMING_SLAVE
195887158 0xbad0036 EEXIST NIC teaming slave
VMK_NICTEAMING_REGULAR_VMNIC
195887159 0xbad0037 EINVAL NIC teaming regular VMNIC
VMK_ABORT_NOT_RUNNING
195887160 0xbad0038 ECANCELED Abort not running
VMK_NOT_READY
195887161 0xbad0039 EIO
Not ready
VMK_CHECKSUM_MISMATCH
195887162 0xbad003a EIO Checksum mismatch
VMK_VLAN_NO_HW_ACCEL
195887163 0xbad003b EINVAL VLan HW Acceleration not
supported
VMK_NO_VLAN_SUPPORT
195887164 0xbad003c EOPNOTSUPP VLan is not supported in vmkernel
VMK_NOT_VLAN_HANDLE
195887165 0xbad003d EINVAL Not a VLan handle
VMK_BAD_VLANID
195887166 0xbad003e EBADF Couldn’t retrieve VLan id
VMK_MIG_CONN_CLOSED
195887167 0xbad003f EINVAL Connection closed by remote host,
possibly due to timeout
VMK_NO_CONNECT
195887168 0xbad0040 EIO No connection
VMK_SEGMENT_OVERLAP
195887169 0xbad0041 EINVAL Segment overlap
VMK_BAD_MPS
195887170 0xbad0042 EIO Error parsing MPS Table
VMK_BAD_ACPI
195887171 0xbad0043 EIO Error parsing ACPI Table
VMK_RESUME_ERROR
195887172 0xbad0044 EIO Failed to resume virtual
machine
VMK_NO_ADDRESS_SPACE
195887173 0xbad0045 ENOMEM Insufficient address space for
operation
VMK_BAD_ADDR_RANGE
195887174 0xbad0046 EINVAL Bad address range
VMK_ENETDOWN 195887175 0xbad0047 ENETDOWN Network is down
VMK_ENETUNREACH
195887176 0xbad0048 ENETUNREACH Network unreachable
VMK_ENETRESET
195887177 0xbad0049 ENETRESET Network dropped connection on
reset
VMK_ECONNABORTED
195887178 0xbad004a ECONNABORTED Software caused connection abort
VMK_ECONNRESET
195887179 0xbad004b ECONNRESET Connection reset by peer
VMK_ENOTCONN
195887180 0xbad004c ENOTCONN Socket is not connected
VMK_ESHUTDOWN
195887181 0xbad004d ESHUTDOWN Cannot send after socket shutdown
VMK_ETOOMANYREFS
195887182 0xbad004e ETOOMANYREFS Too many references: cannot splice
VMK_ECONNREFUSED
195887183 0xbad004f ECONNREFUSED Connection refused
VMK_EHOSTDOWN 195887184 0xbad0050 EHOSTDOWN Host is down
VMK_EHOSTUNREACH
195887185 0xbad0051 EHOSTUNREACH No route to host
VMK_EADDRINUSE
195887186 0xbad0052 EADDRINUSE Address already in use
VMK_BROKEN_PIPE
195887187 0xbad0053 EPIPE Broken pipe
VMK_NOT_A_DIRECTORY
195887188 0xbad0054 ENOTDIR Not a directory
VMK_IS_A_DIRECTORY 195887189 0xbad0055 EISDIR Is a directory
VMK_NOT_EMPTY
195887190 0xbad0056 ENOTEMPTY Directory not empty
VMK_NOT_IMPLEMENTED
195887191 0xbad0057 ENOSYS Not implemented
VMK_NO_SIGNAL_HANDLER
195887192 0xbad0058 EINVAL No signal handler
VMK_FATAL_SIGNAL_BLOCKED
195887193 0xbad0059 EINVAL Fatal signal blocked
VMK_NO_ACCESS
195887194 0xbad005a EACCES Permission denied
VMK_NO_PERMISSION
195887195 0xbad005b EPERM Operation not permitted
VMK_UNDEFINED_SYSCALL
195887196 0xbad005c ENOSYS Undefined syscall
VMK_RESULT_TOO_LARGE
195887197 0xbad005d ERANGE Result too large
VMK_VLAN_FILTERED
195887198 0xbad005e ERANGE Pkts dropped because of VLAN
(support) mismatch
VMK_BAD_EXCFRAME
195887199 0xbad005f EFAULT Unsafe exception frame
VMK_MODULE_NOT_LOADED
195887200 0xbad0060 ENODEV Necessary module isn’t loaded
VMK_NO_SUCH_ZOMBIE
195887201 0xbad0061 ECHILD No dead world by that name
VMK_NO_SUCH_CARTEL
195887202 0xbad0062 ESRCH No cartel by that name
VMK_IS_A_SYMLINK 195887203 0xbad0063 ELOOP Is a symbolic link
VMK_CROSS_DEVICE_LINK
195887204 0xbad0064 EXDEV Cross-device link
VMK_NOT_A_SOCKET
195887205 0xbad0065 ENOTSOCK
Not a socket
VMK_ILLEGAL_SEEK
195887206 0xbad0066 ESPIPE Illegal seek
VMK_ADDRFAM_UNSUPP
195887207 0xbad0067 EAFNOSUPPORT Unsupported address family
VMK_ALREADY_CONNECTED
195887208 0xbad0068 EISCONN Already connected
VMK_DEATH_PENDING
195887209 0xbad0069 ENOENT World is marked for death
VMK_NO_CPU_ASSIGNMENT
195887210 0xbad006a EINVAL No valid scheduler assignment
VMK_CPU_MIN_INVALID
195887211 0xbad006b EINVAL Invalid cpu min
VMK_CPU_MINLIMIT_INVALID
195887212 0xbad006c EINVAL Invalid cpu minLimit
VMK_CPU_MAX_INVALID
195887213 0xbad006d EINVAL Invalid cpu max
VMK_CPU_SHARES_INVALID
195887214 0xbad006e EINVAL Invalid cpu shares
VMK_CPU_MIN_OVERFLOW 195887215 0xbad006f EINVAL CPU min outside valid range
VMK_CPU_MINLIMIT_OVERFLOW
195887216 0xbad0070 EINVAL CPU minLimit outside valid
range
VMK_CPU_MAX_OVERFLOW
195887217 0xbad0071 EINVAL CPU max outside valid range
VMK_CPU_MIN_GT_MINLIMIT
195887218 0xbad0072 EINVAL CPU min exceeds minLimit
VMK_CPU_MIN_GT_MAX
195887219 0xbad0073 EINVAL CPU min exceeds max
VMK_CPU_MINLIMIT_LT_RESERVED
195887220 0xbad0074 ENOSPC CPU minLimit less than cpu already
reserved by children
VMK_CPU_MAX_LT_RESERVED
195887221 0xbad0075 ENOSPC
CPU max less than cpu
already reserved by children
VMK_CPU_ADMIT_FAILED
195887222 0xbad0076 ENOSPC Admission check failed for cpu
resource
VMK_MEM_MIN_INVALID
195887223 0xbad0077 EINVAL Invalid memory min
VMK_MEM_MINLIMIT_INVALID
195887224 0xbad0078 EINVAL Invalid memory minLimit
VMK_MEM_MAX_INVALID
195887225 0xbad0079 EINVAL Invalid memory max
VMK_MEM_MIN_OVERFLOW
195887226 0xbad007a EINVAL Memory min outside valid range
VMK_MEM_MINLIMIT_OVERFLOW
195887227 0xbad007b EINVAL Memory minLimit outside valid
range
VMK_MEM_MAX_OVERFLOW
195887228 0xbad007c EINVAL Memory max outside valid range
VMK_MEM_MIN_GT_MINLIMIT
195887229 0xbad007d EINVAL Memory min exceeds minLimit
VMK_MEM_MIN_GT_MAX
195887230 0xbad007e EINVAL Memory min exceeds max
VMK_MEM_MINLIMIT_LT_RESERVED
195887231 0xbad007f ENOSPC Memory minLimit less than memory
already reserved by children
VMK_MEM_MAX_LT_RESERVED 195887232
0xbad0080 ENOSPC Memory max less than memory already
reserved by children
VMK_MEM_ADMIT_FAILED
195887233 0xbad0081 ENOSPC Admission check failed for memory
resource
VMK_NO_SWAP_FILE 195887234 0xbad0082 ENOENT No swap file
VMK_BAD_PARAM_COUNT
195887235 0xbad0083 EINVAL Bad parameter count
VMK_BAD_PARAM_TYPE
195887236 0xbad0084 EINVAL Bad parameter type
VMK_UNMAP_RETRY
195887237 0xbad0085 ENOMEM Dueling unmaps (ok to retry)
VMK_INVALID_IOCTL
195887238 0xbad0086 ENOTTY Inappropriate ioctl for device
VMK_MAPFAULT_RETRY
195887239 0xbad0087 EBUSY Mmap changed under page fault (ok to
retry)
VMK_EINPROGRESS
195887240 0xbad0088 EINPROGRESS Operation now in progress
VMK_ADDR_UNMAPPED
195887241 0xbad0089 EFAULT Address temporarily unmapped
VMK_INVALID_BUDDY_TYPE
195887242 0xbad008a ENOMEM Invalid buddy type
VMK_LPAGE_INFO_NOT_FOUND 195887243 0xbad008b ENOMEM Large page info not found
VMK_LPAGE_INFO_INVALID
195887244 0xbad008c EINVAL Invalid large page info
VMK_SNAPSHOT_DEV
195887245 0xbad008d EIO SCSI LUN is in snapshot
state
VMK_IN_TRANSITION
195887246 0xbad008e EIO SCSI LUN is in transition
VMK_TXN_FULL
195887247 0xbad008f ENOSPC Transaction ran out of lock space or
log space
VMK_LOCK_NOT_FREE
195887248 0xbad0090 EBUSY Lock was not free
VMK_NUM_FILES_EXCEEDED
195887249 0xbad0091 ENOSPC Exceed maximum number of files on the
filesystem
VMK_MIGRATE_VMX_FAILURE
195887250 0xbad0092 EINVAL Migration determined a failure by the
VMX
VMK_VSI_LIST_OVERFLOW
195887251 0xbad0093 EFBIG VSI GetList handler overflow
VMK_INVALID_WORLD
195887252 0xbad0094 EINVAL Invalid world
VMK_INVALID_VMM
195887253 0xbad0095 EINVAL Invalid vmm
VMK_INVALID_TXN 195887254 0xbad0096 EINVAL Invalid transaction
VMK_FS_RETRY_OPERATION
195887255 0xbad0097 EAGAIN Transient file system condition,
suggest retry
VMK_VCPU_LIMIT_EXCEEDED
195887256 0xbad0098 EINVAL Number of running VCPUs limit
exceeded
VMK_INVALID_METADATA
195887257 0xbad0099 EINVAL Invalid metadata
VMK_INVALID_PAGE_NUMBER
195887258 0xbad009a EINVAL
Invalid page number
VMK_NOT_EXEC
195887259 0xbad009b ENOEXEC Not in executable format
VMK_NFS_CONNECT_FAILURE
195887260 0xbad009c EHOSTDOWN Unable to connect to NFS server
VMK_NFS_MOUNT_NOT_SUPPORTED
195887261 0xbad009d EINVAL The NFS server does not support MOUNT
version 3 over TCP
VMK_NFS_NFS_NOT_SUPPORTED
195887262 0xbad009e EINVAL The NFS server does not support NFS
version 3 over TCP
VMK_NFS_MOUNT_DENIED
195887263 0xbad009f EPERM The NFS server denied the mount
request
VMK_NFS_MOUNT_NOT_DIR
195887264 0xbad00a0 ENOTDIR The specified mount path was not a
directory
VMK_NFS_BAD_FSINFO
195887265 0xbad00a1 EACCES Unable to query remote mount point’s
attributes
VMK_NFS_VOLUME_LIMIT_EXCEEDED
195887266 0xbad00a2 EINVAL NFS has reached the maximum number of
supported volumes
VMK_NO_MEMORY_NICE
195887267 0xbad00a3 ENOMEM Out of nice memory
VMK_MIGRATE_PREEMPTIVE_FAIL
195887268 0xbad00a4 ENOMEM Migration failed to start due to lack
of CPU or memory resources
VMK_CACHE_MISS
195887269 0xbad00a5 EFAULT Cache miss
VMK_STRESS_INDUCED_ERROR
195887270 0xbad00a6 EIO Error induced when stress options
are enabled
VMK_TOO_MANY_LOCK_HOLDERS
195887271 0xbad00a7 EUSERS Maximum number of concurrent hosts
are already accessing this resource
VMK_NO_JOURNAL
195887272 0xbad00a8 EIO Host doesn’t have a journal
VMK_RANK_VIOLATION
195887273 0xbad00a9 EDEADLK Lock rank violation detected
VMK_MODULE_FAILED
195887274 0xbad00aa ENODEV Module failed
VMK_NO_MASTER_PTY
195887275 0xbad00ab ENXIO Unable to open slave if no master
pty
VMK_NOT_IOABLE
195887276 0xbad00ac EFAULT Not IOAble
VMK_NO_FREE_INODES 195887277 0xbad00ad ENOSPC No free inodes
VMK_NO_MEMORY_FOR_FILEDATA
195887278 0xbad00ae ENOSPC No free memory for file data
VMK_NO_TAR_SPACE
195887279 0xbad00af ENOSPC No free space to expand file or meta
data
VMK_NO_FIFO_READER
195887280 0xbad00b0 ENXIO Unable to open writer if no fifo
reader
VMK_NO_SUCH_DEVICE
195887281 0xbad00b1 EINVAL No underlying device for
major,minor
VMK_MEM_MIN_GT_MEMSIZE
195887282 0xbad00b2 EINVAL Memory min exceeds memSize
VMK_NO_SUCH_VT
195887283 0xbad00b3 ENXIO No virtual terminal for number
VMK_TOO_MANY_ELEMENTS
195887284 0xbad00b4 E2BIG Too many elements for list
VMK_SHAREDAREA_MISMATCH
195887285 0xbad00b5 ENOSYS VMM<->VMK shared area
mismatch
VMK_EXEC_FAILURE
195887286 0xbad00b6 ESRCH Failure during exec while original
state already lost
VMK_INVALID_MODULE
195887287 0xbad00b7 EINVAL Invalid module
VMK_UNALIGNED_ADDRESS
195887288 0xbad00b8 EINVAL Address is not aligned on required
boundary
VMK_NOT_MAPPED
195887289 0xbad00b9 ENOMEM Address is not mapped in address space
VMK_NO_MESSAGE_SPACE
195887290 0xbad00ba ENOMEM No space to record a message
VMK_EXCEPTION_HANDLER_INVALID
195887291 0xbad00bb EINVAL Invalid exception handler
VMK_EXCEPTION_NOT_HANDLED
195887292 0xbad00bc EINVAL Exception not handled by exception handler
VMK_INVALID_MULTIWRITER_OBJECT
195887293 0xbad00bd EDEADLK Cannot open sparse/TBZ files in
multiwriter mode
VMK_STORAGE_RETRY_OPERATION
195887294 0xbad00be EAGAIN Transient storage condition, suggest
retry
VMK_HBA_ERROR
195887295 0xbad00bf EIO Storage initiator error
VMK_TIMER_INIT_FAILED 195887296 0xbad00c0 EINVAL Timer initialization failed
VMK_MODULE_NOT_FOUND
195887297 0xbad00c1 ENOENT Module not found
VMK_NOT_SOCKET_OWNER
195887298 0xbad00c2 EINVAL Socket not owned by cartel
VMK_VSI_HANDLER_NOT_FOUND
195887299 0xbad00c3 ENOENT No VSI handler found for the
requested node
VMK_INVALID_MMAPPROTFLAGS
195887300 0xbad00c4 EINVAL Invalid mmap protection flags
VMK_INVALID_MAPCONTIG_SIZE
195887301 0xbad00c5 EINVAL Invalid chunk size for contiguous
mmap
VMK_INVALID_MAPCONTIG_MAX
195887302 0xbad00c6 EINVAL Invalid MPN max for contiguous mmap
VMK_INVALID_MAPCONTIG_FLAG
195887303 0xbad00c7 EINVAL Invalid mmap flag on contiguous mmap
VMK_NOT_LAZY_MMINFO
195887304 0xbad00c8 EINVAL Unexpected fault on pre-faulted
memory region
VMK_MMINFO_WONT_SPLIT
195887305 0xbad00c9 EINVAL Memory region cannot be split
(remap/unmap)
VMK_NO_CACHE_INFO
195887306 0xbad00ca ENOENT Cache Information not available
VMK_CANNOT_REMAP_PINNED_MEMORY
195887307 0xbad00cb EINVAL Cannot remap pinned memory
VMK_NO_SUCH_CARTELGROUP
195887308 0xbad00cc ESRCH No cartel group by that name
VMK_SPLOCKSTATS_DISABLED
195887309 0xbad00cd EINVAL SPLock stats collection
disabled
VMK_BAD_TAR_IMAGE
195887310 0xbad00ce EINVAL Boot image is corrupted
VMK_BRANCHED_ALREADY
195887311 0xbad00cf EPERM Branched file cannot be modified
VMK_NAME_RESERVED_FOR_BRANCH
195887312 0xbad00d0 EPERM Name is reserved for branched
file
VMK_CANNOT_BRANCH_UNLINKED 195887313 0xbad00d1 EPERM Unlinked file cannot be
branched
VMK_MAX_RETRIES_EXCEEDED
195887314 0xbad00d2 EAGAIN Maximum kernel-level retries
exceeded
VMK_OPTLOCK_STOLEN
195887315 0xbad00d3 EAGAIN Optimistic lock acquired by another
host
VMK_NOT_MMAPABLE
195887316 0xbad00d4 ENODEV Object cannot be mmapped
VMK_INVALID_CPU_AFFINITY
195887317 0xbad00d5 EINVAL Invalid cpu affinity
VMK_DEVICE_NOT_PARTOF_LV
195887318 0xbad00d6 ENXIO Device does not contain a logical
volume
VMK_NO_SPACE
195887319 0xbad00d7 ENOSPC
No space left on
device
VMK_VSI_INVALID_NODE_ID
195887320 0xbad00d8 EINVAL Invalid vsi node ID
VMK_TOO_MANY_USERS
195887321 0xbad00d9 EUSERS Too many users accessing this resource
VMK_EALREADY
195887322 0xbad00da EALREADY Operation already in progress
VMK_BUF_TOO_SMALL
195887323 0xbad00db EINVAL Buffer too small to complete the
operation
VMK_SNAPSHOT_DEV_DISALLOWED
195887324 0xbad00dc EACCES Snapshot device disallowed
VMK_LVM_DEVICE_UNREACHABLE
195887325 0xbad00dd EIO LVM device unreachable
VMK_CPU_INVALID_RESOURCE_UNITS 195887326 0xbad00de EINVAL Invalid cpu resource units
VMK_MEM_INVALID_RESOURCE_UNITS
195887327 0xbad00df EINVAL Invalid memory resource units
VMK_ABORTED
195887328 0xbad00e0 ECANCELED IO was aborted
VMK_MEM_MIN_LT_RESERVED
195887329 0xbad00e1 ENOSPC Memory min less than memory already
reserved by children
VMK_MEM_MIN_LT_CONSUMED
195887330 0xbad00e2 ENOSPC Memory min less than memory required
to support current consumption
VMK_MEM_MAX_LT_CONSUMED
195887331 0xbad00e3 ENOSPC Memory max less than memory required
to support current consumption
VMK_TIMEOUT_RETRY_DEPRECATED
195887332 0xbad00e4 ETIMEDOUT Timeout (ok to retry) DEPRECATED
VMK_RESERVATION_LOST
195887333 0xbad00e5 EBUSY Reservation Lost
VMK_FS_STALE_METADATA
195887334 0xbad00e6 ENOENT Cached metadata is stale
VMK_NO_FCNTL_LOCK
195887335 0xbad00e7 ENOLCK No fcntl lock slot left
VMK_NO_FCNTL_LOCK_HOLDER
195887336 0xbad00e8 ENOLCK No fcntl lock holder slot left
VMK_NO_LICENSE
195887337 0xbad00e9 EACCES Not licensed to access VMFS
volumes
VMK_VSI_MODULE_NOT_FOUND
195887338 0xbad00ea ENOENT Vmkernel module necessary for this
vsi call not loaded
VMK_LVM_RETRY_OPERATION
195887339 0xbad00eb EAGAIN Transient LVM device condition,
suggest retry
VMK_SNAPSHOT_LV_INCOMPLETE
195887340 0xbad00ec EAGAIN Snapshot LV incomplete
VMK_MEDIUM_NOT_FOUND
195887341 0xbad00ed EIO Medium not found
VMK_MAX_PATHS_CLAIMED
195887342 0xbad00ee ENOMEM Maximum allowed SCSI paths have
already been claimed
VMK_NOT_MOUNTABLE
195887343 0xbad00ef ENODEV Filesystem is not mountable
VMK_MEMSIZE_GT_MEMSIZELIMIT
195887344 0xbad00f0 EINVAL Memory size exceeds memSizeLimit
VMK_RECORD_WRITE_ERROR
195887345 0xbad00f1 EIO An error occurred trying to write
to the log
VMK_REPLAY_READ_ERROR
195887346 0xbad00f2 EIO An error occurred trying to read from
the log
VMK_REPLAY_TYPE_MISMATCH
195887347 0xbad00f3 EIO There was a type mismatch while
reading from the log
VMK_REPLAY_DIVERGENCE
195887348 0xbad00f4 EIO A divergence was detected during
replay
VMK_FT_NOT_RESPONDING
195887349 0xbad00f5 ENOTCONN The remote side of an FT pair isn’t
responding
VMK_NET_REPLAY_ERROR
195887350 0xbad00f6 EIO An error occurred during replay of
networking.
VMK_VOBERR_INVALID_VOBID
195887351 0xbad00f7 EINVAL Vob ID invalid
VMK_VOBERR_FMT_LIMIT_EXCEEDED
195887352 0xbad00f8 EFBIG Vob format string too long
VMK_VOBERR_INVALID_FMT_STRING
195887353 0xbad00f9 EINVAL Invalid format specifier in VOB
format string
VMK_VOBERR_INVALID_ATTR
195887354 0xbad00fa EINVAL Invalid attribute specifier in VOB
format string
VMK_ELF_CORRUPT
195887355 0xbad00fb EINVAL ELF file is corrupt.
VMK_EADDRNOTAVAIL
195887356 0xbad00fc EADDRNOTAVAIL Address not available
VMK_EDESTADDRREQ 195887357 0xbad00fd EDESTADDRREQ Destination address required
VMK_LVM_STALE_METADATA
195887358 0xbad00fe EPERM Cached LVM metadata is stale.
VMK_NO_RPC_TABLE
195887359 0xbad00ff ENOENT RPC table does not exist
VMK_DUPLICATE_UID
195887360 0xbad0100 EEXIST Device already has UID
VMK_UNRESOLVED_SYMBOL
195887361 0xbad0101 ENOENT Unresolved symbol
VMK_DEVICE_NOT_OWNED
195887362 0xbad0102 EINVAL VMkernel does not own the
device
VMK_DEVICE_NOT_NAMED
195887363 0xbad0103 EINVAL Device has no name
VMK_EPROTONOSUPPORT
195887364 0xbad0104 EPROTONOSUPPORT Protocol not
supported
VMK_EOPNOTSUPP
195887365 0xbad0105 EOPNOTSUPP Operation not supported
VMK_UNDEFINED_VMKCALL
195887366 0xbad0106 ENOSYS Undefined VMKCall
VMK_MIGRATE_MAX_DOWNTIME_EXCEEDED
195887367 0xbad0107 ETIMEDOUT Maximum switchover time for migration
exceeded
VMK_LOCK_EXISTS
195887368 0xbad0108 EEXIST Multiple RO/MW locks held by the same
host
VMK_MIGRATE_PRECOPY_NO_FORWARD_PROGRESS
195887369 0xbad0109 EINVAL Migration failed due to lack of
pre-copy forward progress
VMK_UID_CHANGED
195887370 0xbad010a EEXIST Device UID changed
VMK_VMOTION_CONNECT_FAILED
195887371 0xbad010b ENOTCONN The ESX hosts failed to connect over
the VMotion network
VMK_NO_MIGRATION_IN_PROGRESS
195887372 0xbad010c ENOENT No migration in progress
VMK_EXEC_FILE_BUSY
195887373 0xbad010d ETXTBSY File is being executed, write access
denied
VMK_FS_TIMEOUT_RETRY
195887374 0xbad010e ETIMEDOUT File system timeout (Ok to retry)
VMK_COW_TIMEOUT_RETRY
195887375 0xbad010f ETIMEDOUT COW timeout (Ok to retry)
VMK_FS_LOCKSTATE_IN_TRANSITION_DEPRECATED
195887376 0xbad0110 EBUSY Lock state is in transition (ok to
retry) DEPRECATED
VMK_FS_LOCK_LOST
195887377 0xbad0111 EIO Lost previously held disk
lock
VMK_NO_SPACE_ON_DEVICE
195887378 0xbad0112 ENOSPC Underlying device has no free space
VMK_EOVERFLOW
195887379 0xbad0113 EOVERFLOW Value too large for defined data
type
VMK_MEM_SHARES_INVALID
195887380 0xbad0114 EINVAL Invalid memory shares
VMK_LVM_INCONSISTENT_LOCKLESSOP
195887381 0xbad0115 EAGAIN LVM lockless op reads in an
inconsistent state
VMK_INVALID_SECURITY_LABEL
195887382 0xbad0116 EINVAL Invalid security label
VMK_ACCESS_DENIED
195887383 0xbad0117 EPERM Access denied by vmkernel access
control policy
VMK_WORK_COMPLETED
195887384 0xbad0118 EALREADY Work has already completed
VMK_WORK_RUNNING
195887385 0xbad0119 EINPROGRESS Work is currently running
VMK_WORK_PENDING
195887386 0xbad011a EEXIST Work is already pending
VMK_WORK_INVALID 195887387 0xbad011b EINVAL Work or properties provided
invalid
VMK_VOBERR_OVERFLOW
195887388 0xbad011c EFBIG VOB context overflow
VMK_VOBERR_INVALID_CONTEXT
195887389 0xbad011d EINVAL VOB context invalid
VMK_VOBERR_LOCK_CONFLICT
195887390 0xbad011e EINVAL VOB context conflict for lock
VMK_RETRY
195887391 0xbad011f EINVAL Retry the operation
VMK_NO_MODULE_HEAP
195887392 0xbad0120 ENOMEM Module has no heap to allocate
from
VMK_REMOTE_PAGE_FAULT_FAILURE
195887393 0xbad0121 ENOMEM Remote page fault failure
VMK_VSI_DATA_LENGTH_MISMATCH
195887394 0xbad0122 EIO VSI data length mismatch
VMK_MAPPING_FAILED
195887395 0xbad0123 EFAULT Mapping operation failed
VMK_ATS_MISCOMPARE 195887396 0xbad0124 EINVAL Atomic test and set of disk block
returned false for equality
VMK_NO_BUFFERSPACE
195887397 0xbad0125 ENOBUFS No buffer space available
VMK_FT_NOT_RUNNING 195887398 0xbad0126 EINVAL FT vm is not enabled
VMK_LICENSE_MISMATCH
195887399 0xbad0127 EINVAL Incompatible licenses detected
VMK_ELF_UNKNOWN_RELOCATIONS
195887400 0xbad0128 EINVAL ELF file contains invalid relocation
types
VMK_MESSAGE_TOO_LONG
195887401 0xbad0129 EMSGSIZE Message too long
VMK_INVALID_NAMESPACE
195887402 0xbad012a ENOENT
Invalid or missing
namespace
VMK_SHUTTING_DOWN
195887403 0xbad012b EINVAL Operation not allowed because the
VMKernel is shutting down
VMK_SKIPPED_FREE
195887404 0xbad012c EINVAL Skipped freeing of resource with no
reference
VMK_VMFS_ABORTED
195887405 0xbad012d ECANCELED IO was aborted by VMFS via a virt-reset
on the device
VMK_NO_WRITE_ON_TARDISKS
195887406 0xbad012e EPERM Write not allowed on tardisks
VMK_SVM_IO_RETRY
195887407 0xbad012f EBUSY Re-issue IO at a later time
VMK_MODULE_NO_LICENSE
195887408 0xbad0130 ENOENT
Module does not provide a
license tag
VMK_MODULE_UNKNOWN_LICENSE
195887409 0xbad0131 ENOENT Module provides an unknown license
tag
VMK_PERM_DEV_LOSS
195887410 0xbad0132 EIO Device is permanently
unavailable
VMK_SE_IO_RETRY
195887411 0xbad0133 EBUSY Reissue IO at a later time for SE
disks
VMK_BAD_ADDR_SPACE
195887412 0xbad0134 EINVAL Address space type is not supported
for operation
VMK_DMA_MAPPING_FAILED
195887413 0xbad0135 EINVAL DMA mapping could not be
completed
VMK_RESERVATION_GT_LIMIT
195887414 0xbad0136 EINVAL Memory pool reservation is greater
than limit
VMK_MODULE_NONAMESPACE
195887415 0xbad0137 ENOENT Module tried to export a symbol but
didn’t provide a name space
VMK_FS_OBJECT_UNLINKED
195887416 0xbad0138 EINVAL File system object is unlinked
VMK_HBR_WIRE_INSTANCE_ABORTED
195887417 0xbad0139 ECANCELED Replication instance was aborted
VMK_HBR_WIRE_NEED_FULL_SYNC
195887418 0xbad013a EINVAL
Replicated disk needs full
synchronization
VMK_HBR_WIRE_DISK_SET_MISMATCH
195887419 0xbad013b EINVAL The set of disks on the replication
server doesn’t match
VMK_HBR_WIRE_REQUEST_CHECKSUM_MISMATCH
195887420 0xbad013c EINVAL The checksum for the replication
request was invalid
VMK_HBR_WIRE_RESPONSE_CHECKSUM_MISMATCH
195887421 0xbad013d EINVAL The checksum for the replication
response was invalid
VMK_HBR_WIRE_GROUP_REMOVED 195887422 0xbad013e ENOENT The replication group was removed on
the server side
VMK_HBR_WIRE_GROUP_SESSION_REVOKED
195887423 0xbad013f EEXIST A newer client for this group is
connected to the replication server
VMK_HBR_WIRE_PROTOCOL_CORRUPTED
195887424 0xbad0140 EINVAL Corrupt response from replication
server
VMK_PORTSET_HANDLE_NOT_MUTABLE
195887425 0xbad0141 EINVAL Portset handle is not mutable
VMK_SUSPEND_IO
195887426 0xbad0142 EINVAL Suspend the IO in question
VMK_NO_WORKING_PATHS
195887427 0xbad0143 EINVAL No working paths to select
VMK_EPROTOTYPE 195887428 0xbad0144 EPROTOTYPE Invalid protocol for connection
VMK_MODULE_CONSUMED_RESOURCE_COUNT_NOT_ZERO 195887429 0xbad0145 EBUSY Consumed resource count of module is
not zero
VMK_HBR_SERVER_DOES_NOT_SUPPORT_REQUEST
195887430 0xbad0146 EOPNOTSUPP vSphere Replication Server does not
support request
VMK_STALE_FILEHANDLE
195887431 0xbad0147 ESTALE Stale file handle
VMK_VVOL_UNBOUND
195887432 0xbad0148 ENODEV Virtual volume is not bound
VMK_DEVICE_NOT_READY_FAIL_OPEN
195887433 0xbad0149 EPERM Device open failed with
no-retry
VMK_NOT_THIS_DEVICE
195887434 0xbad014a EINVAL Not for this device
VMK_IGNORE
195887435 0xbad014b EINVAL Ignore
VMK_OBJECT_DESTROYED
195887436 0xbad014c EINVAL Object is being or has been
destroyed
VMK_VVOL_PE_NOT_READY
195887437 0xbad014d EAGAIN Protocol Endpoint not ready for I/O
to given secondary level ID
VMK_SCSI_PI_GUARD_ERROR
195887438 0xbad014e EIO T10 PI GUARD tag check failed
VMK_SCSI_PI_REF_ERROR
195887439 0xbad014f EIO T10 PI REF tag check failed
VMK_RES_META_STALE
195887440 0xbad0150 EAGAIN Cached resource metadata is
stale
VMK_NOT_PINNED
195887441 0xbad0151 ENOENT Page is not pinned
VMK_BAD_SWAP_SCOPE
195887442 0xbad0152 EINVAL Incorrect swap scope
VMK_CONSUMED_GT_ZERO
195887443 0xbad0153
EINVAL Consumed memory is
more than zero
VMK_LOCK_HELD_BY_ZOMBIE_TXN
195887444 0xbad0154 EBUSY Lock held by a transaction in
progress
VMK_HBR_WIRE_FILE_IDENTICAL
195887445 0xbad0155 EEXIST The file being sent already exists
and is identical
VMK_VOL_ALREADY_MOUNTED
195887446 0xbad0156 EBUSY The volume is already mounted
VMK_NO_VOLUMES
195887447 0xbad0157 ENOMEM Out of volumes
VMK_SB_NOT_FOUND
195887448 0xbad0158 ENOENT Super block not found
VMK_NO_PMEM
195887449 0xbad0159 ENOMEM Out of persistent memory
VMK_OTHER
195887450 0xbad015a EINPROGRESS Another operation is in progress
VMK_CANNOT_SHRINK
195887451 0xbad015b EPERM Shrinking is not allowed
VMK_NOT_A_BASE 195887452 0xbad015c EPERM Not a base disk
VMK_HAS_SNAPSHOTS
195887453 0xbad015d ECHILD Disk has snapshots
VMK_LAST_ERR
195887454 0xbad015e EINVAL Invalid error code
VMK_GENERIC_LINUX_ERROR
732758016 0x2bad0000 EIO Generic service console
error
To use vmkerrcode to list details for an individual error code, run the command:
vmkerrcode error_code
Where error_code is either the hexadecimal (hex) or decimal value for the error code of interest. The utility lists the name, decimal, hexadecimal, text error output, and description for that VMK error code.
I had an interesting issue to resolve today, one that I haven’t seen before and one that took a bit of digging to resolve. The problem related to migrating some Exchange mailbox servers from a legacy ESXi 4.1 host onto new ESXi 5.1 host.
This should have been a simple vMotion operation, but the task failed repeatedly at approximately 65% complete. I tried using both high and standard priority migrations, but it failed every time, simply reporting “The VM failed to resume on the destination during early power on”
First thing I did was check the host log files (vmkwarning and vmkernel), as well as the virtual machine log file (vmware.log) located in the virtual machine folder on the datastore;
## START ## Nov 6 13:32:30 src-host vmkernel: 843:05:02:40.662 cpu6:64249)WARNING: Migrate: 296: 1415280590445360 S: Failed: Failed to resume VM (0xbad0044) @0x418023e4b250 Nov 6 13:32:30 src-host vmkernel: 843:05:02:40.664 cpu3:64248)WARNING: VMotionSend: 3857: 1415280590445360 S: failed to send final set of pages to the remote host <xx.xx.xx.xx>: Failure. Nov 6 13:32:30 src-host vmkernel: 843:05:02:40.689 cpu12:48347)WARNING: Migrate: 4328: 1415280590445360 S: Migration considered a failure by the VMX. It is most likely a timeout, but check the VMX log for the true error. Nov 6 14:51:31 src-host vmkernel: 843:06:21:41.945 cpu6:64267)WARNING: Migrate: 296: 1415285312829818 S: Failed: Failed to resume VM (0xbad0044) @0x418023e4b250 Nov 6 14:51:31 src-host vmkernel: 843:06:21:41.953 cpu19:64266)WARNING: VMotionSend: 3857: 1415285312829818 S: failed to send final set of pages to the remote host <xx.xx.xx.xx>: Failure. Nov 6 14:51:31 src-host vmkernel: 843:06:21:41.970 cpu12:48347)WARNING: Migrate: 4328: 1415285312829818 S: Migration considered a failure by the VMX. It is most likely a timeout, but check the VMX log for the true error. ## END ##
## START ## Nov 6 13:35:23 dst-host vmkernel: 501:21:49:25.404 cpu1:63073)WARNING: MemSched: 12625: Non-overhead memory reservation for vmx user-world (worldID=63073) is greater than desired minimum amount of 57344 KB (min=57344 KB, reservedOverhead=0 KB, totalReserved=68812 KB) Nov 6 13:36:05 dst-host vmkernel: 501:21:50:07.143 cpu6:63073)WARNING: MemSched: vm 63073: 5199: Cannot reduce reservation by 2021 pages (total reservation: 27428 pages, consumed reservation: 27428 pages) Nov 6 13:36:28 dst-host vmkernel: 501:21:50:29.836 cpu5:63091)WARNING: MemSched: 12625: Non-overhead memory reservation for vmx user-world (worldID=63091) is greater than desired minimum amount of 57344 KB (min=57344 KB, reservedOverhead=0 KB, totalReserved=68552 KB) Nov 6 13:36:39 dst-host vmkernel: 501:21:50:41.256 cpu6:63091)WARNING: MemSched: vm 63091: 5199: Cannot reduce reservation by 1913 pages (total reservation: 24038 pages, consumed reservation: 24038 pages) Nov 6 13:37:10 dst-host vmkernel: 501:21:51:12.241 cpu5:63106)WARNING: MemSched: 12625: Non-overhead memory reservation for vmx user-world (worldID=63106) is greater than desired minimum amount of 57344 KB (min=57344 KB, reservedOverhead=0 KB, totalReserved=68724 KB) Nov 6 13:37:50 dst-host vmkernel: 501:21:51:51.758 cpu11:63106)WARNING: MemSched: vm 63106: 5199: Cannot reduce reservation by 2021 pages (total reservation: 27327 pages, consumed reservation: 27327 pages) Nov 6 13:38:16 dst-host vmkernel: 501:21:52:18.119 cpu6:63124)WARNING: MemSched: 12625: Non-overhead memory reservation for vmx user-world (worldID=63124) is greater than desired minimum amount of 57344 KB (min=57344 KB, reservedOverhead=0 KB, totalReserved=69464 KB) Nov 6 13:40:23 dst-host vmkernel: 501:21:54:25.336 cpu2:63124)WARNING: MemSched: vm 63124: 5199: Cannot reduce reservation by 2019 pages (total reservation: 38944 pages, consumed reservation: 38944 pages) Nov 6 14:48:34 dst-host vmkernel: 501:23:02:35.673 cpu1:63154)WARNING: MemSched: 12625: Non-overhead memory reservation for vmx user-world (worldID=63154) is greater than desired minimum amount of 57344 KB (min=57344 KB, reservedOverhead=0 KB, totalReserved=69540 KB) Nov 6 14:52:04 dst-host vmkernel: 501:23:06:05.978 cpu15:63154)WARNING: Migrate: 4328: 1415285312829818 D: Migration considered a failure by the VMX. It is most likely a timeout, but check the VMX log for the true error. Nov 6 14:52:04 dst-host vmkernel: 501:23:06:05.978 cpu15:63154)WARNING: Migrate: 296: 1415285312829818 D: Failed: Migration determined a failure by the VMX (0xbad0092) @0x41801fb9acb9 Nov 6 14:52:04 dst-host vmkernel: 501:23:06:05.978 cpu15:63154)WARNING: VMotionUtil: 3548: 1415285312829818 D: timed out waiting 0 ms to transmit data. Nov 6 14:52:04 dst-host vmkernel: 501:23:06:05.978 cpu15:63154)WARNING: VMotionSend: 624: 1415285312829818 D: (9-0x410300002058) failed to receive 72/72 bytes from the remote host <xx.xx.xx.xx>: Timeout ## END ##
So reading through the host log files it looks like there was a problem reserving enough memory resources on the destination host and the operation timed out. This sounds relatively plausible, but the exact same results were observed trying to migrate the VM onto an empty host.
So next step was to review the guest VM’s log file;
## START ## Nov 06 14:52:04.416: vmx| DISKLIB-CTK : Could not open tracking file. File open returned IO error 4. Nov 06 14:52:04.416: vmx| DISKLIB-CTK : Could not open change tracking file "/vmfs/volumes/4f1edf9e-b9c5beba-cd04-0025b30202ac/GUEST-VM/GUEST-VM_3-ctk.vmdk": Could not open/create change tracking file. Nov 06 14:52:04.417: vmx| DISKLIB-LIB : Could not open change tracker /vmfs/volumes/4f1edf9e-b9c5beba-cd04-0025b30202ac/GUEST-VM/GUEST-VM_3-ctk.vmdk: Could not open/create change tracking file. Nov 06 14:52:04.421: vmx| DISKLIB-VMFS : "/vmfs/volumes/4f1edf9e-b9c5beba-cd04-0025b30202ac/GUEST-VM/GUEST-VM_3-rdm.vmdk" : closed. Nov 06 14:52:04.421: vmx| DISKLIB-LIB : Failed to open '/vmfs/volumes/4f1edf9e-b9c5beba-cd04-0025b30202ac/GUEST-VM/GUEST-VM_3.vmdk' with flags 0xa Could not open/create change tracking file (2108). Nov 06 14:52:04.421: vmx| DISK: Cannot open disk "/vmfs/volumes/4f1edf9e-b9c5beba-cd04-0025b30202ac/GUEST-VM/GUEST-VM_3.vmdk": Could not open/create change tracking file (2108). Nov 06 14:52:04.422: vmx| Msg_Post: Error Nov 06 14:52:04.422: vmx| [msg.disk.noBackEnd] Cannot open the disk '/vmfs/volumes/4f1edf9e-b9c5beba-cd04-0025b30202ac/GUEST-VM/GUEST-VM_3.vmdk' or one of the snapshot disks it depends on. Nov 06 14:52:04.423: vmx| [msg.disk.configureDiskError] Reason: Could not open/create change tracking file.---------------------------------------- Nov 06 14:52:04.437: vmx| Module DiskEarly power on failed. Nov 06 14:52:04.439: vmx| VMX_PowerOn: ModuleTable_PowerOn = 0 Nov 06 14:52:04.440: vmx| MigrateSetStateFinished: type=2 new state=11 Nov 06 14:52:04.440: vmx| MigrateSetState: Transitioning from state 10 to 11. Nov 06 14:52:04.441: vmx| Migrate_SetFailure: The VM failed to resume on the destination during early power on. Nov 06 14:52:04.441: vmx| Msg_Post: Error Nov 06 14:52:04.442: vmx| [msg.migrate.resume.fail] The VM failed to resume on the destination during early power on. ## END ##
Interestingly here, we now start getting some hints that perhaps a file lock is occurring and we also see the same error message that was observed in the vSphere client. The VM failed to resume on the destination during early power on.
I decided to have a look at the contents of the virtual machine folder, and found a number of suspicious looking “-ctk.vmdk” files, mostly time stamped from more than two years ago.
## START ## [root@host GUEST-VM]# ls -lh total 73G -rw------- 1 root root 4.3M Feb 28 2012 GUEST-VM_10-ctk.vmdk -rw------- 1 root root 1.1T Feb 27 2012 GUEST-VM_10-rdm.vmdk -rw------- 1 root root 582 Feb 28 2012 GUEST-VM_10.vmdk -rw------- 1 root root 7.7M Nov 6 14:51 GUEST-VM_11-ctk.vmdk -rw------- 1 root root 983G Feb 5 2014 GUEST-VM_11-rdm.vmdk -rw------- 1 root root 559 Feb 5 2014 GUEST-VM_11.vmdk -rw------- 1 root root 65K Jun 22 2013 GUEST-VM_1-ctk.vmdk -rw------- 1 root root 1.0G Nov 3 17:33 GUEST-VM_1-flat.vmdk -rw------- 1 root root 586 Feb 27 2012 GUEST-VM_1.vmdk -rw------- 1 root root 65K Jun 22 2013 GUEST-VM_2-ctk.vmdk -rw------- 1 root root 1.0G Nov 3 17:33 GUEST-VM_2-flat.vmdk -rw------- 1 root root 586 Feb 27 2012 GUEST-VM_2.vmdk -rw------- 1 root root 4.3M Feb 28 2012 GUEST-VM_3-ctk.vmdk -rw------- 1 root root 1.1T Feb 27 2012 GUEST-VM_3-rdm.vmdk -rw------- 1 root root 580 Feb 28 2012 GUEST-VM_3.vmdk -rw------- 1 root root 4.3M Feb 28 2012 GUEST-VM_4-ctk.vmdk -rw------- 1 root root 1.1T Feb 27 2012 GUEST-VM_4-rdm.vmdk -rw------- 1 root root 580 Feb 28 2012 GUEST-VM_4.vmdk -rw------- 1 root root 4.3M Feb 28 2012 GUEST-VM_5-ctk.vmdk -rw------- 1 root root 1.1T Feb 27 2012 GUEST-VM_5-rdm.vmdk -rw------- 1 root root 580 Feb 28 2012 GUEST-VM_5.vmdk -rw------- 1 root root 4.3M Feb 28 2012 GUEST-VM_6-ctk.vmdk -rw------- 1 root root 1.1T Feb 27 2012 GUEST-VM_6-rdm.vmdk -rw------- 1 root root 580 Feb 28 2012 GUEST-VM_6.vmdk -rw------- 1 root root 4.3M Feb 28 2012 GUEST-VM_7-ctk.vmdk -rw------- 1 root root 1.1T Feb 27 2012 GUEST-VM_7-rdm.vmdk -rw------- 1 root root 580 Feb 28 2012 GUEST-VM_7.vmdk -rw------- 1 root root 4.3M Feb 28 2012 GUEST-VM_8-ctk.vmdk -rw------- 1 root root 1.1T Feb 27 2012 GUEST-VM_8-rdm.vmdk -rw------- 1 root root 580 Feb 28 2012 GUEST-VM_8.vmdk -rw-r--r-- 1 root root 48 Nov 6 14:48 GUEST-VM-98045e41.hlog -rw------- 1 root root 4.3M Feb 28 2012 GUEST-VM_9-ctk.vmdk -rw------- 1 root root 1.1T Feb 27 2012 GUEST-VM_9-rdm.vmdk -rw------- 1 root root 580 Feb 28 2012 GUEST-VM_9.vmdk -rw------- 1 root root 4.4M Feb 28 2012 GUEST-VM-ctk.vmdk -rw------- 1 root root 70G Nov 6 15:11 GUEST-VM-flat.vmdk -rw------- 1 root root 8.5K Nov 3 17:36 GUEST-VM.nvram -rw------- 1 root root 585 Feb 27 2012 GUEST-VM.vmdk -rw-r--r-- 1 root root 44 Feb 28 2012 GUEST-VM.vmsd -rwxr-xr-x 1 root root 5.8K Feb 19 2014 GUEST-VM.vmx -rw-r--r-- 1 root root 266 Feb 5 2014 GUEST-VM.vmxf drwxr-xr-x 1 root root 420 Feb 19 2014 phd -rw-r--r-- 1 root root 57K Nov 19 2012 vmware-43.log -rw-r--r-- 1 root root 57K Jun 22 2013 vmware-44.log -rw-r--r-- 1 root root 57K Jun 22 2013 vmware-45.log -rw-r--r-- 1 root root 1.0M Feb 19 2014 vmware-46.log -rw-r--r-- 1 root root 1020K Nov 6 14:51 vmware-47.log -rw-r--r-- 1 root root 57K Nov 6 13:33 vmware-48.log -rw-r--r-- 1 root root 57K Nov 6 14:52 vmware.log ## END ##
You can consolidate this view with a simple grep of just the “-ctk.vmdk” files;
## START ## [root@host GUEST-VM]# ls -al | grep ctk -rw------- 1 root root 4436480 Feb 28 2012 GUEST-VM_10-ctk.vmdk -rw------- 1 root root 8053248 Nov 6 14:51 GUEST-VM_11-ctk.vmdk -rw------- 1 root root 66048 Jun 22 2013 GUEST-VM_1-ctk.vmdk -rw------- 1 root root 66048 Jun 22 2013 GUEST-VM_2-ctk.vmdk -rw------- 1 root root 4436480 Feb 28 2012 GUEST-VM_3-ctk.vmdk -rw------- 1 root root 4436480 Feb 28 2012 GUEST-VM_4-ctk.vmdk -rw------- 1 root root 4436480 Feb 28 2012 GUEST-VM_5-ctk.vmdk -rw------- 1 root root 4436480 Feb 28 2012 GUEST-VM_6-ctk.vmdk -rw------- 1 root root 4436480 Feb 28 2012 GUEST-VM_7-ctk.vmdk -rw------- 1 root root 4436480 Feb 28 2012 GUEST-VM_8-ctk.vmdk -rw------- 1 root root 4436480 Feb 28 2012 GUEST-VM_9-ctk.vmdk -rw------- 1 root root 4588032 Feb 28 2012 GUEST-VM-ctk.vmdk ## END ##
So for ESX/ESXi 3.x/4.x and ESXi 5.0, the lock status of these “-ctk.vmdk” files can be obtained using the vmkfstools command. The process and syntax is explained in detail in KB1003397, titled “Unable to perform operations on a virtual machine with a locked disk.”
## START ## vmkfstools -D /vmfs/volumes/LUN/VM/disk-flat.vmdk ## END ##
You see output similar to this below, and are specifically interested in the mode value returned;
## START ## Lock [type 10c00001 offset 54009856 v 11, hb offset 3198976 gen 9, mode 0, owner 4655cd8b-3c4a19f2-17bc-00145e808070 mtime 114] Addr <4, 116, 4>, gen 5, links 1, type reg, flags 0, uid 0, gid 0, mode 600 len 5368709120, nb 0 tbz 0, cow 0, zla 3, bs 1048576 ## END ##
The mode indicates the type of lock that is on the file, as follows;
## START ## mode 0 = no lock mode 1 = is an exclusive lock (vmx file of a powered on VM, the currently used disk (flat or delta), *vswp, etc.) mode 2 = is a read-only lock (e.g. on the ..-flat.vmdk of a running VM with snapshots) mode 3 = is a multi-writer lock (e.g. used for MSCS clusters disks or FT VMs). ## END ##
In my case, all “-ctk.vmdk” files reported an exclusive mode 1 lock;
## START ## [root@host GUEST-VM]# vmkfstools -D GUEST-VM-ctk.vmdk Lock [type 10c00001 offset 62181376 v 7105, hb offset 3211264 gen 87017, mode 1, owner 5003ce7b-b04b7d3f-f4e5-b499babda354 mtime 14822239] Addr <4, 89, 194>, gen 7012, links 1, type reg, flags 0, uid 0, gid 0, mode 600 len 4588032, nb 1 tbz 0, cow 0, zla 1, bs 8388608 [root@host GUEST-VM]# vmkfstools -D GUEST-VM_1-ctk.vmdk Lock [type 10c00001 offset 62185472 v 7106, hb offset 3211264 gen 87017, mode 1, owner 5003ce7b-b04b7d3f-f4e5-b499babda354 mtime 14822246] Addr <4, 89, 196>, gen 7013, links 1, type reg, flags 0, uid 0, gid 0, mode 600 len 66048, nb 1 tbz 0, cow 0, zla 1, bs 8388608 [root@host GUEST-VM]# vmkfstools -D GUEST-VM_2-ctk.vmdk Lock [type 10c00001 offset 62187520 v 7107, hb offset 3211264 gen 87017, mode 1, owner 5003ce7b-b04b7d3f-f4e5-b499babda354 mtime 14822253] Addr <4, 89, 197>, gen 7014, links 1, type reg, flags 0, uid 0, gid 0, mode 600 len 66048, nb 1 tbz 0, cow 0, zla 1, bs 8388608 [root@host GUEST-VM]# vmkfstools -D GUEST-VM_3-ctk.vmdk Lock [type 10c00001 offset 62189568 v 7052, hb offset 3211264 gen 87017, mode 1, owner 5003ce7b-b04b7d3f-f4e5-b499babda354 mtime 11201757] Addr <4, 89, 198>, gen 7015, links 1, type reg, flags 0, uid 0, gid 0, mode 600 len 4436480, nb 1 tbz 0, cow 0, zla 1, bs 8388608 [root@host GUEST-VM]# vmkfstools -D GUEST-VM_4-ctk.vmdk Lock [type 10c00001 offset 62191616 v 7053, hb offset 3211264 gen 87017, mode 1, owner 5003ce7b-b04b7d3f-f4e5-b499babda354 mtime 11201767] Addr <4, 89, 199>, gen 7016, links 1, type reg, flags 0, uid 0, gid 0, mode 600 len 4436480, nb 1 tbz 0, cow 0, zla 1, bs 8388608 [root@host GUEST-VM]# vmkfstools -D GUEST-VM_5-ctk.vmdk Lock [type 10c00001 offset 61786112 v 7054, hb offset 3211264 gen 87017, mode 1, owner 5003ce7b-b04b7d3f-f4e5-b499babda354 mtime 11201777] Addr <4, 89, 1>, gen 7017, links 1, type reg, flags 0, uid 0, gid 0, mode 600 len 4436480, nb 1 tbz 0, cow 0, zla 1, bs 8388608 [root@host GUEST-VM]# vmkfstools -D GUEST-VM_6-ctk.vmdk Lock [type 10c00001 offset 61792256 v 7055, hb offset 3211264 gen 87017, mode 1, owner 5003ce7b-b04b7d3f-f4e5-b499babda354 mtime 11201787] Addr <4, 89, 4>, gen 7018, links 1, type reg, flags 0, uid 0, gid 0, mode 600 len 4436480, nb 1 tbz 0, cow 0, zla 1, bs 8388608 [root@host GUEST-VM]# vmkfstools -D GUEST-VM_7-ctk.vmdk Lock [type 10c00001 offset 61794304 v 7056, hb offset 3211264 gen 87017, mode 1, owner 5003ce7b-b04b7d3f-f4e5-b499babda354 mtime 11201797] Addr <4, 89, 5>, gen 7019, links 1, type reg, flags 0, uid 0, gid 0, mode 600 len 4436480, nb 1 tbz 0, cow 0, zla 1, bs 8388608 [root@host GUEST-VM]# vmkfstools -D GUEST-VM_8-ctk.vmdk Lock [type 10c00001 offset 61796352 v 7057, hb offset 3211264 gen 87017, mode 1, owner 5003ce7b-b04b7d3f-f4e5-b499babda354 mtime 11201807] Addr <4, 89, 6>, gen 7020, links 1, type reg, flags 0, uid 0, gid 0, mode 600 len 4436480, nb 1 tbz 0, cow 0, zla 1, bs 8388608 [root@host GUEST-VM]# vmkfstools -D GUEST-VM_9-ctk.vmdk Lock [type 10c00001 offset 61798400 v 7058, hb offset 3211264 gen 87017, mode 1, owner 5003ce7b-b04b7d3f-f4e5-b499babda354 mtime 11201817] Addr <4, 89, 7>, gen 7021, links 1, type reg, flags 0, uid 0, gid 0, mode 600 len 4436480, nb 1 tbz 0, cow 0, zla 1, bs 8388608 [root@host GUEST-VM]# vmkfstools -D GUEST-VM_10-ctk.vmdk Lock [type 10c00001 offset 61800448 v 7059, hb offset 3211264 gen 87017, mode 1, owner 5003ce7b-b04b7d3f-f4e5-b499babda354 mtime 11201827] Addr <4, 89, 8>, gen 7022, links 1, type reg, flags 0, uid 0, gid 0, mode 600 len 4436480, nb 1 tbz 0, cow 0, zla 1, bs 8388608 [root@host GUEST-VM]# vmkfstools -D GUEST-VM_11-ctk.vmdk Lock [type 10c00001 offset 12601344 v 46751, hb offset 3211264 gen 87017, mode 1, owner 5003ce7b-b04b7d3f-f4e5-b499babda354 mtime 14822300] Addr <4, 10, 9>, gen 46740, links 1, type reg, flags 0, uid 0, gid 0, mode 600 len 8053248, nb 1 tbz 0, cow 0, zla 1, bs 8388608 ## END ##
So at this stage I created a “tmp” directory in the virtual machine folder and moved all the “-ctk.vmdk” files here. Since this was a live, powered on VM, I felt more comfortable doing this with a GUI than using the shell and used WinSCP to transfer the files.
I then confirmed there were no longer any “-ctk.vmdk” files in the virtual machine folder, and that they were all in the newly created “tmp” folder;
## START ## [root@host GUEST-VM]# ls -al | grep ctk [root@host GUEST-VM]# [root@host GUEST-VM]# cd /vmfs/volumes/datastore/GUEST-VM/tmp/ [root@host tmp]# ls -lh total 96M -rw------- 1 root root 4.3M Feb 28 2012 GUEST-VM_10-ctk.vmdk -rw------- 1 root root 7.7M Nov 6 15:32 GUEST-VM_11-ctk.vmdk -rw------- 1 root root 65K Jun 22 2013 GUEST-VM_1-ctk.vmdk -rw------- 1 root root 65K Jun 22 2013 GUEST-VM_2-ctk.vmdk -rw------- 1 root root 4.3M Feb 28 2012 GUEST-VM_3-ctk.vmdk -rw------- 1 root root 4.3M Feb 28 2012 GUEST-VM_4-ctk.vmdk -rw------- 1 root root 4.3M Feb 28 2012 GUEST-VM_5-ctk.vmdk -rw------- 1 root root 4.3M Feb 28 2012 GUEST-VM_6-ctk.vmdk -rw------- 1 root root 4.3M Feb 28 2012 GUEST-VM_7-ctk.vmdk -rw------- 1 root root 4.3M Feb 28 2012 GUEST-VM_8-ctk.vmdk -rw------- 1 root root 4.3M Feb 28 2012 GUEST-VM_9-ctk.vmdk -rw------- 1 root root 4.4M Feb 28 2012 GUEST-VM-ctk.vmdk [root@host tmp]# ## END ##
Now the vMotion operation completes successfully;
Here are the accompanying entries from the hosts vmkernel logs, which look much healthier than the original logs;
## START ## Nov 6 15:53:18 src-host vmkernel: 843:07:23:28.414 cpu12:48347)Migrate: vm 48348: 3046: Setting VMOTION info: Source ts = 1415289229136448, src ip = <xx.xx.xx.xx> dest ip = <xx.xx.xx.xx> Dest wid = 4359 using SHARED swap Nov 6 15:53:18 src-host vmkernel: 843:07:23:28.415 cpu12:48347)Tcpip_Vmk: 1013: Affinitizing xx.xx.xx.xx to world 64297, Success Nov 6 15:53:18 src-host vmkernel: 843:07:23:28.415 cpu12:48347)VMotion: 2366: 1415289229136448 S: Set ip address 'xx.xx.xx.xx' worldlet affinity to send World ID 64297 Nov 6 15:53:18 src-host vmkernel: 843:07:23:28.415 cpu5:4327)MigrateNet: vm 4327: 1378: Accepted connection from <xx.xx.xx.xx> Nov 6 15:53:18 src-host vmkernel: 843:07:23:28.415 cpu5:4327)MigrateNet: vm 4327: 1422: dataSocket 0x4100a6063410 receive buffer size is 563272 Nov 6 15:53:18 src-host vmkernel: 843:07:23:28.497 cpu23:64298)VMotionDiskOp: 769: 1415289229136448 S: DiskOps handshake successful. Nov 6 15:55:59 src-host vmkernel: 843:07:26:10.006 cpu20:48348)VMotion: 3714: 1415289229136448 S: Another pre-copy iteration needed with 640966 pages left to send (prev2 4194304, prev 4194304, network bandwidth ~91.894 MB/s) Nov 6 15:56:29 src-host vmkernel: 843:07:26:39.611 cpu21:48348)VMotion: 3714: 1415289229136448 S: Another pre-copy iteration needed with 240035 pages left to send (prev2 4194304, prev 640966, network bandwidth ~91.010 MB/s) Nov 6 15:56:42 src-host vmkernel: 843:07:26:53.116 cpu22:48348)VMotion: 3666: 1415289229136448 S: Stopping pre-copy: not enough forward progress (Pages left to send: prev2 640966, prev 240035, cur 185166, network bandwidth ~87.701 MB/s) Nov 6 15:56:42 src-host vmkernel: 843:07:26:53.116 cpu22:48348)VMotion: 3696: 1415289229136448 S: Remaining pages can be sent in 8.445 seconds, which is less than the maximum switchover time of 100 seconds, so proceeding with suspend. Nov 6 15:56:54 src-host vmkernel: 843:07:27:04.521 cpu9:64297)VMotionSend: 3866: 1415289229136448 S: Sent all modified pages to destination (network bandwidth ~81.079 MB/s) ## END ##
## START ## Nov 6 15:57:20 dst-host vmkernel: 0:01:16:26.628 cpu19:4359)VMotion: 4635: 1415289229136448 D: Page-in made enough progress during checkpoint load. Resuming immediately. Nov 6 15:57:20 dst-host vmkernel: 0:01:16:26.664 cpu19:4359)VmMemMigrate: vm 4359: 4786: Regular swap file bitmap checks out. Nov 6 15:57:20 dst-host vmkernel: 0:01:16:26.667 cpu19:4359)VMotion: 4489: 1415289229136448 D: Resume handshake successful Nov 6 15:57:20 dst-host vmkernel: 0:01:16:26.667 cpu13:4374)Swap: vm 4359: 9066: Starting prefault for the migration swap file Nov 6 15:57:20 dst-host vmkernel: 0:01:16:26.736 cpu21:4374)Swap: vm 4359: 9205: Finish swapping in migration swap file. (faulted 0 pages, pshared 0 pages). Success. Nov 6 15:57:20 dst-host vmkernel: 0:01:16:26.824 cpu18:4359)Net: 1421: connected GUEST-VM eth0 to Network xx.xx.xx, portID 0x2000004 Nov 6 15:57:20 dst-host vmkernel: 0:01:16:27.003 cpu21:4359)NetPort: 982: enabled port 0x2000004 with mac 00:00:00:00:00:00 Nov 6 15:57:20 dst-host vmkernel: 0:01:16:27.003 cpu21:4359)Net: 1421: connected GUEST-VM eth0 to Network xx.xx.xx, portID 0x2000005 Nov 6 15:57:20 dst-host vmkernel: 0:01:16:27.003 cpu21:4359)NetPort: 982: enabled port 0x2000005 with mac 00:00:00:00:00:00 Nov 6 15:57:26 dst-host vmkernel: 0:01:16:33.040 cpu16:4359)VmMemMigrate: vm 4359: 1946: pgNum (0x3fce57) changed type to 1 while remotely faulting it in. Nov 6 15:57:26 dst-host vmkernel: 0:01:16:33.061 cpu16:4359)VmMemMigrate: vm 4359: 1946: pgNum (0x3fd43e) changed type to 1 while remotely faulting it in. Nov 6 15:57:26 dst-host vmkernel: 0:01:16:33.096 cpu16:4359)VmMemMigrate: vm 4359: 1946: pgNum (0x3fe6b3) changed type to 1 while remotely faulting it in. Nov 6 15:57:27 dst-host vmkernel: 0:01:16:33.166 cpu1:4367)VMotionRecv: 1984: 1415289229136448 D: DONE paging in Nov 6 15:57:27 dst-host vmkernel: 0:01:16:33.166 cpu1:4367)VMotionRecv: 1992: 1415289229136448 D: Estimated network bandwidth 81.633 MB/s during page-in ## END ##
I’m not sure where these “-ctk.vmdk” files came from, but suspect it may have originated from a legacy backup process from before my time. At least for now the issue is resolved and we know what to look for the next time this happens.
Credits;
Thanks to Jakob Fabritius Nørregaard for posting this blog article which helped identify and resolve this issue.
81,915 total views, 13 views today
An independent IT contractor with a strong focus on VMware virtualisation and infrastructure operations. I am inspired by technology, not afraid to question the status quo and balance my professional commitments with entertaining my three awesome kids (Ashton, Oliver and Lara).
View all posts by Jon Munday