I’ve recently come across an issue whereby performing an upgrade from XenServer 5.6 to XenServer 6 has caused the metadata on an SR to become corrupt. The cause of the problem, believe it or not, is special characters, things like ‘-‘, which often go in server names.
First thing to check, as with anything storage is the SMlog, under /var/logs. You will probably see something which looks like the following:
Error code: SR_BACKEND_FAILURE_181
Error
parameters: , Error in Metadata volume operation for SR. opterr=VDI delete
operation failed for parameters:
/dev/VG_XenStorage-427796a3-6891-dcb7-681f-cc7c5463ea0a/MGT,
7b496cd3-9d9c-4cb4-b510-4613e65e7141. Error: not well-formed (invalid token):
line 1, column 99,
[10146] 2011-11-27 08:05:10.977985 Raising exception
[181, Error in Metadata volume operation for SR. [opterr=VDI delete operation
failed for parameters:
/dev/VG_XenStorage-511f729e-0f0e-a679-9b8e-4d7ed28f6537/MGT,
65f8e675-9416-4909-8cd1-b7d75af06762. Error: not well-formed (invalid token):
line 1, column 65]]
[10146] 2011-11-27 08:05:10.978109 lock: released
/var/lock/sm/511f729e-0f0e-a679-9b8e-4d7ed28f6537/sr [10146] 2011-11-27
08:05:10.979583 ***** sr_scan: EXCEPTION SR.SROSError, Error in Metadata volume
operation for SR. [opterr=VDI delete operation failed for parameters:
/dev/VG_XenStorage-511f729e-0f0e-a679-9b8e-4d7ed28f6537/MGT,
65f8e675-9416-4909-8cd1-b7d75af06762. Error: not well-formed (invalid token):
line 1, column 65]
If you have upgraded to XenServer 6, you require patch http://support.citrix.com/article/CTX135411
If you have upgraded to XenServer 6.0.2 you require patch http://support.citrix.com/article/CTX135623
This issue has not been reported on XenServer 6.1.
If like me, you already have this patch installed, yet still have the issue, the problem is a corruption of the MGT VDI on the affected SR, which holds the metadata. This VDI requires rebuilding, which sounds quite scary, but actually isn’t.
Citrix has a support article regarding this (http://support.citrix.com/article/CTX131660) however it doesn’t really detail the solution correctly, and is far too complex, and involves unplugging storage which really isn’t necessary!
To rebuild your MGT volume, which can be done on a live production system, during operating hours, follow the steps below.
Firstly run the following command to rename the existing MGT VDI on the affected SR
lvrename /dev/VG_Xenstorage- <SR-uuid>/MGT /dev/VG_Xenstorage- <SR-uuid>/MGT.bak
Once complete, browse to the SR in Xencenter and then issue a rescan, and this will create a brand new MGT metadata volume, which in effect will resolve the issue. The special characters can stay in your name labels, as long as you have the patch mentioned above!
«SR BACKEND FAILURE 181» error happens when XenServer does not properly store the incoming virtual machine name-label and description fields in management VDI (MGT).
Here at Ibmi Media, as part of our Server Management Services, we regularly help our Customers to perform related Server Monitoring queries.
What triggers the error «SR BACKEND FAILURE 181» ?
After installation or upgrade to XenServer 6.x, user experiences issues with any storage related operations.
The following entries appears in the log /var/log/SMlog file:
Error code: SR_BACKEND_FAILURE_181
Error parameters: , Error in Metadata volume operation for SR. opterr=VDI delete operation failed for parameters: /dev/VG_XenStorage-427796a3-6891-dcb7-681f-cc7c5463ea0a/MGT, 7b496cd3-9d9c-4cb4-b510-4613e65e7141. Error: not well-formed (invalid token): line 1, column 99,
[10146] 2011-11-27 08:05:10.977985 Raising exception [181, Error in Metadata volume operation for SR. [opterr=VDI delete operation failed for parameters: /dev/VG_XenStorage-511f729e-0f0e-a679-9b8e-4d7ed28f6537/MGT, 65f8e675-9416-4909-8cd1-b7d75af06762. Error: not well-formed (invalid token): line 1, column 65]]
[10146] 2011-11-27 08:05:10.978109 lock: released /var/lock/sm/511f729e-0f0e-a679-9b8e-4d7ed28f6537/sr [10146] 2011-11-27 08:05:10.979583 ***** sr_scan: EXCEPTION SR.SROSError, Error in Metadata volume operation for SR. [opterr=VDI delete operation failed for parameters: /dev/VG_XenStorage-511f729e-0f0e-a679-9b8e-4d7ed28f6537/MGT, 65f8e675-9416-4909-8cd1-b7d75af06762. Error: not well-formed (invalid token): line 1, column 65]
How to fix the error «SR BACKEND FAILURE 181» ?
First and foremost, you can remove all special characters from the virtual machine name-label and description fields.
If the above workaround is unsuccessful, rebuild the SR MGT volume after removing the special characters:
1.Firstly, run lvscan and search for the MGT volume for the respective Storage Repository (SR).
Example: ACTIVE ‘/dev/VG_XenStorage-c14bee2a-44da-656a-5b99-865818c0a700/MGT’ [4.00 MB] inherit
2. Then, run a test to verify that the lv can be renamed:
lvrename -t /dev/VG_Xenstorage- <SR-uuid>/MGT /dev/VG_Xenstorage- <SR-uuid>/MGT.bak
3. When the preceding test is successful, run lvrename without the “-t” option:
lvrename /dev/VG_Xenstorage- <SR-uuid>/MGT /dev/VG_Xenstorage-< SR-uuid>/MGT.bak
4. Then, restart the master host in the pool or unplug and re-plug the Physical Block Device (PBD) on the Master Host for the respective SR to build a new MGT volume.
Example:
Find the Master Host UUID > xe pool-list params=master
master ( RO) : 9079cff1-c457-4e51-b924-8a9e1ec24179
5. To find the master PBD,
Run > xe pbd-list sr-uuid= c14bee2a-44da-656a-5b99-865818c0a700 host-uuid=9079cff1-c457-4e51-b924-8a9e1ec24179 uuid ( RO) : 99c6d7ec-75ef-1071-d925-5ff6b4667ecc host-uuid ( RO): 9079cff1-c457-4e51-b924-8a9e1ec24179 sr-uuid ( RO): 20daa9a1-a9b2-ddc9-30a7-c4e15999a284 device-config (MRO): device: /dev/sdb currently-attached ( RO): true
i. Unplug the PBD for the master (using the PBD UUID from Step 5):
xe pbd-unplug uuid=99c6d7ec-75ef-1071-d925-5ff6b4667ecc
ii. Re-Plug the PBD to attach the host to the SR again:
xe pbd-plug uuid=99c6d7ec-75ef-1071-d925-5ff6b4667ecc
This process will rebuild the MGT volume with the updated information without the special characters in the virtual machine name or description that was previously causing the errors.
[Still stuck with fixing Linux System errors? We can help you. ]
xe sr-create type=lvm content-type=user device-config:device=/dev/sdb1 name-label=StorageGeral shared=false
After I create, I try to create a disk for a machine and give me this error.
SR_BACKEND_FAILURE_181
xe sr-list
uuid ( RO) : bd8037b8-a644-5bfe-79dc-65e83a05e439
name-label ( RW): StorageGeral
name-description ( RW):
host ( RO): xyz
type ( RO): lvm
content-type ( RO): user
xe pbd-list sr-uuid=bd8037b8-a644-5bfe-79dc-65e83a05e439
uuid ( RO) : 38a305bb-3e43-e171-8d4b-91f6d76c4ed9
host-uuid ( RO): 4560aebe-88c3-44c8-9e64-fd4610da3942
sr-uuid ( RO): bd8037b8-a644-5bfe-79dc-65e83a05e439
device-config (MRO): device: /dev/sdb1
currently-attached ( RO): false
xe pbd-plug uuid=38a305bb-3e43-e171-8d4b-91f6d76c4ed9
Error code: SR_BACKEND_FAILURE_181
Error parameters: , Error in Metadata volume operation for SR. [opterr=Error introducing Metadata Volume: Input/output error],
SRInUse
The SR device is currently in use.
16
VDIInUse
The VDI is currently in use.
24
LockErr
The lock or unlock request failed.
37
Unimplemented
The requested method is not supported or implemented.
38
SRNotEmpty
The Storage Repository is not empty.
39
ConfigLUNMissing
The request is missing the LUNid parameter.
87
ConfigSCSIid
The SCSIid parameter is missing or incorrect.
107
ISODconfMissingLocation
‘Location’ parameter must be specified in Device Configuration.
220
ISOMustHaveISOExtension
ISO name must have .iso extension.
221
ISOMountFailure
Could not mount the directory specified in Device Configuration.
222
ISOUnmountFailure
Could not unmount the directory specified in Device Configuration.
223
ISOSharenameFailure
Could not locate the ISO sharename on the target, or the access permissions might be incorrect.
224
ISOLocationStringError
Incorrect Location string format. String must be in the format SERVER:PATH for NFS targets, or \SERVERPATH for CIFS targets.
225
ISOLocalPath
Invalid local path
226
InvalidArg
Invalid argument
1
BadCharacter
A bad character was detected in the dconf string.
2
InvalidDev
No such device
19
InvalidSecret
No such secret
20
SRScan
The Storage Repository scan failed.
40
SRLog
The Storage Repository log operation failed.
41
SRExists
The Storage Repository already exists.
42
VDIExists
The VDI already exists.
43
SRNoSpace
There is insufficient space.
44
VDIUnavailable
The VDI is not available.
46
SRUnavailable
The Storage Repository is not available.
47
SRUnknownType
Unknown repository type
48
SRBadXML
Malformed XML string
49
LVMCreate
Logical Volume creation error
50
LVMDelete
Logical Volume deletion error
51
LVMMount
Logical Volume mount or activate error
52
LVMUnMount
Logical Volume unmounts or deactivate error
53
LVMWrite
Logical Volume write error
54
LVMPartCreate
Logical Volume partition creation error
55
LVMPartInUse
Logical Volume partition in use
56
LVMFilesystem
Logical Volume file system creation error
57
LVMMaster
Logical Volume request must come from master.
58
LVMResize
Logical Volume resize failed.
59
LVMSize
Logical Volume invalid size
60
FileSRCreate
File Storage Repository creation error
61
FileSRRmDir
File Storage Repository failed to remove directory.
62
FileSRDelete
File Storage Repository deletion error
63
VDIRemove
Failed to remove VDI.
64
VDILoad
Failed to load VDI.
65
VDIType
Invalid VDI type
66
ISCSIDevice
ISCSI device failed to appear.
67
ISCSILogin
ISCSI login failed, verify CHAP credentials.
68
ISCSILogout
ISCSI logout failed.
69
ISCSIInitiator
Failed to set ISCSI initiator.
70
ISCSIDaemon
Failed to start ISCSI daemon.
71
NFSVersion
Required NFS server version unsupported.
72
NFSMount
NFS mount error
73
NFSUnMount
NFS unmount error
74
NFSAttached
NFS mount point already attached.
75
NFSDelete
Failed to remove NFS mount point.
76
NFSTarget
Unable to detect an NFS service on this target.
108
LVMGroupCreate
Logical Volume group creation failed.
77
VDICreate
VDI Creation failed.
78
VDISize
VDI Invalid size
79
VDIDelete
Failed to mark VDI hidden.
80
VDIClone
Failed to clone VDI.
81
VDISnapshot
Failed to snapshot VDI.
82
ISCSIDiscovery
ISCSI discovery failed.
83
ISCSIIQN
ISCSI target and received IQNs differ.
84
ISCSIDetach
ISCSI detach failed.
85
ISCSIQueryDaemon
Failed to query the ISCSI daemon.
86
NFSCreate
NFS Storage Repository creation error
88
ConfigLUNIDMissing
The request is missing the LUNid parameter.
89
ConfigDeviceMissing
The request is missing the device parameter.
90
ConfigDeviceInvalid
The device is not a valid path.
91
VolNotFound
The volume cannot be found.
92
PVSfailed
PVS failed
93
ConfigLocationMissing
The request is missing the location parameter.
94
ConfigTargetMissing
The request is missing the target parameter.
95
ConfigTargetIQNMissing
The request is missing or has an incorrect target IQN parameter.
96
ConfigISCSIIQNMissing
Unable to retrieve the host configuration ISCSI IQN parameter.
97
ConfigLUNSerialMissing
The request is missing the LUN serial number.
98
LVMOneLUN
Only 1 LUN may be used with shared LVM.
99
LVMNoVolume
Cannot find volume
100
ConfigServerPathMissing
The request is missing the serverpath parameter.
101
ConfigServerMissing
The request is missing the server parameter.
102
ConfigServerPathBad
The serverpath argument is not valid.
103
LVMRefCount
Unable to open the refcount file
104
Rootdev
Root system device, cannot be used for virtual machine storage.
105
InvalidIQN
The IQN provided is an invalid format.
106
SnapshotChainTooLong
The snapshot chain is too long.
109
VDIResize
VDI resize failed.
110
APISession
Failed to initialize XMLRPC connection.
150
APILocalhost
Failed to query Local Control Domain.
151
APIPBDQuery
A failure occurred querying the PBD entries.
152
APIFailure
A failure occurred accessing an API object.
153
NAPPTarget
Netapp Target parameter is missing in Dconf string.
120
NAPPUsername
Netapp Username parameter is missing in Dconf string.
121
NAPPPassword
Netapp Password parameter is missing in Dconf string.
122
NAPPAggregate
Netapp Aggregate parameter is missing in Dconf string.
123
NAPPTargetFailed
Failed to connect to Netapp target.
124
NAPPAuthFailed
Authentication credentials incorrect
125
NAPPInsufficientPriv
Authentication credentials have insufficient access privileges.
126
NAPPFVolNum
Maximum number of flexvols reached on target. Unable to allocate the requested resource.
127
NAPPSnapLimit
Maximum number of Snapshots reached on target Volume. Unable to create the snapshot.
128
NAPPSnapNoMem
Insufficient space, unable to create the snapshot.
129
NAPPUnsupportedVersion
Netapp Target version unsupported
130
NAPPTargetIQN
Unable to retrieve target IQN
131
NAPPNoISCSIService
ISCSI service is not running on the Netapp target.
132
NAPPAsisLicense
Failed to enable A-SIS for the SR. Requires valid license on the filer.
133
NAPPAsisError
The filer does not support A-SIS on this aggregate. The license is valid however on some filers A-SIS is limited to smaller aggregates, for example, FAS3020 max supported aggregate is 1 TB. Refer to filer support documentation for details on the model. You must either disable A-SIS support, or re-configure the aggregate to the maximum supported size.
134
NAPPExclActivate
Failed to acquire an exclusive lock on the LUN.
135
DNSError
Incorrect DNS name, unable to resolve.
140
ISCSITarget
Unable to connect to ISCSI service on target
141
ISCSIPort
Incorrect value for ISCSI port must be a number between 1 and 65535.
142
BadRequest
Failed to parse the request.
143
VDIMissing
VDI could not be found.
144
EQLTarget
Equallogic Target parameter is missing in Dconf string.
160
EQLUsername
Equallogic Username parameter is missing in Dconf string.
161
EQLPassword
Equallogic Password parameter is missing in Dconf string.
162
EQLStoragePool
Equallogic StoragePool parameter is missing in Dconf string.
163
EQLConnectfail
Failed to connect to Equallogic Array; maximum SSH CLI sessions reached
164
EQLInvalidSnapReserve
Invalid snap-reserver-percentage value must be an integer indicating the amount of space as a percentage of the VDI size, to reserve for snapshots.
165
EQLInvalidSnapDepletionKey
Invalid snap-depletion value must be one of ‘delete-oldest’ or ‘volume-offline’.
166
EQLVolOutofSpace
Volume out of space, probably because to insufficient snapshot reserve allocation.
167
EQLSnapshotOfSnapshot
Cannot create Snapshot of a Snapshot VDI, operation unsupported.
168
EQLPermDenied
Failed to connect to Equallogic Array, Permission denied;username/password invalid
169
EQLUnsupportedVersion
Equallogic Target version unsupported.
170
EQLTargetPort
Unable to logon to Array. Check IP settings.
171
EQLInvalidStoragePool
Equallogic StoragePool parameter specified in Dconf string is Invalid.
172
EQLInvalidTargetIP
Equallogic Target parameter specified in Dconf string is Invalid, specify the correct Group IP address.
173
EQLInvalidSNMPResp
Invalid SNMP response received for a command line interface command.
174
EQLInvalidVolMetaData
Volume metadata stored in the ‘Description’ field is invalid. This field contains encoded data and is not user editable.
175
EQLInvalidEOFRecv
Invalid EOF response received for a CLI command.
176
LVMProvisionAttach
Volume Group out of space. The SR is over-provisioned, and out of space. Unable to grow the underlying volume to accommodate the virtual size of the disk
180
MetadataError
Error in Metadata volume operation for SR.
181
EIO
General IO error
200
EGAIN
Currently unavailable, try again
201
SMGeneral
General backend error
202
FistPoint
An active FIST point was reached that causes the process to exit abnormally.
203
CSLGConfigServerMissing
The CSLG server name or IP address is missing.
400
CSLGConfigSSIDMissing
The Storage System ID is missing.
401
CSLGConfigPoolIDMissing
The Storage Pool ID is missing.
402
CSLGProtocolCheck
The GSSI operation to the CSLG server failed.
410
CSLGLoadSR
The Storage Repository loading operation failed.
411
CSLGInvalidProtocol
An invalid storage protocol was specified
412
CSLGXMLParse
Unable to parse XML
413
CSLGProbe
Failed to probe Storage Repository.
414
CSLGSnapClone
Snapshot/Clone failed.
416
CSLGAssign
Storage assignment failed.
417
CSLGUnassign
Storage un-assignment failed.
418
CSLGAllocate
Storage allocation failed.
419
CSLGDeallocate
Storage deallocation failed.
420
CSLGHBAQuery
HBA Query failed.
421
CSLGISCSIInit
IQN/ISCSI initialization failed.
422
CSLGDeviceScan
SCSI device scan failed.
423
CSLGServer
Failed to connect to CSLG.
424
CSLGConfigSVIDMissing
The Storage Node ID is missing.
425
CSLGIntroduce
The VDI failed to be introduced to the database.
426
CSLGNotInstalled
The CSLG software does not seem to be installed.
427
CSLGPoolCreate
Failed to create multiple sub-pools from parent pool.
428
CSLGOldXML
Current XML definition is newer version.
429
MultipathdCommsFailure
Failed to communicate with the multipath daemon.
430
CSL_Integrated
Error in storage adapter communication
431
CSLGConfigAdapterMissing
The adapter id is missing or unknown
432
CSLGConfigUsernameMissing
The username is missing
433
CSLGConfigPasswordMissing
The password is missing
434
CSLGInvalidSSID
An invalid storage system ID was specified
435
CSLGSysInfo
Failed to collect storage system information
436
CSLGPoolInfo
Failed to collect storage pool information
437
CSLGPoolDelete
Failed to delete storage pool
438
CSLGLunInfo
Failed to collect storage volume information
439
CSLGLunList
Failed to list storage volume
440
CSLGResizeLun
Failed to resize storage volume
441
CSLGTargetPorts
Failed to list storage target ports
442
CSLGPoolList
Failed to list storage pool
443
TapdiskFailed
The tapdisk failed
444
TapdiskAlreadyRunning
The tapdisk is already running
445
CIFSExtendedCharsNotSupported
XenServer does not support extended characters in CIFS paths, usernames, passwords, and file names.
446
IllegalXMLChar
Illegal XML character.
447
Skip to content
Issues Resolved In This Hotfix
This hotfix resolves the following issues:
- When creating a Storage Repository (SR) or a Virtual Disk (VDI), predefined XML entities (for example &, <, >, “, ‘) entered into the Name and Description fields, are written directly into the SR metadata. Any subsequent actions such as creating an SR or a VDI will fail with an error message, as the XML parser fails to parse metadata if it contains predefined XML entities.
- If there is a failure or a change in the number of paths to storage, attempts to create or destroy a VDI on an LVM-based SR may fail and will display the following error message: Error in Metadata Volume Operation on SR. The VDI will be created, but will lack a Name or Description.
- If multipath is active, on some storage arrays an error may occur when reading the Metadata Volume during an SR attach and display the following error message: Error in Metadata Volume Operation on SR.
- Storage may take several minutes to become available when using Dell PowerVault MD32xxi Storage Arrays with LVMoiSCSI
- Xapi frequently hangs pool slaves due to exceeding the number of sessions on the Pool Master.
- An SR may fail to connect when the XenServer host is booted with one failed iSCSI path when using Hitachi or Compellent storage arrays.
- XenServer may not discover all of the available LUNS when using the wildcard IQN. This occurs when using Compellent storage arrays via iSCSI.
- Improvements to the XenCenter Update functionality; customers are correctly notified in XenCenter when new XenServer Tools are available.
In addition, this hotfix includes the following previously released hotfixes.
- CTX131745 – Hotfix XS60E003 – For XenServer 6.0
- CTX133961 – Hotfix XS60E016 – For XenServer 6.0
NOTE: Customers MUST install Hotfix XS60E001 – For XenServer 6.0 before attempting to apply this hotfix.
Installing the Hotfix
Customers should use either XenCenter or the XenServer Command Line Interface (CLI) to install this update. Once the update has installed, the server must be restarted for it to take effect. As with any software update, please back up your data before applying this hotfix. Citrix recommends updating all hosts within a pool sequentially. Upgrading of hosts should be scheduled to minimize the amount of time the pool runs in a “mixed state” where some hosts have been upgraded and some have not. Running a mixed pool of updated and non-updated hosts for general operation is not supported.
NOTE: The attachment to this article is a zip file. It contains both the hotfix update package, and the source code for any modified open source components. The source code is not necessary for hotfix installation: it is provided to fulfil licensing obligations.
Important: This hotfix updates the control domain (dom0) kernel version. The driver disks produced for Hotfix XS60E016 are compatible with this hotfix. For a list of the drivers and their updates, see CTX133960 – Driver Disks for XenServer Hotfix XS60E016.
Installing the update using XenCenter
- Download the update to a known location on a computer that has XenCenter installed.
- In XenCenter, on the Tools menu, select Install New Update. This displays the Install Update wizard.
- Click Nextto start the Wizard.
- Click Add…to upload a new update.
- Browse to the location where you downloaded the hotfix, select it, and then click Open.
- From the list of updates select XS60E017.update and then click Next.
- Select the hosts you wish to apply the hotfix to, and then click Next.
- Follow the recommendations to resolve any upgrade prechecks.
- Click Install to start the installation.
Note: The XenCenter controlled upgrade process reboots each host sequentially starting with the Pool Master, where possible VMs will be migrated to other running hosts to avoid VM downtime. When the Pool Master is being rebooted, XenCenter will be unable to monitor the pool.
You can download the fix here.