Encoding error xml

Please note! This page describes the nature of the error using a hypothetical example and not the erroneous data of the input test file. You should however be able to apply this information to your error case.

Error description:

XML Encoding error is given when validator finds something to be wrong with the encoding of the file. 

There are two aspects when in comes to an encoding of XML file:

• The encoding XML file is using
• The encoding reported in the XML declaration within the file

The actual encoding is selected when the file is saved by the system/program producing the file. The encoding within XML declaration is the attribute «encoding» in the first line of XML. Example below:

It is worth noting that capitalisation does not matter in the value, «UTF-8» is the same as «utf-8». Encoding UTF-8 is used in this example as it is mandated to be used in ISO 20022 payments.

Therefore, possible causes for getting this error are:

• Encoding given in XML declaration does not match with actual file encoding
• No XML declaration is found. Possible reasons for this include:
  • XML declaration is not given in the payment message or is not given in the first line
  • Input file is not a payment message or not in XML format

To save a file in another encoding external program has to be used, e.g. notepad.exe allows saving to UTF-8 and notepad++ allows more exhaustive encoding modification.

It is worth noting that revaldate feature in validator is not able to see the original encoding of the file and treats the file shown in the report to be in UTF-8 format. Therefore it may happen that this error «fixes itself» if the cause of error was a mismatch between file encoding and encoding reported in the file.

Short explanation about encoding here

XML encoding and DOM interface methods

This article describes XML encoding and DOM interface methods.

Original product version: В Extensible Markup Language
Original KB number: В 275883

Summary

One major advantage of Extensible Markup Language (XML) data is that it is an independent platform. However, correct encoding must be specified to ensure proper transfer of XML data between different platforms. The white paper How to Encode XML Data addresses general XML encoding issues in detail: How to Encode XML Data.

Under most scenarios, XML encoding errors originate from the different default encoding settings of the XML parser (MSXML) methods and interfaces. A clear understanding of these default settings will help in preventing the encoding errors.

XML Encodings

MSXML supports all encodings that are supported by Internet Explorer. Internet Explorer’s support depends on which language packs are installed on the computer; this information is stored under the registry key: HKEY_CLASSES_ROOTMIMEDatabaseCharset .

MSXML has native support for the following encodings:

The proper place to specify encoding for the data is the XML declaration. For example, if the data is encoded with ISO-8859-1 standard, you can specify this as follows:

Without this information, the default encoding is UTF-8 or UTF-16, depending on the presence of a UNICODE byte-order mark (BOM) at the beginning of the XML file. If the file starts with a UNICODE byte-order mark (0xFF 0xFE) or (0xFE 0xFF), the document is considered to be in UTF-16 encoding; otherwise, it is in UTF-8. The Save method of the IXMLDOMDocument interface maintains the original encoding of the document. The default for this method is UTF-8.

MSXML DOM Errors

Two common errors that are returned from the XML Document Object Model (DOM) interface methods are:

An Invalid character was found in text content.

Switch from current encoding to specified encoding not supported.

With the load method of the IXMLDOMDocument interface, these errors usually occur under the following conditions:

No encoding is specified, no byte-order mark is found at the beginning of the XML file, and the data contains special characters

The specified encoding does not match the actual encoding of the XML data. A good practice is to always specify the correct encoding inside the XML declaration, rather than accepting the default encoding.

With the MSXML parser versions 2.5, 2.5 SP1 and 2.6, the loadXML method of IXMLDOMDocument can only load UTF-16 or UCS-2 encoded data. Any attempt to load XML data that is encoded with another encoding format results in the following error:

Switch from current encoding to specified encoding not supported. With the release of MSXML 3.0 (Msxml3.dll), this restriction is removed, and the following code runs without error:

The xml property of the IXMLDOMDocument interface writes out the XML data as UTF-16 encoded but without the byte-order mark at the beginning. This may lead to encoding problems.

You may also receive these errors when you call the transformNode method of the IXMLDOMNode interface with an XSL or XSLT file in which the XML encoding information is specified as follows:

The transformNode method returns a BSTR that is UTF-16 encoded data by definition. A better way to retain the encoding is to call the transformNodeToObject method and store the results to a stream or to a new XML document and then save it.

Источник

Xml file encoding error

Error description:

XML Encoding error is given when validator finds something to be wrong with the encoding of the file.

There are two aspects when in comes to an encoding of XML file:

• The encoding XML file is using
• The encoding reported in the XML declaration within the file

The actual encoding is selected when the file is saved by the system/program producing the file. The encoding within XML declaration is the attribute «encoding» in the first line of XML. Example below:

It is worth noting that capitalisation does not matter in the value, «UTF-8» is the same as «utf-8». Encoding UTF-8 is used in this example as it is mandated to be used in ISO 20022 payments.

Therefore, possible causes for getting this error are:

• Encoding given in XML declaration does not match with actual file encoding
• No XML declaration is found. Possible reasons for this include:
  • XML declaration is not given in the payment message or is not given in the first line
  • Input file is not a payment message or not in XML format

To save a file in another encoding external program has to be used, e.g. notepad.exe allows saving to UTF-8 and notepad++ allows more exhaustive encoding modification.

It is worth noting that revaldate feature in validator is not able to see the original encoding of the file and treats the file shown in the report to be in UTF-8 format. Therefore it may happen that this error «fixes itself» if the cause of error was a mismatch between file encoding and encoding reported in the file.

Источник

Xml file encoding error

Answered by:

Question

I need some help regarding this error:

System.Xml.XmlException: Invalid character in the given encoding.

I´m trying to load a Xml:

XmlDocument xDoc = new XmlDocument();
xDoc.Load(«http:// xml Url»);

. but the xml file have word with ã ç õ, and it´s not defined a encode:

Answers

If not specified, the default XML encoding is UTF-8 probably doen’s have the characters you are looking for. The ISO-8859-1 is Western European and may have what you want. Does your XML file begin with.

. if so, try changing it to «ISO-8859-1» and see if it loads.

Les Potter, Xalnix Corporation

Nops, It´s not defined .

I believe that is the problem, right?
Is there a way to force the use of ISO-8859-1 encode?

1. If no encoding is specified, xml-readers are supposed to result to UTF-8 in the first place.

2. Writing on top of your document: encoding=’bla bla’ doesn’t actually SET the encoding on disc. The program that created the xml-file is responsible for that. Imagine the ‘a’ character, in encoding X, on your HardDisk, there will be: 0000 0001. Where in encoding Y, it might be 000 0010 0000 0110. Now if the ‘a’ is part of an xmldocument, and the top of your document says: «encoding=Z», then you will have weird results. Those weird results are exactly what your problem at hand is.

The solution(s):
1. Where does the xml-file come from? Is it generated by another application you wrote? Can you change the encoding your application writes in? Is it a 3rd party (bought, . ) application? Did you search the help-file for the actual encoding?

2. Try re-writing the xml-file yourself. Read it completely, then re-write it in proper encoding. This worked for me:

public static void tryApply( string file)
<
string path = file;
UTF8Encoding enc = new UTF8Encoding ();
string [] result = File .ReadAllLines(path, Encoding .Default);
//^^
FileStream fs = new FileStream (path, FileMode .Create);
BinaryWriter br = new BinaryWriter (fs, Encoding .UTF8);
//^^
for ( int t = 0; t byte [] buffer = enc.GetBytes(result[t]);
br.Write(buffer);
>
br.Flush();
br.Close();
fs.Close();
>

The improbable we do, the impossible just takes a little longer. — Steven Parker

Источник

XML — проблемы с кодировкой

через XDTO создаю текст XML
потом пишу его в файл и проверяю на соответствие схеме

ЗаписьXML = Новый ЗаписьXML;

ЗаписьXML.УстановитьСтроку();
ЗаписьXML.ЗаписатьОбъявлениеXML();
ФабрикаXDTO.ЗаписатьXML(ЗаписьXML, КорневойРаздел);

это возвращает получившийся текст ХМЛ в реквизит документа типа Строка(0)

ИмяФайла = ПолучитьИмяВременногоФайла(«xml»);
ИмяФайлаСхемы = ПолучитьИмяВременногоФайла(«xsd»);

ФайлРеестра = Новый ТекстовыйДокумент;
ФайлРеестра.УстановитьТекст(Объект.ТекстВыгрузки);
ФайлРеестра.Записать(ИмяФайла,»windows-1251″);

ФайлСхемы = Новый ТекстовыйДокумент;
ФайлСхемы.УстановитьТекст(Объект.ТекстСхемыДокумента);
ФайлСхемы.Записать(ИмяФайлаСхемы,»windows-1251″);

Попытка
Схема=Новый COMОбъект(«MSXML2.XMLSchemaCache.6.0»);
Исключение
Сообщить(«Не удалось создать объект XMLSchemaCache (возможно, не установлен MSXML 6)»);
Возврат;
КонецПопытки;

Попытка
Схема.add(«»,ИмяФайлаСхемы);
Исключение
Сообщить(«Не удалось подключить схему: «+ИмяФайлаСхемы);
Схема=Неопределено;
Возврат;
КонецПопытки;

Попытка
ДОМ=Новый COMОбъект(«MSXML2.DOMDocument.6.0»);
Исключение
Сообщить(«Не удалось создать объект DOMDocument (возможно, не установлен MSXML 6)»);
Схема=Неопределено;
Возврат;
КонецПопытки;

ДОМ.schemas=Схема;
ДОМ.async=Ложь;
ДОМ.validateOnParse=Истина;
ДОМ.resolveExternals=Истина;
Если ВыводитьВсеОшибки Тогда
ДОМ.SetProperty(«MultipleErrorMessages»,Истина);
КонецЕсли;
ДОМ.load(ИмяФайла);

Если (ДОМ.parseError.errorCode<>0) Тогда
// ошибки при проверке правильности
Сообщить(«При проверке по схеме выявлены ошибки!»);
Если ВыводитьВсеОшибки Тогда
Для каждого parseError из ДОМ.parseError.AllErrors Цикл
Сообщить(parseError.reason);
Сообщить(parseError.srcText);
КонецЦикла;
Иначе
Сообщить(ДОМ.parseError);
КонецЕсли;
Иначе
Сообщить(«Файл успешно прошёл проверку по схеме!»);
РезПроверки=Истина;
КонецЕсли;

получаю ошибку на первом же атрибуте где встречается кириллица:
В текстовом комментарии обнаружен недопустимый знак.

Источник

Xml file encoding error

This forum has migrated to Microsoft Q&A. Visit Microsoft Q&A to post new questions.

Answered by:

Question

Based on forum posts, it seems that only SQL Server 2019 starts to support UTF-8. Then for early versions of SQL Server, can I insert UTF-8 XML into SQL Server.

I have a database called XMLDB, with a table called XMLTable, with a column called XMLCol, whose type is XML.

Can I use the following two methods to insert XML data with UTF-8 characters:

INSERT INTO XMLTable VALUES (‘

测试
Wheaton
1998
software
‘)

INSERT INTO XMLTable
SELECT * FROM OPENROWSET(BULK N’E:TempTest.xml’, SINGLE_BLOB) AS x;

where Test.xml is a XML file with some UTF-8 characters, as the contents in the method 1.

If yes, then need I always specify the encoding in the XML header explicitly, like this:

Answers

What you do is exactly what I explained. If this is not clear please re-read and I can try to clarify better as well but check it:

Your first Option (under the «— no prolog, non-English characters») works well since you use national character «N» and inside the content you do not use any XML declaration !

therefore, there is no contradiction

Your second option works well since you use national character «N» and inside the content you do use XML declaration encoding=»UTF-16″

Again, national character data types are encoded with «UTF-16» which mean there is no contradiction. This is the preferred option which I mentioned when we use direct INSERT or variable by the way! Using explicit XML declaration encoding=»UTF-16″ and using national character

I do not see exactly what you added to what I said above😃
These are two options which I mentioned

Note! In what you wrote before the samples there are some inaccurate. You must re-read what I wrote before since it is all there!

>> XML prolog is omitted. Accepts both English as well as non-English characters

NOT ALWAYS! If you use XML declaration but you do not use national character «N» and you do not use the new 2019 UTF-8 COLLATE, then your input is not UNICODE. Check the example above or bellow and remember (1) that SQL Server explicitly convert the data to UTF-16 when you insert it to XML data type but you cannot restore characters which were lost when the text was not UNICODE. (2) that this can be impacted by your current COLLATE and the characters you use in the input

>> XML prolog specifies encoding=»UTF-16″. Accepts both English as well as non-English character

If you use encoding=»UTF-16″ and you use national character, then it will work — this is exactly your second sample and this is the option I wrote as preferred.

But if you do not use national character then this will fail! This is the same in any version, and is not related only to non-English character .

The explanation is simple! You have contradiction between the input which is not UNICODE (assuming I am not using the 2019 UTF-8 COLLATE), and the XML DECLARATION which say that this is UTF-16

Check this regarding your next claim.

The magic word is contradiction between the XML DECLARATION and the INPUT DATA.

Ronen Ariely [Personal Site] [Blog] [Facebook] [Linkedin]

• Edited by pituach MVP Thursday, July 11, 2019 12:42 PM
• Marked as answer by tempc Saturday, July 20, 2019 11:31 AM

The upcoming SQL Server 2019 support for UTF-8 is referring to the regular columns with data types VARCHAR(), CHAR(), and the like. I am not sure about XML data type.

You should be able to insert your XML with non-English characters into a column of the XML data type starting with SQL Server 2005 and later.

I would guess that you have DB collation compatible with Chinese ?!

I am guessing that company name element value is in Chinese.

You would need to remove XML prolog with encoding=»UTF-8″ from the XML. It is because currently SQL Server uses UTF-16 internally for the XML data type. The only prolog with encoding=»UTF-16″ is accepted for the Chinese characters.

SQL to deal with the file, encoding utf-16:

Here is my 2 cent

>> it seems that only SQL Server 2019 starts to support UTF-8.

Not accurate. SQL Server 2019 is the first version which support storing text in UTF-8 encoding as a simple text type . This is done by storing text using simple data types like CHAR and VARCHAR and using one of the new UTF-8 COLLATE.

Before version 2019 there are several levels of supports in UTF-8, but if we want to use text data type we could not use simple types and we had to use special types like NCHAR/NVARCHAR and the text was stored in UTF-16 (from 2012 and UCS2 in older versions).

* We could always store the content as binary and not as text
* Export and import flat files which are saved as UTF-8 encoding using BCP utility and the option to the BULK INSERT Transact-SQL command — was added in SQL Server 2014 SP2. This means that the external file is in UTF-8 but while importing the data there is implicit convert to UTF-16 if we store the data in NCHAR, NVARCHAR, or XML data type.

>> for early versions of SQL Server, can I insert UTF-8 XML into SQL Server.

By the way, I usually answer YES for most «can we do» questions! The question should be How and not if we can.

Obviously we can, but before version 2014 SP2 we had to first EXPLICITLY CONVERT the file to UTF-16 and then import it, while from 2014 SP2 we can import the file in UTF-8 directly into SQL Server NCHAR/NVARCHAR/XML data types — in this case there will be IMPLICIT CONVERT to UTF-16

Notice again that the data in the database in this case is in UTF-16 even so the source is in UTF-8

Method 1 is really problematic since INSERT string value directly to a table is totally different from IMPORT DATA using a utility like bcp or using BULK INSERT. Moreover, you did not provide the DDL so we have no idea what is the column’s type.

If you INSERT the data into NCHAR/NVARCHAR then this is totally not relevant. In this case, you can write in the text encoding=»RonenAriely» and this will not change the encoding of the data. This is a simple text which has nothing to do with the encoding of the stored data. In this case the text is inserted as it is.

In both samples in this case, the data is stored in UTF-16 encoding (or UCS2 in older versions). This is true in sql server 2019 as well.

If you are using a data types like CHAR/VARCHAR with a COLLATE that does not support UTF-8 (which only exists in 2019) then it will be stored as extended ASCII encoded according to the COLLATE which you use.

The news in 2019 is that from SQL Server 2019 we have COLLATEs which support UTF-8. If you used one of these COLLATE which support UTF-8 then the data in CHAR/VARCHAR will be stored using UTF-8.

If you INSERT the data into XML data type then this is much more complex story since XML data type has more limitations and rules. In this case, the server actually parse the text and check the content of the text in any manipulation of the data. The server confirms that the value fits to Microsoft SQL Server’s rules regarding this data type, and change the data! The stored data is NOT the same as the INSERTED text.

For example if the content of the text is not valid XML ‘ ‘ then you will get an error which give you some information of the issue. In this sample you will get «unexpected end of input» since we have an opening node without a closer.

If you IMPORT data from a file then the XML declaration must use the same encoding as the file, or no properties for the encoding. If the file is saved using UTF-8 then we need to use encoding=»utf-8″ , and if the file is stored using UTF-16 then you need to use encoding=»utf-16″ .

When you use INSERT like in your Method 1, then you actually use CHAR/VARCHAR as input (since in your sample you did not used the national character «N» before the text). In this case the server will explicitly convert the data into UTF-16 and change the data according to the internal rules I will mention.

In this case if the text includes only ASCII characters then and you use encoding=»utf-8″ then the server will insert the text well

But if the text includes NON-ASCII characters then you will get error about illegal xml character:

If you will use encoding=»utf-8″ and national character «N» then you will get error «unable to switch the encoding», since the input text is now in UTF-16 and since you use XML data type the server parse the text and expect to get text in UTF-8.

Same issue we will get if we will do the opposite: Using UTF-16 for the input (by use the «N» before the text) and inside the content of the file you use encoding = «UTF-8».

In short, THE INPUT MUST FIT THE CONTENT IF YOU ARE USING XML DATA TYPE

The right way to INSERT the value directly into XML data type is to use national character «N» and inside the XML content to tell the server that you are using UTF-16, since the input is in UTF-16

>> If yes, then need I always specify the encoding in the XML header explicitly, like this:

I assume that you speak about XML Data type

The answer is No, but it is recommended.

If you IMPORT the data then the server can get the input encoding from your command. In any case the server will try to use implicit convert to UTF-16. In SQL Server you do not have to use the XML declaration at all (not only the encoding) and in fact the server remove this part when it store the data.

>> I am not sure about XML data type.

In general (not sql server specifically), when we use XML then we usually need to support UNICODE but we can use different encoding.

>> I would guess that you have DB collation compatible with Chinese ?!

This is not relevant for the INSERT of the data.

XML data type use UNICODE UTF-16 encoding for any collate you choose. The COLLATE has other impact like sorting the data, comparing values and so on. But in NCHAR, NVARCHAR, and XML data type the stored data is always in UTF-16 (or UCS2 in older versions)

You can confirm this by using the undocumented commands DBCC IND and DBCC PAGE

>> You would need to remove XML prolog with encoding=»UTF-8″ from the XML.

Check what I wrote above regarding different cases. If this is am XML file that you import then having the XML declaration is probably needed in the file and you do not need to remove it. Using direct INSERT you simply mad a mistake since your INPUT was not in UTF-8 and therefore the XML declaration which declare that this is UTF-8 was wrong and did not fit the real encoding of the data — check what I wrote above

>> SQL to deal with the file, encoding utf-16:

Actually as I explained SQL Server from 2014 SP2 can deal with files in UTF-8 as well.

• XML DATA Type is stored in UTF-16 encoding (this is not related to the server version, exactly as NVARCHAR is UTF-16)
• SQL Server 2014 SP2 and above can import files which are in UTF-8 encoding
• IMPORT from file and INSERT from value does not behave the same
• If we use XML data type then the input source must fit the content text encoding parameter in the XML declaration
• XML data type in SQL Server does not store the data «as it is» in the input text! The stored data will not include the «XML declaration» (the server remove this during the insert), the server might change the order of nodes, the server encode some of the characters and will not store them as they are in the text.

Источник