Internal connection fatal error

I’ve recently started receiving this error: «Internal connection fatal error» from my application. The error occurs randomly. When it happens my application is unusable for the next few minutes.

After careful analysis of the error I’ve concluded that this error happens in only one method within my application. This method fires a series of simple ordinary SQL queries but it does involve multithreading however all threads should be disposed before this block of code. The error happens always on a specific SQL query. For test I’ve eliminated this query which resulted in error happening on next in line query.

This is stack strace:

Internal connection fatal error.

-------------- Stack trace ---------------
   at Microsoft.Practices.EnterpriseLibrary.ExceptionHandling.ExceptionPolicyEntry.IntentionalRethrow(Exception chainException, Exception originalException)
   at Microsoft.Practices.EnterpriseLibrary.ExceptionHandling.ExceptionPolicyEntry.RethrowRecommended(Exception chainException, Exception originalException)
   at Microsoft.Practices.EnterpriseLibrary.ExceptionHandling.ExceptionPolicyEntry.Handle(Exception exceptionToHandle)
   at Microsoft.Practices.EnterpriseLibrary.ExceptionHandling.ExceptionPolicyImpl.HandleException(Exception exceptionToHandle)
   at Microsoft.Practices.EnterpriseLibrary.ExceptionHandling.ExceptionPolicy.HandleException(Exception exceptionToHandle, String policyName, ExceptionPolicyFactory policyFactory)
   at Microsoft.Practices.EnterpriseLibrary.ExceptionHandling.ExceptionPolicy.HandleException(Exception exceptionToHandle, String policyName)
   at Base.Sql.ExecuteScalar()

Application is built on .NET 3.5 and it obviously uses Enterprise Library Data Access.
Application is run on Win2003 Terminal Server and it’s using Sql Server 2005 database which is on different server.

The method which causes this error is not triggered from GUI but from command line if that is of any significence.

If someone would suggest where to go from here I would be very grateful.

I’ve recently started receiving this error: «Internal connection fatal error» from my application. The error occurs randomly. When it happens my application is unusable for the next few minutes.

After careful analysis of the error I’ve concluded that this error happens in only one method within my application. This method fires a series of simple ordinary SQL queries but it does involve multithreading however all threads should be disposed before this block of code. The error happens always on a specific SQL query. For test I’ve eliminated this query which resulted in error happening on next in line query.

This is stack strace:

Internal connection fatal error.

-------------- Stack trace ---------------
   at Microsoft.Practices.EnterpriseLibrary.ExceptionHandling.ExceptionPolicyEntry.IntentionalRethrow(Exception chainException, Exception originalException)
   at Microsoft.Practices.EnterpriseLibrary.ExceptionHandling.ExceptionPolicyEntry.RethrowRecommended(Exception chainException, Exception originalException)
   at Microsoft.Practices.EnterpriseLibrary.ExceptionHandling.ExceptionPolicyEntry.Handle(Exception exceptionToHandle)
   at Microsoft.Practices.EnterpriseLibrary.ExceptionHandling.ExceptionPolicyImpl.HandleException(Exception exceptionToHandle)
   at Microsoft.Practices.EnterpriseLibrary.ExceptionHandling.ExceptionPolicy.HandleException(Exception exceptionToHandle, String policyName, ExceptionPolicyFactory policyFactory)
   at Microsoft.Practices.EnterpriseLibrary.ExceptionHandling.ExceptionPolicy.HandleException(Exception exceptionToHandle, String policyName)
   at Base.Sql.ExecuteScalar()

Application is built on .NET 3.5 and it obviously uses Enterprise Library Data Access.
Application is run on Win2003 Terminal Server and it’s using Sql Server 2005 database which is on different server.

The method which causes this error is not triggered from GUI but from command line if that is of any significence.

If someone would suggest where to go from here I would be very grateful.

System.InvalidOperationException: ‘Internal connection fatal error.’ #1116

I’m looking for some help troubleshooting an issue that I’m having in one of my Development machines.

The following code is throwing a System.InvalidOperationException: ‘Internal connection fatal error.’ when opening the connection.

Here is the stach trace

On the same machine, using the legacy library ‘System.Data.SqlClient’, the following code works fine

I have tried a few things like:
Repair .NET 5
Repair SQL Server 2019 Developer
Repair VS2019

Does anyone have suggestions that could help to understand the source of the issue?

The text was updated successfully, but these errors were encountered:

You could try running something like WireShark to capture the difference between the two cases at the TCP level, because there’s probably not enough information in the stacktrace to help.

Hi @figueiredorui, what version of Microsoft.Data.SqlClient are you using? Is this a .NET Core or .NET Framework application? You can try enabling event tracing and posting the event log here.

Hi @johnnypham
Using Microsoft.Data.SqlClient 3.0.0, .NET Core 5

1, ‘Server=localhost;Database=Test02;Trusted_Connection=True;’ : Unable to load custom AppContextSwitchOverrides . Default value of T type returns. Entry point. Exit point. Neither SqlClientAuthenticationProviders nor SqlAuthenticationProviders configuration section found. TdsParserStateObjectFactory.CreateTdsParserStateObject | Info | AppContext switch ‘Switch.Microsoft.Data.SqlClient.UseManagedNetworkingOnWindows’ not enabled, native networking implementation will be used. TdsParser.Connect | SEC | Connection Object Id 4, Authentication Mode: SqlPassword TdsParser.Connect | SEC | SSPI or Active Directory Authentication Library loaded for SQL Server based integrated authentication Sending prelogin handshake ClientConnectionID 3f6323bc-78bd-418e-aae6-5204682d11ac, ActivityID 099fff25-893b-4144-921b-1c2062a11fc1:2 Consuming prelogin handshake 4, Breaking connection. Potential multi-threaded misuse of connection, unexpected TDS token found 1 ‘System.InvalidOperationException: Internal connection fatal error.’ 4

Similar issue here using 3.0 with .NET 5, version 2.1.3 works fine.

Could you also try out the 3 preview versions of v3.0.0 to get the first version to contain this issue?

@cheenamalhotra preview 1 seems to be working. 2, 3 and stable are not working.

@figueiredorui @r-work can you provide as much details as you can about your client/server environment (version, platform, etc) so we can get a local repro to investigate this?

I’m setting up a new development environment and installing the various tools/frameworks to find out which one makes my project fail running. So far so good .

I’m experiencing very similar with .NET Core 3.1 and Microsoft.Data.SqlClient 3.0.0:

I assume its relevant that this issue started for me when I was forced to apply the latest Windows Update (KB5003637). I’m working on Win 10 Pro with VS2019 (16.9.4).

. any thoughts on ways to investigate and resolve this?

.NET Core 3.1 and Microsoft.Data.SqlClient 3.0.0:
Windows Update (KB5003637). Win 10 Pro with VS2019 (16.9.4).

I wasn’t able to reproduce it with this same configuration. Can you confirm if the issue is present starting with 3.0.0-preview2?

Thank you for your reply. I’ve just enabled prerelease in NuGet, and installed 3.0.0-preview2.21106.5 which is exhibiting the same behaviour.

Is there any other information I can share so that your system can reproduce the same error? One of my colleagues updated his system to the same Window Update Version and is now also getting the same error while previously everything worked fine, so we have managed to reproduced this at least twice.

Can you try 3.0.0-preview1?

Unfortunately exactly the same error message.

I assume its relevant that this issue started for me when I was forced to apply the latest Windows Update (KB5003637). I’m working on Win 10 Pro with VS2019 (16.9.4).

Your issue seems different that what is being discussed here from earlier customer. In your case it seems to be related to some sort of configuration issue or an environment related problem. Could you open a new ticket with all related information:

Источник

“Internal connection fatal error” from TdsParser when calling native compiled stored procedure in SQL Server 2019 #365

Describe the bug

I am having huge problems with the current SQL Server 2019 (15.0.2070.41) and native compiled stored procedures when calling them from the Microsoft.Data.SqlClient (1.1.0 and lower).

It is working when either using SQL Server 2017 or System.Data.SqlClient , or just not using a native compiled procedure.

Apparently the TdsParser is receiving data it doesn’t expect from the SQL Server connection and is throwing an exception during the parsing.

To reproduce

Expected behavior

No exception and the reader should provide the result of the query.

Further technical details

Microsoft.Data.SqlClient version: 1.1.0 and lower
.NET target: Core 3.1, Framework 4.7.2
SQL Server version: SQL Server 2019 (15.0.2070.41)
Operating system: Windows 10, Windows Server 2016 Standard

Additional context
I have the same problem with JDBC and it’s TDSParser (7.4.1.jre8), see here. I also posted the problem on Stackoverflow

The text was updated successfully, but these errors were encountered:

I’m able to reproduce issue with Microsoft.Data.SqlClient and am looking into it.
Thanks for reporting this with us.

The token which can’t be parsed seems to be the same as the one which JDBC has a problem with, and that also doesn’t get recogniced by Wireshark: The 0xA3 token.

Introduced in TDS 7.4, the DATACLASSIFICATION token SHOULD describe the data
classification of the query result set.
Token Stream Comments:
 The token value is 0xA3.
 This token only be sent by the server if the client sends a DATACLASSIFICATION FeatureExt in the
Login message and the server responds with a DATACLASSIFICATION FeatureExtAck. When this
token is used, the token is sent by the server in response to every SQLBatch request.

Yes you’re right, this token is received because both JDBC and Microsoft.Data.SqlClient drivers support Data Classification and we send DATACLASSIFICATION FeatureExt token during login. System.Data.SqlClient does not support this feature hence is not impacted.

The glitch is this token is not expected to be in the stream at a location where it’s currently being received for this use-case, as per feature design. We’re following up with product team internally to gather more details on the same. Will keep you posted!

Client sends a 0x09 (DATACLASSIFICATION) as FeatureExt during Login (link), server sends acknowledgement with 0x09 (DATACLASSIFICATION) as FeatureExtAck .

Wireshark may not translate TDS login packets from client correctly.

I’d like to confirm this was identified as a bug in the SQL Server 2019 and we have received confirmation on the fix made to address the same.

The upcoming SQL Server 2019 CU2 release will address this issue.
There will be no changes required in SqlClient and JDBC drivers.

I know this is not the right place to ask this kind of question, but I’m experiencing the same issue once I migrated from System.Data.SqlClient to Microsoft.Data.SqlClient while trying to execute a native compiled stored procedure on an SQL 2019 compatiblity Azure SQL Database through a linux-based dotNet Core 2.2 application on AKS, but I couldn’t find any other information on the internet.

The question is: Do you have an information when this fix (CU2) would be released for Azure SQL Database (PAAS)?

@Genmutant thanks for linking the release update, we’ll close the issue!

Hello @cheenamalhotra! How are you?
Do you know if this issue was also fixed in Azure SQL service?
Best,
Augusto.

The fix is currently not yet deployed in Azure instances.
Estimate of deployment is in April.

Hello,
@cheenamalhotra:
Can anybody confirm that this is roled out on every azure DB instance? I have a strange behaviour that the same code is running on nearly all DBs, but on one database i get this excecption.

Or can be there an ohter issue?

@cheenamalhotra Can you confirm if the fix has been deployed to Azure SQL Service, and if not when it is scheduled?

Do you have a repro app for Azure DB to reproduce errors you are facing?
As of the above code, «With Native Compilation» is not supported in Azure DB, so the above example is not applicable to Azure.

But the fix was deployed to Azure in April 2020 as per our information.

I’m also getting this error, I have an Azure SQL, and running a stored procedure through .net core 3.1 / EF Core (3.1.2)
Works locally but not in Azure.

The code is simple, something like this:
var result = context.Database .ExecuteSqlInterpolated($»EXEC [dbo].[ConvertUser] «); ;

Can you confirm you’re able to run the stored procedure from SSMS?
And if yes, can you attach your scripts/repro here to look into it? (Ideally definition of «ConvertUser» stored proc in your case)

Can you confirm you’re able to run the stored procedure from SSMS?
And if yes, can you attach your scripts/repro here to look into it? (Ideally definition of «ConvertUser» stored proc in your case)

@cheenamalhotra Sorry I didn’t see the reply.

Yes I just tested from SSMS and it ran fine using:

DECLARE @return_value int
EXEC @return_value = [dbo].[ConvertUser]
@id = 32261
SELECT «Return Value» = @return_value
GO

The code is very simple:
CREATE PROCEDURE[dbo].[ConvertUser]
@id bigint
AS
BEGIN
SET NOCOUNT ON;
UPDATE Users
SET Discriminator = «Participant»
WHERE > END;
GO

Hope this helps?
Chris

Still having issues on Azure SQL+ Data Classification + TdsParser as of 8/18/2020:

If i remove all the data classification tags on that database the problem goes away.

System.InvalidOperationException: Internal connection fatal error. at Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady) at Microsoft.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj) at Microsoft.Data.SqlClient.SqlBulkCopy.RunParser(BulkCopySimpleResultSet bulkCopyHandler) at Microsoft.Data.SqlClient.SqlBulkCopy.CreateAndExecuteInitialQueryAsync(BulkCopySimpleResultSet& result) at Microsoft.Data.SqlClient.SqlBulkCopy.WriteToServerInternalRestAsync(CancellationToken cts, TaskCompletionSource 1 source) at Microsoft.Data.SqlClient.SqlBulkCopy.WriteToServerInternalAsync(CancellationToken ctoken) at Microsoft.Data.SqlClient.SqlBulkCopy.WriteRowSourceToServerAsync(Int32 columnCount, CancellationToken ctoken) at Microsoft.Data.SqlClient.SqlBulkCopy.WriteToServer(IDataReader reader) at CallSite.Target(Closure , CallSite , Object , IDataReader ) at System.Dynamic.UpdateDelegates.UpdateAndExecuteVoid2[T0,T1](CallSite site, T0 arg0, T1 arg1) at .( , DbConnection , DbTransaction ) at .( ) at .( ) at .Execute(List 1 actions) at .(List 1 ) at Z.BulkOperations.BulkOperation.Execute() at Z.BulkOperations.BulkOperation.BulkInsert() at .BulkInsert[T](DbContext this, IEntityType entityType, IEnumerable 1 list, Action 1 options, Boolean forceSpecificTypeMapping) at .BulkInsert[T](DbContext this, IEnumerable 1 entities, Action 1 options) at .(DbContext this, StateManager , IReadOnlyList 1 , Action 1 ) at .(DbContext this, StateManager , IReadOnlyList 1 , Action 1 ) at .(DbContext this, Action 1 , DbContext ) at DbContextExtensions.BulkSaveChanges(DbContext this, Action 1 options) at DbContextExtensions..() at System.Threading.Tasks.Task.InnerInvoke() at System.Threading.Tasks.Task.<>c. b__274_0(Object obj) at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state) — End of stack trace from previous location where exception was thrown — at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state) at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread) — End of stack trace from previous location where exception was thrown — at DbContextExtensions.BulkSaveChangesAsync(DbContext this, Action 1 options, CancellationToken cancellationToken) at CafComplianceWebLib.Controllers.CensusController.CensusFileUpload(CensusFileUploadViewModel model) in D:UsersjakehSourceNBS2CafComplianceCafComplianceCafComplianceWebLibControllersCensusController.cs:line 224 at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker. g__Logged|12_1(ControllerActionInvoker invoker) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker. g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker. g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker. g__Awaited|24_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker. g__Awaited|19_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker. g__Logged|17_1(ResourceInvoker invoker) at Microsoft.AspNetCore.Routing.EndpointMiddleware. g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger) at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context) at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context) at Serilog.AspNetCore.RequestLoggingMiddleware.Invoke(HttpContext httpContext) at NSwag.AspNetCore.Middlewares.SwaggerUiIndexMiddleware.Invoke(HttpContext context) at NSwag.AspNetCore.Middlewares.RedirectToIndexMiddleware.Invoke(HttpContext context) at NSwag.AspNetCore.Middlewares.OpenApiDocumentMiddleware.Invoke(HttpContext context) at NSwag.AspNetCore.Middlewares.OpenApiDocumentMiddleware.Invoke(HttpContext context) at Joonasw.AspNetCore.SecurityHeaders.Csp.CspMiddleware.Invoke(HttpContext context) at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware. g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)

@jumpingjackson That is due to a bug in SQLClient when using Data Classification with Bulk Copy. I believe that the bug was fixed in SQLCLient V2.

@jumpingjackson That is due to a bug in SQLClient when using Data Classification with Bulk Copy. I believe that the bug was fixed in SQLCLient V2.

Thanks! that fixed it. Its not obvious that you need to install 2.0.0 because if you install EFCore stuff sqlClient doesnt get referenced directly in your project so you dont get prompted to upgrade like most packagews.

Источник

@cheenamalhotra the issue is not yet resolved we are still facing the same issue in System.Data.SqlClient
Can you please help with this Bug

at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
at System.Data.SqlClient.TdsParser.DrainData(TdsParserStateObject stateObj)
at System.Data.SqlClient.TdsParser.ThrowUnsupportedCollationEncountered(TdsParserStateObject stateObj)
at System.Data.SqlClient.TdsParser.GetCodePage(SqlCollation collation, TdsParserStateObject stateObj)
at System.Data.SqlClient.TdsParser.TryProcessEnvChange(Int32 tokenLength, TdsParserStateObject stateObj, SqlEnvChange& sqlEnvChange)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
at System.Data.SqlClient.SqlInternalConnectionTds.CompleteLogin(Boolean enlistOK)
at System.Data.SqlClient.SqlInternalConnectionTds.AttemptOneLogin(ServerInfo serverInfo, String newPassword, SecureString newSecurePassword, Boolean ignoreSniOpenTimeout, TimeoutTimer timeout, Boolean withFailover)
at System.Data.SqlClient.SqlInternalConnectionTds.LoginNoFailover(ServerInfo serverInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString connectionOptions, SqlCredential credential, TimeoutTimer timeout)
at System.Data.SqlClient.SqlInternalConnectionTds.OpenLoginEnlist(TimeoutTimer timeout, SqlConnectionString connectionOptions, SqlCredential credential, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance)
at System.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, SqlCredential credential, Object providerInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData, Boolean applyTransientFaultHandling, String accessToken)
at System.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection, DbConnectionOptions userOptions)
at System.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(DbConnectionPool pool, DbConnection owningObject, DbConnectionOptions options, DbConnectionPoolKey poolKey, DbConnectionOptions userOptions)
at System.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
at System.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection) at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource1 retry, DbConnectionOptions userOptions) at System.Data.ProviderBase.DbConnectionClosed.TryOpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource1 retry, DbConnectionOptions userOptions)