Gssapi operation failed with error an unsupported mechanism was requested

Hi All,

I am facing this error message when my api (dotnet core 3) tries to call another api endpoint.
I am packaging my apis as docker images on the on-prem k8s docker enterprise cluster.

My docker file is as follows:

`
#FROM microsoft/dotnet:2.2-aspnetcore-runtime-alpine AS runtime
FROM ddcdtr.bayer.cnb/microsoft/dotnet-core-aspnet:3.0 AS runtime

WORKDIR /app

Only Metadata

#EXPOSE 80
ENV PORT=8080
ENV ASPNETCORE_URLS=http://+:${PORT}

ENV HTTP_PROXY=${http_proxy}
ENV HTTPS_PROXY=${https_proxy}

EXPOSE $PORT

#ARG ASPNETCORE_ENVIRONMENT
#ENV ASPNETCORE_ENVIRONMENT=${ASPNETCORE_ENVIRONMENT}

https://rehansaeed.com/securing-asp-net-core-in-docker/

ENV COMPlus_EnableDiagnostics=0

#FROM microsoft/dotnet:2.2-sdk AS build
FROM ddcdtr.bayer.cnb/microsoft/dotnet-core-sdk:3.0 AS build

RUN apt-get update
#RUN apt-get -y install gss-ntlmssp
RUN apt-get update && apt-get install -y —no-install-recommends gss-ntlmssp

WORKDIR /src
#ARG NUGET_CONFIG
#RUN echo $NUGET_CONFIG > NuGet.config

COPY Bardo.Data/Bardo.Data.csproj Bardo.Data/
COPY GBoxApi/GBoxApi.csproj GBoxApi/
COPY GBoxApi.IntegrationTests/GBoxApi.IntegrationTests.csproj GBoxApi.IntegrationTests/
COPY GBoxData/GBoxData.csproj GBoxData/

RUN dotnet restore GBoxApi/GBoxApi.csproj

COPY . .
WORKDIR /src/GBoxApi

RUN dotnet build -c Release -o /app

FROM build AS publish

RUN dotnet publish -c Release -o /app

FROM runtime AS final

Create a group and user

#RUN addgroup -S -g 1000 appgroup
#&& adduser -S -u 1000 -G appgroup -s /bin/sh appuser

WORKDIR /app

RUN mkdir -p /local/

COPY —from=publish /app .

COPY —from=build /src/GBoxApi/GBoxApi.xml .
#COPY —from=build . .

#RUN chown appuser:appgroup /local
#RUN chown appuser:appgroup /app

USER root:root

Tell docker that all future commands should run as the appuser user

#USER appuser

ENTRYPOINT [«dotnet», «GBoxApi.dll»]

`

build and deployment is all fine, but at runtime, my application throws this error after (log entry 4) at client.SendAsync(request);

`
public async Task GetPermissionsAsync(string identity, string context)
{
string log = «/n Start: «;
try
{
Permission result = null;
var permissionUri = $»{ _config[«PermissionAPI:BaseUrl»] }?digitalIdentity={identity}&apiContext={context}»;
log += «1 «;
var request = new HttpRequestMessage(HttpMethod.Get, permissionUri);
log += «2 «;
request.Headers.Add(«Accept», «application/json»);
log += «3 «;

            var client = _clientFactory.CreateClient();
            log += "4 PermissionUri: " + permissionUri;

            var response = await client.SendAsync(request);

`

i have also tried other base images for (sdk, runtime) like dotnet-core-aspnet:3.0-alpine and dotnet-core-sdk:3.0-alpine, but still same problem.

Carry on with what we have been doing in the previous post regarding deploying containerized apps to AKS. This post addresses some of the issues and how we are going to solve it. My main goal is to allow the app to have access to the on-prem resources using Windows Authentication for Linux containers, just like we would normally do with our apps running on an intranet network. I briefly mentioned our approach to achieve this using Azure VNet. In case you haven’t seen that post, here is the link.

So now we have the app up and running in AKS but it still isn’t able to talk to the database hosted on the on-prem network.

Two issues I have encountered:

• My Navision instance was configured to use NTLM authentication protocol. This means I can only call its OData/SOAP endpoints with this protocol type when using Windows Authentication.
• Containers are unable to connect to the on-prem SQL Server.

Fixing the NTLM authentication issue in NAV

The app I was deploying is a .Net Core 3.1 console,  a Worker Service app to be more specific. I have SEQ logging setup in Azure so I can view the logs from there.

These lines of code allow me to send http requests to the OData endpoints using Windows Authentication. It’s a simple HttpClient class that was injected to the repository.

services.AddHttpClient<INavODataRepository, NavODataRepository>(http =>
{
    http.BaseAddress = new Uri(options.NavODataEndpoint);
    http.DefaultRequestHeaders.Add("OData-MaxVersion", "4.0");
    http.DefaultRequestHeaders.Add("OData-Version", "4.0");
    http.DefaultRequestHeaders.Accept.Add(
        new MediaTypeWithQualityHeaderValue("application/json"));
}).ConfigurePrimaryHttpMessageHandler(() => new HttpClientHandler()
{
    Credentials = new NetworkCredential("username", "password", "your_domain")
});

The issue is that out of the box, Linux containers don’t use NTLM be default but my Navision instance running on the on-prem server was configured to accept NTLM coming from any source. The app in AKS cluster keeps throwing this error message:

Error: GSSAPI operation failed with error - An unsupported mechanism was requested. NTLM authentication requires the GSSAPI plugin 'gss-ntlmssp'

This was a hint so I googled gss-ntlmssp and found out what I need to do. You have two ways to install this plugin:

1. Either include it when you build the docker image, or
2. You can install after the image has been deployed to AKS

The first approach is easier, you simply add these lines to the Dockerfile . Remember to put the them inside the runtime stage.  This link talks about multi-stage in Docker image build, and this one for the .Net core build.

RUN apt-get update
RUN apt-get -y install gss-ntlmssp

#See https://aka.ms/containerfastmode to understand how Visual Studio uses this Dockerfile to build your images for faster debugging.
FROM mcr.microsoft.com/dotnet/core/runtime:3.1-buster-slim AS base
LABEL maintainer="gpham@example.com"
# Workaround as per https://github.com/dotnet/corefx/issues/28961
RUN apt-get update
RUN apt-get -y install gss-ntlmssp
WORKDIR /app

#ENV NETCORE_ENVIRONMENT Development
FROM mcr.microsoft.com/dotnet/core/sdk:3.1-buster AS build
WORKDIR /src
COPY ["Diamond.ItemSync.WorkerService/Diamond.ItemSync.WorkerService.csproj", "Diamond.ItemSync.WorkerService/"]
COPY ["Diamond.ItemSync.DataAccess/Diamond.ItemSync.DataAccess.csproj", "Diamond.ItemSync.DataAccess/"]
COPY ["Diamond.ItemSync.Domain/Diamond.ItemSync.Domain.csproj", "Diamond.ItemSync.Domain/"]
RUN dotnet restore "Diamond.ItemSync.WorkerService/Diamond.ItemSync.WorkerService.csproj"
COPY . .
WORKDIR "/src/Diamond.ItemSync.WorkerService"
RUN dotnet build "Diamond.ItemSync.WorkerService.csproj" -c Release -o /app/build
FROM build AS publish
RUN dotnet publish "Diamond.ItemSync.WorkerService.csproj" -c Release -o /app/publish
FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "Diamond.ItemSync.WorkerService.dll"]

If somehow, you can’t add via the Dockerfile, you can login to the Linux container and install the plugin from there. I found using Visual Studio code is the simplest way to achieve this or you can just SSH to the container. Make sure you have Kubernetes extension installed on your VS Code.

With just a few steps, we can now have the app talking back to the on-prem Navision instance through its OData or Soap endpoints.

Fixing the issue: containers are unable to connect to the on-prem SQL Server

Normally, you could get a SQL Connection by having something similar to this:

public IDbConnection CreateOpenConnection()
{
    try
    {
        SqlConnection sqlConnection = new SqlConnection(
            "Data Source=SQL-INSTANCE;Initial Catalog=DB_NAME;User Id=readonly;Password=PASS_WORD;");
        sqlConnection.Open();
        return sqlConnection;
    }
    catch (System.Exception ex)
    {
        Log.Error("Failed to open SQL connection. Error {@Error}", ex);
        throw ex;
    }
}

This readonly user was configured to be authenticated using SQL Authentication (as we can’t pass user credentials through a connection for Windows auth). When I tried to execute this code inside my docker container, I got a time out exception. The issue is this line of code: sqlConnection.Open(); takes  too long to response. It just keeps hanging forevever.

Here is why.

• My AKS node was created as a Ubuntu-based VM and my AKS container deployment was set to use Ubuntu but the docker image was built for Debian.

If you pay attention to the Dockerfile:

FROM mcr.microsoft.com/dotnet/core/runtime:3.1-buster-slim AS base 
FROM mcr.microsoft.com/dotnet/core/sdk:3.1-buster AS build WORKDIR /src

Both the base and the build images are for Debian docker-image build. This will still work but an issue was raised specific to the SQL connection. To fix this, we just need to pull the 3.1-bionic base images from Microsoft. These are specific to building docker images for Ubuntu:

FROM mcr.microsoft.com/dotnet/core/runtime:3.1-bionic AS base
WORKDIR /app
ARG BUILD_ENV_ARG=development
ENV DOTNET_ENVIRONMENT=$BUILD_ENV_ARG
RUN echo "Installing gss-ntlmssp"
RUN apt-get update
RUN apt-get -y install gss-ntlmssp
RUN echo "Installing gss-ntlmssp complete"
FROM mcr.microsoft.com/dotnet/core/sdk:3.1-bionic AS build

So I was pulled onto a different project for a couple of weeks and I finally got a chance to get back to this. It took me a little while to figure out that DebugConsole was a class from the SDK, but I managed to get an output of the websocket (below):

[2020-Oct-29T21:31:45.563] OPENING WEBSOCKET CONNECTION 50280062
[2020-Oct-29T21:31:45.632] HANDLING REDIRECT:
{
  "jsonrpc": "2.0",
  "method": "OnAuthenticationInformation",
  "params": {
    "loginUri": "https://myurl.com/internal_windows_authentication/?targetId=5c4541ee-8622-4a66-ac6e-c1a38e77ef6e",
    "mustAuthenticate": true
  }
}
[2020-Oct-29T21:31:45.690] ### Failed attempt 1 ### 50280062
[2020-Oct-29T21:31:45.690] ### Failure message: Ticket retrieval failed.
[2020-Oct-29T21:31:45.708] ### Failure stack:    at Qlik.Sense.JsonRpc.WebSocket.SystemNetWebSocketSession.RedirectionPerformed(ConnectionSettings connectionSettings, WebSocket theSocket, Action`1 onMessage, CancellationToken token)
   at Qlik.Sense.JsonRpc.WebSocket.SystemNetWebSocketSession.Open(ConnectionSettings connectionSettings, Action`1 log, CancellationToken token)
   at Qlik.Sense.JsonRpc.WebSocketSessionContainer.OpenAsync(Action`1 onMessage, Action`1 onError, CancellationToken cancellationToken)
   at Qlik.Sense.JsonRpc.RpcConnection.WithTimeout[T](Func`1 func, Int32 timeout, CancellationToken token)
   at Qlik.Sense.JsonRpc.RpcConnection.RepeatAttempt(Int32 maxAttempts, Func`2 f, CancellationToken token)
[2020-Oct-29T21:31:46.578] HANDLING REDIRECT:
{
  "jsonrpc": "2.0",
  "method": "OnAuthenticationInformation",
  "params": {
    "loginUri": "https://myurl.com/internal_windows_authentication/?targetId=8a714e62-d147-4b3f-b57e-3b2154b5c483",
    "mustAuthenticate": true
  }
}
[2020-Oct-29T21:31:46.596] ### Failed attempt 2 ### 50280062
[2020-Oct-29T21:31:46.596] ### Failure message: Ticket retrieval failed.
[2020-Oct-29T21:31:46.597] ### Failure stack:    at Qlik.Sense.JsonRpc.WebSocket.SystemNetWebSocketSession.RedirectionPerformed(ConnectionSettings connectionSettings, WebSocket theSocket, Action`1 onMessage, CancellationToken token)
   at Qlik.Sense.JsonRpc.WebSocket.SystemNetWebSocketSession.Open(ConnectionSettings connectionSettings, Action`1 log, CancellationToken token)
   at Qlik.Sense.JsonRpc.WebSocketSessionContainer.OpenAsync(Action`1 onMessage, Action`1 onError, CancellationToken cancellationToken)
   at Qlik.Sense.JsonRpc.RpcConnection.WithTimeout[T](Func`1 func, Int32 timeout, CancellationToken token)
   at Qlik.Sense.JsonRpc.RpcConnection.RepeatAttempt(Int32 maxAttempts, Func`2 f, CancellationToken token)
[2020-Oct-29T21:31:46.607] HANDLING REDIRECT:
{
  "jsonrpc": "2.0",
  "method": "OnAuthenticationInformation",
  "params": {
    "loginUri": "https://myurl.com/internal_windows_authentication/?targetId=d91fcf91-8ca2-4f6e-9130-d073f758fce7",
    "mustAuthenticate": true
  }
}
[2020-Oct-29T21:31:46.622] ### Failed attempt 3 ### 50280062
[2020-Oct-29T21:31:46.622] ### Failure message: Ticket retrieval failed.
[2020-Oct-29T21:31:46.623] ### Failure stack:    at Qlik.Sense.JsonRpc.WebSocket.SystemNetWebSocketSession.RedirectionPerformed(ConnectionSettings connectionSettings, WebSocket theSocket, Action`1 onMessage, CancellationToken token)
   at Qlik.Sense.JsonRpc.WebSocket.SystemNetWebSocketSession.Open(ConnectionSettings connectionSettings, Action`1 log, CancellationToken token)
   at Qlik.Sense.JsonRpc.WebSocketSessionContainer.OpenAsync(Action`1 onMessage, Action`1 onError, CancellationToken cancellationToken)
   at Qlik.Sense.JsonRpc.RpcConnection.WithTimeout[T](Func`1 func, Int32 timeout, CancellationToken token)
   at Qlik.Sense.JsonRpc.RpcConnection.RepeatAttempt(Int32 maxAttempts, Func`2 f, CancellationToken token)
[2020-Oct-29T21:31:46.633] HANDLING REDIRECT:
{
  "jsonrpc": "2.0",
  "method": "OnAuthenticationInformation",
  "params": {
    "loginUri": "https://myurl.com/internal_windows_authentication/?targetId=c9cf46de-30b8-484b-9c0c-e73c6edc5de1",
    "mustAuthenticate": true
  }
}
[2020-Oct-29T21:31:46.650] ### Failed attempt 4 ### 50280062
[2020-Oct-29T21:31:46.650] ### Failure message: Ticket retrieval failed.
[2020-Oct-29T21:31:46.650] ### Failure stack:    at Qlik.Sense.JsonRpc.WebSocket.SystemNetWebSocketSession.RedirectionPerformed(ConnectionSettings connectionSettings, WebSocket theSocket, Action`1 onMessage, CancellationToken token)
   at Qlik.Sense.JsonRpc.WebSocket.SystemNetWebSocketSession.Open(ConnectionSettings connectionSettings, Action`1 log, CancellationToken token)
   at Qlik.Sense.JsonRpc.WebSocketSessionContainer.OpenAsync(Action`1 onMessage, Action`1 onError, CancellationToken cancellationToken)
   at Qlik.Sense.JsonRpc.RpcConnection.WithTimeout[T](Func`1 func, Int32 timeout, CancellationToken token)
   at Qlik.Sense.JsonRpc.RpcConnection.RepeatAttempt(Int32 maxAttempts, Func`2 f, CancellationToken token)
[2020-Oct-29T21:31:46.651] ### Disposing ### 50280062

It is also worth noting that one difference between my local environment and out QA server is that the latter is behind a nginx reverse proxy. I tried tinkering with that to see if it was keeping the websocket from forming, but I started getting the following error instead:

   AggregateException
   One or more errors occurred. (Unable to establish websocket connection.) (Unable to establish websocket connection.) (Unable to establish websocket connection.) (Unable to establish websocket connection.)
   at Qlik.Sense.JsonRpc.RpcConnection.RepeatAttempt(Int32 maxAttempts, Func`2 f, CancellationToken token)
   at Qlik.Sense.JsonRpc.RpcConnection.OpenAsync(CancellationToken token)
   at Qlik.Sense.JsonRpc.RpcConnection.EnsureConnectionIsOpen()
   at Qlik.Sense.JsonRpc.RpcConnection.SendAsync(Request request, TaskCompletionSource`1 tcs)
   at Qlik.Engine.Communication.QlikConnection.PingAsync()
   at Qlik.Engine.LocationExtensions.DisposeOnErrorAsync(IDisposable o, Func`1 f)
   at Qlik.Engine.LocationExtensions.HubAsync(ILocation location, ISession session)

This leads me to believe that the «Ticket retrieval failed» error occurs AFTER the websocket connection has been established, so it might not be the reverse proxy blocking the websocket, but I could be wrong.

I am going to try the rest_sdk, but I am curious if there is a way to interact with the engine service with plain HTTP requests instead of establishing a websocket?