Remote error tls unknown certificate

«Remote error» means error sent from client (firefox in this case).

Solution is to figure out why firefox doesn’t like the certificate and fix it.

There is something that firefox doesn’t like about the certificate. Open up firefox dev tools and see if you can find any warnings regarding the certificate. If you had to manually accept the certificate in Firefox then it is possible that Firefox is still reporting to the server that it doesn’t like the certificate even though you have told firefox to load the page anyway (see chrome example below).

However as this is a self generated/signed certificate the warning is probably because firefox doesn’t trust this certificate. Solution is to either add this certificate to firefox’s trusted certificates if this server is only for your own personal use…or to get a certificate signed by a commercial CA or letsencrypt.

«Fixing this server-side» means fixing whatever it is about how your certificate/app is served in order to make it trusted by firefox. Or I suppose ignoring the errors if you are just doing development.

More details…

Key thing here is that this is a «remote error» meaning it is an error from the tls CLIENT connecting to your server. In your case it is Firefox complaining during the TLS handshake that the certificate is invalid in some way.

I noticed the same thing with chrome. Cert is signed by a public CA (ie a CA that should be trusted by most browsers) but as I am developing on my local machine, the certificate is invalid because hostname (localhost) doesn’t match the certificate CommonName (CN) or Subject Alternative Names (SAN).

Easiest thing to do is just look at the TLS handshake in wireshark.

I told chrome to accept the certificate, started capture and did a SINGLE refresh of the page (https://localhost:8081 in this case). Chrome does not present me with a warning page and shows the content. There is a red warning in the address bar however.

The interesting thing (to me) is that it appears (I’m no TLS expert) that there are two TLS handshakes.

Client Hello
Server Hello
Alert (client error)
Client Hello
Server Hello
Client complete handshake
…encrypted application data…

Since chrome doesn’t interrupt my refresh of the page I am not sure why chrome does the handshake twice (first time with Alert/failure) for a single page refresh.

The interesting bit I learned here (obvious in hindsight) is that it is possible to get reporting from browsers/clients when they reject your certificate. This can be used on the server side (if monitored) to discover subtle certificate mis-configurations in production that your test cases might not cover. Unfortunately as this is pre-http you won’t get user-agent or anything useful to help with reproducing….but a large number of these errors as a percentage of server traffic indicates you need to do some serious cross browser/os/device testing

I been bashing my head on this problem but both my pacience and google-fu failed me … so i m turning to anyone out there who might encountered this issue.

I had this working on a previous server (before anyone says, then go get the old files from it, the disk died and i failed at backups … lesson learned) and i m trying rebuild my server from scratch and trying to make Traefik 2.0 working with Letsencrypt DNS challenge, used the documentation page over here as reference to build this configuration file.

So far i m out of luck, i cannot get any certificate from letsencrypt and resulting traefik starting to use a self sign certificate (and that’s why i get the error on the title and in the container log):

time="2019-12-30T00:49:54Z" level=info msg="Configuration loaded from flags."
time="2019-12-30T00:49:54Z" level=info msg="Traefik version 2.1.1 built on 2019-12-12T19:01:37Z"
time="2019-12-30T00:49:54Z" level=debug msg="Static configuration loaded {"global":{"checkNewVersion":true},"serversTransport":{"maxIdleConnsPerHost":200},"entryPoints":{"web":{"address":":80","transport":{"lifeCycle":{"graceTimeOut":10000000000},"respondingTimeouts":{"idleTimeout":180000000000}},"forwardedHeaders":{}},"websecure":{"address":":443","transport":{"lifeCycle":{"graceTimeOut":10000000000},"respondingTimeouts":{"idleTimeout":180000000000}},"forwardedHeaders":{}}},"providers":{"providersThrottleDuration":2000000000},"api":{"dashboard":true},"log":{"level":"DEBUG","format":"common"},"certificatesResolvers":{"le":{"acme":{"email":"username@domain.com","caServer":"https://acme-v02.api.letsencrypt.org/directory","storage":"/letsencrypt/acme.json","keyType":"RSA4096","dnsChallenge":{"provider":"provider","resolvers":["1.1.1.1:53","8.8.8.8:53"]},"httpChallenge":{"entryPoint":"web"},"tlsChallenge":{}}}}}"
time="2019-12-30T00:49:54Z" level=info msg="nStats collection is disabled.nHelp us improve Traefik by turning this feature on :)nMore details on: https://docs.traefik.io/v2.0/contributing/data-collection/n"
time="2019-12-30T00:49:54Z" level=info msg="Starting provider aggregator.ProviderAggregator {}"
time="2019-12-30T00:49:54Z" level=debug msg="Start TCP Server" entryPointName=websecure
time="2019-12-30T00:49:54Z" level=debug msg="Start TCP Server" entryPointName=web
time="2019-12-30T00:49:54Z" level=info msg="Starting provider *acme.Provider {"email":"username@domain.com","caServer":"https://acme-v02.api.letsencrypt.org/directory","storage":"/letsencrypt/acme.json","keyType":"RSA4096","dnsChallenge":{"provider":"provider","resolvers":["1.1.1.1:53","8.8.8.8:53"]},"httpChallenge":{"entryPoint":"web"},"tlsChallenge":{},"ResolverName":"le","store":{},"ChallengeStore":{}}"
time="2019-12-30T00:49:54Z" level=info msg="Testing certificate renew..." providerName=le.acme
time="2019-12-30T00:49:54Z" level=info msg="Starting provider *traefik.Provider {}"
time="2019-12-30T00:49:54Z" level=debug msg="Configuration received from provider le.acme: {"http":{},"tls":{}}" providerName=le.acme
time="2019-12-30T00:49:54Z" level=debug msg="Configuration received from provider internal: {"http":{"services":{"api":{},"dashboard":{}}},"tcp":{},"tls":{}}" providerName=internal
time="2019-12-30T00:49:54Z" level=debug msg="No default certificate, generating one"
time="2019-12-30T00:51:03Z" level=debug msg="http: TLS handshake error from 192.168.0.253:54501: remote error: tls: bad certificate"
time="2019-12-30T00:51:05Z" level=debug msg="Serving default certificate for request: "domain.com""

My docker-compose for traefik is the following:

  traefik:
    image: "traefik:latest"
    container_name: traefik
    hostname: traefik
    restart: always
    command:
      - --log.level=DEBUG
      - --entrypoints.web.address=:80
      - --entrypoints.websecure.address=:443
      - --api
      # Stating server - --certificatesresolvers.le.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory
      # Prod Server
      - --certificatesResolvers.le.acme.caServer=https://acme-v02.api.letsencrypt.org/directory
      - --certificatesresolvers.le.acme.email=username@domain.com
      - --certificatesresolvers.le.acme.storage=/letsencrypt/acme.json
      - --certificatesResolvers.le.acme.httpChallenge.entryPoint=web
      - --certificatesresolvers.le.acme.tlschallenge=true
      - --certificatesResolvers.le.acme.keyType=RSA4096
      - --certificatesResolvers.le.acme.dnsChallenge=true
      - --certificatesResolvers.le.acme.dnsChallenge.provider=dreamhost
      - --certificatesResolvers.le.acme.dnsChallenge.resolvers=1.1.1.1:53,8.8.8.8:53
      - --certificatesResolvers.le.acme.dnsChallenge.delayBeforeCheck=0
    labels:
      - traefik.http.routers.blog.rule=Host('domain.com')
      - traefik.http.routers.blog.tls=true
      - traefik.http.routers.blog.tls.certresolver=le
      - traefik.http.routers.blog.tls.domains[0].main=domain.com
      - traefik.http.routers.blog.tls.domains[0].sans=*.domain.com
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - ${DOCKERDIR}/traefik/config:/etc/traefik
      - ${DOCKERDIR}/traefik/letsencrypt:/letsencrypt
      - ${DOCKERDIR}/shared:/shared
    environment:
      - DREAMHOST_API_KEY=${DREAMHOST_API_KEY}

Anyone can point me to the right way to enable traefik to get the certificates from Letsencrypt? thank you …

UPDATE: Got tired of getting nowhere, i really enjoyed Traefik was but i prefer something that i can manage and not spend 3 days bashing my head and trying to make it to work. Moved back to nginx, but not the way i used to do it (editing files) for anyone interested i m using Nginx Proxy Manager, Pros dead simple to use takes 10 min to set up.. Cons for me, unable to set letsencrypt wildcard certificate, need to ask 1 certificate for all my sub domains

This article will be useful to you if you want to create a self signed server in golang. There are many ways to use certificates to build and run a https server, this article will approach one of them: self-signed using openssl tool. You can see all the source code used in the current article in the public github repository.

Why https?

Firstly let’s remember some concepts. The Hypertext Transfer Protocol (Http) specifies a standard track protocol for the internet community and has been in use since 1990. The problem with the use of only http is that the exchanged messages between server and client will not be encrypted, so everyone who intercepts those messages will know exactly what that messages means and also can modify the data to masquerade as one of the peers involved. To avoid attacks like man-in-the-middle and to provide private communication, the Secure Sockets Layer (SSL) was first introduced by Netscape in 1994 being the pioneer in secure communications protocols, but it was succeeded later by the Transport Layer Security (TLS). TLS is the channel-oriented security protocol currently in use on the internet and it’s composed basically by two protocols: the handshake and the record protocol.

The TLS Handshake protocol is responsible to authenticate two end-points, besides that, it also negotiates cryptographic parameters and generates keying material. The record protocol uses the parameters established by the handshake protocol to protect traffic between the end-points.

Why self signed?

By default your operation system trusts in a set of certification authorities (CA) like GlobalSign, Let’s Encrypt, Digicert, GoDaddy, etc. The self-signed certificates are those that aren’t signed by any CA, in this case, the certificate is signed with its own private key, instead of requesting it from a CA. So in that case, the client should trust in the certificate issued by the server.

The first thing you need to ask yourself is: why do I need a self signed certificate? There are few reasons for that and I’ve never faced a reason to use it in a production environment. So maybe you’re thinking: so on, what is this article for? There are some scenarios which demands you to provide a https endpoint (to run your application locally for example). In my case, I needed it to run an application locally to integrate it with a cloud service that requires a https endpoint. There are some frameworks, SDKs or tool kits written in other languages that provides to you an https endpoint natively with self-signed certificates (it’s the case of JDK and .NET Core), but I didn’t find anything like that in golang.

NOTE: if you’re gonna use https in production, I strongly recommend you to use a certificate signed by a CA (try to use a cloud solution like AWS Certificate Manager, or an open source tool like certbot). There are some security risks that you should be aware of when using self-signed, you can see more about it here.

Running the project

«Talk is cheap, show me the code» — Linus Torvalds

To proceed with the next steps, you’re gonna need to clone this github repo. The current example is composed by a server and a client called https-server and https-client respectively. Each one runs in its specific container, the server provides a REST API written in golang and is responsible to create the self signed certificate. That certificate protects two hostnames: localhost and https-server, that multi-domain approach is possible thanks to the Subject Alternative Names (SANs). Take a look at the diagram below that represents the current example:

As you can see above, the server generates the certificate and the clients trust that certificate (client container or a client running in the host). So, to up the client and server containers, run the command below:

docker-compose up

Server

The command above will firstly up the server container and run some commands from a file called generate-certificate.sh. That bash file contains some openssl commands to create the self signed certificate. First, it generates a servercert.key and servercert.csr which are respectively: the private key and the certificate signing request (CSR) that contains the public key. The CN field in -subj is very important because some browsers like chrome require that information (CN means Common Name, that’s the domain name you would like to have SSL secured). Then, the certificate file will be generated also, this file, named servercert.crt, is generated by the last command in the bash file. That’s the self-signed certificate signed by your own servercert.key private key. The x509 flag states the standard format of an SSL/TLS certificate, the X.509 format. Finally, the https server are gonna get up by the go run main.go command. Take a look at the bash commands bellow:

apk update && apk add openssl && rm -rf /var/cache/apk/*
openssl req -new -subj "/C=US/ST=California/CN=localhost" 
    -newkey rsa:2048 -nodes -keyout "$FILE_CERT_NAME.key" -out "$FILE_CERT_NAME.csr"
openssl x509 -req -days 365 -in "$FILE_CERT_NAME.csr" -signkey "$FILE_CERT_NAME.key" -out "certificates/$FILE_CERT_NAME.crt" -extfile "self-signed-cert.ext"

the ext file has all tha SANs protected by the certificate:

subjectAltName = @alt_names
[alt_names]
DNS.1 = localhost
DNS.2 = https-server

Now that you already have the certificate, you need to serve your https server. Inside the main.go file, the ListenAndServeTLS method is responsible for use the cert and key to serve the https self signed server:

func handleRequests() {

  tlsCert := os.Getenv("tls-certificate")
  tlsKey := os.Getenv("tls-key")
  serverPort := os.Getenv("server-port")

  router := mux.NewRouter().StrictSlash(true)
  controllers.HandleHomeRoutes(router, "https")

  log.Fatal(http.ListenAndServeTLS(serverPort, tlsCert, tlsKey, router))
}

Along with that, as the cert and key was gotten from the .env file, you should declare both paths:

tls-certificate="certificates/servercert.crt"
tls-key="servercert.key"

Client

The client container has a volume with the path where the server certificate was generated: ./server/certificates:/certificates. That’s because the client needs to trust that certificate to make https calls to the server. The command update-ca-certificates is responsible to add that certificate to the system’s trust store, it was executed in trust-server-certificate.sh bash file. After that, the client will be able to call the server with https (the handshake will happen normally). The https-client container calls the /home endpoint from the server with https two times after trusting its certificate, take a look at the curl calls in get-server-home.sh file:

#!/bin/ash
echo "Installing curl package"
apk update && apk add curl && rm -rf /var/cache/apk/*
echo "Two requests below to get https server home"
sleep 10
curl https://https-server:8081/home
sleep 20
curl https://https-server:8081/home

Call the server with a client running locally (localhost)

As mentioned before, you need to trust the server certificate in your local trust store if you want to use https. If you’re using a linux based OS, you should run the commands shown in trust-server-certificate.sh file. Otherwise, follow one of the steps below:

• Mac Os

• Windows

• Linux

If you call a server endpoint before trusting the server certificate, you’ll get an error like the following in your browser:

after trusting the certificate locally, you’ll get the response with a 200 Ok status code:

if you expand the certificate, you will see all the domains secured by the self-signed certificate:

that behavior is also shown in the server stdout, before trusting the certificate there is a handshake error, but after trusting it, the handshake is successful:

https-server | 2022/02/07 00:59:53 http: TLS handshake error from 172.19.0.1:55672: remote error: tls: unknown certificate
https-server | Home page endepoint hit

References

Digicert; Multi-Domain (SAN) Certificates — Using Subject Alternative Names

Globalsign; The Dangers of Self-Signed SSL Certificates

Keyfactor; What is a Self-Signed Certificate? Advantages, Risks & Alternatives

OpenSSL; Cryptography and SSL/TLS Toolkit

RFC 2616; Hypertext Transfer Protocol — HTTP/1.1

RFC 4949; Internet Security Glossary, Version 2

RFC 6101; The Secure Sockets Layer (SSL) Protocol Version 3.0

RFC 8446; The Transport Layer Security (TLS) Protocol Version 1.3

Trying to get the Dashboard UI working in a kubeadm cluster using kubectl proxy for remote access. Getting

Error: 'dial tcp 192.168.2.3:8443: connect: connection refused'
Trying to reach: 'https://192.168.2.3:8443/'

when accessing http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/ via remote browser.

Looking at API logs, I see that I’m getting the following errors:

I1215 20:18:46.601151       1 log.go:172] http: TLS handshake error from 10.21.72.28:50268: remote error: tls: unknown certificate authority
I1215 20:19:15.444580       1 log.go:172] http: TLS handshake error from 10.21.72.28:50271: remote error: tls: unknown certificate authority
I1215 20:19:31.850501       1 log.go:172] http: TLS handshake error from 10.21.72.28:50275: remote error: tls: unknown certificate authority
I1215 20:55:55.574729       1 log.go:172] http: TLS handshake error from 10.21.72.28:50860: remote error: tls: unknown certificate authority
E1215 21:19:47.246642       1 watch.go:233] unable to encode watch object *v1.WatchEvent: write tcp 134.84.53.162:6443->134.84.53.163:38894: write: connection timed out (&streaming.encoder{writer:(*metrics.fancyResponseWriterDelegator)(0xc42d6fecb0), encoder:(*versioning.codec)(0xc429276990), buf:(*bytes.Buffer)(0xc42cae68c0)})

I presume this is related to not being able to get the Dashboard working, and if so am wondering what the issue with the API server is. Everything else in the cluster appears to be working.

NB, I have admin.conf running locally and am able to access the cluster via kubectl with no issue.

Also, of note is that this had been working when I first got the cluster up. However, I was having networking issues, and had to apply this in order to get CoreDNS to work Coredns service do not work,but endpoint is ok the other SVCs are normal only except dns, so I am wondering if this maybe broke the proxy service?

* EDIT *

Here is output for the dashboard pod:

[gms@thalia0 ~]$ kubectl describe pod kubernetes-dashboard-77fd78f978-tjzxt --namespace=kube-system
Name:               kubernetes-dashboard-77fd78f978-tjzxt
Namespace:          kube-system
Priority:           0
PriorityClassName:  <none>
Node:               thalia2.hostdoman/hostip<redacted>
Start Time:         Sat, 15 Dec 2018 15:17:57 -0600
Labels:             k8s-app=kubernetes-dashboard
                    pod-template-hash=77fd78f978
Annotations:        cni.projectcalico.org/podIP: 192.168.2.3/32
Status:             Running
IP:                 192.168.2.3
Controlled By:      ReplicaSet/kubernetes-dashboard-77fd78f978
Containers:
  kubernetes-dashboard:
    Container ID:  docker://ed5ff580fb7d7b649d2bd1734e5fd80f97c80dec5c8e3b2808d33b8f92e7b472
    Image:         k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.0
    Image ID:      docker-pullable://k8s.gcr.io/kubernetes-dashboard-amd64@sha256:1d2e1229a918f4bc38b5a3f9f5f11302b3e71f8397b492afac7f273a0008776a
    Port:          8443/TCP
    Host Port:     0/TCP
    Args:
      --auto-generate-certificates
    State:          Running
      Started:      Sat, 15 Dec 2018 15:18:04 -0600
    Ready:          True
    Restart Count:  0
    Liveness:       http-get https://:8443/ delay=30s timeout=30s period=10s #success=1 #failure=3
    Environment:    <none>
    Mounts:
      /certs from kubernetes-dashboard-certs (rw)
      /tmp from tmp-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kubernetes-dashboard-token-mrd9k (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  kubernetes-dashboard-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  kubernetes-dashboard-certs
    Optional:    false
  tmp-volume:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
  kubernetes-dashboard-token-mrd9k:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  kubernetes-dashboard-token-mrd9k
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node-role.kubernetes.io/master:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>

I checked the service:

[gms@thalia0 ~]$ kubectl -n kube-system get service kubernetes-dashboard
NAME                   TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
kubernetes-dashboard   ClusterIP   10.103.93.93   <none>        443/TCP   4d23h

And also of note, if I curl http://localhost:8001/api from the master node, I do get a valid response.

So, in summary, I’m not sure which if any of these errors are the source of not being able to access the dashboard.