Internal error occurred failed calling webhook validate nginx ingress kubernetes io

Table of contents

Background information

Ingress controllers acquire Ingress rule changes from the API server and dynamically
generate configuration files, such as nginx.conf. These configuration files are required by a load balancer, such as NGINX. Then,
the Ingress controllers reload the load balancer. For example, the Ingress controllers
run the nginx -s load command to reload NGINX.and then generate new Ingress rules.

Diagnostic procedure

1. You can perform the following steps to check whether an issue is caused by the Ingress.
   Make sure that the configuration of the Ingress controller is valid.
   1. Check whether you can access a specific pod from the controller pod. For more information,
      see Manually access the Ingress and backend pod by using the Ingress controller pod.
   2. Check whether the NGINX Ingress controller is properly configured. For more information,
      see Documentation for the NGINX Ingress controller.
2. Use the Ingress diagnostics feature to check the configurations of the Ingress and
   components. Then, modify the configurations based on the prompts. For more information,
   see Use the Ingress diagnostics feature.
3. Locate the cause of the issue and refer to the relevant solution based on Troubleshooting.
4. If the issue persists, perform the following steps:
   • Issues that are related to TLS certificates:
     1. Check whether the domain name is added to Web Application Firewall (WAF) in CNAME
        record mode or transparent proxy mode.
        • If the domain name is added to WAF, check whether the domain name has a TLS certificate.
        • If the domain name is not added to WAF, proceed to the next step.
     2. Check whether a Layer 7 listener is added for the Server Load Balancer (SLB) instance.
        • If a Layer 7 listener is added for the SLB instance, check whether a TLS certificate
          is associated with the listener.
        • If no Layer 7 listener is added for the SLB instance, proceed to the next step.
   • If the issue is not related to TLS certificates, diagnose the error log of the controller
     pod. For more information, see Diagnose the error log of the controller pod.
5. If the issue persists, locate the cause of the issue by capturing packets in the controller
   pod and the backend pod. For more information, see Capture packets.
6. If the issue persists, Submit a ticket for troubleshooting.

Commonly used diagnostic methods

Use the Ingress diagnostics feature

1. Log on to the ACK console.
2. In the left-side navigation pane of the ACK console, click Clusters.
3. On the Clusters page, find the cluster on which you want to perform a check and choose in the Actions column.
4. In the left-side navigation pane of the Container Intelligence Service page, choose
   .
5. On the Diagnosis page, click Ingress Diagnosis.
6. In the Ingress Diagnosis panel, enter the URL that cannot be accessed, such as https://www.example.com. Select I know and agree and then click Create diagnosis.

   After the diagnostic is completed, you can view the diagnostic result and try to fix
   the issue.

Diagnose the access log of the NGINX Ingress controller pod in Log Service

You can check the access log format of the NGINX Ingress controller in the nginx-configuration
ConfigMap in the kube-system namespace.

$remote_addr - [$remote_addr] - $remote_user [$time_local]
    "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $request_length
    $request_time [$proxy_upstream_name] $upstream_addr $upstream_response_length
    $upstream_response_time $upstream_status $req_id $host [$proxy_alternative_upstream_name]

The following figure shows the page on which the access log of the NGINX Ingress controller
is displayed in the Log Service console. For more information, see Step 4: View log data.

kubectl logs <controller pod name> -n <namespace> | less

42.11.**.** - [42.11.**.**]--[25/Nov/2021:11:40:30 +0800]"GET / HTTP/1.1" 200 615 "_" "curl/7.64.1" 76 0.001 [default-nginx-svc-80] 172.16.254.208:80 615 0.000 200 46b79dkahflhakjhdhfkah**** 47.11.**.**[]
42.11.**.** - [42.11.**.**]--[25/Nov/2021:11:40:31 +0800]"GET / HTTP/1.1" 200 615 "_" "curl/7.64.1" 76 0.001 [default-nginx-svc-80] 172.16.254.208:80 615 0.000 200 fadgrerthflhakjhdhfkah**** 47.11.**.**[]

Diagnose the error log of the NGINX Ingress controller pod

You can diagnose the error log of the NGINX Ingress controller pod to narrow down
the scope of troubleshooting. The error log of the Ingress controller pod includes
the following types:

• The log that records errors of the Ingress controller. Typically, this type of error
  log is generated due to invalid Ingress configurations. You can run the following
  command to query this type of error log:

  kubectl logs <controller pod name> -n <namespace> | grep -E ^[WE]

  Note During the initialization of an Ingress controller, a few warning events may be generated.
  For example, if you do not specify the kubeconfig file or IngressClass resource, warning
  events are generated. These events do not affect the Ingress controller and can be
  ignored.

• The log that records errors of the NGINX application. Typically, this type of error
  log is generated due to request processing failures. You can run the following command
  to query this type of error log:

  kubectl logs <controller pod name> -n <namespace> | grep error

Manually access the Ingress and backend pod by using the Ingress controller pod

1. Run the following command to log on to the Ingress controller pod:

   kubectl exec <controller pod name> -n <namespace> -it -- bash

2. The Ingress controller pod is preinstalled with curl and OpenSSL, which allow you
   to test network connectivity and verify certificates.
   • Run the following command to test the network connectivity between the Ingress and
     the backend pod:

     # Replace your.domain.com with the actual domain name of the Ingress. 
     curl -H "Host: your.domain.com" http://127.0.**.**/ # for http
     curl --resolve your.domain.com:443:127.0.0.1 https://127.0.0.1/ # for https

   • Run the following command to verify the certificate:

     openssl s_client -servername your.domain.com -connect 127.0.0.1:443

   • Test access to the backend pod.

     Note An Ingress controller directly connects to the IP address of the backend pod instead
     of using a ClusterIP Service.

     1. Run the following kubectl command to query the IP address of the backend pod:

        kubectl get pod -n <namespace> <pod name> -o wide

        Expected output:

        NAME                      READY    STATUS    RESTARTS   AGE    IP            NODE                        NOMINATED NODE    READINESS GATES
        nginx-dp-7f5fcc7f-****    1/1      Running   0          23h    10.71.0.146   cn-beijing.192.168.**.**    <none>            <none>

        The output shows that the IP address of the backend pod is 10.71.0.146.

     2. To test the network connectivity between the Ingress controller pod and the backend
        pod, run the following command to connect to the IP address by using the Ingress controller
        pod:

        curl http://<your pod ip>:<port>/path

Capture packets

If you cannot identify the issue, capture and diagnose packets.

1. Check whether the issue is related to the Ingress controller pod or the application
   pod. If this cannot be done, capture packets for both the Ingress controller pod and
   the application pod.
2. Log on to the nodes on which the application pod and Ingress controller pod run.
3. Run the following command on the Elastic Compute Service (ECS) instances to capture
   all recent packets that are received by the Ingress:

   tcpdump -i any host <Application pod IP or Ingress controller pod IP> -C 20 -W 200 -w /tmp/ingress.pcap

4. If an error is identified in the log data, stop capturing packets.
5. Diagnose the packets that are transferred during the time period in which the error
   occurred.

   Note

   • Packet capture does not affect your service and only causes a slight increase in the
     CPU utilization and disk I/O.
   • The preceding command rotates the captured packets and can generate at most 200 .pcap files, each of which is 20 MB in size.

Why do I fail to access the IP address of the LoadBalancer from within the Kubernetes
cluster?

Symptom

In a Kubernetes cluster, only specific nodes can access the IP address of the LoadBalancer
if externalTrafficPolicy is set to Local for the LoadBalancer.

Causes

externalTrafficPolicy: Local is set for the LoadBalancer. In Local mode, the IP address of the LoadBalancer is
accessible only from pods that are provisioned on the local node (the node that runs
the LoadBalancer). The IP address of the LoadBalancer is inaccessible from pods on
other nodes in the cluster. The IP address of the LoadBalancer is external to the
Kubernetes cluster. If nodes or pods in the ACK cluster cannot access the IP address
without using a second hop, requests do not pass through the LoadBalancer. As a result,
the IP address of the LoadBalancer is considered an extended IP address of the Service
that uses the LoadBalancer. Requests are forwarded by kube-proxy based on iptables
or IP Virtual Server (IPVS).

In this scenario, if the requested pod is not provisioned on the local node, a connectivity
issue occurs. The IP address of the LoadBalancer is accessible only if the requested
pod is provisioned on the local node. For more information about external-lb, see
kube-proxy adds the IP address of external-lb to the node local iptables rules.

Solutions

• We recommend that you access the IP address of the LoadBalancer from within the Kubernetes
  cluster by using the ClusterIP Service or the Ingress name.

  The Ingress name is nginx-ingress-lb in the kube-system namespace.

• Run the kubectl edit svc nginx-ingress-lb -n kube-system command to modify the Ingress. Set externalTrafficPolicy to Cluster for the LoadBalancer. After you change the setting, client IP addresses cannot be
  preserved.
• If the network plug-in of the cluster is set to Terway and the exclusive or inclusive
  ENI mode is selected, you can set externalTrafficPolicy to Cluster for the LoadBalancer and add the ENI annotation. The annotation adds pods that are
  signed elastic network interfaces (ENIs) as the backend servers of the LoadBalancer.
  This way, client IP addresses can be preserved and the IP address of the LoadBalancer
  can be accessed from within the cluster.

  Example:

  apiVersion: v1
  kind: Service
  metadata:
    annotations:
      service.beta.kubernetes.io/backend-type: eni   # Add pods that are signed ENIs as the backend servers of the LoadBalancer. 
    labels:
      app: nginx-ingress-lb
    name: nginx-ingress-lb
    namespace: kube-system
  spec:
    externalTrafficPolicy: Cluster

  For more information about the annotations of Services, see Use annotations to configure load balancing.

Why does the Ingress controller pod fail to access the Ingress controller?

Symptom

In a cluster for which Flannel is used, when the Ingress controller pod accesses the
Ingress controller through a domain name, an SLB IP address, or cluster IP address,
some or all of the requests sent from the Ingress controller pod are dropped.

Causes

By default, Flannel does not allow loopback requests.

Solutions

• We recommend that you create a new cluster that uses the Terway network plug-in and
  migrate your workloads to the new cluster.
• If you do not want to create a new cluster, you can enable hairpinMode in the configurations of Flannel. After you modify the configurations, recreate the
  Flannel pod for the modification to take effect.
  1. Run the following command to modify the configurations of Flannel:

     kubectl edit cm kube-flannel-cfg -n kube-system

  2. In the cni-config.json file that is returned, add "hairpinMode": true in the delegate field.

     Example:

     cni-conf.json: |
         {
           "name": "cb0",
           "cniVersion":"0.3.1",
           "type": "flannel",
           "delegate": {
             "isDefaultGateway": true,
             "hairpinMode": true
           }
         }

  3. Run the following command to restart Flannel:

     kubectl delete pod -n kube-system -l app=flannel   

  4. Delete and recreate the pod.

Why is the default TLS certificate or previous TLS certificate used after I add a
TLS certificate to the cluster or change the TLS certificate?

Symptom

You added a Secret to the cluster or modified a Secret in the cluster, and updated
the Secret name in the secretName field in the Ingress. When you access the cluster,
the default certificate (Kubernetes Ingress Controller Fake Certificate) or the previous
certificate is used.

Causes

• The certificate is not returned by the Ingress controller in the cluster.
• The certificate is invalid and the Ingress controller cannot load the certificate.
• The certificate is returned by the Ingress controller based on the Server Name Indication
  (SNI) field. The SNI field may not be sent as a part of the TLS handshake.

Solutions

• Use one of the following methods to check whether the SNI field is sent as a part
  of the TLS handshake:
  • Upgrade your browser to a version that supports SNI.
  • When you run the openssl s_client command to check whether the certificate is in use, specify the -servername parameter.
  • When you run curl commands, add hosts or use the --resolve parameter to map the domain name, other than using the host request header.
• Make sure that no TLS certificate is specified when you connect the website to WAF
  in CNAME record mode or transparent proxy mode, or no TLS certificate is associated
  with the Layer 7 listener of the SLB instance. The TLS certificate must be returned
  by the Ingress controller in the cluster.
• Navigate to the Container Intelligence Service console and diagnose the Ingress. In
  the diagnostic report, check whether the configurations of the Ingress are valid and
  whether the log data shows errors. For more information, see Use the Ingress diagnostics feature.
• Run the following command to view the error log of the Ingress controller pod and
  troubleshoot the issue based on the log data.

  kubectl logs <ingress pod name> -n <pod namespace> | grep -E ^[EW]

Why do I fail to access the Ingress by using the test domain name provided in the
ACK console?

Symptom

You cannot access the Ingress by using the test domain name (*.cxxxxxxxxxxx.xxx.alicontainer.com)
provided in the ACK console.

Causes

The test domain name is not resolved to the IP address of the Service that is associated
with the Ingress. Possible causes:

• The Service is not nginx-ingress-lb in the kube-system namespace.
• The external IP address of the Service is changed after the Ingress is created.

Solutions

• Associate the desired IP address with the nginx-ingress-lb Service in the kube-system
  namespace.
• On the cluster details page of the Container Service for Kubernetes (ACK) console, click the Basic Information tab and then click Rebind Domain Name.

Why do I fail to access gRPC Services that are exposed by an Ingress?

Symptom

You cannot access gRPC Services that are exposed by an Ingress.

Causes

• You do not set annotations in the Ingress to specify the backend protocol.
• gRPC Services can be accessed only by using Transport Layer Security (TLS).

Solutions

• Set the following annotation in the Ingress: nginx.ingress.kubernetes.io/backend-protocol:"GRPC".
• Make sure that clients use HTTPS ports to send requests and the traffic is encrypted
  by using TLS.

Why do I fail to access backend HTTPS services?

Symptom

• You fail to access backend HTTPS services through the Ingress.
• A 400 error code may be returned and the following error message may be prompted:
  The plain HTTP request was sent to HTTPS port.

Causes

The Ingress controller sends HTTP requests to the backend pods. This is the default
setting.

Solutions

Set the following annotation in the Ingress: nginx.ingress.kubernetes.io/backend-protocol:"HTTPS".

Why does the Ingress controller pod fail to preserve client IP addresses?

Symptom

The Ingress controller pod cannot preserve client IP addresses. Only the node IP,
the CIDR blocks 100.XX.XX.XX, or other IP addresses are preserved.

Causes

• externalTrafficPolicy is set to Cluster for the Service that is associated with the Ingress.
• A Layer 7 proxy is used by the SLB instance.
• Your website is connected to WAF in CNAME record mode or transparent proxy mode.

Solutions

• If externalTrafficPolicy is set to Cluster for the Service and a Layer 4 SLB instance is used, perform the following steps:

  Set externalTrafficPolicy to Local. However, you may fail to access the Ingress by using the IP address of the SLB instance
  from within the cluster. For more information, see Why do I fail to access the IP address of the LoadBalancer from within the Kubernetes cluster?.

• Perform the following steps if a Layer 7 proxy is used, for example, a Layer 7 SLB
  instance is used or your website is connected to WAF in CNAME record mode or transparent
  proxy mode:
  1. Make sure that the X-Forwarded-For header is enabled for the Layer 7 proxy.
  2. Add enable-real-ip: "true" to the ConfigMap of the Ingress controller. By default, the ConfigMap is named nginx-configuration
     and belongs to the kube-system namespace.
  3. Analyze the log data to check whether client IP addresses can be preserved.
• If a client request traverses multiple hops before it reaches the Ingress controller
  pod, for example, the request must pass through a reverse proxy before it reaches
  the Ingress controller pod, you can check the value of remote_addr after you set enable-real-ip to true. If the value is a client IP address, this indicates that the X-Forwarded-For
  header is enabled to pass client IP addresses to the Ingress controller pod. If the
  X-Forwarded-For header is disabled, enable the X-Forwarded-For header or use other
  methods to add client IP addresses to requests before the requests reach the Ingress
  controller pod.

Why do canary release rules fail to take effect?

Symptom

You set canary release rules in a cluster but the rules do not take effect.

Causes

Possible causes:

• When you add canary-* annotations, you do not set nginx.ingress.kubernetes.io/canary: "true".
• The version of the NGINX Ingress controller is earlier than 0.47.0. When you add canary-* annotations, you do not specify the domain name of your application in the host field.

Solutions

• Set nginx.ingress.kubernetes.io/canary: "true" or specify the domain name of your application in the host field.
• If the issue persists, see Why are requests not distributed based on the specified canary release rules or why do the canary release rules affect other Ingresses that are associated with the same Service?.

Why are requests not distributed based on the specified canary release rules or why
do the canary release rules affect other Ingresses that are associated with the same
Service?

Symptom

Requests are not distributed based on the canary release rules that you set, or the
canary release rules affect other Ingresses that are associated with the same Service.

Causes

Canary release rules in an NGINX Ingress controller take effect on all Ingresses that
are associated with the Service for which the canary release rules are created.

For more information about this issue, see An Ingress with canary annotations affects other Ingresses that are associated with the same Service.

Solutions

Canary Ingresses include Ingresses that are assigned the service-match or canary-* annotations. Before you create a canary Ingress, create two same Services that are
used for canary releases, and then map the Services to the backend pods that you want
to access.

Why does the following error occur when you create an Ingress: «failed calling webhook»?

Symptom

The following error occurs when you create an Ingress: «Internal error occurred: failed
calling webhook…».

Causes

When you create an Ingress resource, a Service is used to check whether the Ingress
is valid. By default, the Service named ingress-nginx-controller-admission is used.
If Webhook link errors occur, for example, the Service or the Ingress controller is
deleted, the Ingress cannot be created.

Solutions

• Check whether the resource exists and works as expected based on the following Webhook
  link: ValidatingWebhookConfiguration > Service > Pod.
• Make sure that the admission feature is enabled for the Ingress controller pod and
  the pod can be accessed from outside the cluster.
• If the Ingress controller is deleted or you do not want to use the Webhook feature,
  you can delete the ValidatingWebhookConfiguration resource.

Why is the following error returned for HTTPS requests: SSL_ERROR_RX_RECORD_TOO_LONG?

Symptom

One of the following errors is returned for HTTPS requests: SSL_ERROR_RX_RECORD_TOO_LONG or routines:CONNECT_CR_SRVR_HELLO:wrong version number.

Causes

HTTPS requests are distributed to a non-HTTPS port, such as an HTTP port.

Solutions

Modify the configurations of the SLB instance or Service to ensure that HTTPS requests
can be distributed to the proper port.

Common HTTP status codes

Symptom

HTTP status codes other than 2xx and 3xx are returned, such as 502, 503, 413, and
499.

Causes and solutions

View the log and check whether the error is returned by the Ingress controller. For
more information, see Diagnose the access log of the NGINX Ingress controller pod in Log Service. If the error is returned by the Ingress controller, use the following solutions:

• 413 error
  • Cause: The request size exceeds the upper limit.
  • Solution: Increase the value of proxy-body-size in the ConfigMap of the Ingress controller.
    The default value of proxy-body-size is 1 MB for the NGINX Ingress controller of open
    source Kubernetes and the default value of proxy-body-size is 20 MB for the NGINX
    Ingress controller of ACK.
• 499 error
  • Cause: The client terminates the connection in advance. The error may not be caused
    by the Ingress controller or backend services.
  • Solution:
    • If the 499 error does not occur frequently and your workloads are not affected, you
      can ignore the error.
    • If the 499 error occurs frequently, you must check whether the amount of time that
      the backend pods cost to process requests exceeds the request timeout period that
      is set on the client.
• 502 error
  • Cause: The Ingress controller cannot connect to backend pods.
  • Solution:
    • The issue occurs occasionally:
      • Check whether the backend pods work as expected. If the backend pods are overloaded,
        add more backend pods.
      • By default, the Ingress controller sends HTTP 1.1 requests to backend services and
        HTTP keep-alive is enabled. Make sure that the keep-alive timeout period configured
        for the backend pods is greater than that configured for the Ingress controller. By
        default, the timeout period is set to 60 seconds.
    • The issue occurs every time:
      • Check whether the Service port is valid and whether the Service can be accessed from
        the Ingress controller pod.
      • In the Container Intelligence Service console, choose . On the Diagnosis page, click Network diagnosis to check the network connectivity.
    • If the issue persists, capture and analyze packets, and then Submit a ticket.
• 503 error
  • Symptom: The Ingress controller cannot discover the backend pods, or the Ingress controller
    fails to access all backend pods.
  • Solution:
    • The issue occurs occasionally:
      • Refer to the solution for resolving the 502 error.
      • Check the status of the backend pods and configure health checks.
    • The issue occurs every time:

      Check whether the Service configuration is valid and whether the endpoint exists.

    • If you cannot locate the cause by using the preceding methods, Submit a ticket.

The following error occurs: net::ERR_HTTP2_SERVER_REFUSED_STREAM.

Symptom

When you access the website, some resources cannot be loaded and one of the following
errors is prompted in the console: net::ERR_HTTP2_SERVER_REFUSED_STREAM or net::ERR_FAILED.

Causes

The number of concurrent HTTP/2 streams to the resource has reached the upper limit.

Solutions

• We recommend that you change http2-max-concurrent-streams in the ConfigMap to a greater value. The default value is 128. For more information,
  see http2-max-concurrent-streams.
• Disable HTTP/2 by setting use-http2 to false in the ConfigMap. For more information, see use-http2.

Why does the following error occur: The param of ServerGroupName is illegal?

Causes

ServerGroupName is generated in the following format: namespace+svcName+port. The server group name must be 2 to 128 characters in length and can contain letters,
digits, periods (.), underscores (_), and hyphens (-). The name must start with a
letter.

Solutions

Modify the server group name based on the required format.

Why does the «certificate signed by unknown authority» error occur when I create an
Ingress?

Causes

If multiple Ingresses are deployed in the cluster and the Ingresses use the same resources,
such as Secrets, Services, or webhook configurations, the preceding error occurs because
different SSL certificates are used to communicate with backend servers when webhooks
are triggered.

Solutions

Redeploy two Ingresses and make sure that the Ingresses use different resources. For
more information about the resources used by Ingresses, see What are the system updates after I update the NGINX Ingress controller on the Add-ons page of the ACK console?.

Why does the Ingress controller pod restart after it fails the health check?

Symptom

The Ingress controller pod restarts after it fails the health check.

Causes

• The Ingress controller pod or the node where the pod is deployed is overloaded. As
  a result, the pod failed to pass the health check.
• Kernel parameters such as tcp_tw_reuse or tcp_tw_timestamps may be configured for the cluster node where the Ingress controller pod is deployed.
  This may cause health check failures.

Solutions

• Add more Ingress controller pods and check whether the issue persists. For more information,
  see Deploy an Ingress access layer with high reliability.
• Disable tcp_tw_reuse or set the value of the parameter to 2, disable tcp_tw_timestamps, and then check whether the issue persists.

How do I add Services that use TCP or UDP?

1. Add specific entries to the tcp-services and udp-services ConfigMaps. By default,
   the ConfigMaps belong to the kube-system namespace.

   The following code block shows an example on how to map port 8080 of example-go in
   the default namespace to port 9000:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: tcp-services
     namespace: ingress-nginx
   data:
     9000: "default/example-go:8080"  # Map port 8080 to port 9000. 

2. Add port 9000 to the Deployment of the Ingress. By default, the Deployment is named
   nginx-ingress-controller and belongs to the kube-system namespace.
3. Add the port 9000 to the Service that is associated with the Ingress.

   Example:

   apiVersion: v1
   kind: Service
   metadata:
     name: ingress-nginx
     namespace: ingress-nginx
     labels:
       app.kubernetes.io/name: ingress-nginx
       app.kubernetes.io/part-of: ingress-nginx
   spec:
     type: LoadBalancer
     ports:
       - name: http
         port: 80
         targetPort: 80
         protocol: TCP
       - name: https
         port: 443
         targetPort: 443
         protocol: TCP
       - name: proxied-tcp-9000
         port: 9000
         targetPort: 9000
         protocol: TCP
     selector:
       app.kubernetes.io/name: ingress-nginx
       app.kubernetes.io/part-of: ingress-nginx

   For more information about how to add Services that use TCP or UDP, see Expose Services that use TCP or UDP.

Why do Ingress rules fail to take effect?

Symptom

After you add or modify Ingress rules, the rules do not take effect.

Causes

• The configuration of the Ingress contains errors. As a result, the Ingress failed
  to load the Ingress rules.
• The configurations of Ingress resources contain errors.
• The Ingress controller does not have the required permissions. As a result, the Ingress
  controller cannot monitor the changes made to Ingress resources.
• The previous Ingress uses a domain name specified in the server-alias field. The domain name is in conflict with that of the new Ingress. As a result,
  the Ingress rules are ignored.

Why does the system fail to load some web page resources or return a blank white screen
when requests are redirected to the root directory?

Symptom

After you set the rewrite-target annotation in the Ingress to rewrite requests, some web page resources cannot be
loaded or a blank white screen is displayed when you access the backend service.

Causes

• You do not set rewrite-target in regular expressions.
• The path of the requested resource is set to the root directory.

Solutions

• Check whether rewrite-target is set in regular expressions and whether capture groups are used. For more information,
  see Rewrite.
• Check whether requests are redirected to the expected path.

How do I fix the issue that Log Service cannot parse logs as expected after ingress-nginx-controller
is upgraded?

Symptom

The ingress-nginx-controller component has two commonly used versions: ingress-nginx-controller
0.20 and 0.30. After you upgrade ingress-nginx-controller from 0.20 to 0.30 on the
Add-ons page in the console, the Ingress dashboard may not show the actual statistics of
requests to the backend servers when you perform canary releases or blue-green releases
with an Ingress.

Causes

The default log format of ingress-nginx-controller 0.20 is different from that of
ingress-nginx-controller 0.30. Therefore, the Ingress dashboard may not show the actual
statistics of requests to the backend servers when you perform canary releases or
blue-green releases with an Ingress.

Solutions

To fix the issue, perform the following steps to update the nginx-configuration ConfigMap and the configuration of k8s-nginx-ingress.

1. Update the nginx-configuration ConfigMap.
   • If you have not modified the nginx-configuration ConfigMap, copy the following content to a file named nginx-configuration.yaml and run the kubectl apply -f nginx-configuration.yaml command to deploy the file.

     apiVersion: v1
     kind: ConfigMap
     data:
       allow-backend-server-header: "true"
       enable-underscores-in-headers: "true"
       generate-request-id: "true"
       ignore-invalid-headers: "true"
       log-format-upstream: $remote_addr - [$remote_addr] - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $request_length $request_time [$proxy_upstream_name] $upstream_addr $upstream_response_length $upstream_response_time $upstream_status $req_id $host [$proxy_alternative_upstream_name]
       max-worker-connections: "65536"
       proxy-body-size: 20m
       proxy-connect-timeout: "10"
       reuse-port: "true"
       server-tokens: "false"
       ssl-redirect: "false"
       upstream-keepalive-timeout: "900"
       worker-cpu-affinity: auto
     metadata:
       labels:
         app: ingress-nginx
       name: nginx-configuration
       namespace: kube-system

   • If you have modified the nginx-configuration ConfigMap, run the following command to update the configuration. This ensures that
     your previous modifications are not overwritten.

     kubectl edit configmap nginx-configuration -n kube-system

   Append [$proxy_alternative_upstream_name] to the log-format-upstream field, save the changes, and then exit.

2. Update the configuration of k8s-nginx-ingress.

   Copy the following content to a file named k8s-nginx-ingress.yaml and run the kubectl apply -f k8s-nginx-ingress.yaml command to start the deployment.

   apiVersion: log.alibabacloud.com/v1alpha1
   kind: AliyunLogConfig
   metadata:
       namespace: kube-system
     # your config name, must be unique in you k8s cluster
     name: k8s-nginx-ingress
   spec:
     # logstore name to upload log
     logstore: nginx-ingress
     # product code, only for k8s nginx ingress
     productCode: k8s-nginx-ingress
     # logtail config detail
     logtailConfig:
       inputType: plugin
       # logtail config name, should be same with [metadata.name]
       configName: k8s-nginx-ingress
       inputDetail:
         plugin:
           inputs:
           - type: service_docker_stdout
             detail:
               IncludeLabel:
                 io.kubernetes.container.name: nginx-ingress-controller
               Stderr: false
               Stdout: true
           processors:
           - type: processor_regex
             detail:
               KeepSource: false
               Keys:
               - client_ip
               - x_forward_for
               - remote_user
               - time
               - method
               - url
               - version
               - status
               - body_bytes_sent
               - http_referer
               - http_user_agent
               - request_length
               - request_time
               - proxy_upstream_name
               - upstream_addr
               - upstream_response_length
               - upstream_response_time
               - upstream_status
               - req_id
               - host
               - proxy_alternative_upstream_name
               NoKeyError: true
               NoMatchError: true
               Regex: ^(S+)s-s[([^]]+)]s-s(S+)s[(S+)sS+s"(w+)s(S+)s([^"]+)"s(d+)s(d+)s"([^"]*)"s"([^"]*)"s(S+)s(S+)+s[([^]]*)]s(S+)s(S+)s(S+)s(S+)s(S+)s*(S*)s*[*([^]]*)]*.*
               SourceKey: content