Rpc error code canceled desc context canceled

2019/10/16 11:05:14 [TRACE] Executing graph transform *terraform.OrphanResourceCountTransformer
2019/10/16 11:05:14 [TRACE] Completed graph transform *terraform.OrphanResourceCountTransformer (no changes)
2019/10/16 11:05:14 [TRACE] Executing graph transform *terraform.AttachStateTransformer
2019/10/16 11:05:14 [DEBUG] Resource state not found for node "module.docker_master.module.docker.null_resource.swarm_cluster", instance module.docker_master.module.docker.null_resource.swarm_cluster
2019/10/16 11:05:14 [TRACE] Completed graph transform *terraform.AttachStateTransformer (no changes)
2019/10/16 11:05:14 [TRACE] Executing graph transform *terraform.TargetsTransformer
2019/10/16 11:05:14 [TRACE] Completed graph transform *terraform.TargetsTransformer (no changes)
2019/10/16 11:05:14 [TRACE] Executing graph transform *terraform.ReferenceTransformer
2019/10/16 11:05:14 [WARN] no schema for provisioner "remote-exec" is attached to module.docker_master.module.docker.null_resource.swarm_cluster, so provisioner block references cannot be detected
2019/10/16 11:05:14 [WARN] no schema for provisioner "local-exec" is attached to module.docker_master.module.docker.null_resource.swarm_cluster, so provisioner block references cannot be detected
2019/10/16 11:05:14 [WARN] no schema for provisioner "file" is attached to module.docker_master.module.docker.null_resource.swarm_cluster, so provisioner block references cannot be detected
2019/10/16 11:05:14 [WARN] no schema for provisioner "file" is attached to module.docker_master.module.docker.null_resource.swarm_cluster, so provisioner block references cannot be detected
2019/10/16 11:05:14 [WARN] no schema for provisioner "file" is attached to module.docker_master.module.docker.null_resource.swarm_cluster, so provisioner block references cannot be detected
2019/10/16 11:05:14 [WARN] no schema for provisioner "remote-exec" is attached to module.docker_master.module.docker.null_resource.swarm_cluster, so provisioner block references cannot be detected
2019/10/16 11:05:14 [WARN] no schema for provisioner "file" is attached to module.docker_master.module.docker.null_resource.swarm_cluster, so provisioner block references cannot be detected
2019/10/16 11:05:14 [WARN] no schema for provisioner "file" is attached to module.docker_master.module.docker.null_resource.swarm_cluster, so provisioner block references cannot be detected
2019/10/16 11:05:14 [WARN] no schema for provisioner "remote-exec" is attached to module.docker_master.module.docker.null_resource.swarm_cluster, so provisioner block references cannot be detected
2019/10/16 11:05:14 [WARN] no schema for provisioner "remote-exec" is attached to module.docker_master.module.docker.null_resource.swarm_cluster, so provisioner block references cannot be detected
2019/10/16 11:05:14 [WARN] no schema for provisioner "remote-exec" is attached to module.docker_master.module.docker.null_resource.swarm_cluster, so provisioner block references cannot be detected
2019/10/16 11:05:14 [WARN] no schema for provisioner "remote-exec" is attached to module.docker_master.module.docker.null_resource.swarm_cluster, so provisioner block references cannot be detected
2019/10/16 11:05:14 [WARN] no schema for provisioner "remote-exec" is attached to module.docker_master.module.docker.null_resource.swarm_cluster, so provisioner block references cannot be detected
2019/10/16 11:05:14 [WARN] no schema for provisioner "local-exec" is attached to module.docker_master.module.docker.null_resource.swarm_cluster, so provisioner block references cannot be detected
2019/10/16 11:05:14 [WARN] no schema for provisioner "file" is attached to module.docker_master.module.docker.null_resource.swarm_cluster, so provisioner block references cannot be detected
2019/10/16 11:05:14 [WARN] no schema for provisioner "file" is attached to module.docker_master.module.docker.null_resource.swarm_cluster, so provisioner block references cannot be detected
2019/10/16 11:05:14 [WARN] no schema for provisioner "file" is attached to module.docker_master.module.docker.null_resource.swarm_cluster, so provisioner block references cannot be detected
2019/10/16 11:05:14 [WARN] no schema for provisioner "remote-exec" is attached to module.docker_master.module.docker.null_resource.swarm_cluster, so provisioner block references cannot be detected
2019/10/16 11:05:14 [WARN] no schema for provisioner "file" is attached to module.docker_master.module.docker.null_resource.swarm_cluster, so provisioner block references cannot be detected
2019/10/16 11:05:14 [WARN] no schema for provisioner "file" is attached to module.docker_master.module.docker.null_resource.swarm_cluster, so provisioner block references cannot be detected
2019/10/16 11:05:14 [WARN] no schema for provisioner "remote-exec" is attached to module.docker_master.module.docker.null_resource.swarm_cluster, so provisioner block references cannot be detected
2019/10/16 11:05:14 [WARN] no schema for provisioner "remote-exec" is attached to module.docker_master.module.docker.null_resource.swarm_cluster, so provisioner block references cannot be detected
2019/10/16 11:05:14 [WARN] no schema for provisioner "remote-exec" is attached to module.docker_master.module.docker.null_resource.swarm_cluster, so provisioner block references cannot be detected
2019/10/16 11:05:14 [WARN] no schema for provisioner "remote-exec" is attached to module.docker_master.module.docker.null_resource.swarm_cluster, so provisioner block references cannot be detected
2019/10/16 11:05:14 [DEBUG] ReferenceTransformer: "module.docker_master.module.docker.null_resource.swarm_cluster" references: []
2019/10/16 11:05:14 [TRACE] Completed graph transform *terraform.ReferenceTransform
Error: rpc error: code = Unavailable desc = transport is closing
er (no changes)
2019/10/16 11:05:14 [TRACE] Executing graph transform *terraform.RootTransformer
2019/10/16 11:05:14 [TRACE] Completed graph transform *terraform.RootTransformer (no changes)
2019/10/16 11:05:14 [TRACE] vertex "module.docker_master.module.docker.null_resource.swarm_cluster": entering dynamic subgraph
2019/10/16 11:05:14 [TRACE] dag/walk: updating graph
2019/10/16 11:05:14 [TRACE] dag/walk: added new vertex: "module.docker_master.module.docker.null_resource.swarm_cluster"
2019/10/16 11:05:14 [TRACE] dag/walk: visiting "module.docker_master.module.docker.null_resource.swarm_cluster"
2019/10/16 11:05:14 [TRACE] vertex "module.docker_master.module.docker.null_resource.swarm_cluster": starting visit (*terraform.NodeRefreshableManagedResourceInstance)
2019/10/16 11:05:14 [TRACE] NodeRefreshableManagedResourceInstance: module.docker_master.module.docker.null_resource.swarm_cluster has no existing state to refresh
2019/10/16 11:05:14 [TRACE] vertex "module.docker_master.module.docker.null_resource.swarm_cluster": evaluating
2019/10/16 11:05:14 [TRACE] [walkRefresh] Entering eval tree: module.docker_master.module.docker.null_resource.swarm_cluster
2019/10/16 11:05:14 [TRACE] module.docker_master.module.docker: eval: *terraform.EvalSequence
2019/10/16 11:05:14 [TRACE] module.docker_master.module.docker: eval: *terraform.EvalGetProvider

2019-10-16T11:05:14.109+0200 [DEBUG] plugin: plugin process exited: path=/infrastructure/.terraform/plugins/darwin_amd64/terraform-provider-scaleway_v1.11.0_x4 pid=59952
2019-10-16T11:05:14.109+0200 [DEBUG] plugin: plugin exited
2019/10/16 11:05:14 [TRACE] [walkRefresh] Exiting eval tree: module.docker_master.module.docker.module.node.provider.scaleway (close)
2019/10/16 11:05:14 [TRACE] vertex "module.docker_master.module.docker.module.node.provider.scaleway (close)": visit complete
2019/10/16 11:05:14 [TRACE] module.docker_master.module.docker: eval: *terraform.EvalReadState
2019/10/16 11:05:14 [TRACE] EvalReadState: reading state for module.docker_master.module.docker.null_resource.swarm_cluster
2019/10/16 11:05:14 [TRACE] EvalReadState: no state present for module.docker_master.module.docker.null_resource.swarm_cluster
2019/10/16 11:05:14 [TRACE] module.docker_master.module.docker: eval: *terraform.EvalDiff
2019/10/16 11:05:14 [TRACE] Re-validating config for "module.docker_master.module.docker.null_resource.swarm_cluster"
2019/10/16 11:05:14 [TRACE] GRPCProvider: ValidateResourceTypeConfig
2019/10/16 11:05:14 [TRACE] GRPCProvider: PlanResourceChange
2019/10/16 11:05:14 [WARN] Provider "null" produced an invalid plan for module.docker_master.module.docker.null_resource.swarm_cluster, but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .triggers: planned value cty.UnknownVal(cty.Map(cty.String)) does not match config value cty.MapVal(map[string]cty.Value{"server_ids":cty.UnknownVal(cty.String)})
2019/10/16 11:05:14 [TRACE] module.docker_master.module.docker: eval: *terraform.EvalWriteState
2019/10/16 11:05:14 [TRACE] EvalWriteState: writing current state object for module.docker_master.module.docker.null_resource.swarm_cluster
2019/10/16 11:05:14 [TRACE] module.docker_master.module.docker: eval: *terraform.EvalWriteDiff
2019/10/16 11:05:14 [TRACE] EvalWriteDiff: recorded
 Create change for module.docker_master.module.docker.null_resource.swarm_cluster
2019/10/16 11:05:14 [TRACE] [walkRefresh] Exiting eval tree: module.docker_master.module.docker.null_resource.swarm_cluster
2019/10/16 11:05:14 [TRACE] vertex "module.docker_master.module.docker.null_resource.swarm_cluster": visit complete
2019/10/16 11:05:14 [TRACE] vertex "module.docker_master.module.docker.null_resource.swarm_cluster": dynamic subgraph completed successfully
2019/10/16 11:05:14 [TRACE] vertex "module.docker_master.module.docker.null_resource.swarm_cluster": visit complete
2019/10/16 11:05:14 [TRACE] dag/walk: visiting "provider.null (close)"
2019/10/16 11:05:14 [TRACE] vertex "provider.null (close)": starting visit (*terraform.graphNodeCloseProvider)
2019/10/16 11:05:14 [TRACE] vertex "provider.null (close)": evaluating
2019/10/16 11:05:14 [TRACE] [walkRefresh] Entering eval tree: provider.null (close)
2019/10/16 11:05:14 [TRACE] <root>: eval: *terraform.EvalCloseProvider
2019/10/16 11:05:14 [TRACE] GRPCProvider: Close
2019-10-16T11:05:14.115+0200 [DEBUG] plugin: plugin process exited: path=/infrastructure/.terraform/plugins/darwin_amd64/terraform-provider-null_v2.1.2_x4 pid=59943
2019-10-16T11:05:14.115+0200 [DEBUG] plugin: plugin exited
2019/10/16 11:05:14 [TRACE] [walkRefresh] Exiting eval tree: provider.null (close)
2019/10/16 11:05:14 [TRACE] vertex "provider.null (close)": visit complete
2019/10/16 11:05:14 [TRACE] dag/walk: upstream of "root" errored, so skipping
2019/10/16 11:05:14 [TRACE] statemgr.Filesystem: removing lock metadata file .terraform.tfstate.lock.info
2019/10/16 11:05:14 [TRACE] statemgr.Filesystem: unlocking terraform.tfstate using fcntl flock
2019-10-16T11:05:14.118+0200 [DEBUG] plugin: plugin exited

Error: rpc error: code = Canceled desc = context canceled



Error: rpc error: code = Canceled desc = context canceled



Error: rpc error: code = Unavailable desc = transport is closing



Error: rpc error: code = Unavailable desc = transport is closing



Error: rpc error: code = Canceled desc = context canceled



Error: rpc error: code = Unavailable desc = transport is closing



Error: rpc error: code = Canceled desc = context canceled



Error: rpc error: code = Canceled desc = context canceled



Error: rpc error: code = Unavailable desc = transport is closing



Error: rpc error: code = Unavailable desc = transport is closing



Error: rpc error: code = Canceled desc = context canceled



Error: rpc error: code = Canceled desc = context canceled



Error: rpc error: code = Canceled desc = context canceled



Error: rpc error: code = Canceled desc = context canceled



Error: rpc error: code = Unavailable desc = transport is closing



Error: rpc error: code = Unavailable desc = transport is closing



Error: rpc error: code = Unavailable desc = transport is closing



Error: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: authentication handshake failed: write unix ->/var/folders/z5/90ptswb91rsb7lb29jc3lwsm0000gn/T/plugin251160601: write: broken pipe"



Error: rpc error: code = Unavailable desc = transport is closing

Содержание

1. rpc error: code = Canceled desc = context canceled #10280
2. Comments
3. Terraform Version
4. Affected Resource(s)
5. Terraform Configuration Files
6. Debug Output
7. Expected Behavior
8. Actual Behavior
9. Steps to Reproduce
10. Community Note
11. Footer
12. loki returned 500 and «rpc error: code = Canceled desc = context canceled» when handling large data query #3244
13. Comments
14. Error: rpc error: code = Canceled desc = context canceled #10084
15. Comments

Fault Locating

If the pod status is ImagePullBackOff, the image fails to be pulled. For details about how to view Kubernetes events, see Viewing Pod Events.

Troubleshooting Process

Determine the cause based on the event information, as listed in Table 1.

Figure 1 Troubleshooting process

Check Item 1: Whether imagePullSecret Is Specified When You Use kubectl to Create a Workload

If the workload status is abnormal and a Kubernetes event is displayed indicating that the pod fails to pull the image, check whether the imagePullSecrets field exists in the YAML file.

Items to Check

• If an image needs to be pulled from SWR, the name parameter must be set to default-secret.

  apiVersion: extensions/v1beta1
  kind: Deployment
  metadata:
    name: nginx
  spec:
    replicas: 1
    selector:
      matchLabels:
        app: nginx
    strategy:
      type: RollingUpdate
    template:
      metadata:
        labels:
          app: nginx
      spec:
        containers:
        - image: nginx 
          imagePullPolicy: Always
          name: nginx
        imagePullSecrets:
        - name: default-secret

• If an image needs to be pulled from a third-party image repository, the imagePullSecrets parameter must be set to the created secret name.

  When you use kubectl to create a workload from a third-party image, specify the imagePullSecret field, in which name indicates the name of the secret used to pull the image. For details about how to create a secret, see Using kubectl.

Check Item 2: Whether the Image Address Is Correct When a Third-Party Image Is Used

CCE allows you to create workloads using images pulled from third-party image repositories.

Enter the third-party image address according to requirements. The format must be ip:port/path/name:version or name:version. If no tag is specified, latest is used by default.

• For a private repository, enter an image address in the format of ip:port/path/name:version.
• For an open-source Docker repository, enter an image address in the format of name:version, for example, nginx:latest.

  Figure 2 Using a third-party image
  
  

The following information is displayed when you fail to pull an image due to incorrect image address provided.

Failed to pull image "nginx:v1.1": rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io: no such host

Solution

You can either edit your YAML file to modify the image address or log in to the CCE console to replace the image on the Upgrade tab page on the workload details page.

Check Item 3: Whether an Incorrect Secret Is Used When a Third-Party Image Is Used

Generally, a third-party image repository can be accessed only after authentication (using your account and password). CCE uses the secret authentication mode to pull images. Therefore, you need to create a secret for an image repository before pulling images from the repository.

Solution

If your secret is incorrect, images will fail to be pulled. In this case, create a new secret.

To create a secret, see Using kubectl.

Check Item 4: Whether the Node Disk Space Is Insufficient

A 100 GB data disk dedicated for Docker is attached to the new node. If the data disk space is insufficient, image fails to be pulled.

Figure 3 Data disk capacity (GB)

If the Kubernetes event contains the following information, the node has no disk space left for storing images. You need to clean up images or expand the disk capacity.

Run the lvs command to check the disk space for storing images on the node.

Run the following command to clean up images:

docker rmi -f {Image ID}

To expand the disk capacity, perform the following steps:

1. Expand the capacity of the data disk on the EVS console.
2. Log in to the CCE console and click the cluster. In the navigation pane, choose Nodes. Click More > Sync Server Data at the row containing the target node.
3. Log in to the target node.
4. Run the lsblk command to check the block device information of the node.

   A data disk is divided depending on the container storage Rootfs:

   • Overlayfs: No independent thin pool is allocated. Image data is stored in the dockersys disk.

     # lsblk
     NAME                MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
     sda                   8:0    0   50G  0 disk 
     └─sda1                8:1    0   50G  0 part /
     sdb                   8:16   0  200G  0 disk 
     ├─vgpaas-dockersys  253:0    0   90G  0 lvm  /var/lib/docker               # Space used by Docker.
     └─vgpaas-kubernetes 253:1    0   10G  0 lvm  /mnt/paas/kubernetes/kubelet  # Space used by Kubernetes.

     Run the following commands on the node to add the new disk capacity to the dockersys disk:

     pvresize /dev/sdb 
     lvextend -l+100%FREE -n vgpaas/dockersys
     resize2fs /dev/vgpaas/dockersys

   • Devicemapper: A thin pool is allocated to store image data.

     # lsblk
     NAME                                MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
     sda                                   8:0    0   50G  0 disk 
     └─sda1                                8:1    0   50G  0 part /
     sdb                                   8:16   0  200G  0 disk 
     ├─vgpaas-dockersys                  253:0    0   18G  0 lvm  /var/lib/docker    
     ├─vgpaas-thinpool_tmeta             253:1    0    3G  0 lvm                   
     │ └─vgpaas-thinpool                 253:3    0   67G  0 lvm                   # Thin pool space.
     │   ...
     ├─vgpaas-thinpool_tdata             253:2    0   67G  0 lvm  
     │ └─vgpaas-thinpool                 253:3    0   67G  0 lvm  
     │   ...
     └─vgpaas-kubernetes                 253:4    0   10G  0 lvm  /mnt/paas/kubernetes/kubelet

     Run the following commands on the node to add the new disk capacity to the thinpool disk:

     pvresize /dev/sdb 
     lvextend -l+100%FREE -n vgpaas/thinpool

Check Item 5: Whether the Remote Image Repository Uses an Unknown or Insecure Certificate

When a pod pulls an image from a third-party image repository that uses an unknown or insecure certificate, the image fails to be pulled from the node. The pod event list contains the event «Failed to pull the image» with the cause «x509: certificate signed by unknown authority».

Solution

1. Check the IP address and port number of the third-party image server for which the error message «unknown authority» is displayed.

   You can see the IP address and port number of the third-party image server for which the error is reported in the event information «Failed to pull image».

   Failed to pull image "bitnami/redis-cluster:latest": rpc error: code = Unknown desc = error pulling image configuration: Get https://production.cloudflare.docker.com/registry-v2/docker/registry/v2/blobs/sha256/e8/e83853f03a2e792614e7c1e6de75d63e2d6d633b4e7c39b9d700792ee50f7b56/data?verify=1636972064-AQbl5RActnudDZV%2F3EShZwnqOe8%3D: x509: certificate signed by unknown authority

   The IP address of the third-party image server is production.cloudflare.docker.com, and the default HTTPS port number is 443.

2. Load the root certificate of the third-party image server to the node where the third-party image is to be downloaded.

   Run the following commands on the EulerOS and CentOS nodes with {server_url}:{server_port} replaced with the IP address and port number obtained in Step 1, for example, production.cloudflare.docker.com:443:

   If the container engine of the node is containerd, replace systemctl restart docker with systemctl restart containerd.

   openssl s_client -showcerts -connect {server_url}:{server_port} < /dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > /etc/pki/ca-trust/source/anchors/tmp_ca.crt
   update-ca-trust
   systemctl restart docker

   Run the following command on Ubuntu nodes:

   openssl s_client -showcerts -connect {server_url}:{server_port} < /dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > /usr/local/share/ca-certificates/tmp_ca.crt
   update-ca-trust
   systemctl restart docker

Check Item 6: Whether the Image Size Is Too Large

The pod event list contains the event «Failed to pull image». This may be caused by a large image size.

Failed to pull image "XXX": rpc error: code = Unknown desc = context canceled

Log in to the node and run the docker pull command to manually pull the image. The image is successfully pulled.

Root Cause

The default value of image-pull-progress-deadline is 1 minute. If the image pull progress is not updated within 1 minute, image pull is canceled. If the node performance is poor or the image size is too large, the image may fail to be pulled and the workload may fail to be started.

Solution

• (Recommended) Method 1: Log in to the node, run the docker pull command to manually pull the image, and check whether imagePullPolicy of the workload is IfNotPresent (default policy configuration). In this case, the image that has been pulled to the local host is used to create the workload.
• Method 2: Modify the kubelet configuration parameters.

  For a cluster of v1.15 or later, run the following command:

  vi /opt/cloud/cce/kubernetes/kubelet/kubelet

  For a cluster earlier than v1.15, run the following command:

  vi /var/paas/kubernetes/kubelet/kubelet

  Add —image-pull-progress-deadline=30m to the end of the DAEMON_ARGS parameter. 30m indicates 30 minutes. You can change the value as required. The added configuration and the existing configuration are separated by a space.

  

  Run the following command to restart kubelet:

  systemctl restart kubelet

  Wait for a while and check whether the kubelet status is running.

  systemctl status kubelet

  

  The workload is started properly, and the image is successfully pulled.

Check Item 7: Connection to the Image Repository

Symptom

The following error message is displayed during workload creation:

Failed to pull image "docker.io/bitnami/nginx:1.22.0-debian-11-r3": rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Cause

Failed to connect to the image repository due to the disconnected network. SWR allows you to pull images from the official Docker repository. For image pulls from other repositories, you need to connect to the repositories first.

Solution

• Bind a public IP address (EIP) to the node that pulls the image.
• Upload the image to SWR and then pull the image from SWR.