Error response from daemon: layer does not exist

2022-12-27 · 12 min read

今天发现一堆pod处于pending状态，经过排查发现这些pod都被调度到同一个node上...

一、K8S Event信息

apiVersion: v1
kind: Event
metadata:
  name: bifrost-9f96c78fb-8b25m.173474bc390d2ea3
  namespace: prom
  creationTimestamp: '2022-12-26T21:21:53Z'
count: 1447
eventTime: null
firstTimestamp: '2022-12-26T21:21:53Z'
involvedObject:
  kind: Pod
  namespace: prom
  name: bifrost-9f96c78fb-8b25m
lastTimestamp: '2022-12-27T02:37:03Z'
message: >-
  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to
  inspect sandbox image "easzlab/pause-amd64:3.2": Error response from daemon:
  layer does not exist
reason: FailedCreatePodSandBox
reportingComponent: ''
reportingInstance: ''
source:
  component: kubelet
  host: 192.168.1.2
type: Warning

二、问题docker状态

[root@uat-node7 ~]# systemctl status docker.service
● docker.service - Docker Application Container Engine
   Loaded: loaded (/etc/systemd/system/docker.service; enabled; vendor preset: disabled)
   Active: active (running) since 二 2022-12-27 05:21:32 CST; 5h 18min ago
     Docs: http://docs.docker.io
  Process: 939 ExecStartPost=/sbin/iptables -I FORWARD -s 0.0.0.0/0 -j ACCEPT (code=exited, status=0/SUCCESS)
 Main PID: 938 (dockerd)
    Tasks: 38
   Memory: 152.6M
   CGroup: /system.slice/docker.service
           ├─ 938 /opt/kube/bin/dockerd
           └─1163 containerd --config /var/run/docker/containerd/containerd.toml --log-level warn

12月 27 10:39:34 uat-node7 dockerd[938]: time="2022-12-27T10:39:34.633134251+08:00" level=error msg="Handler for GET /v1.40/images/easzlab/pause-amd64:3.2/json returned error: layer does not exist"
12月 27 10:39:34 uat-node7 dockerd[938]: time="2022-12-27T10:39:34.633216593+08:00" level=error msg="Handler for GET /v1.40/images/easzlab/pause-amd64:3.2/json returned error: layer does not exist"
12月 27 10:39:35 uat-node7 dockerd[938]: time="2022-12-27T10:39:35.632983336+08:00" level=error msg="Handler for GET /v1.40/images/easzlab/pause-amd64:3.2/json returned error: layer does not exist"
12月 27 10:39:36 uat-node7 dockerd[938]: time="2022-12-27T10:39:36.633004314+08:00" level=error msg="Handler for GET /v1.40/images/easzlab/pause-amd64:3.2/json returned error: layer does not exist"
12月 27 10:39:36 uat-node7 dockerd[938]: time="2022-12-27T10:39:36.633028299+08:00" level=error msg="Handler for GET /v1.40/images/easzlab/pause-amd64:3.2/json returned error: layer does not exist"
12月 27 10:39:36 uat-node7 dockerd[938]: time="2022-12-27T10:39:36.633011326+08:00" level=error msg="Handler for GET /v1.40/images/easzlab/pause-amd64:3.2/json returned error: layer does not exist"
12月 27 10:39:37 uat-node7 dockerd[938]: time="2022-12-27T10:39:37.632943042+08:00" level=error msg="Handler for GET /v1.40/images/easzlab/pause-amd64:3.2/json returned error: layer does not exist"
12月 27 10:39:37 uat-node7 dockerd[938]: time="2022-12-27T10:39:37.632943242+08:00" level=error msg="Handler for GET /v1.40/images/easzlab/pause-amd64:3.2/json returned error: layer does not exist"
12月 27 10:39:38 uat-node7 dockerd[938]: time="2022-12-27T10:39:38.632928270+08:00" level=error msg="Handler for GET /v1.40/images/easzlab/pause-amd64:3.2/json returned error: layer does not exist"
12月 27 10:39:40 uat-node7 dockerd[938]: time="2022-12-27T10:39:40.632954191+08:00" level=error msg="Handler for GET /v1.40/images/easzlab/pause-amd64:3.2/json returned error: layer does not exist"

三、问题node日志

[root@uat-node7 lib]# journalctl --since "2022-12-27 05:00:00" | grep dockerd
12月 27 05:21:34 uat-node7 dockerd[938]: time="2022-12-27T05:21:34.103094137+08:00" level=warning msg="could not change group /var/run/docker.sock to docker: group docker not found"
12月 27 05:21:34 uat-node7 dockerd[938]: time="2022-12-27T05:21:34.622615445+08:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.devmapper" error="devmapper not configured"
12月 27 05:21:34 uat-node7 dockerd[938]: time="2022-12-27T05:21:34.634027294+08:00" level=warning msg="could not use snapshotter devmapper in metadata plugin" error="devmapper not configured"
12月 27 05:21:34 uat-node7 dockerd[938]: time="2022-12-27T05:21:34.822922309+08:00" level=warning msg="Your kernel does not support cgroup blkio weight"
12月 27 05:21:34 uat-node7 dockerd[938]: time="2022-12-27T05:21:34.822961932+08:00" level=warning msg="Your kernel does not support cgroup blkio weight_device"
12月 27 05:21:47 uat-node7 dockerd[938]: time="2022-12-27T05:21:47.151382213+08:00" level=error msg="(*service).Write failed" error="rpc error: code = OutOfRange desc = write @1052 must occur at current offset 9994" expected="sha256:4a1c4b21597c1b4415bdbecb28a3296c6b5e23ca4f9feeb599860a1dac6a0108" ref="manifest-sha256:4a1c4b21597c1b4415bdbecb28a3296c6b5e23ca4f9feeb599860a1dac6a0108" total=526
12月 27 05:21:47 uat-node7 dockerd[938]: time="2022-12-27T05:21:47.158780666+08:00" level=error msg="(*service).Write failed" error="rpc error: code = OutOfRange desc = write @1052 must occur at current offset 10520" expected="sha256:4a1c4b21597c1b4415bdbecb28a3296c6b5e23ca4f9feeb599860a1dac6a0108" ref="manifest-sha256:4a1c4b21597c1b4415bdbecb28a3296c6b5e23ca4f9feeb599860a1dac6a0108" total=526
12月 27 05:21:47 uat-node7 dockerd[938]: time="2022-12-27T05:21:47.163083352+08:00" level=error msg="(*service).Write failed" error="rpc error: code = OutOfRange desc = write @1052 must occur at current offset 11046" expected="sha256:4a1c4b21597c1b4415bdbecb28a3296c6b5e23ca4f9feeb599860a1dac6a0108" ref="manifest-sha256:4a1c4b21597c1b4415bdbecb28a3296c6b5e23ca4f9feeb599860a1dac6a0108" total=526
12月 27 05:21:47 uat-node7 dockerd[938]: time="2022-12-27T05:21:47.167259709+08:00" level=warning msg="Error persisting manifest" digest="sha256:4a1c4b21597c1b4415bdbecb28a3296c6b5e23ca4f9feeb599860a1dac6a0108" error="error committing manifest to content store: commit failed: write @1052 must occur at current offset 9994: unknown" remote="docker.io/easzlab/pause-amd64:3.2"
12月 27 05:21:47 uat-node7 dockerd[938]: time="2022-12-27T05:21:47.167294683+08:00" level=warning msg="error aborting content ingest" digest="sha256:4a1c4b21597c1b4415bdbecb28a3296c6b5e23ca4f9feeb599860a1dac6a0108" error="reference manifest-sha256:4a1c4b21597c1b4415bdbecb28a3296c6b5e23ca4f9feeb599860a1dac6a0108: not found" remote="docker.io/easzlab/pause-amd64:3.2"
12月 27 05:21:47 uat-node7 dockerd[938]: time="2022-12-27T05:21:47.167412619+08:00" level=warning msg="Error persisting manifest" digest="sha256:4a1c4b21597c1b4415bdbecb28a3296c6b5e23ca4f9feeb599860a1dac6a0108" error="error committing manifest to content store: commit failed: write @1052 must occur at current offset 11046: unknown" remote="docker.io/easzlab/pause-amd64:3.2"
12月 27 05:21:47 uat-node7 dockerd[938]: time="2022-12-27T05:21:47.167303140+08:00" level=warning msg="error aborting content ingest" digest="sha256:4a1c4b21597c1b4415bdbecb28a3296c6b5e23ca4f9feeb599860a1dac6a0108" error="reference manifest-sha256:4a1c4b21597c1b4415bdbecb28a3296c6b5e23ca4f9feeb599860a1dac6a0108: not found" remote="docker.io/easzlab/pause-amd64:3.2"
12月 27 05:21:47 uat-node7 dockerd[938]: time="2022-12-27T05:21:47.167485709+08:00" level=warning msg="Error persisting manifest" digest="sha256:4a1c4b21597c1b4415bdbecb28a3296c6b5e23ca4f9feeb599860a1dac6a0108" error="error committing manifest to content store: commit failed: write @1052 must occur at current offset 10520: unknown" remote="docker.io/easzlab/pause-amd64:3.2"
12月 27 05:21:47 uat-node7 dockerd[938]: time="2022-12-27T05:21:47.194273283+08:00" level=warning msg="Published ports are discarded when using host network mode"
12月 27 05:21:47 uat-node7 dockerd[938]: time="2022-12-27T05:21:47.194649449+08:00" level=error msg="Handler for POST /v1.40/containers/create returned error: layer does not exist"
12月 27 05:21:47 uat-node7 dockerd[938]: time="2022-12-27T05:21:47.195477445+08:00" level=error msg="Handler for POST /v1.40/containers/create returned error: layer does not exist"
12月 27 05:21:47 uat-node7 dockerd[938]: time="2022-12-27T05:21:47.197857910+08:00" level=warning msg="Published ports are discarded when using host network mode"
12月 27 05:21:47 uat-node7 dockerd[938]: time="2022-12-27T05:21:47.198217431+08:00" level=error msg="Handler for POST /v1.40/containers/create returned error: layer does not exist"
12月 27 05:21:47 uat-node7 dockerd[938]: time="2022-12-27T05:21:47.273908435+08:00" level=error msg="(*service).Write failed" error="rpc error: code = OutOfRange desc = write @1052 must occur at current offset 11572" expected="sha256:4a1c4b21597c1b4415bdbecb28a3296c6b5e23ca4f9feeb599860a1dac6a0108" ref="manifest-sha256:4a1c4b21597c1b4415bdbecb28a3296c6b5e23ca4f9feeb599860a1dac6a0108" total=526
12月 27 05:21:47 uat-node7 dockerd[938]: time="2022-12-27T05:21:47.274287874+08:00" level=warning msg="error aborting content ingest" digest="sha256:4a1c4b21597c1b4415bdbecb28a3296c6b5e23ca4f9feeb599860a1dac6a0108" error="reference manifest-sha256:4a1c4b21597c1b4415bdbecb28a3296c6b5e23ca4f9feeb599860a1dac6a0108: not found" remote="docker.io/easzlab/pause-amd64:3.2"
12月 27 05:21:47 uat-node7 dockerd[938]: time="2022-12-27T05:21:47.274319283+08:00" level=warning msg="Error persisting manifest" digest="sha256:4a1c4b21597c1b4415bdbecb28a3296c6b5e23ca4f9feeb599860a1dac6a0108" error="error committing manifest to content store: commit failed: write @1052 must occur at current offset 11572: unknown" remote="docker.io/easzlab/pause-amd64:3.2"
12月 27 05:21:47 uat-node7 dockerd[938]: time="2022-12-27T05:21:47.277633332+08:00" level=error msg="Handler for POST /v1.40/containers/create returned error: layer does not exist"
12月 27 05:21:53 uat-node7 dockerd[938]: time="2022-12-27T05:21:53.408680415+08:00" level=error msg="Handler for GET /v1.40/images/easzlab/pause-amd64:3.2/json returned error: layer does not exist"
12月 27 05:21:53 uat-node7 dockerd[938]: time="2022-12-27T05:21:53.408681500+08:00" level=error msg="Handler for GET /v1.40/images/easzlab/pause-amd64:3.2/json returned error: layer does not exist"

系统上一次系统启动时间

[root@uat-node7 ~]# who -b
         系统引导 2022-12-27 05:21

四、原因 & 解决

12月 27 05:21:47 uat-node7 dockerd[938]: time="2022-12-27T05:21:47.151382213+08:00" level=error msg="(*service).Write failed" error="rpc error: code = OutOfRange desc = write @1052 must occur at current offset 9994" expected="sha256:4a1c4b21597c1b4415bdbecb28a3296c6b5e23ca4f9feeb599860a1dac6a0108" ref="manifest-sha256:4a1c4b21597c1b4415bdbecb28a3296c6b5e23ca4f9feeb599860a1dac6a0108" total=526

经合日志分析，发现 uat-node7 在当天早晨 05:21 发生了意外重启，导致 docker imagefs 部分数据损坏，kubelet 无法正常创建 PodSandbox。重启docker后恢复正常。