crictl¶
crictl
是CRI兼容容器runtime的命令行接口。可以通过 crictl
在Kubernetes节点检查( inspect
)和debug容器runtime和应用程序。
安装cri-tools¶
安装crictl:
VERSION="v1.17.0" wget https://github.com/kubernetes-sigs/cri-tools/releases/download/$VERSION/crictl-$VERSION-linux-amd64.tar.gz sudo tar zxvf crictl-$VERSION-linux-amd64.tar.gz -C /usr/local/bin rm -f crictl-$VERSION-linux-amd64.tar.gz
安装critest:
VERSION="v1.17.0" wget https://github.com/kubernetes-sigs/cri-tools/releases/download/$VERSION/critest-$VERSION-linux-amd64.tar.gz sudo tar zxvf critest-$VERSION-linux-amd64.tar.gz -C /usr/local/bin rm -f critest-$VERSION-linux-amd64.tar.gz
注解
我在部署 私有云架构 的Kubernetes集群 基于DNSRR的高可用Kubernetes集群 时,完全摈弃了 Docker Atlas 改为推荐的 Kubernetes 容器运行时(Container Runtimes) containerd运行时(runtime) 。此时无法使用 docker
命令,需要采用Kubernetes规范的CRI工具,也就是 crictl
。这个工具安装采用了 安装containerd官方执行程序 步骤完成,详细步骤也可参考 z-k8s高可用Kubernetes集群准备
使用crictl¶
crictl
命令默认连接到 unix:///var/run/dockershim.sock
如果要连接到其他runtimes,需要设置 endpoint:
可以通过命令行参数
--runtime-endpoint
和--image-endpoint
可以通过设置环境变量
CONTAINER_RUNTIME_ENDPOINT
和CONTAINER_RUNTIME_ENDPOINT
可以通过配置文件的endpoint设置
--config=/etc/crictl.yaml
当前配置为 /etc/crictl.yaml
:
runtime-endpoint: unix:///var/run/containerd/containerd.sock
image-endpoint: unix:///var/run/containerd/containerd.sock
timeout: 10
#debug: true
debug: false
crictl命令案例¶
注解
实践案例见 基于DNS轮询构建高可用Kubernetes
列出pods:
crictl pods
输出显示类似:
POD ID CREATED STATE NAME NAMESPACE ATTEMPT RUNTIME
173ffdaeefab9 13 hours ago Ready kube-proxy-vwqsn kube-system 0 (default)
0952e1399e340 13 hours ago Ready kube-scheduler-z-k8s-m-1 kube-system 0 (default)
424cc7a5a9bfc 13 hours ago Ready kube-controller-manager-z-k8s-m-1 kube-system 0 (default)
7249ac0122d31 13 hours ago Ready kube-apiserver-z-k8s-m-1 kube-system 0 (default)
可以指定某个pods检查:
crictl pods --name kube-apiserver-z-k8s-m-1
列出所有本地镜像:
crictl images
显示类似:
IMAGE TAG IMAGE ID SIZE
k8s.gcr.io/coredns/coredns v1.8.6 a4ca41631cc7a 13.6MB
k8s.gcr.io/kube-apiserver v1.24.2 d3377ffb7177c 33.8MB
k8s.gcr.io/kube-apiserver v1.24.3 d521dd763e2e3 33.8MB
k8s.gcr.io/kube-controller-manager v1.24.2 34cdf99b1bb3b 31MB
k8s.gcr.io/kube-controller-manager v1.24.3 586c112956dfc 31MB
k8s.gcr.io/kube-proxy v1.24.2 a634548d10b03 39.5MB
k8s.gcr.io/kube-proxy v1.24.3 2ae1ba6417cbc 39.5MB
k8s.gcr.io/kube-scheduler v1.24.2 5d725196c1f47 15.5MB
k8s.gcr.io/kube-scheduler v1.24.3 3a5aa3a515f5d 15.5MB
k8s.gcr.io/pause 3.5 ed210e3e4a5ba 301kB
k8s.gcr.io/pause 3.6 6270bb605e12e 302kB
k8s.gcr.io/pause 3.7 221177c6082a8 311kB
使用 crictl images -a
还可以进一步显示完整的镜像ID( sha256
签名 )
列出主机上容器(这个命令类似 Docker Atlas 的
docker ps -a
: 注意是所有容器,包括了运行状态和停止状态的所有容器 ):crictl ps -a
显示输出:
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD
fd65e2a037600 2ae1ba6417cbc 16 hours ago Running kube-proxy 0 173ffdaeefab9 kube-proxy-vwqsn
5922644848149 3a5aa3a515f5d 16 hours ago Running kube-scheduler 0 0952e1399e340 kube-scheduler-z-k8s-m-1
58e48e8bc6861 586c112956dfc 16 hours ago Running kube-controller-manager 0 424cc7a5a9bfc kube-controller-manager-z-k8s-m-1
901b1dc06eed1 d521dd763e2e3 16 hours ago Running kube-apiserver 0 7249ac0122d31 kube-apiserver-z-k8s-m-1
如果只需要查看正在运行的容器(不显示停止的容器),则去掉 -a
参数
执行容器中的命令( 类似
docker
或者kubectl
提供的exec
指令,直接在容器内部运行命令 ):crictl exec -it fd65e2a037600 ls
举例显示:
bin boot dev etc go-runner home lib lib64 media mnt opt proc root run sbin srv sys tmp usr var
但是,并不是所有容器都可以执行,例如上文中只有 kube-proxy-vwqsn
这个pod提供都容器执行成功;而我尝试对 kube-apiserver-z-k8s-m-1
等对应容器执行 ls
命令都是失败:
crictl exec -it 901b1dc06eed1 ls
提示不能打开 /dev/pts/0
设备:
FATA[0000] execing command in container: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "3152d5e5f78f25a91d5e2a659c6e8036bf07978dbb8db5c95d1470089b968c9d": OCI runtime exec failed: exec failed: unable to start container process: open /dev/pts/0: operation not permitted: unknown
为何在容器内部没有 /dev/pts/0
设备? (我暂时没有找到解决方法)
检查容器日志(这里检查 apiserver 日志):
crictl logs 901b1dc06eed1
可以看到容器的详细日志
并且可以指定最后多少行日志,例如查看最后20行日志:
crictl logs --tail=20 901b1dc06eed1
运行pod sandbox¶
使用 crictl
运行pod sandobx可以用来debug容器运行时。
创建
pod-config.json
{
"metadata": {
"name": "nginx-sandbox",
"namespace": "default",
"attempt": 1,
"uid": "hdishd83djaidwnduwk28bcsb"
},
"log_directory": "/tmp",
"linux": {
}
}
执行以下命令运行sandbox:
crictl runp pod-config.json
注解
crictl runp
表示运行一个新pod(Run a new pod)crictl run
表示在一个sandbox内部运行一个新容器(Run a new container inside a sandbox)
我遇到报错:
E0719 20:50:53.394867 2319718 remote_runtime.go:201] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: expected cgroupsPath to be of format \"slice:prefix:name\" for systemd cgroups, got \"/k8s.io/81805934b02f89c87f1babefc460beb28679e184b777ebb8082942c3776a8d5b\" instead: unknown"
FATA[0001] run pod sandbox: rpc error: code = Unknown desc = failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: expected cgroupsPath to be of format "slice:prefix:name" for systemd cgroups, got "/k8s.io/81805934b02f89c87f1babefc460beb28679e184b777ebb8082942c3776a8d5b" instead: unknown
注解
Configuring a cgroup driver: Migrating to the systemd driver 提到将就节点转为使用 systemd
driver,这个步骤是在节点加入是完成
排查无法运行pod¶
原因是 kubelet
/ crictl
/ containerd
所使用的 cgroup 驱动不一致导致的: 有两种 cgroup 驱动,一种是 cgroupfs cgroup driver
,另一种是 systemd cgroup driver
:
我在 z-k8s高可用Kubernetes集群准备 时采用了
cgroup v2
(通过 Systemd进程管理器 ),同时在 配置 systemd cgroup驱动 ,这样 containerd运行时(runtime) 就会使用systemd cgroup driver
配置第一个管控节点(control plane ndoe) 默认已经激活了
kubelet
使用systemd cgroup driver
(从 Kubernetes 1.22 开始,默认kubelet
就是使用systemd cgroup driver
无需特别配置: 这点可以通过kubectl edit cm kubelet-config -n kube-system
查看,可以看到集群当前配置就是cgroupDriver: systemd
如果kubelet确实没有使用
systemd
的cgroup
的话,可以参考 Update the cgroup driver on all nodes 方法进行修订
但是
crictl
默认配置没有采用systemd cgroup driver
,可以通过以下命令检查:crictl info | grep system
可以看到输出:
"SystemdCgroup": true
"systemdCgroup": false,
crictl info
显示信息错误,这个异常可以参考 crictl info get wrong container runtime cgroupdrives when use containerd. #728 ,但是这个bug应该在 Change the type of CRI runtime option #5300 已经修复
我仔细看了以下 crictl info | grep systemd
输出,发现有2个地方和cgroup有关:
"SystemdCgroup": true
"systemdCgroup": false,
为何一个是 true
一个是 false
检查 /etc/containerd/config.toml
中也有几处和 systemdCgroup
有关:
[plugins]
...
[plugins."io.containerd.grpc.v1.cri"]
...
systemd_cgroup = false
...
...
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
...
SystemdCgroup = true
修订 containerd 配置 systemd_cgoup(失败,无需特别配置)¶
我尝试修改上面 systemd_cgroup = false
改为 systemd_cgroup = true
,结果重启 containerd
就挂掉了,节点 NotReady。
检查 containerd
的服务状态显示:
Jul 19 23:28:15 z-k8s-n-1 containerd[2321243]: time="2022-07-19T23:28:15.548636577+08:00" level=info msg="loading plugin \"io.containerd.tracing.processor.v1.otlp\"..." type=io.containerd.tracing.processor.v1
Jul 19 23:28:15 z-k8s-n-1 containerd[2321243]: time="2022-07-19T23:28:15.548669143+08:00" level=info msg="skip loading plugin \"io.containerd.tracing.processor.v1.otlp\"..." error="no OpenTelemetry endpoint: skip plugin" type=io.containerd.tracing.processor.v1
Jul 19 23:28:15 z-k8s-n-1 containerd[2321243]: time="2022-07-19T23:28:15.548696451+08:00" level=info msg="loading plugin \"io.containerd.internal.v1.tracing\"..." type=io.containerd.internal.v1
Jul 19 23:28:15 z-k8s-n-1 containerd[2321243]: time="2022-07-19T23:28:15.548761787+08:00" level=error msg="failed to initialize a tracing processor \"otlp\"" error="no OpenTelemetry endpoint: skip plugin"
Jul 19 23:28:15 z-k8s-n-1 containerd[2321243]: time="2022-07-19T23:28:15.548852549+08:00" level=info msg="loading plugin \"io.containerd.grpc.v1.cri\"..." type=io.containerd.grpc.v1
Jul 19 23:28:15 z-k8s-n-1 containerd[2321243]: time="2022-07-19T23:28:15.549391266+08:00" level=warning msg="failed to load plugin io.containerd.grpc.v1.cri" error="invalid plugin config: `systemd_cgroup` only works for runtime io.containerd.runtime.v1.linux"
Jul 19 23:28:15 z-k8s-n-1 containerd[2321243]: time="2022-07-19T23:28:15.549774946+08:00" level=info msg=serving... address=/run/containerd/containerd.sock.ttrpc
Jul 19 23:28:15 z-k8s-n-1 containerd[2321243]: time="2022-07-19T23:28:15.549940533+08:00" level=info msg=serving... address=/run/containerd/containerd.sock
此时也无法执行 crictl ps
,提示错误:
E0722 09:59:01.808419 2346431 remote_runtime.go:536] "ListContainers with filter from runtime service failed" err="rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService" filter="&ContainerFilter{Id:,State:&ContainerStateValue{State:CONTAINER_RUNNING,},PodSandboxId:,LabelSelector:map[string]string{},}"
FATA[0000] listing containers: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
注解
我仔细核对 containerd
配置 config.toml
,其实有2个地方配置 systemd cgroup driver
,之前在 z-k8s高可用Kubernetes集群准备 参考官方文档仅修订一处
参考 deprecated (?) systemd_cgroup still printed by “containerd config default” #4574 :
containerd 1.3.7 已经不再支持
io.containerd.grpc.v1.cri
这个插件配置成systemd_cgroup = true
,也就是如此操作是失败的:containerd config default > /etc/containerd/config.toml sed -i -e 's/systemd_cgroup = false/systemd_cgroup = true/' /etc/containerd/config.toml 启动 containerd 失败报错: Jul 19 23:28:15 z-k8s-n-1 containerd[2321243]: time="2022-07-19T23:28:15.549391266+08:00" level=warning msg="failed to load plugin io.containerd.grpc.v1.cri" error="invalid plugin config: `systemd_cgroup` only works for runtime io.containerd.runtime.v1.linux"
配置方法需要参考 how to configure systemd cgroup with 1.3.X #4203 的comment ,原来Kubernetes官方文档是正确的,现在配置
io.containerd.runc.v2
确实如 安装containerd官方执行程序 中配置的:
[plugins]
[plugins."io.containerd.grpc.v1.cri"]
[plugins."io.containerd.grpc.v1.cri".containerd]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
...
runtime_type = "io.containerd.runc.v2"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
...
SystemdCgroup = true
另外一个 how to configure systemd cgroup with 1.3.X #4203 的另一个comment 说明了有2个
systemd_cgroup
值:systemd_cgroup 是用于 shim v1 的配置,当 shim v1 废弃以后将会移除。也就是说,现在使用 containerd 不再需要修订这个配置
systemdCgroup 是现在真正需要调整当v2版本配置,对应当是
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
看起来不需要修改 containerd 的 config.toml 配置
注解
参考 config.toml SystemdCgroup not work #4900 讨论 验证containerd是否使用 systemd cgroups
不要看 crictl info
输出,这个输出非常让人困惑。
正确方法是查看 systemctl status containerd
可以看到启动的服务显示:
CGroup: /system.slice/containerd.service
├─2306099 /usr/local/bin/containerd-shim-runc-v2 -namespace k8s.io -id 2cd1b6245
├─2306139 /usr/local/bin/containerd-shim-runc-v2 -namespace k8s.io -id c380743d5
└─2346698 /usr/local/bin/containerd
此外检查 systemd-cgls
输出,就可以看到容器都是位于 systemd
的 cgroups
之下,这就表明正确使用了 systemd cgroups
创建容器¶
下载busybox镜像:
crictl pull busybox
显示:
Image is up to date for sha256:62aedd01bd8520c43d06b09f7a0f67ba9720bdc04631a8242c65ea995f3ecac8
创建 pod 配置:
{
"metadata": {
"name": "nginx-sandbox",
"namespace": "default",
"attempt": 1,
"uid": "hdishd83djaidwnduwk28bcsb"
},
"log_directory": "/tmp",
"linux": {
}
}
{
"metadata": {
"name": "busybox"
},
"image":{
"image": "busybox"
},
"command": [
"top"
],
"log_path":"busybox.log",
"linux": {
}
}
创建:
crictl create f84dd361f8dc51518ed291fbadd6db537b0496536c1d2d6c05ff943ce8c9a54f container-config.json pod-config.json
这里的字串是之前创建 sandbox返回的
参考¶
how to configure systemd cgroup with 1.3.X #4203 containerd新版支持systemd cgroup配置方法