概述

Container Runtime 位于 Node 上,负责容器的整个生命周期,其中 Docker 应用最为广泛

image-20230726235313869

CRI 是 Kubernetes 定义的一组 gRPC 服务

  1. Dockershim 支持 CRI,代码耦合kubelet
  2. Docker 本身是不支持 CRI 的,但 Docker 内部的 containerd 是支持 CRI

image-20230726235627377

  1. Kubelet 作为客户端,基于 gRPC 框架,通过 Socket 和 Container Runtime 通信
  2. Container Runtime 提供 gRPC 服务
    • Image Service - 下载、检查和删除镜像
    • Runtime Service - 容器生命周期管理 + 与容器交互
      • 区分了 SandBox + Container

Push Image 并不在 CRI 中 – 开发环境用 Docker,生产环境用 containerd

image-20230727003001485

运行时分层

Runtime Impl
CRI - High-level - gRPC Dockershim / containerd / CRI-O
OCI - Low-level runC / kata-runtime / gVisor

OCI = Image Specification + Runtime Specification

  1. Image Specification
    • 定义了 OCI 镜像的标准
    • 高层级运行时下载一个 OCI 镜像,并把它解压成 OCI 运行时文件系统包(Filesystem bundle)
  2. Runtime Specification - 区分了 SandBoxContainer
    • 描述如何从 OCI 运行时文件系统包运行容器程序,并且定义它的配置、运行环境和生命周期
    • 另外为新容器设置 NamespaceCgroups,以及挂载 RootFS

image-20230727002131671

crictl

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
$ sudo crictl -h
NAME:
crictl - client for CRI

USAGE:
crictl [global options] command [command options] [arguments...]

VERSION:
v1.26.0

COMMANDS:
attach Attach to a running container
create Create a new container
exec Run a command in a running container
version Display runtime version information
images, image, img List images
inspect Display the status of one or more containers
inspecti Return the status of one or more images
imagefsinfo Return image filesystem info
inspectp Display the status of one or more pods
logs Fetch the logs of a container
port-forward Forward local port to a pod
ps List containers
pull Pull an image from a registry
run Run a new container inside a sandbox
runp Run a new pod
rm Remove one or more containers
rmi Remove one or more images
rmp Remove one or more pods
pods List pods
start Start one or more created containers
info Display information of the container runtime
stop Stop one or more running containers
stopp Stop one or more running pods
update Update one or more running containers
config Get and set crictl client configuration options
stats List container(s) resource usage statistics
statsp List pod resource usage statistics
completion Output shell completion code
checkpoint Checkpoint one or more running containers
help, h Shows a list of commands or help for one command

GLOBAL OPTIONS:
--config value, -c value Location of the client config file. If not specified and the default does not exist, the program's directory is searched as well (default: "/etc/crictl.yaml") [$CRI_CONFIG_FILE]
--debug, -D Enable debug mode (default: false)
--image-endpoint value, -i value Endpoint of CRI image manager service (default: uses 'runtime-endpoint' setting) [$IMAGE_SERVICE_ENDPOINT]
--runtime-endpoint value, -r value Endpoint of CRI container runtime service (default: uses in order the first successful one of [unix:///var/run/dockershim.sock unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock unix:///var/run/cri-dockerd.sock]). Default is now deprecated and the endpoint should be set instead. [$CONTAINER_RUNTIME_ENDPOINT]
--timeout value, -t value Timeout of connecting to the server in seconds (e.g. 2s, 20s.). 0 or less is set to default (default: 2s)
--help, -h show help (default: false)
--version, -v print the version (default: false)
1
2
$ cat /etc/crictl.yaml
runtime-endpoint: unix:///run/containerd/containerd.sock

version

1
2
3
4
5
6
7
8
$ sudo crictl version
Version: 0.1.0
RuntimeName: containerd
RuntimeVersion: 1.6.21
RuntimeApiVersion: v1

$ sudo crictl --version
crictl version v1.26.0

imagefsinfo

1
2
3
4
5
6
7
8
9
10
11
12
13
14
{
"status": {
"timestamp": "1690708414155867149",
"fsId": {
"mountpoint": "/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs"
},
"usedBytes": {
"value": "1971908608"
},
"inodesUsed": {
"value": "16668"
}
}
}

info

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
{
"status": {
"conditions": [
{
"type": "RuntimeReady",
"status": true,
"reason": "",
"message": ""
},
{
"type": "NetworkReady",
"status": true,
"reason": "",
"message": ""
}
]
},
"cniconfig": {
"PluginDirs": [
"/opt/cni/bin"
],
"PluginConfDir": "/etc/cni/net.d",
"PluginMaxConfNum": 1,
"Prefix": "eth",
"Networks": [
{
"Config": {
"Name": "cni-loopback",
"CNIVersion": "0.3.1",
"Plugins": [
{
"Network": {
"type": "loopback",
"ipam": {},
"dns": {}
},
"Source": "{\"type\":\"loopback\"}"
}
],
"Source": "{\n\"cniVersion\": \"0.3.1\",\n\"name\": \"cni-loopback\",\n\"plugins\": [{\n \"type\": \"loopback\"\n}]\n}"
},
"IFName": "lo"
},
{
"Config": {
"Name": "k8s-pod-network",
"CNIVersion": "0.3.1",
"Plugins": [
{
"Network": {
"type": "calico",
"ipam": {
"type": "calico-ipam"
},
"dns": {}
},
"Source": "{\"container_settings\":{\"allow_ip_forwarding\":false},\"datastore_type\":\"kubernetes\",\"ipam\":{\"assign_ipv4\":\"true\",\"assign_ipv6\":\"false\",\"type\":\"calico-ipam\"},\"kubernetes\":{\"k8s_api_root\":\"https://10.96.0.1:443\",\"kubeconfig\":\"/etc/cni/net.d/calico-kubeconfig\"},\"log_file_max_age\":30,\"log_file_max_count\":10,\"log_file_max_size\":100,\"log_file_path\":\"/var/log/calico/cni/cni.log\",\"log_level\":\"Info\",\"mtu\":0,\"nodename_file_optional\":false,\"policy\":{\"type\":\"k8s\"},\"type\":\"calico\"}"
},
{
"Network": {
"type": "bandwidth",
"capabilities": {
"bandwidth": true
},
"ipam": {},
"dns": {}
},
"Source": "{\"capabilities\":{\"bandwidth\":true},\"type\":\"bandwidth\"}"
},
{
"Network": {
"type": "portmap",
"capabilities": {
"portMappings": true
},
"ipam": {},
"dns": {}
},
"Source": "{\"capabilities\":{\"portMappings\":true},\"snat\":true,\"type\":\"portmap\"}"
}
],
"Source": "{\n\t\t\t \"name\": \"k8s-pod-network\",\n\t\t\t \"cniVersion\": \"0.3.1\",\n\t\t\t \"plugins\": [{\"container_settings\":{\"allow_ip_forwarding\":false},\"datastore_type\":\"kubernetes\",\"ipam\":{\"assign_ipv4\":\"true\",\"assign_ipv6\":\"false\",\"type\":\"calico-ipam\"},\"kubernetes\":{\"k8s_api_root\":\"https://10.96.0.1:443\",\"kubeconfig\":\"/etc/cni/net.d/calico-kubeconfig\"},\"log_file_max_age\":30,\"log_file_max_count\":10,\"log_file_max_size\":100,\"log_file_path\":\"/var/log/calico/cni/cni.log\",\"log_level\":\"Info\",\"mtu\":0,\"nodename_file_optional\":false,\"policy\":{\"type\":\"k8s\"},\"type\":\"calico\"},{\"capabilities\":{\"bandwidth\":true},\"type\":\"bandwidth\"},{\"capabilities\":{\"portMappings\":true},\"snat\":true,\"type\":\"portmap\"}] \n\t\t\t}"
},
"IFName": "eth0"
}
]
},
"config": {
"containerd": {
"snapshotter": "overlayfs",
"defaultRuntimeName": "runc",
"defaultRuntime": {
"runtimeType": "",
"runtimePath": "",
"runtimeEngine": "",
"PodAnnotations": [],
"ContainerAnnotations": [],
"runtimeRoot": "",
"options": {},
"privileged_without_host_devices": false,
"baseRuntimeSpec": "",
"cniConfDir": "",
"cniMaxConfNum": 0
},
"untrustedWorkloadRuntime": {
"runtimeType": "",
"runtimePath": "",
"runtimeEngine": "",
"PodAnnotations": [],
"ContainerAnnotations": [],
"runtimeRoot": "",
"options": {},
"privileged_without_host_devices": false,
"baseRuntimeSpec": "",
"cniConfDir": "",
"cniMaxConfNum": 0
},
"runtimes": {
"runc": {
"runtimeType": "io.containerd.runc.v2",
"runtimePath": "",
"runtimeEngine": "",
"PodAnnotations": [],
"ContainerAnnotations": [],
"runtimeRoot": "",
"options": {
"BinaryName": "",
"CriuImagePath": "",
"CriuPath": "",
"CriuWorkPath": "",
"IoGid": 0,
"IoUid": 0,
"NoNewKeyring": false,
"NoPivotRoot": false,
"Root": "",
"ShimCgroup": "",
"SystemdCgroup": true
},
"privileged_without_host_devices": false,
"baseRuntimeSpec": "",
"cniConfDir": "",
"cniMaxConfNum": 0
}
},
"noPivot": false,
"disableSnapshotAnnotations": true,
"discardUnpackedLayers": false,
"ignoreRdtNotEnabledErrors": false
},
"cni": {
"binDir": "/opt/cni/bin",
"confDir": "/etc/cni/net.d",
"maxConfNum": 1,
"confTemplate": "",
"ipPref": ""
},
"registry": {
"configPath": "/etc/containerd/certs.d",
"mirrors": {},
"configs": {},
"auths": {},
"headers": {
"User-Agent": [
"containerd/1.6.21"
]
}
},
"imageDecryption": {
"keyModel": "node"
},
"disableTCPService": true,
"streamServerAddress": "127.0.0.1",
"streamServerPort": "0",
"streamIdleTimeout": "4h0m0s",
"enableSelinux": false,
"selinuxCategoryRange": 1024,
"sandboxImage": "registry.aliyuncs.com/google_containers/pause:3.6",
"statsCollectPeriod": 10,
"systemdCgroup": false,
"enableTLSStreaming": false,
"x509KeyPairStreaming": {
"tlsCertFile": "",
"tlsKeyFile": ""
},
"maxContainerLogSize": 16384,
"disableCgroup": false,
"disableApparmor": false,
"restrictOOMScoreAdj": false,
"maxConcurrentDownloads": 3,
"disableProcMount": false,
"unsetSeccompProfile": "",
"tolerateMissingHugetlbController": true,
"disableHugetlbController": true,
"device_ownership_from_security_context": false,
"ignoreImageDefinedVolumes": false,
"netnsMountsUnderStateDir": false,
"enableUnprivilegedPorts": false,
"enableUnprivilegedICMP": false,
"containerdRootDir": "/var/lib/containerd",
"containerdEndpoint": "/run/containerd/containerd.sock",
"rootDir": "/var/lib/containerd/io.containerd.grpc.v1.cri",
"stateDir": "/run/containerd/io.containerd.grpc.v1.cri"
},
"golang": "go1.19.9",
"lastCNILoadStatus": "OK",
"lastCNILoadStatus.default": "OK"
}

stats

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
$ sudo crictl stats
CONTAINER CPU % MEM DISK INODES
053271e88c086 0.27 18.48MB 12.29kB 7
2dba506d8859e 0.00 13.53MB 94.21kB 25
311fe197dc860 0.16 31.01MB 49.15kB 16
31e6e9e4e6dc7 0.31 13.06MB 45.06kB 14
5047e03048a4b 4.17 54.32MB 73.73kB 21
5427ea6a4c67e 0.16 19.15MB 57.34kB 18
711b90ec6cbbe 0.15 13.71MB 45.06kB 14
8062467827d14 0.19 14.23MB 65.54kB 19
84b3edfddbec4 0.18 31.48MB 49.15kB 15
c0c4f851a53bb 1.05 89.09MB 864.3kB 203
e179d884842e5 0.00 4.997MB 57.34kB 15
e559dfdc8a723 1.67 31.09MB 49.15kB 16
e8368ffed5039 13.64 374.3MB 53.25kB 15
e9b168211a83a 1.95 37.25MB 36.86kB 11
f18a6adf5b23c 0.00 3.846MB 32.77kB 8

$ sudo crictl statsp
POD POD ID CPU % MEM
kube-scheduler-mac-k8s 1165bab25ae21 0.75 19.25MB
coredns-7f6cbbb7b8-rr9dr 14994dea79206 0.21 13.98MB
calico-apiserver-5bc5cf4559-l7m8r 6cdce974ca5b8 0.14 31.76MB
calico-kube-controllers-6f6579ddff-sq7hz 704b36894705f 0.00 14.91MB
csi-node-driver-j62g7 7d6e14cf214eb 0.00 9.638MB
tigera-operator-84469c6b57-62x2b 88103cf807ae5 0.19 32.32MB
kube-controller-manager-mac-k8s 95fb6fc590c28 2.30 55.23MB
coredns-7f6cbbb7b8-78bwt a31bfee878b49 0.30 13.73MB
kube-proxy-tfxn8 a732a1530dd18 0.01 14.87MB
calico-typha-64bcb7cddc-mgdnz bc8c16a440f5c 0.08 19.86MB
calico-node-tsjrx c1d2f125b07f6 1.65 178.6MB
kube-apiserver-mac-k8s d115ca9d37d81 10.70 375.2MB
etcd-mac-k8s d69c146236495 1.53 50.41MB
calico-apiserver-5bc5cf4559-9d6dq ecddecbd8bb0b 0.24 31.93MB

push - Push 并非 CRI 规范

1
2
$ sudo crictl push
No help topic for 'push'

主流 Runtime

一开始 Dockershim 的代码耦合Kubelet

image-20230727003529992

Docker vs containerd

  1. Docker 关于 Container Runtime 的核心组件为 containerd
    • 后来 containerd 也实现了 CRI(可以直接对接 Kubelet
    • Kubernetes 1.24 后,Kubelet 移除 Dockershim 代码,直接对接同样支持 CRIcontainerd
  2. 优势
    • 架构简化 - containerd 减少了 Docker 所需的处理模块 DockerdDockershim
    • 性能更优 - 对 Docker 支持的存储进行了优化
  3. 劣势 - 瑕不掩瑜
    • 不支持 zfs 存储驱动
    • 不支持对日志大小数量进行限制

Dockercontainerd镜像管理分开的,所以从 Docker 切换到 containerd 后,镜像需要重新拉取

image-20230727004037371

containerd 和 CRI-O 都符合 CRIOCI 标准,containerd稳定性性能上更优

image-20230727004408319

containerd CRI-O
性能 更优
功能 符合 CRIOCI 标准
稳定 稳定 未知

containerd

Docker 切换成 containerd

containerd 是符合 CRI 规范的 Runtime,crictl pods - SandBoxcrictl ps - Container

1
2
3
$ sudo systemctl stop kubelet
$ sudo systemctl stop docker
$ sudo systemctl stop containerd
1
2
$ sudo mkdir -p /etc/containerd
$ containerd config default | sudo tee /etc/containerd/config.toml
1
2
$ sudo sed -i s#registry.k8s.io/pause:3.6#registry.aliyuncs.com/google_containers/pause:3.6#g /etc/containerd/config.toml
$ sudo sed -i s#'SystemdCgroup = false'#'SystemdCgroup = true'#g /etc/containerd/config.toml
/etc/containerd/config.toml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
disabled_plugins = []
imports = []
oom_score = 0
plugin_dir = ""
required_plugins = []
root = "/var/lib/containerd"
state = "/run/containerd"
temp = ""
version = 2

[cgroup]
path = ""

[debug]
address = ""
format = ""
gid = 0
level = ""
uid = 0

[grpc]
address = "/run/containerd/containerd.sock"
gid = 0
max_recv_message_size = 16777216
max_send_message_size = 16777216
tcp_address = ""
tcp_tls_ca = ""
tcp_tls_cert = ""
tcp_tls_key = ""
uid = 0

[metrics]
address = ""
grpc_histogram = false

[plugins]

[plugins."io.containerd.gc.v1.scheduler"]
deletion_threshold = 0
mutation_threshold = 100
pause_threshold = 0.02
schedule_delay = "0s"
startup_delay = "100ms"

[plugins."io.containerd.grpc.v1.cri"]
device_ownership_from_security_context = false
disable_apparmor = false
disable_cgroup = false
disable_hugetlb_controller = true
disable_proc_mount = false
disable_tcp_service = true
enable_selinux = false
enable_tls_streaming = false
enable_unprivileged_icmp = false
enable_unprivileged_ports = false
ignore_image_defined_volumes = false
max_concurrent_downloads = 3
max_container_log_line_size = 16384
netns_mounts_under_state_dir = false
restrict_oom_score_adj = false
sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.6"
selinux_category_range = 1024
stats_collect_period = 10
stream_idle_timeout = "4h0m0s"
stream_server_address = "127.0.0.1"
stream_server_port = "0"
systemd_cgroup = false
tolerate_missing_hugetlb_controller = true
unset_seccomp_profile = ""

[plugins."io.containerd.grpc.v1.cri".cni]
bin_dir = "/opt/cni/bin"
conf_dir = "/etc/cni/net.d"
conf_template = ""
ip_pref = ""
max_conf_num = 1

[plugins."io.containerd.grpc.v1.cri".containerd]
default_runtime_name = "runc"
disable_snapshot_annotations = true
discard_unpacked_layers = false
ignore_rdt_not_enabled_errors = false
no_pivot = false
snapshotter = "overlayfs"

[plugins."io.containerd.grpc.v1.cri".containerd.default_runtime]
base_runtime_spec = ""
cni_conf_dir = ""
cni_max_conf_num = 0
container_annotations = []
pod_annotations = []
privileged_without_host_devices = false
runtime_engine = ""
runtime_path = ""
runtime_root = ""
runtime_type = ""

[plugins."io.containerd.grpc.v1.cri".containerd.default_runtime.options]

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes]

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
base_runtime_spec = ""
cni_conf_dir = ""
cni_max_conf_num = 0
container_annotations = []
pod_annotations = []
privileged_without_host_devices = false
runtime_engine = ""
runtime_path = ""
runtime_root = ""
runtime_type = "io.containerd.runc.v2"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
BinaryName = ""
CriuImagePath = ""
CriuPath = ""
CriuWorkPath = ""
IoGid = 0
IoUid = 0
NoNewKeyring = false
NoPivotRoot = false
Root = ""
ShimCgroup = ""
SystemdCgroup = true

[plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime]
base_runtime_spec = ""
cni_conf_dir = ""
cni_max_conf_num = 0
container_annotations = []
pod_annotations = []
privileged_without_host_devices = false
runtime_engine = ""
runtime_path = ""
runtime_root = ""
runtime_type = ""

[plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime.options]

[plugins."io.containerd.grpc.v1.cri".image_decryption]
key_model = "node"

[plugins."io.containerd.grpc.v1.cri".registry]
config_path = "/etc/containerd/certs.d"

[plugins."io.containerd.grpc.v1.cri".registry.auths]

[plugins."io.containerd.grpc.v1.cri".registry.configs]

[plugins."io.containerd.grpc.v1.cri".registry.headers]

[plugins."io.containerd.grpc.v1.cri".registry.mirrors]

[plugins."io.containerd.grpc.v1.cri".x509_key_pair_streaming]
tls_cert_file = ""
tls_key_file = ""

[plugins."io.containerd.internal.v1.opt"]
path = "/opt/containerd"

[plugins."io.containerd.internal.v1.restart"]
interval = "10s"

[plugins."io.containerd.internal.v1.tracing"]
sampling_ratio = 1.0
service_name = "containerd"

[plugins."io.containerd.metadata.v1.bolt"]
content_sharing_policy = "shared"

[plugins."io.containerd.monitor.v1.cgroups"]
no_prometheus = false

[plugins."io.containerd.runtime.v1.linux"]
no_shim = false
runtime = "runc"
runtime_root = ""
shim = "containerd-shim"
shim_debug = false

[plugins."io.containerd.runtime.v2.task"]
platforms = ["linux/arm64/v8"]
sched_core = false

[plugins."io.containerd.service.v1.diff-service"]
default = ["walking"]

[plugins."io.containerd.service.v1.tasks-service"]
rdt_config_file = ""

[plugins."io.containerd.snapshotter.v1.aufs"]
root_path = ""

[plugins."io.containerd.snapshotter.v1.btrfs"]
root_path = ""

[plugins."io.containerd.snapshotter.v1.devmapper"]
async_remove = false
base_image_size = ""
discard_blocks = false
fs_options = ""
fs_type = ""
pool_name = ""
root_path = ""

[plugins."io.containerd.snapshotter.v1.native"]
root_path = ""

[plugins."io.containerd.snapshotter.v1.overlayfs"]
root_path = ""
upperdir_label = false

[plugins."io.containerd.snapshotter.v1.zfs"]
root_path = ""

[plugins."io.containerd.tracing.processor.v1.otlp"]
endpoint = ""
insecure = false
protocol = ""

[proxy_plugins]

[stream_processors]

[stream_processors."io.containerd.ocicrypt.decoder.v1.tar"]
accepts = ["application/vnd.oci.image.layer.v1.tar+encrypted"]
args = ["--decryption-keys-path", "/etc/containerd/ocicrypt/keys"]
env = ["OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf"]
path = "ctd-decoder"
returns = "application/vnd.oci.image.layer.v1.tar"

[stream_processors."io.containerd.ocicrypt.decoder.v1.tar.gzip"]
accepts = ["application/vnd.oci.image.layer.v1.tar+gzip+encrypted"]
args = ["--decryption-keys-path", "/etc/containerd/ocicrypt/keys"]
env = ["OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf"]
path = "ctd-decoder"
returns = "application/vnd.oci.image.layer.v1.tar+gzip"

[timeouts]
"io.containerd.timeout.bolt.open" = "0s"
"io.containerd.timeout.shim.cleanup" = "5s"
"io.containerd.timeout.shim.load" = "5s"
"io.containerd.timeout.shim.shutdown" = "3s"
"io.containerd.timeout.task.state" = "2s"

[ttrpc]
address = ""
gid = 0
uid = 0
1
2
$ sudo vim /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
Environment="KUBELET_EXTRA_ARGS=--container-runtime=remote --container-runtime-endpoint=unix:///run/containerd/containerd.sock --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.6"
1
2
3
$ sudo systemctl daemon-reload
$ sudo systemctl restart containerd
$ sudo systemctl restart kubelet
1
2
3
$ cat <<EOF | sudo tee /etc/crictl.yaml
runtime-endpoint: unix:///run/containerd/containerd.sock
EOF

容器销毁 + 镜像失效

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
nginx latest 2002d33a54f7 3 weeks ago 192MB
quay.io/tigera/operator v1.30.4 ad14f750fd47 5 weeks ago 64.9MB
calico/typha v3.26.1 1846b2c90378 5 weeks ago 61.9MB
calico/kube-controllers v3.26.1 01cf521462df 5 weeks ago 68.8MB
calico/apiserver v3.26.1 00af1d6faf68 5 weeks ago 87.9MB
calico/cni v3.26.1 750dd81f472a 5 weeks ago 199MB
calico/node-driver-registrar v3.26.1 1075da421911 5 weeks ago 23.2MB
calico/csi v3.26.1 38afdba79f37 5 weeks ago 18.7MB
calico/pod2daemon-flexvol v3.26.1 07cc45ad90a6 5 weeks ago 10.7MB
calico/node v3.26.1 d5dd9023bb47 5 weeks ago 258MB
registry.aliyuncs.com/google_containers/kube-apiserver v1.22.2 9789bbb9890f 22 months ago 119MB
registry.aliyuncs.com/google_containers/kube-controller-manager v1.22.2 e596def24758 22 months ago 113MB
registry.aliyuncs.com/google_containers/kube-proxy v1.22.2 464a2b742200 22 months ago 97.4MB
registry.aliyuncs.com/google_containers/kube-scheduler v1.22.2 0e95bd29ff69 22 months ago 49.3MB
registry.aliyuncs.com/google_containers/pause 3.6 7d46a07936af 23 months ago 484kB
registry.aliyuncs.com/google_containers/etcd 3.5.0-0 2252d5eb703b 2 years ago 364MB
registry.aliyuncs.com/google_containers/coredns v1.8.4 6d3ffc2696ac 2 years ago 44.4MB
registry.aliyuncs.com/google_containers/pause 3.5 f7ff3c404263 2 years ago 484kB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
$ sudo crictl pods
POD ID CREATED STATE NAME NAMESPACE ATTEMPT RUNTIME
6cdce974ca5b8 About a minute ago Ready calico-apiserver-5bc5cf4559-l7m8r calico-apiserver 1 (default)
a31bfee878b49 About a minute ago Ready coredns-7f6cbbb7b8-78bwt kube-system 1 (default)
704b36894705f About a minute ago Ready calico-kube-controllers-6f6579ddff-sq7hz calico-system 1 (default)
14994dea79206 About a minute ago Ready coredns-7f6cbbb7b8-rr9dr kube-system 1 (default)
7d6e14cf214eb About a minute ago Ready csi-node-driver-j62g7 calico-system 1 (default)
ecddecbd8bb0b About a minute ago Ready calico-apiserver-5bc5cf4559-9d6dq calico-apiserver 1 (default)
a732a1530dd18 5 minutes ago Ready kube-proxy-tfxn8 kube-system 0 (default)
88103cf807ae5 5 minutes ago Ready tigera-operator-84469c6b57-62x2b tigera-operator 0 (default)
bc8c16a440f5c 5 minutes ago Ready calico-typha-64bcb7cddc-mgdnz calico-system 0 (default)
c1d2f125b07f6 5 minutes ago Ready calico-node-tsjrx calico-system 0 (default)
95fb6fc590c28 7 minutes ago Ready kube-controller-manager-mac-k8s kube-system 0 (default)
1165bab25ae21 7 minutes ago Ready kube-scheduler-mac-k8s kube-system 0 (default)
d69c146236495 7 minutes ago Ready etcd-mac-k8s kube-system 0 (default)
d115ca9d37d81 7 minutes ago Ready kube-apiserver-mac-k8s kube-system 0 (default)

$ sudo crictl ps
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD
e559dfdc8a723 00af1d6faf684 3 seconds ago Running calico-apiserver 1 ecddecbd8bb0b calico-apiserver-5bc5cf4559-9d6dq
84b3edfddbec4 ad14f750fd47c 4 minutes ago Running tigera-operator 3 88103cf807ae5 tigera-operator-84469c6b57-62x2b
2dba506d8859e 464a2b7422007 4 minutes ago Running kube-proxy 1 a732a1530dd18 kube-proxy-tfxn8
5427ea6a4c67e 1846b2c903785 4 minutes ago Running calico-typha 1 bc8c16a440f5c calico-typha-64bcb7cddc-mgdnz
053271e88c086 0e95bd29ff690 5 minutes ago Running kube-scheduler 4 1165bab25ae21 kube-scheduler-mac-k8s
5047e03048a4b e596def247585 6 minutes ago Running kube-controller-manager 3 95fb6fc590c28 kube-controller-manager-mac-k8s
e9b168211a83a 2252d5eb703b0 6 minutes ago Running etcd 1 d69c146236495 etcd-mac-k8s
e8368ffed5039 9789bbb9890f7 6 minutes ago Running kube-apiserver 2 d115ca9d37d81 kube-apiserver-mac-k8s

$ sudo crictl images
IMAGE TAG IMAGE ID SIZE
docker.io/calico/apiserver v3.26.1 00af1d6faf684 36.4MB
docker.io/calico/cni v3.26.1 750dd81f472ae 85.5MB
docker.io/calico/csi v3.26.1 38afdba79f37f 9.5MB
docker.io/calico/kube-controllers v3.26.1 01cf521462df2 29.2MB
docker.io/calico/node v3.26.1 d5dd9023bb474 84.7MB
docker.io/calico/pod2daemon-flexvol v3.26.1 07cc45ad90a68 5.58MB
docker.io/calico/typha v3.26.1 1846b2c903785 25.8MB
quay.io/tigera/operator v1.30.4 ad14f750fd47c 19.1MB
registry.aliyuncs.com/google_containers/coredns v1.8.4 6d3ffc2696ac2 12.3MB
registry.aliyuncs.com/google_containers/etcd 3.5.0-0 2252d5eb703b0 158MB
registry.aliyuncs.com/google_containers/kube-apiserver v1.22.2 9789bbb9890f7 28.4MB
registry.aliyuncs.com/google_containers/kube-controller-manager v1.22.2 e596def247585 27MB
registry.aliyuncs.com/google_containers/kube-proxy v1.22.2 464a2b7422007 34.4MB
registry.aliyuncs.com/google_containers/kube-scheduler v1.22.2 0e95bd29ff690 13.5MB
registry.aliyuncs.com/google_containers/pause 3.6 7d46a07936af9 254kB

$ k get po -A
NAMESPACE NAME READY STATUS RESTARTS AGE
calico-apiserver calico-apiserver-5bc5cf4559-9d6dq 1/1 Running 1 3d5h
calico-apiserver calico-apiserver-5bc5cf4559-l7m8r 1/1 Running 1 3d5h
calico-system calico-kube-controllers-6f6579ddff-sq7hz 1/1 Running 1 3d5h
calico-system calico-node-tsjrx 1/1 Running 1 3d5h
calico-system calico-typha-64bcb7cddc-mgdnz 1/1 Running 1 3d5h
calico-system csi-node-driver-j62g7 2/2 Running 2 3d5h
kube-system coredns-7f6cbbb7b8-78bwt 1/1 Running 1 3d5h
kube-system coredns-7f6cbbb7b8-rr9dr 1/1 Running 1 3d5h
kube-system etcd-mac-k8s 1/1 Running 1 3d5h
kube-system kube-apiserver-mac-k8s 1/1 Running 2 (10m ago) 3d5h
kube-system kube-controller-manager-mac-k8s 1/1 Running 3 3d5h
kube-system kube-proxy-tfxn8 1/1 Running 1 3d5h
kube-system kube-scheduler-mac-k8s 1/1 Running 4 (9m16s ago) 3d5h
tigera-operator tigera-operator-84469c6b57-62x2b 1/1 Running 3 3d5h

$ sudo crictl pull nginx
Image is up to date for sha256:2002d33a54f72d1333751d4d1b4793a60a635eac6e94a98daf0acea501580c4f

$ k apply -f deployment.yaml
deployment.apps/nginx-deployment created

$ k get deployments.apps
NAME READY UP-TO-DATE AVAILABLE AGE
nginx-deployment 1/1 1 1 72s

$ k get po -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-deployment-6799fc88d8-hlsdg 1/1 Running 0 89s 192.168.185.15 mac-k8s <none> <none>