在扩展etcd环境安装cilium

cilium快速起步 介绍了快速安装cilium的方法，但是只是适合比较简单环境，即采用堆叠etcd模式环境，而在采用外部独立etcd集群，则需要做一些调整，把 etcd 集群配置传递给cilium安装程序

cilium使用外部KV存储(通常是 etcd - 分布式kv存储 )优点:

  • 使用外部扩展KV存储可以解决Kubernetes大量事件传播开销极大的问题

  • 使用外部扩展KV存储可以避免cilium将状态存储在Kubernetes自定义资源(CRD)中

  • 使用外部扩展KV存储可以支持更多pod和节点

运行cilium环境要求:

  • Kubernetes >= 1.16

  • Linux kernel >= 4.9

  • Kubernetes使用CNI模式

  • 在所有工作节点挂载了 eBPF 文件系统

  • 建议 : 在 kube-controller-manager 上激活 PodCIDR 分配( --allocate-node-cidrs )

配置Cilium

Cilium需要在ConfigMap中配置扩展外部KV存储，这个配置是通过 helm 完成的，所以需要首先安装 helm3 :

在Linux平台安装helm
version=3.12.2
wget https://get.helm.sh/helm-v${version}-linux-amd64.tar.gz
tar -zxvf helm-v${version}-linux-amd64.tar.gz
sudo mv linux-amd64/helm /usr/local/bin/helm

  • 设置cilium Helm仓库:

设置cilium Helm仓库
helm repo add cilium https://helm.cilium.io/

此时提示:

"cilium" has been added to your repositories

  • 通过 helm 部署Cilium:

为cilium配置访问etcd的Kubernetes secret，安装cilium采用SSL模式访问etcd
VERSION=1.11.7

ETCD_0_IP=192.168.6.204
ETCD_1_IP=192.168.6.205
ETCD_2_IP=192.168.6.206

kubectl create secret generic -n kube-system cilium-etcd-secrets \
    --from-file=etcd-client-ca.crt=/etc/kubernetes/pki/etcd/ca.crt \
    --from-file=etcd-client.key=/etc/kubernetes/pki/apiserver-etcd-client.key \
    --from-file=etcd-client.crt=/etc/kubernetes/pki/apiserver-etcd-client.crt

helm install cilium cilium/cilium --version ${VERSION} \
  --namespace kube-system \
  --set etcd.enabled=true \
  --set etcd.ssl=true \
  --set "etcd.endpoints[0]=https://${ETCD_0_IP}:2379" \
  --set "etcd.endpoints[1]=https://${ETCD_1_IP}:2379" \
  --set "etcd.endpoints[2]=https://${ETCD_2_IP}:2379"

执行安装以后提示信息:

W0719 01:01:50.091851  995900 warnings.go:70] spec.template.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[1].matchExpressions[0].key: beta.kubernetes.io/os is deprecated since v1.14; use "kubernetes.io/os" instead
NAME: cilium
LAST DEPLOYED: Tue Jul 19 01:01:48 2022
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
You have successfully installed Cilium with Hubble.

Your release version is 1.11.7.

For any further help, visit https://docs.cilium.io/en/v1.11/gettinghelp

此时，在安装了 cilium 这样的 CNI 之后，在 基于DNS轮询构建高可用Kubernetes 部署过程中没有运行起来的coredns容器就能够分配IP地址并运行起来:

安装cilium CNI网络之后coredns就可以运行，此时 kubectl get pods 输出可以看到所有pods已分配IP并运行
NAME                                READY   STATUS    RESTARTS   AGE     IP              NODE        NOMINATED NODE   READINESS GATES
cilium-7c5nv                        1/1     Running   0          8m40s   192.168.6.101   z-k8s-m-1   <none>           <none>
cilium-operator-68dffdc9f7-cqvqr    1/1     Running   0          8m40s   192.168.6.101   z-k8s-m-1   <none>           <none>
cilium-operator-68dffdc9f7-rph4w    0/1     Pending   0          8m40s   <none>          <none>      <none>           <none>
coredns-6d4b75cb6d-jnfmj            1/1     Running   0          25h     10.0.0.241      z-k8s-m-1   <none>           <none>
coredns-6d4b75cb6d-nm5fz            1/1     Running   0          25h     10.0.0.141      z-k8s-m-1   <none>           <none>
kube-apiserver-z-k8s-m-1            1/1     Running   0          25h     192.168.6.101   z-k8s-m-1   <none>           <none>
kube-controller-manager-z-k8s-m-1   1/1     Running   0          25h     192.168.6.101   z-k8s-m-1   <none>           <none>
kube-proxy-vwqsn                    1/1     Running   0          25h     192.168.6.101   z-k8s-m-1   <none>           <none>
kube-scheduler-z-k8s-m-1            1/1     Running   0          25h     192.168.6.101   z-k8s-m-1   <none>           <none>

验证安装

  • 安装最新版本Cilium CLI:

安装cilium CLI
curl -L --remote-name-all https://github.com/cilium/cilium-cli/releases/latest/download/cilium-linux-amd64.tar.gz{,.sha256sum}
sha256sum --check cilium-linux-amd64.tar.gz.sha256sum
sudo tar xzvfC cilium-linux-amd64.tar.gz /usr/local/bin
rm cilium-linux-amd64.tar.gz{,.sha256sum}

  • 检查:

    cilium status

此时屏幕会输出:

cilium安装完成后状态验证
    /¯¯\
 /¯¯\__/¯¯\    Cilium:         OK
 \__/¯¯\__/    Operator:       1 errors, 1 warnings
 /¯¯\__/¯¯\    Hubble:         disabled
 \__/¯¯\__/    ClusterMesh:    disabled
    \__/

Deployment        cilium-operator    Desired: 2, Ready: 1/2, Available: 1/2, Unavailable: 1/2
DaemonSet         cilium             Desired: 1, Ready: 1/1, Available: 1/1
Containers:       cilium-operator    Running: 1, Pending: 1
                  cilium             Running: 1
Cluster Pods:     2/2 managed by Cilium
Image versions    cilium             quay.io/cilium/cilium:v1.11.7@sha256:66a6f72a49e55e21278d07a99ff2cffa7565ed07f2578d54b5a92c1a492a6597: 1
                  cilium-operator    quay.io/cilium/operator-generic:v1.11.7@sha256:0f8ed5d815873d20848a360df3f2ebbd4116481ff817d3f295557801e0b45900: 2
Errors:           cilium-operator    cilium-operator                     1 pods of Deployment cilium-operator are not ready
Warnings:         cilium-operator    cilium-operator-68dffdc9f7-rph4w    pod is pending

备注

这里显示 cilium-operator 配置了2个pod，但只有1个pod运行是因为目前我正在bootstrap管控节点，当前只运行了一个管控节点，所以deployment配置了2个replicas只能先运行1个。稍后完成管控节点扩容后就能保证有足够master节点运行 cilium-operator

  • 基于DNS轮询构建高可用Kubernetes 第一个管控节点安装好 cilium 之后 CoreDNS 就可以分配到IP地址运行起来。接下来可以完成第二、第三个管控节点以及各个工作节点添加，直到整个集群建立

备注

只需要在第一个管控节点上安装 helm 以及 cilium 客户端(验证)，通过 helm 就可以为整个集群安装部署 cilium 网络，扩展安装非常方便。

cilium connectivity test 是一个非常 赞 的探针功能，自动构建了同一个worker节点和不同worker节点上的 cilium-test 容器相互间进行网络联通测试:

$ kubectl get pods -o wide -n cilium-test
NAME                              READY   STATUS    RESTARTS   AGE     IP           NODE        NOMINATED NODE   READINESS GATES
client-7df6cfbf7b-kc4mz           1/1     Running   0          5m50s   10.0.3.103   z-k8s-n-1   <none>           <none>
client2-547996d7d8-pv4n5          1/1     Running   0          4m38s   10.0.3.210   z-k8s-n-1   <none>           <none>
echo-other-node-d79544ccf-hxjzb   2/2     Running   0          57s     10.0.7.132   z-k8s-n-4   <none>           <none>
echo-same-node-5d466d5444-kbgzl   2/2     Running   0          2m5s    10.0.3.68    z-k8s-n-1   <none>           <none>

完成测试后会在终端显示测试结果

参考