Cilium Hubble可观测性¶

在初步完成在扩展etcd环境安装cilium 之后，建议部署Cilium的核心组件 Hubble 来实现网络流量的观测。

Hubble是Cilium的可观测层，用于获得集群范围观察Kubernetes集群的网络和安全层。

备注

我在基于DNS轮询构建高可用Kubernetes 采用在扩展etcd环境安装cilium ，完成后默认还没有激活 hubble 。此时，通过以下命令检查:

cilium status

输出:

cilium安装完成后状态验证¶

    /¯¯\
 /¯¯\__/¯¯\    Cilium:         OK
 \__/¯¯\__/    Operator:       1 errors, 1 warnings
 /¯¯\__/¯¯\    Hubble:         disabled
 \__/¯¯\__/    ClusterMesh:    disabled
    \__/

Deployment        cilium-operator    Desired: 2, Ready: 1/2, Available: 1/2, Unavailable: 1/2
DaemonSet         cilium             Desired: 1, Ready: 1/1, Available: 1/1
Containers:       cilium-operator    Running: 1, Pending: 1
                  cilium             Running: 1
Cluster Pods:     2/2 managed by Cilium
Image versions    cilium             quay.io/cilium/cilium:v1.11.7@sha256:66a6f72a49e55e21278d07a99ff2cffa7565ed07f2578d54b5a92c1a492a6597: 1
                  cilium-operator    quay.io/cilium/operator-generic:v1.11.7@sha256:0f8ed5d815873d20848a360df3f2ebbd4116481ff817d3f295557801e0b45900: 2
Errors:           cilium-operator    cilium-operator                     1 pods of Deployment cilium-operator are not ready
Warnings:         cilium-operator    cilium-operator-68dffdc9f7-rph4w    pod is pending

激活Cilium的Hubble¶

有两种方式可以激活Hubble:

方法一: 使用 Cilcium CLI :

Cilium CLI激活Hubble¶

cilium hubble enable

这里可能会出现报错:

Error: Unable to enable Hubble: unable to retrieve helm values secret kube-system/cilium-cli-helm-values: secrets "cilium-cli-helm-values" not found

这是因为安装前通过 helm 安装，所以我改为方法二是可以直接成功等。但是，实际上 cilium 客户端如果调用 helm 还是需要使用 kube-system/cilium-cli-helm-values 这个secret密钥，例如后续 cilium hubble port-forward ，所以还是需要补充完整。参考 Cilium connectivity test fails: unable to retrieve helm values secret kube-system/cilium-cli-helm-values #927 ，这个问题似乎是版本bug，所以我采用升级Cilium 修复(我升级以后发现还是没有解决，所以最后还是采用helm方式)

方法二: 使用 helm :

使用Helm激活Hubble¶

helm upgrade cilium cilium/cilium --version 1.12.0 \
   --namespace kube-system \
   --reuse-values \
   --set hubble.relay.enabled=true \
   --set hubble.ui.enabled=true

我最初在扩展etcd环境安装cilium 安装的是 1.11.7 版本，不过上述尝试 cilium hubble enable 报错促使我先做升级Cilium ，然后再返回过来完成激活 hubble

提示信息:

使用Helm激活Hubble输出信息¶

Release "cilium" has been upgraded. Happy Helming!
NAME: cilium
LAST DEPLOYED: Tue Aug 16 16:40:26 2022
NAMESPACE: kube-system
STATUS: deployed
REVISION: 5
TEST SUITE: None
NOTES:
You have successfully installed Cilium with Hubble Relay and Hubble UI.

Your release version is 1.12.0.

For any further help, visit https://docs.cilium.io/en/v1.12/gettinghelp

激活 Hubble 之后，再次使用 cilium status 可以验证

cilium激活Hubble之后状态¶

    /¯¯\
 /¯¯\__/¯¯\    Cilium:         OK
 \__/¯¯\__/    Operator:       OK
 /¯¯\__/¯¯\    Hubble:         OK
 \__/¯¯\__/    ClusterMesh:    disabled
    \__/

Deployment        cilium-operator    Desired: 2, Ready: 2/2, Available: 2/2
Deployment        hubble-relay       Desired: 1, Ready: 1/1, Available: 1/1
DaemonSet         cilium             Desired: 8, Ready: 8/8, Available: 8/8
Deployment        hubble-ui          Desired: 1, Ready: 1/1, Available: 1/1
Containers:       cilium             Running: 8
                  cilium-operator    Running: 2
                  hubble-relay       Running: 1
                  hubble-ui          Running: 1
Cluster Pods:     11/11 managed by Cilium
Image versions    hubble-ui          quay.io/cilium/hubble-ui:v0.9.0@sha256:0ef04e9a29212925da6bdfd0ba5b581765e41a01f1cc30563cef9b30b457fea0: 1
                  hubble-ui          quay.io/cilium/hubble-ui-backend:v0.9.0@sha256:000df6b76719f607a9edefb9af94dfd1811a6f1b6a8a9c537cba90bf12df474b: 1
                  cilium             quay.io/cilium/cilium:v1.12.0@sha256:079baa4fa1b9fe638f96084f4e0297c84dd4fb215d29d2321dcbe54273f63ade: 8
                  cilium-operator    quay.io/cilium/operator-generic:v1.12.0@sha256:bb2a42eda766e5d4a87ee8a5433f089db81b72dd04acf6b59fcbb445a95f9410: 2
                  hubble-relay       quay.io/cilium/hubble-relay:v1.12.0@sha256:ca8033ea8a3112d838f958862fa76c8d895e3c8d0f5590de849b91745af5ac4d: 1

安装Hubble客户端¶

安装hubble客户端¶

export HUBBLE_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/hubble/master/stable.txt)
HUBBLE_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then HUBBLE_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-linux-${HUBBLE_ARCH}.tar.gz{,.sha256sum}
sha256sum --check hubble-linux-${HUBBLE_ARCH}.tar.gz.sha256sum
sudo tar xzvfC hubble-linux-${HUBBLE_ARCH}.tar.gz /usr/local/bin
rm hubble-linux-${HUBBLE_ARCH}.tar.gz{,.sha256sum}

验证Hubble API¶

要访问Hubble API，需要在本地主机创建一个端口转发，这样就能够让Hubble客户端通过本地 4245 访问Kubernetes集群的Hubble Relay 服务(原理是通过使用端口转发来访问Kubernetes集群应用 )

启动hubble端口转发:
```
cilium hubble port-forward&
```

这里报错:

Error: Unable to port forward: unable to retrieve helm values secret kube-system/cilium-cli-helm-values: secrets "cilium-cli-helm-values" not found

检查 kube-system 的 secret:
```
kubectl -n kube-system get secret
```

可以看到:

NAME                           TYPE                 DATA   AGE
cilium-ca                      Opaque               2      68m
cilium-etcd-secrets            Opaque               3      28d
hubble-ca-secret               Opaque               2      28d
hubble-relay-client-certs      kubernetes.io/tls    3      17m
hubble-server-certs            kubernetes.io/tls    3      28d
sh.helm.release.v1.cilium.v1   helm.sh/release.v1   1      28d
sh.helm.release.v1.cilium.v2   helm.sh/release.v1   1      3d1h
sh.helm.release.v1.cilium.v3   helm.sh/release.v1   1      2d17h
sh.helm.release.v1.cilium.v4   helm.sh/release.v1   1      68m
sh.helm.release.v1.cilium.v5   helm.sh/release.v1   1      17m

确实没有 cilium-cli-helm-values

参考 Cilium connectivity test fails: unable to retrieve helm values secret kube-system/cilium-cli-helm-values #927 仔细看了下代码，原来是修复了即使不存在 cilium-cli-helm-values 也可以，所以需要升级 cilium-cli 客户端

再次按照 cilium快速起步重新安装最新的 cilium-cli : 版本从 v0.11.11 升级到 v0.12.1

升级完成后，再次执行:

cilium hubble port-forward&

就可以看到成功输出信息:

[1] 1299844

验证CLI访问Hubble API:
```
hubble status
```

输出显示:

Healthcheck (via localhost:4245): Ok
Current/Max Flows: 32,760/32,760 (100.00%)
Flows/s: 52.29
Connected Nodes: 8/8

现在可以查询 flow API:
```
hubble observe
```

输出类似:

Aug 16 09:09:35.968: 10.0.6.185:57828 (remote-node) <> 10.0.0.160:4240 (health) to-overlay FORWARDED (TCP Flags: ACK)
Aug 16 09:09:35.969: 10.0.6.185:47590 (remote-node) <> 10.0.1.85:4240 (health) to-overlay FORWARDED (TCP Flags: ACK)
Aug 16 09:09:35.969: 10.0.6.185:33860 (remote-node) <> 10.0.5.52:4240 (health) to-overlay FORWARDED (TCP Flags: ACK)
...

下一步¶

在完成了Cilium Hubble激活之后，我们就可以:

命令行解析网络流量
通过 Cilium服务地图和Hubble UI 观测网络

参考¶

Setting up Hubble Observability