02.28 擴展 Kubernetes 之 Kubectl Plugin

簡介

kubectl 是 重要的 kubernetes 管理/運維工具

kubectl 功能非常強大, 常見的命令使用方式可以參考 kubectl --help，或者這篇文章

這篇文章首先會簡單介紹幾個 kubectl 你可能不知道的小技巧，主要篇幅介紹 kubectl 的 plugin.

kubectl 小技巧

kubectl plguin

kubectl 支持一種簡單的 plugin 機制，支持通過 kubectl 調用另一個二進制，完成 kubernetes 相關的一些功能（其實對二進制執行的功能沒有任何限制）

目前這種機制並沒有在 kubectl 和 plugin 傳遞任何信息，只對 plugin 有兩點要求

krew

本地安裝方式很簡單，只需要把 可執行文件移動到比如 /usr/local/bin ，並且命名為 kubectl-$plugin_name即可。但是做好到插件如何分享，以及如何獲取別人安裝到插件呢。kubectl 提供了一個 krew（他本身也是一個插件） 工具提供了相應到功能

<code>Available Commands: help Help about any command info Show information about a kubectl plugin install Install kubectl plugins list List installed kubectl plugins search Discover kubectl plugins uninstall Uninstall plugins update Update the local copy of the plugin index upgrade Upgrade installed plugins to newer versions version Show krew version and diagnostics/<code>

檢索插件

可以使用命令 kubectl krew search, 但是這上面到介紹比較簡介，更好到方式是到 這個 index頁面 查看介紹和去對應的 github 倉庫查看詳細介紹。

<code>➜ kubectl krew searchNAME DESCRIPTION INSTALLEDaccess-matrix Show an RBAC access matrix for server resources noadvise-psp Suggests PodSecurityPolicies for cluster. noauth-proxy Authentication proxy to a pod or service nobulk-action Do bulk actions on Kubernetes resources. noca-cert Print the PEM CA certificate of the current clu... nocapture Triggers a Sysdig capture to troubleshoot the r... no.../<code>

安裝插件

使用 kubectl krew install

<code>➜ kubectl krew install custom-colsUpdated the local copy of plugin index.Installing plugin: custom-colsInstalled plugin: custom-cols\\ | Use this plugin: | kubectl custom-cols | Documentation: | https://github.com/webofmars/kubectl-custom-cols | Caveats: | \\ | | The list of templates is for now limited and can be retrieved with the --help option. | | Please feel free to submit any PR upstream (see github repo) to add more. | //WARNING: You installed a plugin from the krew-index plugin repository. These plugins are not audited for security by the Krew maintainers. Run them at your own risk.(base)/<code>

推薦插件介紹

change-ns

切換 ns, 用於切換 namespace，切換後會設置在 kubeconfig 中，後續的操作就不用再加 --namespaces 了。不過設置了 namespace 之後需要注意的是後續的命令默認 namespace 都是這個設置值了，如果你在 yaml 中沒有寫名 namespace，資源可能不會創建到你期望的 default 目錄下面了.

<code>➜ kubectl change-ns kube-systemnamespace changed to "kube-system"/<code>

cssh

ssh 到 kubernetes node 上面去，會自動從 node 信息中提取 外網 ip，並連接 tmux 嘗試做 ssh 登陸.

<code> > kubectl cssh --helpAllows users to SSH into Kubernetes nodes by opening a new tmux pane for each matching nodeOptions: -a, --address-type='ExternalIP': Node address type to query for (e.g. InternalIP/ExternalIP) -i, --identity-file='': Selects a file from which the identity (private key) for public key authentication is read -l, --selector='': Selector (label query) to filter on, supports '=', '==', and '!='.(e.g. -l key1=value1,key2=value2) -p, --port='': SSH port -u, --username='': SSH Username > kubectl cssh --username=ubuntu/<code>

image

debug/spy

兩個插件的作用差不多，目的都是進去 container 的命名空間進行 debug，不同到是debug 依賴一個 EphemeralContainers feature, 而 spy 不依賴

<code>➜ kubectl spy kube-dns-d5876cbfd-r8kh4loading spy pod...If you don't see a command prompt, try pressing enter./ # psPID USER TIME COMMAND 1 root 12:56 /dnsmasq-nanny -v=2 -logtostderr -configDir=/etc/k8s/dns/dnsmasq-nanny -restartDnsmasq=true -- -k --cache-size=1000 --log-facility=- --server=/cluster.local/12 16 root 14:55 /usr/sbin/dnsmasq -k --cache-size=1000 --log-facility=- --server=/cluster.local/127.0.0.1#10053 --server=/in-addr.arpa/127.0.0.1#10053 --server=/ip6.arpa/127.0 21 root 0:00 sh 28 root 0:00 ps/<code>

還有一個常用的 debug 命令也可以設做 alias

<code>kubectl run --rm -i -t test --image=byrnedo/alpine-curl --restart=Never --limits=cpu=10m,memory=10Mi --command=true /bin/sh/<code>

tree

用 tree的形式展示 Kubernetes objects

<code>➜ kubectl tree deployment kube-dnsNAMESPACE NAME READY REASON AGEkube-system Deployment/kube-dns - 286dkube-system ├─ReplicaSet/kube-dns-898dbbfc6 - 286dkube-system └─ReplicaSet/kube-dns-d5876cbfd - 141dkube-system ├─Pod/kube-dns-d5876cbfd-r8kh4 True 141dkube-system └─Pod/kube-dns-d5876cbfd-w8xvh True 141d/<code>

trace/sniff

分別用 bpftrace/tcpdump 工具對 pod 進行debug

<code>kubectl trace run ip-180-12-0-152.ec2.internal -f read.bt/<code>

image.png

<code>➜ kubectl sniff prometheus-k8s-0 -n monitoringINFO[0000] sniffing method: upload static tcpdumpINFO[0000] using tcpdump path at: '/Users/leiwang/.krew/store/sniff/v1.3.1/static-tcpdump'INFO[0000] no container specified, taking first container we found in pod.INFO[0000] selected container: 'prometheus'INFO[0000] sniffing on pod: 'prometheus-k8s-0' [namespace: 'monitoring', container: 'prometheus', filter: '', interface: 'any']INFO[0000] uploading static tcpdump binary from: '/Users/leiwang/.krew/store/sniff/v1.3.1/static-tcpdump' to: '/tmp/static-tcpdump'INFO[0000] uploading file: '/Users/leiwang/.krew/store/sniff/v1.3.1/static-tcpdump' to '/tmp/static-tcpdump' on container: 'prometheus'INFO[0000] executing command: '[/bin/sh -c ls -alt /tmp/static-tcpdump]' on container: 'prometheus', pod: 'prometheus-k8s-0', namespace: 'monitoring'INFO[0000] command: '[/bin/sh -c ls -alt /tmp/static-tcpdump]' executing successfully exitCode: '0', stdErr :''/<code>

dig

獲取關於 kubernetes node 的一切信息

<code># 這個工具並沒有在 krew index 發佈，所以用 go get 安裝go get -u github.com/sysdiglabs/kubectl-dig/cmd/kubectl-dig/<code>

warp

是 kubectl run + sshd-rsync 命令的合成，可以方便在在 pod 中執行一個本地文件.

<code># Start nodejs project in node containercd examples/nodejskubectl warp -i -t --image node testing-node -- npm run watch/<code>

get-all

get all, 慎用，和 kubectl 的 get --all 不同，這個命令是真的 all

<code>➜ kubectl get-allNAME NAMESPACE AGEcomponentstatus/scheduler <unknown>componentstatus/controller-manager <unknown>componentstatus/etcd-0 <unknown>configmap/token-100001343833 100001343833 69dconfigmap/token-100002873007 100002873007 77d....../<unknown>/<unknown>/<unknown>/<code>

grep

grep by name

<code>➜ kubectl grep pods -A nginxNAMESPACE NAME READY STATUS RESTART AGEdefault nginx-6dc5bfc797-vwdz7 1/1 Running 0 31dmonitoring prometheus-nginx-656ddc9c86-nq9dk 1/1 Running 1 181d/<code>

konfig

當你頻繁創建 kubernetes 集群需要 import 配置的時候很有用

<code>kubectl konfig import --save ~/Downloads/cls-5en24mcc-config/<code>

doctor

診斷 kubernetes 集群，目前做了一下的一些檢查

open-svc

利用 kubectl proxy 開啟遠程轉發，方便debug

<code>➜ kubectl open-svc prometheus-k8s -n monitoringStarting to serve on 127.0.0.1:8001Opening service/prometheus-k8s in the default browser.../<code>

image

resource-capacity/view-utilization

觀察 node/pod/namespace 等的資源申請和使用情況

<code>➜ kubectl resource-capacity --podsNODE NAMESPACE POD CPU REQUESTS CPU LIMITS MEMORY REQUESTS MEMORY LIMITS* * * 39728m (45%) 52108m (59%) 85069Mi (43%) 111368Mi (56%)10.0.0.10 * * 762m (19%) 1252m (31%) 585Mi (8%) 2399Mi (34%)10.0.0.10 kube-system ccs-log-collector-r4w6m 300m (7%) 1000m (25%) 238Mi (3%) 1907Mi (27%)10.0.0.10 kube-system etcd-10.0.0.10 0m (0%) 0m (0%) 0Mi (0%) 0Mi (0%)10.0.0.10 kube-system gpu-quota-admission-10.0.0.10 0m (0%) 0m (0%) 0Mi (0%) 0Mi (0%)10.0.0.10 kube-system kube-apiserver-10.0.0.10 0m (0%) 0m (0%) 0Mi (0%) 0Mi (0%)10.0.0.10 kube-system kube-controller-manager-10.0.0.10 0m (0%) 0m (0%) 0Mi (0%) 0Mi (0%)10.0.0.10 kube-system kube-dns-d5876cbfd-w8xvh 260m (6%) 0m (0%) 66Mi (0%) 162Mi (2%)10.0.0.10 kube-system kube-proxy-8m2cf 0m (0%) 0m (0%) 0Mi (0%) 0Mi (0%)10.0.0.10 kube-system kube-router-n86t5 100m (2%) 150m (3%) 100Mi (1%) 150Mi (2%)10.0.0.10 monitoring node-exporter-hdspv 102m (2%) 102m (2%) 180Mi (2%) 180Mi (2%)10.0.0.10 kube-system tke-bridge-agent-dlmhs 0m (0%) 0m (0%) 0Mi (0%) 0Mi (0%)10.0.0.10 kube-system tke-cni-agent-cdfhs 0m (0%) 0m (0%) 0Mi (0%) 0Mi (0%)10.0.0.13 * * 7632m (48%) 9782m (61%) 24290Mi (80%) 28981Mi (96%)10.0.0.13 100010987341 a2sc75rcc7x8x664-64ff96c4d5-wsks6 2000m (12%) 2000m (12%) 2048Mi (6%) 2048Mi (6%)10.0.0.13 100010987341 a2sc75rcc7x8x664-8459df9d4d-dc48m 2000m (12%) 2000m (12%) 10240Mi (34%) 10240Mi (34%)10.0.0.13 100010987341 a2sc75rcc7x8x664-8459df9d4d-kktgx 2000m (12%) 2000m (12%) 10240Mi (34%) 10240Mi (34%)10.0.0.13 kube-system ccs-log-collector-flmcq 300m (1%) 1000m (6%) 238Mi (0%) 1907Mi (6%)10.0.0.13 kube-system ip-masq-agent-lnq4c 0m (0%) 0m (0%) 0Mi (0%) 0Mi (0%)10.0.0.13 kube-system kube-proxy-sjn7j 0m (0%) 0m (0%) 0Mi (0%) 0Mi (0%)10.0.0.13 kube-system kube-router-d8dgw 100m (0%) 150m (0%) 100Mi (0%) 150Mi (0%)10.0.0.13 monitoring node-exporter-jths9 102m (0%) 102m (0%) 180Mi (0%) 180Mi (0%)10.0.0.13 monitoring prometheus-k8s-test-3-0 65m (0%) 15m (0%) 110Mi (0%) 60Mi (0%)10.0.0.13 monitoring prometheus-k8s-test-5-0 65m (0%) 15m (0%) 110Mi (0%) 60Mi (0%)10.0.0.13 kube-system service-controller-6fcb5fc4f4-cvmfz 250m (1%) 1000m (6%) 256Mi (0%) 1024Mi (3%)10.0.0.13 541004974 test-75c58cd5f7-qgm7m 750m (4%) 1500m (9%) 768Mi (2%) 3072Mi (10%)10.0.0.13 kube-system tke-bridge-agent-9n2k9 0m (0%) 0m (0%) 0Mi (0%) 0Mi (0%)10.0.0.13 kube-system tke-cni-agent-pmwpf 0m (0%) 0m (0%) 0Mi (0%) 0Mi (0%)/<code>

<code>➜ kubectl view-utilization node -hCPU : ▄▆▄▇▃▃▆▃▅▂▆▃Memory: ▇▅▂▆▁▃▃▄▇▂▇▃ CPU MemoryNode Req %R Lim %L Req %R Lim %L10.0.0.10 0.76 19% 1.3 31% 600M 8% 2.5G 36%10.0.0.13 6.9 43% 8.3 52% 23G 78% 25G 86%10.0.0.17 1 25% 3.3 82% 1G 15% 4.3G 63%10.0.0.31 1.5 38% 2.3 57% 2.5G 13% 4.3G 23%10.0.0.4 11 75% 13 86% 23G 74% 25G 83%10.0.0.44 0.57 14% 1.3 32% 640M 3% 2.4G 13%/<code>

實踐

實踐部分我們開發一個方便多 cluster 操作的簡單插件。在操作多集群到時候 切換 context 可能會比較麻煩，這裡我們參考 ansible 的 inventory 設計，支持寫入一個 配置文件，把 cluster 進行分組管理，選擇一個或者所有 group 的 cluster 執行 kubectl 命令。

插件代碼在 kubectl-clusters

下載腳本，安裝到 /usr/local/bin, 運行

<code># exec in all groups ➜ kubectl clusters all get pod[GROUP]: test --------------------------------------------------------------------------------[CLUSTER]: cls-test1 --------------------------------------------------[DEBUG] kubectl --context=cls-test1 --namespace=default get podNAME READY STATUS RESTARTS AGEnginx-6dc5bfc797-vwdz7 1/1 Running 0 32d[GROUP]: prod --------------------------------------------------------------------------------[CLUSTER]: cls-prod --------------------------------------------------[DEBUG] kubectl --context=cls-qcvhpqog get podNAME READY STATUS RESTARTS AGEa4cgfxv7srbfhbsn-78479b5cf7-f85d8 2/2 Running 0 37d# exec in single group➜ kubectl clusters prod get pod[GROUP]: prod --------------------------------------------------------------------------------[CLUSTER]: cls-prod --------------------------------------------------[DEBUG] kubectl --context=cls-qcvhpqog get podNAME READY STATUS RESTARTS AGEa4cgfxv7srbfhbsn-78479b5cf7-f85d8 2/2 Running 0 37d/<code>

提交到 krew index 目錄

參考