簡介
kubectl 是 重要的 kubernetes 管理/運維工具
kubectl 功能非常強大, 常見的命令使用方式可以參考 kubectl --help，或者這篇文章
這篇文章首先會簡單介紹幾個 kubectl 你可能不知道的小技巧，主要篇幅介紹 kubectl 的 plugin.
kubectl 小技巧
- 設置自動補全 kubectl completion zsh
- 檢查資源 SPEC (有沒有遇到過 想看SPEC 只能去查API文檔或者翻代碼的情況?) kubectl explain [--recursive]
- 給常用的命令設置 alias, 比如筆者常用的: kns="kubectl -n kube-system", kna="kubectl --all-namespaces=true, kcc="kubectl config use-context, kgy="kubectl get -o yaml", 或者直接使用這個項目 生成的 alias, 這個項目使用一套規則生成了 800 多個 aliases
kubectl plguin
kubectl 支持一種簡單的 plugin 機制，支持通過 kubectl 調用另一個二進制，完成 kubernetes 相關的一些功能（其實對二進制執行的功能沒有任何限制）
目前這種機制並沒有在 kubectl 和 plugin 傳遞任何信息，只對 plugin 有兩點要求
- plugin 為可執行文件
- plugin 可執行文件的名字為 kubectl-$plugin_name
krew
本地安裝方式很簡單，只需要把 可執行文件移動到比如 /usr/local/bin ，並且命名為 kubectl-$plugin_name即可。但是做好到插件如何分享，以及如何獲取別人安裝到插件呢。kubectl 提供了一個 krew（他本身也是一個插件） 工具提供了相應到功能
<code>Available Commands: help Help about any command info Show information about a kubectl plugin install Install kubectl plugins list List installed kubectl plugins search Discover kubectl plugins uninstall Uninstall plugins update Update the local copy of the plugin index upgrade Upgrade installed plugins to newer versions version Show krew version and diagnostics/<code>
檢索插件
可以使用命令 kubectl krew search, 但是這上面到介紹比較簡介，更好到方式是到 這個 index頁面 查看介紹和去對應的 github 倉庫查看詳細介紹。
<code>➜ kubectl krew searchNAME DESCRIPTION INSTALLEDaccess-matrix Show an RBAC access matrix for server resources noadvise-psp Suggests PodSecurityPolicies for cluster. noauth-proxy Authentication proxy to a pod or service nobulk-action Do bulk actions on Kubernetes resources. noca-cert Print the PEM CA certificate of the current clu... nocapture Triggers a Sysdig capture to troubleshoot the r... no.../<code>
安裝插件
使用 kubectl krew install
<code>➜ kubectl krew install custom-colsUpdated the local copy of plugin index.Installing plugin: custom-colsInstalled plugin: custom-cols\\ | Use this plugin: | kubectl custom-cols | Documentation: | https://github.com/webofmars/kubectl-custom-cols | Caveats: | \\ | | The list of templates is for now limited and can be retrieved with the --help option. | | Please feel free to submit any PR upstream (see github repo) to add more. | //WARNING: You installed a plugin from the krew-index plugin repository. These plugins are not audited for security by the Krew maintainers. Run them at your own risk.(base)/<code>
推薦插件介紹
change-ns
切換 ns, 用於切換 namespace，切換後會設置在 kubeconfig 中，後續的操作就不用再加 --namespaces 了。不過設置了 namespace 之後需要注意的是後續的命令默認 namespace 都是這個設置值了，如果你在 yaml 中沒有寫名 namespace，資源可能不會創建到你期望的 default 目錄下面了.
<code>➜ kubectl change-ns kube-systemnamespace changed to "kube-system"/<code>
cssh
ssh 到 kubernetes node 上面去，會自動從 node 信息中提取 外網 ip，並連接 tmux 嘗試做 ssh 登陸.
<code> > kubectl cssh --helpAllows users to SSH into Kubernetes nodes by opening a new tmux pane for each matching nodeOptions: -a, --address-type='ExternalIP': Node address type to query for (e.g. InternalIP/ExternalIP) -i, --identity-file='': Selects a file from which the identity (private key) for public key authentication is read -l, --selector='': Selector (label query) to filter on, supports '=', '==', and '!='.(e.g. -l key1=value1,key2=value2) -p, --port='': SSH port -u, --username='': SSH Username > kubectl cssh --username=ubuntu/<code>
image
debug/spy
兩個插件的作用差不多，目的都是進去 container 的命名空間進行 debug，不同到是debug 依賴一個 EphemeralContainers feature, 而 spy 不依賴
<code>➜ kubectl spy kube-dns-d5876cbfd-r8kh4loading spy pod...If you don't see a command prompt, try pressing enter./ # psPID USER TIME COMMAND 1 root 12:56 /dnsmasq-nanny -v=2 -logtostderr -configDir=/etc/k8s/dns/dnsmasq-nanny -restartDnsmasq=true -- -k --cache-size=1000 --log-facility=- --server=/cluster.local/12 16 root 14:55 /usr/sbin/dnsmasq -k --cache-size=1000 --log-facility=- --server=/cluster.local/127.0.0.1#10053 --server=/in-addr.arpa/127.0.0.1#10053 --server=/ip6.arpa/127.0 21 root 0:00 sh 28 root 0:00 ps/<code>
還有一個常用的 debug 命令也可以設做 alias
<code>kubectl run --rm -i -t test --image=byrnedo/alpine-curl --restart=Never --limits=cpu=10m,memory=10Mi --command=true /bin/sh/<code>
tree
用 tree的形式展示 Kubernetes objects
<code>➜ kubectl tree deployment kube-dnsNAMESPACE NAME READY REASON AGEkube-system Deployment/kube-dns - 286dkube-system ├─ReplicaSet/kube-dns-898dbbfc6 - 286dkube-system └─ReplicaSet/kube-dns-d5876cbfd - 141dkube-system ├─Pod/kube-dns-d5876cbfd-r8kh4 True 141dkube-system └─Pod/kube-dns-d5876cbfd-w8xvh True 141d/<code>
trace/sniff
分別用 bpftrace/tcpdump 工具對 pod 進行debug
<code>kubectl trace run ip-180-12-0-152.ec2.internal -f read.bt/<code>
image.png
<code>➜ kubectl sniff prometheus-k8s-0 -n monitoringINFO[0000] sniffing method: upload static tcpdumpINFO[0000] using tcpdump path at: '/Users/leiwang/.krew/store/sniff/v1.3.1/static-tcpdump'INFO[0000] no container specified, taking first container we found in pod.INFO[0000] selected container: 'prometheus'INFO[0000] sniffing on pod: 'prometheus-k8s-0' [namespace: 'monitoring', container: 'prometheus', filter: '', interface: 'any']INFO[0000] uploading static tcpdump binary from: '/Users/leiwang/.krew/store/sniff/v1.3.1/static-tcpdump' to: '/tmp/static-tcpdump'INFO[0000] uploading file: '/Users/leiwang/.krew/store/sniff/v1.3.1/static-tcpdump' to '/tmp/static-tcpdump' on container: 'prometheus'INFO[0000] executing command: '[/bin/sh -c ls -alt /tmp/static-tcpdump]' on container: 'prometheus', pod: 'prometheus-k8s-0', namespace: 'monitoring'INFO[0000] command: '[/bin/sh -c ls -alt /tmp/static-tcpdump]' executing successfully exitCode: '0', stdErr :''/<code>
dig
獲取關於 kubernetes node 的一切信息
<code># 這個工具並沒有在 krew index 發佈，所以用 go get 安裝go get -u github.com/sysdiglabs/kubectl-dig/cmd/kubectl-dig/<code>
warp
是 kubectl run + sshd-rsync 命令的合成，可以方便在在 pod 中執行一個本地文件.
<code># Start nodejs project in node containercd examples/nodejskubectl warp -i -t --image node testing-node -- npm run watch/<code>
get-all
get all, 慎用，和 kubectl 的 get --all 不同，這個命令是真的 all
<code>➜ kubectl get-allNAME NAMESPACE AGEcomponentstatus/scheduler <unknown>componentstatus/controller-manager <unknown>componentstatus/etcd-0 <unknown>configmap/token-100001343833 100001343833 69dconfigmap/token-100002873007 100002873007 77d....../<unknown>/<unknown>/<unknown>/<code>
grep
grep by name
<code>➜ kubectl grep pods -A nginxNAMESPACE NAME READY STATUS RESTART AGEdefault nginx-6dc5bfc797-vwdz7 1/1 Running 0 31dmonitoring prometheus-nginx-656ddc9c86-nq9dk 1/1 Running 1 181d/<code>
konfig
當你頻繁創建 kubernetes 集群需要 import 配置的時候很有用
<code>kubectl konfig import --save ~/Downloads/cls-5en24mcc-config/<code>
doctor
診斷 kubernetes 集群，目前做了一下的一些檢查
- core component health (etcd cluster members, scheduler, controller-manager)
- orphan endpoints (endpoints with no ipv4 attached)
- persistent-volume available & unclaimed
- persistent-volume-claim in lost state
- k8s nodes that are not in ready state
- orphan replicasets (desired number of replicas are bigger than 0 but the available replicas are 0)
- leftover replicasets (desired number of replicas and the available # of replicas are 0)
- orphan deployments (desired number of replicas are bigger than 0 but the available replicas are 0)
- leftover deployments (desired number of replicas and the available # of replicas are 0)
- leftover cronjobs (last active date is more than 30 days)
open-svc
利用 kubectl proxy 開啟遠程轉發，方便debug
<code>➜ kubectl open-svc prometheus-k8s -n monitoringStarting to serve on 127.0.0.1:8001Opening service/prometheus-k8s in the default browser.../<code>
image
resource-capacity/view-utilization
觀察 node/pod/namespace 等的資源申請和使用情況
<code>➜ kubectl resource-capacity --podsNODE NAMESPACE POD CPU REQUESTS CPU LIMITS MEMORY REQUESTS MEMORY LIMITS* * * 39728m (45%) 52108m (59%) 85069Mi (43%) 111368Mi (56%)10.0.0.10 * * 762m (19%) 1252m (31%) 585Mi (8%) 2399Mi (34%)10.0.0.10 kube-system ccs-log-collector-r4w6m 300m (7%) 1000m (25%) 238Mi (3%) 1907Mi (27%)10.0.0.10 kube-system etcd-10.0.0.10 0m (0%) 0m (0%) 0Mi (0%) 0Mi (0%)10.0.0.10 kube-system gpu-quota-admission-10.0.0.10 0m (0%) 0m (0%) 0Mi (0%) 0Mi (0%)10.0.0.10 kube-system kube-apiserver-10.0.0.10 0m (0%) 0m (0%) 0Mi (0%) 0Mi (0%)10.0.0.10 kube-system kube-controller-manager-10.0.0.10 0m (0%) 0m (0%) 0Mi (0%) 0Mi (0%)10.0.0.10 kube-system kube-dns-d5876cbfd-w8xvh 260m (6%) 0m (0%) 66Mi (0%) 162Mi (2%)10.0.0.10 kube-system kube-proxy-8m2cf 0m (0%) 0m (0%) 0Mi (0%) 0Mi (0%)10.0.0.10 kube-system kube-router-n86t5 100m (2%) 150m (3%) 100Mi (1%) 150Mi (2%)10.0.0.10 monitoring node-exporter-hdspv 102m (2%) 102m (2%) 180Mi (2%) 180Mi (2%)10.0.0.10 kube-system tke-bridge-agent-dlmhs 0m (0%) 0m (0%) 0Mi (0%) 0Mi (0%)10.0.0.10 kube-system tke-cni-agent-cdfhs 0m (0%) 0m (0%) 0Mi (0%) 0Mi (0%)10.0.0.13 * * 7632m (48%) 9782m (61%) 24290Mi (80%) 28981Mi (96%)10.0.0.13 100010987341 a2sc75rcc7x8x664-64ff96c4d5-wsks6 2000m (12%) 2000m (12%) 2048Mi (6%) 2048Mi (6%)10.0.0.13 100010987341 a2sc75rcc7x8x664-8459df9d4d-dc48m 2000m (12%) 2000m (12%) 10240Mi (34%) 10240Mi (34%)10.0.0.13 100010987341 a2sc75rcc7x8x664-8459df9d4d-kktgx 2000m (12%) 2000m (12%) 10240Mi (34%) 10240Mi (34%)10.0.0.13 kube-system ccs-log-collector-flmcq 300m (1%) 1000m (6%) 238Mi (0%) 1907Mi (6%)10.0.0.13 kube-system ip-masq-agent-lnq4c 0m (0%) 0m (0%) 0Mi (0%) 0Mi (0%)10.0.0.13 kube-system kube-proxy-sjn7j 0m (0%) 0m (0%) 0Mi (0%) 0Mi (0%)10.0.0.13 kube-system kube-router-d8dgw 100m (0%) 150m (0%) 100Mi (0%) 150Mi (0%)10.0.0.13 monitoring node-exporter-jths9 102m (0%) 102m (0%) 180Mi (0%) 180Mi (0%)10.0.0.13 monitoring prometheus-k8s-test-3-0 65m (0%) 15m (0%) 110Mi (0%) 60Mi (0%)10.0.0.13 monitoring prometheus-k8s-test-5-0 65m (0%) 15m (0%) 110Mi (0%) 60Mi (0%)10.0.0.13 kube-system service-controller-6fcb5fc4f4-cvmfz 250m (1%) 1000m (6%) 256Mi (0%) 1024Mi (3%)10.0.0.13 541004974 test-75c58cd5f7-qgm7m 750m (4%) 1500m (9%) 768Mi (2%) 3072Mi (10%)10.0.0.13 kube-system tke-bridge-agent-9n2k9 0m (0%) 0m (0%) 0Mi (0%) 0Mi (0%)10.0.0.13 kube-system tke-cni-agent-pmwpf 0m (0%) 0m (0%) 0Mi (0%) 0Mi (0%)/<code>
<code>➜ kubectl view-utilization node -hCPU : ▄▆▄▇▃▃▆▃▅▂▆▃Memory: ▇▅▂▆▁▃▃▄▇▂▇▃ CPU MemoryNode Req %R Lim %L Req %R Lim %L10.0.0.10 0.76 19% 1.3 31% 600M 8% 2.5G 36%10.0.0.13 6.9 43% 8.3 52% 23G 78% 25G 86%10.0.0.17 1 25% 3.3 82% 1G 15% 4.3G 63%10.0.0.31 1.5 38% 2.3 57% 2.5G 13% 4.3G 23%10.0.0.4 11 75% 13 86% 23G 74% 25G 83%10.0.0.44 0.57 14% 1.3 32% 640M 3% 2.4G 13%/<code>
實踐
實踐部分我們開發一個方便多 cluster 操作的簡單插件。在操作多集群到時候 切換 context 可能會比較麻煩，這裡我們參考 ansible 的 inventory 設計，支持寫入一個 配置文件，把 cluster 進行分組管理，選擇一個或者所有 group 的 cluster 執行 kubectl 命令。
插件代碼在 kubectl-clusters
下載腳本，安裝到 /usr/local/bin, 運行
<code># exec in all groups ➜ kubectl clusters all get pod[GROUP]: test --------------------------------------------------------------------------------[CLUSTER]: cls-test1 --------------------------------------------------[DEBUG] kubectl --context=cls-test1 --namespace=default get podNAME READY STATUS RESTARTS AGEnginx-6dc5bfc797-vwdz7 1/1 Running 0 32d[GROUP]: prod --------------------------------------------------------------------------------[CLUSTER]: cls-prod --------------------------------------------------[DEBUG] kubectl --context=cls-qcvhpqog get podNAME READY STATUS RESTARTS AGEa4cgfxv7srbfhbsn-78479b5cf7-f85d8 2/2 Running 0 37d# exec in single group➜ kubectl clusters prod get pod[GROUP]: prod --------------------------------------------------------------------------------[CLUSTER]: cls-prod --------------------------------------------------[DEBUG] kubectl --context=cls-qcvhpqog get podNAME READY STATUS RESTARTS AGEa4cgfxv7srbfhbsn-78479b5cf7-f85d8 2/2 Running 0 37d/<code>
提交到 krew index 目錄
參考
- Boosting your kubectl productivity
- KubectlPlugins
- The Top 19 Kubectl Plugins Open Source Projects
- GithubTopic-kubectl-plugins
- kubectl-plugins-index
分享到:
閱讀更多 Echa攻城獅 的文章
