我在github上準備好了相關文件的下載：https://github.com/anakin/k8s-install
開始改造我們之前搭建的基礎集群，加上安全機制。
準備工作
1. 停止master節點的所有服務
查看已有的service
kubectl get services
刪除service（例如有一個叫xxx的service）
kubectl delete service xxx
查看已有的deployments
kubectl get deploy
刪除deployments
kubectl delete deploy xxx
停止master的服務
service kube-calico stop service kube-scheduler stop service kube-controller-manager stop service kube-apiserver stop service etcd stop && rm -fr /var/lib/etcd/*
2. 停止worker節點的所有服務
service kubelet stop rm -fr /var/lib/kubelet/* service kube-proxy stop rm -fr /var/lib/kube-proxy/* service kube-calico stop
3. 安裝需要的工具
在所有節點上安裝cfssl:
CFSSL是CloudFlare開源的一款PKI/TLS工具。 CFSSL 包含一個命令行工具 和一個用於 簽名，驗證並且捆綁TLS證書的 HTTP API 服務。 用Go寫的。具體相關信息自行google。
下載
wget -q --show-progress --https-only --timestamping \ https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 \ https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
修改權限
chmod +x cfssl_linux-amd64 cfssljson_linux-amd64
放到可執行目錄
mv cfssl_linux-amd64 /usr/local/bin/cfssl mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
在worker節點上安裝conntrack
apt install conntrack
4.在master上生成根證書
mkdir -p /etc/kubernetes/ca cd /etc/kubernetes/ca
編輯ca-config.json文件，內容如下：
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"kubernetes": {
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
],
"expiry": "87600h"
}
}
}
}
編輯ca-csr.json文件，內容如下：
{
"CN": "kubernetes",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "Beijing",
"L": "XS",
"O": "k8s",
"OU": "System"
}
]
}
執行生成證書的命令：
cfssl gencert -initca ca-csr.json | cfssljson -bare ca
OK。
master節點設置
1. etcd設置
編輯/lib/systemd/system/etcd.service，內容如下：
[Unit] Description=Etcd Server After=network.target After=network-online.target Wants=network-online.target Documentation=https://github.com/coreos [Service] Type=notify WorkingDirectory=/var/lib/etcd/ ExecStart=/home/anakin/bin/etcd \ --name=192.168.32.131 \ --listen-client-urls=https://192.168.32.131:2379,http://127.0.0.1:2379 \ --advertise-client-urls=https://192.168.32.131:2379 \ --data-dir=/var/lib/etcd \ --listen-peer-urls=https://192.168.32.131:2380 \ --initial-advertise-peer-urls=https://192.168.32.131:2380 \ --cert-file=/etc/kubernetes/ca/etcd/etcd.pem \ --key-file=/etc/kubernetes/ca/etcd/etcd-key.pem \ --peer-cert-file=/etc/kubernetes/ca/etcd/etcd.pem \ --peer-key-file=/etc/kubernetes/ca/etcd/etcd-key.pem \ --trusted-ca-file=/etc/kubernetes/ca/ca.pem \ --peer-trusted-ca-file=/etc/kubernetes/ca/ca.pem Restart=on-failure RestartSec=5 LimitNOFILE=65536 [Install] WantedBy=multi-user.target
然後啟動服務：
systemctl daemon-reload service etcd start
2. apiserver設置
創建一個存放證書的目錄：
mkdir -p /etc/kubernetes/ca/kubernetes cd /etc/kubernetes/ca/kubernetes/
編輯kubernetes-csr.json文件，內容如下：
{
"CN": "kubernetes",
"hosts": [
"127.0.0.1",
"192.168.32.131",
"10.68.0.1",
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "Beijing",
"L": "XS",
"O": "k8s",
"OU": "System"
}
]
}
簽發證書：
cfssl gencert \ -ca=/etc/kubernetes/ca/ca.pem \ -ca-key=/etc/kubernetes/ca/ca-key.pem \ -config=/etc/kubernetes/ca/ca-config.json \ -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes
生成一個token文件：
head -c 16 /dev/urandom | od -An -t x | tr -d ' ' c81fb8ce5502f19d510d159ff8a1cf12
把這個token存入文件：
echo "c81fb8ce5502f19d510d159ff8a1cf12,kubelet-bootstrap,10001,"system:kubelet-bootstrap"" > /etc/kubernetes/ca/kubernetes/token.csv
編輯/lib/systemd/system/kube-apiserver.service，內容如下：
[Unit] Description=Kubernetes API Server Documentation=https://github.com/GoogleCloudPlatform/kubernetes After=network.target [Service] ExecStart=/home/anakin/bin/kube-apiserver \ --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota,NodeRestriction \ --insecure-bind-address=127.0.0.1 \ --kubelet-https=true \ --bind-address=192.168.32.131 \ --authorization-mode=Node,RBAC \ --runtime-config=rbac.authorization.k8s.io/v1 \ --enable-bootstrap-token-auth \ --token-auth-file=/etc/kubernetes/ca/kubernetes/token.csv \ --tls-cert-file=/etc/kubernetes/ca/kubernetes/kubernetes.pem \ --tls-private-key-file=/etc/kubernetes/ca/kubernetes/kubernetes-key.pem \ --client-ca-file=/etc/kubernetes/ca/ca.pem \ --service-account-key-file=/etc/kubernetes/ca/ca-key.pem \ --etcd-cafile=/etc/kubernetes/ca/ca.pem \ --etcd-certfile=/etc/kubernetes/ca/kubernetes/kubernetes.pem \ --etcd-keyfile=/etc/kubernetes/ca/kubernetes/kubernetes-key.pem \ --service-cluster-ip-range=10.68.0.0/16 \ --service-node-port-range=20000-40000 \ --etcd-servers=https://192.168.32.131:2379 \ --enable-swagger-ui=true \ --allow-privileged=true \ --audit-log-maxage=30 \ --audit-log-maxbackup=3 \ --audit-log-maxsize=100 \ --audit-log-path=/var/lib/audit.log \ --event-ttl=1h \ --v=2 Restart=on-failure RestartSec=5 Type=notify LimitNOFILE=65536 [Install] WantedBy=multi-user.target
然後啟動服務：
systemctl daemon-reload service kube-apiserver start
3. controller-manager設置
編輯/lib/systemd/system/kube-controller-manager.service文件，內容如下：
[Unit] Description=Kubernetes Controller Manager Documentation=https://github.com/GoogleCloudPlatform/kubernetes [Service] ExecStart=/home/anakin/bin/kube-controller-manager \ --address=127.0.0.1 \ --master=http://127.0.0.1:8080 \ --allocate-node-cidrs=true \ --service-cluster-ip-range=10.68.0.0/16 \ --cluster-cidr=172.20.0.0/16 \ --cluster-name=kubernetes \ --leader-elect=true \ --cluster-signing-cert-file=/etc/kubernetes/ca/ca.pem \ --cluster-signing-key-file=/etc/kubernetes/ca/ca-key.pem \ --service-account-private-key-file=/etc/kubernetes/ca/ca-key.pem \ --root-ca-file=/etc/kubernetes/ca/ca.pem \ --v=2 Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target
啟動服務
systemctl daemon-reload service kube-controller-manager start
4. scheduler設置
scheduler不需要重新設置，直接啟動就ok了。
5. kubectl設置先創建一個叫做admin的系統管理員：
mkdir -p /etc/kubernetes/ca/admin cd /etc/kubernetes/ca/admin/
編輯admin-csr.json文件，內容如下：
{
"CN": "admin",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "Beijing",
"L": "XS",
"O": "system:masters",
"OU": "System"
}
]
}
簽發證書：
cfssl gencert \ -ca=/etc/kubernetes/ca/ca.pem \ -ca-key=/etc/kubernetes/ca/ca-key.pem \ -config=/etc/kubernetes/ca/ca-config.json \ -profile=kubernetes admin-csr.json | cfssljson -bare admin
然後開始設置kubectl
kubectl config set-cluster kubernetes \ --certificate-authority=/etc/kubernetes/ca/ca.pem \ --embed-certs=true \ --server=https://192.168.32.131:6443 kubectl config set-credentials admin \ --client-certificate=/etc/kubernetes/ca/admin/admin.pem \ --embed-certs=true \ --client-key=/etc/kubernetes/ca/admin/admin-key.pem kubectl config set-context kubernetes \ --cluster=kubernetes --user=admin kubectl config use-context kubernetes
6. calico設置生成證書：
mkdir -p /etc/kubernetes/ca/calico cd /etc/kubernetes/ca/calico/
編輯calico-csr.json文件，內容如下：
{
"CN": "calico",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "Beijing",
"L": "XS",
"O": "k8s",
"OU": "System"
}
]
}
簽發證書：
cfssl gencert \ -ca=/etc/kubernetes/ca/ca.pem \ -ca-key=/etc/kubernetes/ca/ca-key.pem \ -config=/etc/kubernetes/ca/ca-config.json \ -profile=kubernetes calico-csr.json | cfssljson -bare calico
編輯/lib/systemd/system/kube-calico.service文件，內容如下：
[Unit] Description=calico node After=docker.service Requires=docker.service [Service] User=root PermissionsStartOnly=true ExecStart=/usr/bin/docker run --net=host --privileged --name=calico-node \ -e NODENAME="calico1" \ -e ETCD_ENDPOINTS=https://192.168.32.131:2379 \ -e ETCD_CA_CERT_FILE=/etc/kubernetes/ca/ca.pem \ -e ETCD_CERT_FILE=/etc/kubernetes/ca/calico/calico.pem \ -e ETCD_KEY_FILE=/etc/kubernetes/ca/calico/calico-key.pem \ -e CALICO_LIBNETWORK_ENABLED=true \ -e CALICO_NETWORKING_BACKEND=bird \ -e CALICO_DISABLE_FILE_LOGGING=true \ -e CALICO_IPV4POOL_CIDR=172.20.0.0/16 \ -e CALICO_IPV4POOL_IPIP=off \ -e FELIX_DEFAULTENDPOINTTOHOSTACTION=ACCEPT \ -e FELIX_IPV6SUPPORT=false \ -e FELIX_LOGSEVERITYSCREEN=info \ -e FELIX_IPINIPMTU=1440 \ -e FELIX_HEALTHENABLED=true \ -e IP=192.168.32.131 \ -v /etc/kubernetes/ca:/etc/kubernetes/ca \ -v /var/run/calico:/var/run/calico \ -v /lib/modules:/lib/modules \ -v /run/docker/plugins:/run/docker/plugins \ -v /var/run/docker.sock:/var/run/docker.sock \ -v /var/log/calico:/var/log/calico \ calico/node:release-v2.6 ExecStop=/usr/bin/docker rm -f calico-node Restart=always RestartSec=10 [Install] WantedBy=multi-user.target
啟動服務：
systemctl daemon-reload service kube-calico start
還有一步，就是把/etc/kubernetes/ca/calico目錄下的這幾個證書拷貝到worker節點對應的位置上去備用
7. 設置kubelet的角色綁定
kubectl -n kube-system get clusterrole kubectl create clusterrolebinding kubelet-bootstrap \ --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap
worker節點設置
1. calico設置
編輯/lib/systemd/system/kube-calico.service文件，參考master節點的內容，只需要修改一下NODENAME就可以了，然後啟動服務
2. kubelet設置
設置kubelet參數：
kubectl config set-cluster kubernetes \ --certificate-authority=/etc/kubernetes/ca/ca.pem \ --embed-certs=true \ --server=https://192.168.32.131:6443 \ --kubeconfig=bootstrap.kubeconfig kubectl config set-credentials kubelet-bootstrap \ --token=c81fb8ce5502f19d510d159ff8a1cf12\ --kubeconfig=bootstrap.kubeconfig kubectl config set-context default \ --cluster=kubernetes \ --user=kubelet-bootstrap \ --kubeconfig=bootstrap.kubeconfig kubectl config use-context default --kubeconfig=bootstrap.kubeconfig mv bootstrap.kubeconfig /etc/kubernetes/
設置cni：
cd /etc/cni/net.d/
編輯10-calico.conf文件，內容如下：
{
"name": "calico-k8s-network",
"cniVersion": "0.1.0",
"type": "calico",
"etcd_endpoints": "https://192.168.32.131:2379",
"etcd_key_file": "/etc/kubernetes/ca/calico/calico-key.pem",
"etcd_cert_file": "/etc/kubernetes/ca/calico/calico.pem",
"etcd_ca_cert_file": "/etc/kubernetes/ca/ca.pem",
"log_level": "info",
"ipam": {
"type": "calico-ipam"
},
"kubernetes": {
"kubeconfig": "/etc/kubernetes/kubelet.kubeconfig"
}
}
編輯/lib/systemd/system/kubelet.service文件，內容如下：
[Unit] Description=Kubernetes Kubelet Documentation=https://github.com/GoogleCloudPlatform/kubernetes After=docker.service Requires=docker.service [Service] WorkingDirectory=/var/lib/kubelet ExecStart=/home/anakin/bin/kubelet \ --address=192.168.32.132 \ --hostname-override=192.168.32.132 \ --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/imooc/pause-amd64:3.0 \ --kubeconfig=/etc/kubernetes/kubelet.kubeconfig \ --experimental-bootstrap-kubeconfig=/etc/kubernetes/bootstrap.kubeconfig \ --cert-dir=/etc/kubernetes/ca \ --hairpin-mode hairpin-veth \ --network-plugin=cni \ --cni-conf-dir=/etc/cni/net.d \ --cni-bin-dir=/home/anakin/bin \ --cluster-dns=10.68.0.2 \ --cluster-domain=cluster.local. \ --allow-privileged=true \ --fail-swap-on=false \ --logtostderr=true \ --v=2 #kubelet cAdvisor 默認在所有接口監聽 4194 端口的請求, 以下iptables限制內網訪問 ExecStartPost=/sbin/iptables -A INPUT -s 10.0.0.0/8 -p tcp --dport 4194 -j ACCEPT ExecStartPost=/sbin/iptables -A INPUT -s 172.16.0.0/12 -p tcp --dport 4194 -j ACCEPT ExecStartPost=/sbin/iptables -A INPUT -s 192.168.0.0/16 -p tcp --dport 4194 -j ACCEPT ExecStartPost=/sbin/iptables -A INPUT -p tcp --dport 4194 -j DROP Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target
啟動服務：
systemctl daemon-reload service kubelet start
啟動之後，需要去master節點進行授權操作：
kubectl get csr|grep 'Pending' | awk '{print $1}'| xargs kubectl certificate approve
3. kube-proxy設置
準備證書：
mkdir -p /etc/kubernetes/ca/kube-proxy cd /etc/kubernetes/ca/kube-proxy
編輯kube-proxy-csr.json文件，內容如下：
{
"CN": "system:kube-proxy",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "Beijing",
"L": "XS",
"O": "k8s",
"OU": "System"
}
]
}
簽發證書：
cfssl gencert \ -ca=/etc/kubernetes/ca/ca.pem \ -ca-key=/etc/kubernetes/ca/ca-key.pem \ -config=/etc/kubernetes/ca/ca-config.json \ -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy
生成各種配置文件：
kubectl config set-cluster kubernetes \ --certificate-authority=/etc/kubernetes/ca/ca.pem \ --embed-certs=true \ --server=https://192.168.32.131:6443 \ --kubeconfig=kube-proxy.kubeconfig kubectl config set-credentials kube-proxy \ --client-certificate=/etc/kubernetes/ca/kube-proxy/kube-proxy.pem \ --client-key=/etc/kubernetes/ca/kube-proxy/kube-proxy-key.pem \ --embed-certs=true \ --kubeconfig=kube-proxy.kubeconfig kubectl config set-context default \ --cluster=kubernetes \ --user=kube-proxy \ --kubeconfig=kube-proxy.kubeconfig kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig mv kube-proxy.kubeconfig /etc/kubernetes/kube-proxy.kubeconfig
編輯/lib/systemd/system/kube-proxy.service文件，內容如下：
[Unit] Description=Kubernetes Kube-Proxy Server Documentation=https://github.com/GoogleCloudPlatform/kubernetes After=network.target [Service] WorkingDirectory=/var/lib/kube-proxy ExecStart=/home/anakin/bin/kube-proxy \ --bind-address=192.168.32.132 \ --hostname-override=192.168.32.132 \ --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig \ --logtostderr=true \ --v=2 Restart=on-failure RestartSec=5 LimitNOFILE=65536 [Install] WantedBy=multi-user.target
啟動服務：
systemctl daemon-reload service kube-proxy start
總結
中間可能會遇到的問題：
1. calico
calico是以docker的方式運行的，有時候重啟服務的時候，之前的container還沒殺死，日誌裡會報錯誤：
Error response from daemon: Conflict. The container name "/aaa" is already in use by conta iner "xxx".
這時候只要刪除這個container就可以了:
docker rm -f xxx
如果是報node的名字被佔用的錯誤，到master節點刪除就ok了：
calicoctl delete node
2. 一些常用的命令:
查看集群節點：
kubectl get nodes
查看calico節點：
calicoctl node status
每個服務啟動之後，都可以用下面的命令查看服務的日誌信息：
journalctl -f -u xxx
ok。寫的好累。
分享到:
關鍵字: 搭建 delete CloudFlare
