本文參照紅帽官方文檔，在裸機安裝Openshift4.3文檔進行。因為只有一臺64G內存的PC機，安裝vmware vsphere 6.7免費版進行本測試，所以嘗試在OCP官方文檔要求的最低內存需求基礎上，內存減半安裝，記錄如下。
發現頭條號不支持markdown，對程序員太不又好了，拷貝過來的代碼格式都沒了，簡書的這一點就要好一些。
https://www.jianshu.com/p/7c0c2affadb8
1、ocp安裝的過程
紅帽官方文檔記載的安裝過程如下：
- bootstrap啟動並從準備好master需要的資源
- master從bootstrap獲取需要的資源並完成啟動
- master通過bootstrap構建etcd集群
- bootstrap使用剛才構建的etcd集群啟動一個臨時的kubernetes control plane
- 臨時control plane在master節點啟動生產control plane
- 臨時control plane關閉並將控制權移交給生產control plane
- bootstrap將ocp組建注入到生產control plane
- 安裝程序關閉bootstrap
- control plane 部署計算節點
- control plane 通過operator方式安裝其他服務
2、準備服務器資源
服務器規劃如下：
- 3臺control plane節點，安裝etcd、control plane組件和infras基礎組件，因為資源緊張，不部署dns服務器，通過hosts文件解析域名；
- 2臺compute 節點，運行實際負載；
- 1臺bootstrap節點，執行安裝任務；
- 1臺misc/lb節點，用於準備安裝資源、啟動bootstrap，並作為lb節點使用。
Hostname vcpu ram hdd ip fqdn misc/lb 4 8g 120g 192.168.128.30 misc.ocptest.ipingcloud.com/lb.ocptest.ipincloud.com bootstrap 4 8g 120g 192.168.128.31 bootstrap.ocptest.ipincloud.com master1 4 8g 120g 192.168.128.32 master1.ocptest.ipincloud.com master2 4 8g 120g 192.168.128.33 master2.ocptest.ipincloud.com master3 4 8g 120g 192.168.128.34 master3.ocptest.ipincloud.com worker1 2 4g 120g 192.168.128.35 worker1.ocptest.ipincloud.com worker2 2 4g 120g 192.168.128.36 worker2.ocptest.ipincloud.com
3、準備網絡資源
api server和ingress公用一個lb,即misc/lb 以為dns配置記錄，ocptest是cluster名，ipingcloud.com是基礎域名.這些配置，需要修改ansi-playbook文件的tasks/相應模板。 參見 https://github.com/scwang18/ocp4-upi-helpernode.git
- dns配置
組件 dns記錄 描述 Kubernetes API api.ocptest.ipincloud.com 該DNS記錄指向control plane節點的負載平衡器。群集外部和群集中所有節點都必須可以解析此記錄。 Kubernetes API api-int.ocptest.ipincloud.com 該DNS記錄指向control plane節點的負載平衡器。該記錄必須可從群集中的所有節點上解析。 Routes *.apps.ocptest.ipincloud.com 通配符DNS記錄指向ingress slb。群集外部和群集中所有節點都必須可以解析此記錄。 etcd etcd-.ocptest.ipincloud.com DNS記錄指向etcd節點，群集所有節點都必須可以解析此記錄。 etcd _etcd-server-ssl._tcp.ocptest.ipincloud.com 因為etcd使用2380對外服務，因此，需要建立對應每臺etcd節點的srv dns記錄，優先級0，權重10和端口2380，如下表
- etcd srv dns記錄表
#一下激怒是必須的，用於bootstrap創建etcd服務器上，自動配置etcd服務解析
#_service._proto.name. TTL class SRV priority weight port target. _etcd-server-ssl._tcp.<cluster>.<base> 86400 IN SRV 0 10 2380 etcd-0.<cluster>.<base>. _etcd-server-ssl._tcp.<cluster>.<base> 86400 IN SRV 0 10 2380 etcd-1.<cluster>.<base>. _etcd-server-ssl._tcp.<cluster>.<base> 86400 IN SRV 0 10 2380 etcd-2.<cluster>.<base>./<base>/<cluster>/<base>/<cluster>/<base>/<cluster>/<base>/<cluster>/<base>/<cluster>/<base>/<cluster>
- 創建ssh私鑰並加入ssh agent
通過免登陸ssh私鑰，可以用core用戶身份登錄到master節點，在集群上進行安裝調試和災難恢復。
（1）在misc節點上執行一下命令創建sshkey
<code>ssh-keygen -t rsa -b 4096 -N '' /<code>
以上命令在~/.ssh/文件夾下創建id_rsa和id_rsa.pub兩個文件。
（2）啟動ssh agent進程並把將無密碼登錄的私鑰加入ssh agent
<code>eval "$(ssh-agent -s)"ssh-add ~/.ssh/id_rsa/<code>
下一步安裝ocp時，需要將ssh公鑰提供給安裝程序配置文件。
因為我們採用自己手動準備資源方式，因此，需要將ssh公鑰放到集群各節點，本機就可以免密碼登錄集群節點
<code>#將剛才生成的 ~/.ssh目錄中的 id_rsa.pub 這個文件拷貝到你要登錄的集群節點 的~/.ssh目錄中scp ~/.ssh/id_rsa.pub [email protected]:~/.ssh/#然後在集群節點上運行以下命令來將公鑰導入到~/.ssh/authorized_keys這個文件中cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys/<code>
4、獲取安裝程序
需要註冊紅帽官網賬號，下載測試版安裝程序，下載鏈接具體過程略。 https://cloud.redhat.com/openshift/install/metal/user-provisioned
- 下載安裝程序
<code>rm -rf /data/pkgmkdir -p /data/pkgcd /data/pkg#ocp安裝程序#wget https://mirror.openshift.com/pub/openshift-v4/clients/ocp/latest/openshift-install-linux-4.3.0.tar.gz#ocp 客戶端#wget https://mirror.openshift.com/pub/openshift-v4/clients/ocp/latest/openshift-client-linux-4.3.0.tar.gz#rhcos安裝程序wget https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/latest/latest/rhcos-4.3.0-x86_64-installer.iso#rhcos bios raw文件wget https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/latest/latest/rhcos-4.3.0-x86_64-metal.raw.gz#如果採用iso文件方式安裝，相面兩個文件都不需要下載#rhcos安裝程序內核文件，用於使用ipex方式安裝wget https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/latest/latest/rhcos-4.3.0-x86_64-installer-kernel#rhcos初始化鏡像文件，用於使用ipex方式安裝wget https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/latest/latest/rhcos-4.3.0-x86_64-installer-initramfs.img/<code>
5、準備工具機misc
參照王徵的腳本修改的工具機準備工具，可以方便的在工具機上啟動 LB、DHCP、PXE、DNS和HTTP服務 （1）安裝ansible和git
<code>yum -y install ansible git/<code>
（2）從github拉取playbook
<code>cd /data/pkggit clone https://github.com/scwang18/ocp4-upi-helpernode.git/<code>
（3）修改playbook的參數文件 根據自己的網絡規劃修改參數文件
<code>[root@centos75 pkg]# cd /data/pkg/ocp4-upi-helpernode/[root@centos75 ocp4-upi-helpernode]# cat vars-static.yaml[root@misc pkg]# cat vars-static.yaml---staticips: truenamed: truehelper: name: "helper" ipaddr: "192.168.128.30" networkifacename: "ens192"dns: domain: "ipincloud.com" clusterid: "ocptest" forwarder1: "192.168.128.30" forwarder2: "192.168.128.30" registry: name: "registry" ipaddr: "192.168.128.30" yum: name: "yum" ipaddr: "192.168.128.30"bootstrap: name: "bootstrap" ipaddr: "192.168.128.31"masters: - name: "master1" ipaddr: "192.168.128.32" - name: "master2" ipaddr: "192.168.128.33" - name: "master3" ipaddr: "192.168.128.34"workers: - name: "worker1" ipaddr: "192.168.128.35" - name: "worker2" ipaddr: "192.168.128.36"force_ocp_download: falseocp_bios: "file:///data/pkg/rhcos-4.3.0-x86_64-metal.raw.gz"ocp_initramfs: "file:///data/pkg/rhcos-4.3.0-x86_64-installer-initramfs.img"ocp_install_kernel: "file:///data/pkg/rhcos-4.3.0-x86_64-installer-kernel"ocp_client: "file:///data/pkg/openshift-client-linux-4.3.0.tar.gz"ocp_installer: "file:///data/pkg/openshift-install-linux-4.3.0.tar.gz"ocp_filetranspiler: "file:///data/pkg/filetranspiler-master.zip"registry_server: "registry.ipincloud.com:8443"[root@misc pkg]#/<code>
（4）執行ansible安裝
<code>ansible-playbook -e @vars-static.yaml tasks/main.yml/<code>
6、準備docker env
<code># 在可以科學上網的機器上打包必要的鏡像文件#rm -rf /data/ocp4mkdir -p /data/ocp4cd /data/ocp4# 這個腳本不好用，不下載，使用下面自己修改過# wget https://raw.githubusercontent.com/wangzheng422/docker_env/dev/redhat/ocp4/4.3/scripts/build.dist.shyum -y install podman docker-distribution pigz skopeo docker buildah jq python3-pip pip3 install yq# https://blog.csdn.net/ffzhihua/article/details/85237411wget http://mirror.centos.org/centos/7/os/x86_64/Packages/python-rhsm-certificates-1.19.10-1.el7_4.x86_64.rpmrpm2cpio python-rhsm-certificates-1.19.10-1.el7_4.x86_64.rpm | cpio -iv --to-stdout ./etc/rhsm/ca/redhat-uep.pem | tee /etc/rhsm/ca/redhat-uep.pemsystemctl start dockerdocker login -u wuliangye2019 -p Red@123! registry.redhat.iodocker login -u wuliangye2019 -p Red@123! registry.access.redhat.comdocker login -u wuliangye2019 -p Red@123! registry.connect.redhat.compodman login -u wuliangye2019 -p Red@123! registry.redhat.iopodman login -u wuliangye2019 -p Red@123! registry.access.redhat.compodman login -u wuliangye2019 -p Red@123! registry.connect.redhat.com# to download the pull-secret.json, open following link# https://cloud.redhat.com/openshift/install/metal/user-provisionedcat << 'EOF' > /data/pull-secret.json{"auths":{"cloud.openshift.com":{"auth":"xxxxxxxxxxx}}}EOF/<code>創建 build.dist.sh文件
<code>#!/usr/bin/env bashset -eset -xvar_date=$(date '+%Y-%m-%d')echo $var_date#以下不用每次都執行#cat << EOF >> /etc/hosts#127.0.0.1 registry.ipincloud.com#EOF#mkdir -p /etc/crts/#cd /etc/crts#openssl req \\# -newkey rsa:2048 -nodes -keyout ipincloud.com.key \\# -x509 -days 3650 -out ipincloud.com.crt -subj \\# "/C=CN/ST=GD/L=SZ/O=Global Security/OU=IT Department/CN=*.ipincloud.com"#cp /etc/crts/ipincloud.com.crt /etc/pki/ca-trust/source/anchors/#update-ca-trust extractsystemctl stop docker-distributionrm -rf /data/registrymkdir -p /data/registrycat << EOF > /etc/docker-distribution/registry/config.ymlversion: 0.1log: fields: service: registrystorage: cache: layerinfo: inmemory filesystem: rootdirectory: /data/registry delete: enabled: truehttp: addr: :8443 tls: certificate: /etc/crts/ipincloud.com.crt key: /etc/crts/ipincloud.com.keyEOFsystemctl restart dockersystemctl enable docker-distributionsystemctl restart docker-distributionbuild_number_list=$(cat << EOF4.3.0EOF)mkdir -p /data/ocp4cd /data/ocp4install_build() { BUILDNUMBER=$1 echo ${BUILDNUMBER} mkdir -p /data/ocp4/${BUILDNUMBER} cd /data/ocp4/${BUILDNUMBER} #下載並安裝openshift客戶端和安裝程序 第一次需要運行，工具機ansi初始化時，已經完成這些動作了 #wget https://mirror.openshift.com/pub/openshift-v4/clients/ocp/${BUILDNUMBER}/release.txt #wget https://mirror.openshift.com/pub/openshift-v4/clients/ocp/${BUILDNUMBER}/openshift-client-linux-${BUILDNUMBER}.tar.gz #wget https://mirror.openshift.com/pub/openshift-v4/clients/ocp/${BUILDNUMBER}/openshift-install-linux-${BUILDNUMBER}.tar.gz #解壓安裝程序和客戶端到用戶執行目錄 第一次需要運行 #tar -xzf openshift-client-linux-${BUILDNUMBER}.tar.gz -C /usr/local/bin/ #tar -xzf openshift-install-linux-${BUILDNUMBER}.tar.gz -C /usr/local/bin/ export OCP_RELEASE=${BUILDNUMBER} export LOCAL_REG='registry.ipincloud.com:8443' export LOCAL_REPO='ocp4/openshift4' export UPSTREAM_REPO='openshift-release-dev' export LOCAL_SECRET_JSON="/data/pull-secret.json" export OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE=${LOCAL_REG}/${LOCAL_REPO}:${OCP_RELEASE} export RELEASE_NAME="ocp-release" oc adm release mirror -a ${LOCAL_SECRET_JSON} \\ --from=quay.io/${UPSTREAM_REPO}/${RELEASE_NAME}:${OCP_RELEASE}-x86_64 \\ --to-release-image=${LOCAL_REG}/${LOCAL_REPO}:${OCP_RELEASE} \\ --to=${LOCAL_REG}/${LOCAL_REPO}}while read -r line; do install_build $linedone <<< "$build_number_list"cd /data/ocp4#wget -O ocp4-upi-helpernode-master.zip https://github.com/wangzheng422/ocp4-upi-helpernode/archive/master.zip#以下注釋，因為quay.io/wangzheng422這個倉庫的registry版本是v1不能與v2共存#podman pull quay.io/wangzheng422/filetranspiler#podman save quay.io/wangzheng422/filetranspiler | pigz -c > filetranspiler.tgz#podman pull docker.io/library/registry:2#podman save docker.io/library/registry:2 | pigz -c > registry.tgzsystemctl start dockerdocker login -u wuliangye2019 -p Red@123! registry.redhat.iodocker login -u wuliangye2019 -p Red@123! registry.access.redhat.comdocker login -u wuliangye2019 -p Red@123! registry.connect.redhat.compodman login -u wuliangye2019 -p Red@123! registry.redhat.iopodman login -u wuliangye2019 -p Red@123! registry.access.redhat.compodman login -u wuliangye2019 -p Red@123! registry.connect.redhat.com# 以下命令要運行 2-3個小時，耐心等待。。。# build operator catalogpodman login registry.ipincloud.com:8443 -u root -p Scwang18oc adm catalog build \\ --appregistry-endpoint https://quay.io/cnr \\ --appregistry-org redhat-operators \\ --to=${LOCAL_REG}/ocp4-operator/redhat-operators:v1 oc adm catalog mirror \\ ${LOCAL_REG}/ocp4-operator/redhat-operators:v1 \\ ${LOCAL_REG}/operator#cd /data#tar cf - registry/ | pigz -c > registry.tgz#cd /data#tar cf - ocp4/ | pigz -c > ocp4.tgz/<code>執行build.dist.sh腳本
這裡有個巨坑，因為從quay.io拉取image鏡像到本地時，拉取的文件有5G多，通常一次拉取不完，會出錯，每次出錯後，重新運行build.dist.sh會把以前的registry刪除掉，從頭再來，浪費很多時間，實際上可以不用刪除，執行oc adm release mirror時會自動跳過已經存在的image。血淚教訓。
<code>bash build.dist.sh/<code>
oc adm release mirror執行完畢後，回根據官方鏡像倉庫生成本地鏡像倉庫，返回的信息需要記錄下來，特別是imageContentSource信息，後面 install-config.yaml 文件裡配置進去
<code>SuccessUpdate image: registry.ipincloud.com:8443/ocp4/openshift4:4.3.0Mirror prefix: registry.ipincloud.com:8443/ocp4/openshift4To use the new mirrored repository to install, add the following section to the install-config.yaml:imageContentSources:- mirrors: - registry.ipincloud.com:8443/ocp4/openshift4 source: quay.io/openshift-release-dev/ocp-release- mirrors: - registry.ipincloud.com:8443/ocp4/openshift4 source: quay.io/openshift-release-dev/ocp-v4.0-art-devTo use the new mirrored repository for upgrades, use the following to create an ImageContentSourcePolicy:apiVersion: operator.openshift.io/v1alpha1kind: ImageContentSourcePolicymetadata: name: examplespec: repositoryDigestMirrors: - mirrors: - registry.ipincloud.com:8443/ocp4/openshift4 source: quay.io/openshift-release-dev/ocp-release - mirrors: - registry.ipincloud.com:8443/ocp4/openshift4 source: quay.io/openshift-release-dev/ocp-v4.0-art-dev/<code>
以下命令不需要執行，在build.dish.sh裡已經執行了
<code>oc adm release mirror -a /data/pull-secret.json --from=quay.io/openshift-release-dev/ocp-release:4.3.0-x86_64 --to-release-image=registry.ipincloud.com:8443/ocp4/openshift4:4.3.0 --to=registry.ipincloud.com:8443/ocp4/openshift4 podman login registry.ipincloud.com:8443 -u root -p Scwang18oc adm catalog build \\ --appregistry-endpoint https://quay.io/cnr \\ --appregistry-org redhat-operators \\ --to=registry.ipincloud.com:8443/ocp4-operator/redhat-operators:v1 oc adm catalog mirror \\ registry.ipincloud.com:8443/ocp4-operator/redhat-operators:v1 \\ registry.ipincloud.com:8443/operator#如果oc adm catalog mirror執行不成功，會生成一個mapping.txt的文件，可以根據這個文件，執行不成功的行刪除，再以下面的方式執行oc image mirror -a /data/pull-secret.json -f /data/mapping-ok.txtoc image mirror quay.io/external_storage/nfs-client-provisioner:latest registry.ipincloud.com:8443/ocp4/openshift4/nfs-client-provisioner:latestoc image mirror quay.io/external_storage/nfs-client-provisioner:latest registry.ipincloud.com:8443/quay.io/external_storage/nfs-client-provisioner:latest#查看鏡像的shacurl -v --silent -H "Accept: application/vnd.docker.distribution.manifest.v2+json" -X GET https://registry.ipincloud.com:8443/v2/ocp4/openshift4/nfs-client-provisioner/manifests/latest 2>&1 | grep Docker-Content-Digest | awk '{print ($3)}'#刪除鏡像摘要curl -v --silent -H "Accept: application/vnd.docker.distribution.manifest.v2+json" -X DELETE https://registry.ipincloud.com:8443/v2/ocp4/openshift4/nfs-client-provisioner/manifests/sha256:022ea0b0d69834b652a4c53655d78642ae23f0324309097be874fb58d09d2919#回收鏡像空間podman exec -it mirror-registry /bin/registry garbage-collect /etc/docker/registry/config.yml/<code>7、創建installer配置文件
（1）創建installer文件夾
<code>rm -rf /data/installmkdir -p /data/installcd /data/install/<code>
（2）定製install-config.yaml文件
- 補充pullSecret
<code>[root@misc data]# cat /data/pull-secret.json{"auths":{"cloud.openshift.com":{"auth":"省略"}}}/<code>- 添加sshKey(3.1創建的公鑰文件內容)
<code>cat ~/.ssh/id_rsa.pub/<code>
- additionalTrustBundle（Mirror registry創建是生成的csr)
<code>[root@misc crts]# cat /etc/crts/ipincloud.com.crt-----BEGIN CERTIFICATE-----xxx省略-----END CERTIFICATE-----/<code>
- 添加代理
生產環境可以不用直連外網，通過在install-config.yaml文件為集群設置代理。
本次測試，為了加速外網下載，我在aws上事先搭建了一個v2ray server，misc服務器作為v2ray客戶端，具體搭建過程另文敘述。
- 在反覆試驗時，比如 install-config.yaml 所在的目錄是 config，必須 rm -rf install 而不是 rm -rf install/*，後者未刪除其中的隱藏文件 .openshift_install_state.json，有可能引起：x509: certificate has expired or is not yet valid。
- 在文檔和博客示例中 install-config.yaml 的 cidr 配置為 10 網段，由於未細看文檔理解成了節點機網段，這造成了整個過程中最莫名其妙的錯誤：no matches for kind MachineConfig。
- 最終文件內容如下：
<code>[root@centos75 install]# vi install-config.yamlapiVersion: v1baseDomain: ipincloud.comproxy: httpProxy: http://192.168.128.30:8001 httpsProxy: http://192.168.128.30:8001compute:- hyperthreading: Enabled name: worker replicas: 0controlPlane: hyperthreading: Enabled name: master replicas: 3metadata: name: ocptestnetworking: clusterNetwork: - cidr: 10.128.0.0/14 hostPrefix: 23 networkType: OpenShiftSDN serviceNetwork: - 172.30.0.0/16platform: none: {}fips: falsepullSecret: '{"auths":{"省略'additionalTrustBundle: | -----BEGIN CERTIFICATE----- 省略，注意這裡要前面空兩格 -----END CERTIFICATE-----imageContentSources:- mirrors: - registry.ipincloud.com:8443/ocp4/openshift4 source: quay.io/openshift-release-dev/ocp-release- mirrors: - registry.ipincloud.com:8443/ocp4/openshift4 source: quay.io/openshift-release-dev/ocp-v4.0-art-dev/<code>（3）備份定製install-config.yaml文件，便於以後可以重複使用
<code>cd /data/installcp install-config.yaml ../install-config.yaml.20200205/<code>
8、創建Kubernetes manifest和Ignition配置文件
（1）生成Kubernetes manifests文件
<code>openshift-install create manifests --dir=/data/install/<code>
注意：指定install-config.yaml所在目錄是，需要使用絕的路徑
（2）修改 manifests/cluster-scheduler-02-config.yml文件以防止pod調度到control plane節點
紅帽官方安裝文檔說明，kubernetes不支持ingress的load balancer訪問control-plane節點的pod
<code>a.打開manifests/cluster-scheduler-02-config.ymlb.找到mastersSchedulable參數，設置為Falsec.保存並退出。vi /data/install/manifests/cluster-scheduler-02-config.yml/<code>
（3）創建Ignition配置文件
注意：創建Ignition配置文件完成後，install-config.yaml文件將被刪除，請務必先備份此文件。
<code>openshift-install create ignition-configs --dir=/data/install/<code>
（4）將Ignition配置文件拷貝到http服務器目錄，待安裝時使用
<code>cd /data/install\\cp -f bootstrap.ign /var/www/html/ignition/bootstrap.ign\\cp -f master.ign /var/www/html/ignition/master1.ign\\cp -f master.ign /var/www/html/ignition/master2.ign\\cp -f master.ign /var/www/html/ignition/master3.ign\\cp -f worker.ign /var/www/html/ignition/worker1.ign\\cp -f worker.ign /var/www/html/ignition/worker2.igncd /var/www/html/ignition/chmod 755 *.ign/<code>
至此，已完成必要的配置文件設置，開始進入下一步創建節點。
9、定製RHCOS ISO
安裝時需要修改啟動參數，只能手動錄入，每臺機器修改很麻煩，容易出錯，因此我們採用genisoimage來定製每臺機器的安裝鏡像。
<code>#安裝鏡像創建工具yum -y install genisoimage libguestfs-toolssystemctl start libvirtd#設置環境變量export NGINX_DIRECTORY=/data/pkgexport RHCOSVERSION=4.3.0export VOLID=$(isoinfo -d -i ${NGINX_DIRECTORY}/rhcos-${RHCOSVERSION}-x86_64-installer.iso | awk '/Volume id/ { print $3 }')#生成一個臨時文件目錄，用於放置過程文件TEMPDIR=$(mktemp -d)echo $VOLIDecho $TEMPDIRcd ${TEMPDIR}# Extract the ISO content using guestfish (to avoid sudo mount)#使用guestfish可以將不用sudo mount將iso文件解壓出來guestfish -a ${NGINX_DIRECTORY}/rhcos-${RHCOSVERSION}-x86_64-installer.iso \\ -m /dev/sda tar-out / - | tar xvf -#定義修改配置文件的函數modify_cfg(){ for file in "EFI/redhat/grub.cfg" "isolinux/isolinux.cfg"; do # 添加恰當的 image 和 ignition url sed -e '/coreos.inst=yes/s|$| coreos.inst.install_dev=sda coreos.inst.image_url='"${URL}"'\\/install\\/'"${BIOSMODE}"'.raw.gz coreos.inst.ignition_url='"${URL}"'\\/ignition\\/'"${NODE}"'.ign ip='"${IP}"'::'"${GATEWAY}"':'"${NETMASK}"':'"${FQDN}"':'"${NET_INTERFACE}"':none:'"${DNS}"' nameserver='"${DNS}"'|' ${file} > $(pwd)/${NODE}_${file##*/} # 修改參數裡的啟動等待時間 sed -i -e 's/default vesamenu.c32/default linux/g' -e 's/timeout 600/timeout 10/g' $(pwd)/${NODE}_${file##*/} done}#設置url，網關、dns等iso啟動通用參數變量URL="http://192.168.128.30:8080"GATEWAY="192.168.128.254"NETMASK="255.255.255.0"DNS="192.168.128.30"#設置bootstrap節點變量NODE="bootstrap"IP="192.168.128.31"FQDN="bootstrap"BIOSMODE="bios"NET_INTERFACE="ens192"modify_cfg#設置master1節點變量NODE="master1"IP="192.168.128.32"FQDN="master1"BIOSMODE="bios"NET_INTERFACE="ens192"modify_cfg#設置master2節點變量NODE="master2"IP="192.168.128.33"FQDN="master2"BIOSMODE="bios"NET_INTERFACE="ens192"modify_cfg#設置master3節點變量NODE="master3"IP="192.168.128.34"FQDN="master3"BIOSMODE="bios"NET_INTERFACE="ens192"modify_cfg#設置master4節點變量NODE="worker1"IP="192.168.128.35"FQDN="worker1"BIOSMODE="bios"NET_INTERFACE="ens192"modify_cfg#設置master5節點變量NODE="worker2"IP="192.168.128.36"FQDN="worker2"BIOSMODE="bios"NET_INTERFACE="ens192"modify_cfg# 為每個節點創建不同的安裝鏡像# https://github.com/coreos/coreos-assembler/blob/master/src/cmd-buildextend-installer#L97-L103for node in bootstrap master1 master2 master3 worker1 worker2; do # 為每個節點創建不同的 grub.cfg and isolinux.cfg 文件 for file in "EFI/redhat/grub.cfg" "isolinux/isolinux.cfg"; do /bin/cp -f $(pwd)/${node}_${file##*/} ${file} done # 創建iso鏡像 genisoimage -verbose -rock -J -joliet-long -volset ${VOLID} \\ -eltorito-boot isolinux/isolinux.bin -eltorito-catalog isolinux/boot.cat \\ -no-emul-boot -boot-load-size 4 -boot-info-table \\ -eltorito-alt-boot -efi-boot images/efiboot.img -no-emul-boot \\ -o ${NGINX_DIRECTORY}/${node}.iso .done# 清除過程文件cdrm -Rf ${TEMPDIR}cd ${NGINX_DIRECTORY}/<code>9、在節點機器上安裝RHCOS
（1）將定製的ISO文件拷貝到vmware esxi主機上，準備裝節點
<code>[root@misc pkg]# scp bootstrap.iso [email protected]:/vmfs/volumes/hdd/iso[root@misc pkg]# scp m*.iso [email protected]:/vmfs/volumes/hdd/iso[root@misc pkg]# scp w*.iso [email protected]:/vmfs/volumes/hdd/iso/<code>
（2）按規劃創建master，設置從iso啟動安裝
- 進入啟動界面後，直接點擊安裝，系統自動回自動下載bios和配置文件，完成安裝
- 安裝完成後，需要將iso文件退出來，避免再次進入安裝界面
- 安裝順序是bootstrap,master1,master2,master3，待master安裝並啟動完成後，再進行worker安裝
- 安裝過程中可以通過proxy查看進度 http://registry.ipincloud.com:9000/
- 安裝過程中可以在misc節點查看詳細的bootstrap進度。
<code>openshift-install --dir=/data/install wait-for bootstrap-complete --log-level debug/<code>
注意事項：
- ignition和iso文件的正確匹配
- 我在安裝的時候，master1提示etcdmain: member ab84b6a6e4a3cc9a has already been bootstrapped，花了很多時間分析和解決問題，因為master1在安裝完成後，etcd組件會自動安裝並註冊為member,我再次使用iso文件重新安裝master1後，etcd自動安裝註冊時，會檢測到etcd及集群裡已經有這個member，無法重新註冊，因此這個節點的etcd一直無法正常啟動，解決辦法是：
手工修改-aster1節點的etcd的yaml文件,在exec etcd命令末尾增加–initial-cluster-state=existing參數，再刪除問題POD後,系統會自動重新安裝etcd pod，恢復正常。 正常啟動以後，要把這個改回去，否則machine-config回一直無法完成
<code>#[root@master1 /]# vi /etc/kubernetes/manifests/etcd-member.yaml exec etcd \\ --initial-advertise-peer-urls=https://${ETCD_IPV4_ADDRESS}:2380 \\ --cert-file=/etc/ssl/etcd/system:etcd-server:${ETCD_DNS_NAME}.crt \\ --key-file=/etc/ssl/etcd/system:etcd-server:${ETCD_DNS_NAME}.key \\ --trusted-ca-file=/etc/ssl/etcd/ca.crt \\ --client-cert-auth=true \\ --peer-cert-file=/etc/ssl/etcd/system:etcd-peer:${ETCD_DNS_NAME}.crt \\ --peer-key-file=/etc/ssl/etcd/system:etcd-peer:${ETCD_DNS_NAME}.key \\ --peer-trusted-ca-file=/etc/ssl/etcd/ca.crt \\ --peer-client-cert-auth=true \\ --advertise-client-urls=https://${ETCD_IPV4_ADDRESS}:2379 \\ --listen-client-urls=https://0.0.0.0:2379 \\ --listen-peer-urls=https://0.0.0.0:2380 \\ --listen-metrics-urls=https://0.0.0.0:9978 \\ --initial-cluster-state=existing [root@master1 /]# crictl podsPOD ID CREATED STATE NAME NAMESPACE ATTEMPTc4686dc3e5f4f 38 minutes ago Ready etcd-member-master1.ocptest.ipincloud.com openshift-etcd 5 [root@master1 /]# crictl rmp xxx/<code>- 檢查是否安裝完成
如果出現INFO It is now safe to remove the bootstrap resources，表示master節點安裝完成，控制面轉移到master集群。
<code>[root@misc install]# openshift-install --dir=/data/install wait-for bootstrap-complete --log-level debugDEBUG OpenShift Installer v4.3.0DEBUG Built from commit 2055609f95b19322ee6cfdd0bea73399297c4a3eINFO Waiting up to 30m0s for the Kubernetes API at https://api.ocptest.ipincloud.com:6443...INFO API v1.16.2 upINFO Waiting up to 30m0s for bootstrapping to complete...DEBUG Bootstrap status: completeINFO It is now safe to remove the bootstrap resources[root@misc install]#/<code>
（3）安裝worker
- 進入啟動界面後，直接點擊安裝，系統自動回自動下載bios和配置文件，完成安裝
- 安裝完成後，需要將iso文件退出來，避免再次進入安裝界面
- 安裝順序是bootstrap,master1,master2,master3，待master安裝並啟動完成後，再進行worker安裝
- 安裝過程中可以通過proxy查看進度 http://registry.ipincloud.com:9000/
- 也可以在misc節點是查看詳細安裝節點
<code>[root@misc redhat-operators-manifests]# openshift-install --dir=/data/install wait-for install-complete --log-level debugDEBUG OpenShift Installer v4.3.0DEBUG Built from commit 2055609f95b19322ee6cfdd0bea73399297c4a3eINFO Waiting up to 30m0s for the cluster at https://api.ocptest.ipincloud.com:6443 to initialize...DEBUG Cluster is initializedINFO Waiting up to 10m0s for the openshift-console route to be created...DEBUG Route found in openshift-console namespace: consoleDEBUG Route found in openshift-console namespace: downloadsDEBUG OpenShift console route is createdINFO Install complete!INFO To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=/data/install/auth/kubeconfig'INFO Access the OpenShift web-console here:https://console-openshift-console.apps.ocptest.ipincloud.comINFO Login to the console with user: kubeadmin, password: pubmD-8Baaq-IX36r-WIWWf/<code>
- 需要審批worker節點的加入申請
查看待審批的csr
<code>[root@misc ~]# oc get csrNAME AGE REQUESTOR CONDITIONcsr-7lln5 70m system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issuedcsr-d48xk 69m system:node:master1.ocptest.ipincloud.com Approved,Issuedcsr-f2g7r 69m system:node:master2.ocptest.ipincloud.com Approved,Issuedcsr-gbn2n 69m system:node:master3.ocptest.ipincloud.com Approved,Issuedcsr-hwxwx 13m system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Pendingcsr-ppgxx 13m system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Pendingcsr-wg874 70m system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issuedcsr-zkp79 70m system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issued[root@misc ~]#/<code>
執行審批
<code>oc get csr -ojson | jq -r '.items[] | select(.status == {} ) | .metadata.name' | xargs oc adm certificate approve/<code>（3）在misc上啟動nfs
<code>bash /data/pkg/ocp4-upi-helpernode/files/nfs-provisioner-setup.sh#查看狀態oc get pods -n nfs-provisioner（4）ocp內部registry使用nfs作為存儲oc patch configs.imageregistry.operator.openshift.io cluster -p '{"spec":{"storage":{"pvc":{"claim":""}}}}' --type=mergeoc get clusteroperator image-registry/<code>10 配置登錄
（1）配置普通管理員賬號
<code>#在misc機器上創建admin tokenmkdir -p ~/authhtpasswd -bBc ~/auth/admin-passwd admin scwang18#拷貝到本地mkdir -p ~/authscp -P 20030 [email protected]:/root/auth/admin-passwd ~/auth/#在 OAuth Details 頁面添加 HTPasswd 類型的 Identity Providers 並上傳admin-passwd 文件。https://console-openshift-console.apps.ocptest.ipincloud.com#授予新建的admin用戶集群管理員權限oc adm policy add-cluster-role-to-user cluster-admin admin/<code>
分享到:
閱讀更多 雲原生學習班 的文章
