1.文檔編寫目的
Cloudera在2019年3月30日，對外宣佈正式發佈Cloudera Enterprise 6.2，相關介紹可以參考Fayson昨天的文章《 》，《 》和《 》。本文檔Fayson主要描述如何在Redhat7.4安裝CDH6.2。CDH6與CDH5的安裝步驟一致，主要包括以下四部分：
1.安全前置準備，包括安裝操作系統、關閉防火牆、同步服務器時鐘等；
2.外部數據庫如MySQL安裝
3.安裝Cloudera Manager；
4.安裝CDH集群；
請務必注意CDH6的安裝前置條件包括如下：
- 外部數據庫支持：
MySQL 5.7或更高
MariaDB 5.5或更高
PostgreSQL 8.4或更高
Oracle 12c或更高
- JDK
Oracle JDK1.8，將不再支持JDK1.7
- 操作系統支持
RHEL 6.8或更高
RHEL 7.2或更高
SLES 12 SP2或更高
Ubuntu 16或更高
- 本次Fayson的測試環境為
1.CM和CDH版本為6.2
2.Redhat7.4
3.JDK1.8.0_181
4.MariaDB-5.5.56
5.root用戶安裝
2.前置準備
2.1.hostname及hosts配置
集群中各個節點之間能互相通信使用靜態IP地址。IP地址和主機名通過/etc/hosts配置，主機名通過/etc/hostname進行配置。
以cm節點（172.31.6.83）為例：
- hostname配置
/etc/hostname文件如下：
ip-172-31-6-83.ap-southeast-1.compute.internal
或者你可以通過命令修改立即生效
[root@ip-172-31-6-83 ~]$ hostnamectl set-hostname ip-172-31-6-83.ap-southeast-1.compute.internal
- hosts配置
/etc/hosts文件如下:
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
172.31.6.83 ip-172-31-6-83.ap-southeast-1.compute.internal
172.31.12.142 ip-172-31-12-142.ap-southeast-1.compute.internal
172.31.9.113 ip-172-31-9-113.ap-southeast-1.compute.internal
172.31.4.105 ip-172-31-4-105.ap-southeast-1.compute.internal
以上兩步操作，在集群中其它節點做相應配置。確認需要安裝的4臺主機的hosts文件：
2.2.禁用SELinux
在所有節點執行setenforce 0 命令，此處使用批處理shell執行：
[root@ip-172-31-6-83 shell]# sh ssh_do_all.sh node.list "setenforce 0"
集群所有節點修改/etc/selinux/config文件如下：
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of three two values:
# targeted - Targeted processes are protected,
# minimum - Modification of targeted policy. Only selected processes are protected.
# mls - Multi Level Security protection.
SELINUXTYPE=targeted
2.3.關閉防火牆
集群所有節點執行 systemctl stop命令，此處通過shell批量執行命令如下：
[root@ip-172-31-6-83 shell]# sh ssh_do_all.sh node.list "systemctl stop firewalld"
[root@ip-172-31-6-83 shell]# sh ssh_do_all.sh node.list "systemctl disable firewalld"
[root@ip-172-31-6-83 shell]# sh ssh_do_all.sh node.list "systemctl status firewalld"
2.4.集群時鐘同步
在Redhat7.x的操作系統上，已經默認的安裝了chrony，我們這裡先卸載chrony，然後安裝ntp。使用ntp來配置各臺機器的時鐘同步，將cm（172.31.6.83）服務作為本地ntp服務器，其它3臺服務器與其保持同步。
1.所有機器卸載chrony
[root@ip-172-31-6-83 shell]# sh ssh_do_all.sh node.list "yum -y remove chrony"
2.所有機器安裝ntp
[root@ip-172-31-6-83 shell]# sh ssh_do_all.sh node.list "yum -y install ntp"
3.cm機器配置時鐘與自己同步
[root@ ip-172-31-6-83 shell]# vim /etc/ntp.conf
#server 0.rhel.pool.ntp.org iburst
#server 1.rhel.pool.ntp.org iburst
#server 2.rhel.pool.ntp.org iburst
#server 3.rhel.pool.ntp.org iburst
server 127.127.1.0 # local clock
fudge 127.127.1.0 stratum 10
4.集群其它節點，配置找cm機器去同步
#server 0.rhel.pool.ntp.org iburst
#server 1.rhel.pool.ntp.org iburst
#server 2.rhel.pool.ntp.org iburst
#server 3.rhel.pool.ntp.org iburst
server 172.31.6.83
5.重啟所有機器的ntp服務
[root@ ip-172-31-6-83 shell]# sh ssh_do_all.sh node.list "systemctl restart ntpd"
[root@ ip-172-31-6-83 shell]# sh ssh_do_all.sh node.list "systemctl status ntpd"
6.驗證始終同步，在所有節點執行ntpq -p命令，如下使用腳本批量執行
[root@ ip-172-31-6-83 shell]# sh ssh_do_all.sh node.list "ntpq -p"
左邊出現*號表示同步成功。
2.5.設置swap
所有節點執行
[root@ ip-172-31-6-83 shell]# sh ssh_do_all.sh node.list "echo vm.swappiness = 1 >> /etc/sysctl.conf"
[root@ ip-172-31-6-83 shell]#
2.6.設置透明大頁面
所有節點執行：
[root@ ip-172-31-6-83 shell]# sh ssh_do_all.sh node.list "echo never > /sys/kernel/mm/transparent_hugepage/defrag "
[root@ ip-172-31-6-83 shell]# sh ssh_do_all.sh node.list "echo never > /sys/kernel/mm/transparent_hugepage/enabled"
設置開機自關閉
將如下腳本添加到/etc/rc.d/rc.local文件中
if test -f /sys/kernel/mm/transparent_hugepage/enabled; then echo never > /sys/kernel/mm/transparent_hugepage/enabled fi if test -f /sys/kernel/mm/transparent_hugepage/defrag; then echo never > /sys/kernel/mm/transparent_hugepage/defrag fi
同步到所有節點
2.7.配置操作系統repo
Fayson用的是AWS的環境，這步是可以省略的，放在這裡供物理機部署的兄弟們參考。
- 掛載操作系統iso文件
[ec2-user@ip-172-31-2-159 ~]$ sudo mkdir /media/DVD1
[ec2-user@ip-172-31-2-159 ~]$ sudo mount -o loop
CentOS-7-x86_64-DVD-1611.iso /media/DVD1/
- 配置操作系統repo
[ec2-user@ip-172-31-2-159 ~]$ sudo vim /etc/yum.repos.d/local_os.repo
[local_iso]
name=CentOS-$releasever - Media
baseurl=file:///media/DVD1
gpgcheck=0
enabled=1
[ec2-user@ip-172-31-2-159 ~]$ sudo yum repolist
2.8.安裝http服務
- 安裝httpd服務
[ec2-user@ip-172-31-2-159 ~]$ sudo yum -y install httpd
- 啟動httpd服務
[ec2-user@ip-172-31-2-159 ~]$ sudo systemctl start httpd
- 安裝完httpd後，重新制作操作系統repo，換成http的方式方便其它服務器也可以訪問
[ec2-user@ip-172-31-2-159 ~]$ sudo mkdir /var/www/html/iso
[ec2-user@ip-172-31-2-159 ~]$ sudo scp -r /media/DVD1/* /var/www/html/iso/
[ec2-user@ip-172-31-2-159 ~]$ sudo vim /etc/yum.repos.d/os.repo
[osrepo]
name=os_repo
baseurl=http://172.31.2.159/iso/
enabled=true
gpgcheck=false
[ec2-user@ip-172-31-2-159 ~]$ sudo yum repolist
1.修改/etc/httpd/conf/httpd.conf配置文件，在<ifmodule>中修改以下內容/<ifmodule>
AddType application/x-gzip .gz .tgz .parcel
2.保存httpd.conf的修改，並重啟httpd服務
[root@ip-172-31-6-83 java]# systemctl restart httpd
2.9.安裝MariaDB
1.安裝MariaDB
[root@ ip-172-31-6-83 ~]# yum -y install mariadb
[root@ ip-172-31-6-83 ~]# yum -y install mariadb-server
2.啟動並配置MariaDB
[root@ ip-172-31-6-83 ~]# systemctl start mariadb
[root@ ip-172-31-6-83 ~]# /usr/bin/mysql_secure_installation
NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB
SERVERS IN PRODUCTION USE! PLEASE READ EACH STEP CAREFULLY!
In order to log into MariaDB to secure it, we'll need the current
password for the root user. If you've just installed MariaDB, and
you haven't set the root password yet, the password will be blank,
so you should just press enter here.
Enter current password for root (enter for none):
OK, successfully used password, moving on...
Setting the root password ensures that nobody can log into the MariaDB
root user without the proper authorisation.
Set root password? [Y/n] Y
New password:
Re-enter new password:
Password updated successfully!
Reloading privilege tables..
... Success!
By default, a MariaDB installation has an anonymous user, allowing anyone
to log into MariaDB without having to have a user account created for
them. This is intended only for testing, and to make the installation
go a bit smoother. You should remove them before moving into a
production environment.
Remove anonymous users? [Y/n] Y
... Success!
Normally, root should only be allowed to connect from 'localhost'. This
ensures that someone cannot guess at the root password from the network.
Disallow root login remotely? [Y/n] n
... skipping.
By default, MariaDB comes with a database named 'test' that anyone can
access. This is also intended only for testing, and should be removed
before moving into a production environment.
Remove test database and access to it? [Y/n] Y
- Dropping test database...
... Success!
- Removing privileges on test database...
... Success!
Reloading the privilege tables will ensure that all changes made so far
will take effect immediately.
Reload privilege tables now? [Y/n] Y
... Success!
Cleaning up...
All done! If you've completed all of the above steps, your MariaDB
installation should now be secure.
Thanks for using MariaDB!
3.建立CM，Hive等需要的表
[root@ip-172-31-6-83 ~]# mysql -u root -p
Enter password:
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 9
Server version: 5.5.56-MariaDB MariaDB Server
Copyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]>
create database metastore default character set utf8; CREATE USER 'hive'@'%' IDENTIFIED BY 'password'; GRANT ALL PRIVILEGES ON metastore. * TO 'hive'@'%'; FLUSH PRIVILEGES; create database cm default character set utf8; CREATE USER 'cm'@'%' IDENTIFIED BY 'password'; GRANT ALL PRIVILEGES ON cm. * TO 'cm'@'%'; FLUSH PRIVILEGES; create database am default character set utf8; CREATE USER 'am'@'%' IDENTIFIED BY 'password'; GRANT ALL PRIVILEGES ON am. * TO 'am'@'%'; FLUSH PRIVILEGES; create database rm default character set utf8; CREATE USER 'rm'@'%' IDENTIFIED BY 'password'; GRANT ALL PRIVILEGES ON rm. * TO 'rm'@'%'; FLUSH PRIVILEGES;
create database hue default character set utf8; CREATE USER 'hue'@'%' IDENTIFIED BY 'password'; GRANT ALL PRIVILEGES ON hue. * TO 'hue'@'%'; FLUSH PRIVILEGES;
create database oozie default character set utf8; CREATE USER 'oozie'@'%' IDENTIFIED BY 'password'; GRANT ALL PRIVILEGES ON oozie. * TO 'oozie'@'%'; FLUSH PRIVILEGES;
create database sentry default character set utf8; CREATE USER 'sentry'@'%' IDENTIFIED BY 'password'; GRANT ALL PRIVILEGES ON sentry. * TO 'sentry'@'%'; FLUSH PRIVILEGES;
create database nav_ms default character set utf8; CREATE USER 'nav_ms'@'%' IDENTIFIED BY 'password'; GRANT ALL PRIVILEGES ON nav_ms. * TO 'nav_ms'@'%'; FLUSH PRIVILEGES;
create database nav_as default character set utf8;
CREATE USER 'nav_as'@'%' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON nav_as. * TO 'nav_as'@'%';
FLUSH PRIVILEGES;
- 安裝JDBC驅動
[root@ip-172-31-6-83 shell]# mkdir -p /usr/share/java/
[root@ip-172-31-6-83 ~]# mv mysql-connector-java-5.1.34.jar /usr/share/java/
[root@ip-172-31-6-83 ~]# cd /usr/share/java
[root@ip-172-31-6-83 java]# chmod 777 mysql-connector-java-5.1.34.jar
[root@ip-172-31-6-83 java]# ln -s mysql-connector-java-5.1.34.jar mysql-connector-java.jar
[root@ip-172-31-6-83 java]# ll
total 940
-rwxrwxrwx. 1 root root 960372 May 16 15:53 mysql-connector-java-5.1.34.jar
lrwxrwxrwx. 1 root root 31 May 16 15:53 mysql-connector-java.jar -> mysql-connector-java-5.1.34.jar
3.Cloudera Manager安裝
3.1.配置本地repo源
1.下載CM6.2的安裝包，地址為：
https://archive.cloudera.com/cm6/6.2.0/redhat7/yum/RPMS/x86_64/cloudera-manager-agent-6.2.0-968826.el7.x86_64.rpm
https://archive.cloudera.com/cm6/6.2.0/redhat7/yum/RPMS/x86_64/cloudera-manager-daemons-6.2.0-968826.el7.x86_64.rpm
https://archive.cloudera.com/cm6/6.2.0/redhat7/yum/RPMS/x86_64/cloudera-manager-server-6.2.0-968826.el7.x86_64.rpm
https://archive.cloudera.com/cm6/6.2.0/redhat7/yum/RPMS/x86_64/cloudera-manager-server-db-2-6.2.0-968826.el7.x86_64.rpm
https://archive.cloudera.com/cm6/6.2.0/redhat7/yum/RPMS/x86_64/enterprise-debuginfo-6.2.0-968826.el7.x86_64.rpm
https://archive.cloudera.com/cm6/6.2.0/redhat7/yum/RPMS/x86_64/oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm
https://archive.cloudera.com/cm6/6.2.0/allkeys.asc
2.下載CDH6.2的安裝包，地址為：
https://archive.cloudera.com/cdh6/6.2.0/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel
https://archive.cloudera.com/cdh6/6.2.0/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel.sha1
https://archive.cloudera.com/cdh6/6.2.0/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel.sha256
https://archive.cloudera.com/cdh6/6.2.0/parcels/manifest.json
3.將Cloudera Manager安裝需要的6個rpm包以及一個asc文件下載到本地，放在同一目錄，執行createrepo命令生成rpm元數據。
[root@ip-172-31-6-83 cm6.2]# ll
total 1364096
-rw-r--r-- 1 root root 14041 Mar 28 17:13 allkeys.asc
-rw-r--r-- 1 root root 10215488 Mar 28 17:13 cloudera-manager-agent-6.2.0-968826.el7.x86_64.rpm
-rw-r--r-- 1 root root 1187380436 Mar 28 17:13 cloudera-manager-daemons-6.2.0-968826.el7.x86_64.rpm
-rw-r--r-- 1 root root 9984 Mar 28 17:13 cloudera-manager-server-6.2.0-968826.el7.x86_64.rpm
-rw-r--r-- 1 root root 10992 Mar 28 17:13 cloudera-manager-server-db-2-6.2.0-968826.el7.x86_64.rpm
-rw-r--r-- 1 root root 14200108 Mar 28 17:13 enterprise-debuginfo-6.2.0-968826.el7.x86_64.rpm
-rw-r--r-- 1 root root 184988341 Mar 28 17:13 oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm
[root@ip-172-31-6-83 cm6.2]# createrepo .
Spawning worker 0 with 2 pkgs
Spawning worker 1 with 2 pkgs
Spawning worker 2 with 1 pkgs
Spawning worker 3 with 1 pkgs
Workers Finished
Saving Primary metadata
Saving file lists metadata
Saving other metadata
Generating sqlite DBs
Sqlite DBs complete
4.配置Web服務器
將上述cdh6.2/cm6.2目錄移動到/var/www/html目錄下, 使得用戶可以通過HTTP訪問這些rpm包。
[root@ip-172-31-6-83 ~]# mv cm6.2/ cdh6.2/ /var/www/html/
[root@ip-172-31-6-83 ~]# cd /var/www/html/
[root@ip-172-31-6-83 html]# ll *6.2
驗證瀏覽器能否正常訪問
5.製作Cloudera Manager的repo源
[root@ip-172-31-6-83 ~]# vim /etc/yum.repos.d/cm.repo
[cmrepo]
name = cm_repo
baseurl = baseurl=http://172.31.6.83/cm6.2
enable = true
gpgcheck = false
[root@ip-172-31-6-83 yum.repos.d]# yum repolist
Loaded plugins: amazon-id, rhui-lb, search-disabled-repos
repo id repo name status
cmrepo cm_repo 6
rhui-REGION-client-config-server-7/x86_64 Red Hat Update Infrastructure 2.0 Client Configuration Server 7 2
rhui-REGION-rhel-server-releases/7Server/x86_64 Red Hat Enterprise Linux Server 7 (RPMs) 23,947
rhui-REGION-rhel-server-rh-common/7Server/x86_64 Red Hat Enterprise Linux Server 7 RH Common (RPMs) 235
repolist: 24,190
6.驗證安裝JDK
[root@ip-172-31-6-83 cm6.2]# yum -y install oracle-j2sdk1.8-1.8.0+update181-1.x86_64
3.2.安裝Cloudera Manager Server
1.通過yum安裝Cloudera Manager Server
yum -y install cloudera-manager-server
2.初始化數據庫
[root@ip-172-31-6-83 cm6.2]# /opt/cloudera/cm/schema/scm_prepare_database.sh mysql cm cm password
JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
Verifying that we can write to /etc/cloudera-scm-server
Creating SCM configuration file in /etc/cloudera-scm-server
Executing: /usr/java/jdk1.8.0_181-cloudera/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/opt/cloudera/cm/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.
[ main] DbCommandExecutor INFO Successfully connected to database.
All done, your SCM database is configured correctly!
[root@ip-172-31-6-83 cm6.2]#
3.啟動Cloudera Manager Server
[root@ ip-172-31-6-83 ~]# systemctl start cloudera-scm-server
4.檢查端口是否監聽
[root@ip-172-31-6-83 cm6.2]# netstat -lnpt | grep 7180
tcp 0 0 0.0.0.0:7180 0.0.0.0:* LISTEN 1308/java
5.通過http://cm_ip:7180/cmf/login訪問CM
分享到:
閱讀更多 Hadoop實操 的文章
關鍵字: Hat Enterprise Cloudera
