Docker安装Hadoop3

1、准备需要的软件：

jdk-8u161-linux-x64.tar

hadoop-3.1.0.tar

2、docker安装centos7、启动

docker pull centos

docker run -d -t centos

3、安装ssh

docker container exec -it a8d4b8936b29 /bin/bash 进入镜像系统

yum -y install openssh-server

yum -y install openssh-clients

/usr/sbin/sshd -D启动sshd报错

运行以下命令解决错误

ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key -N ""

ssh-keygen -t ecdsa -f /etc/ssh/ssh_host_ecdsa_key -N ""

ssh-keygen -t ed25519 -f /etc/ssh/ssh_host_ed25519_key -N ""

vi /etc/ssh/sshd_config 修改配置文件把UsePAM yes 改为#UsePAM no

修改root的密码

passwd root

退出镜像系统，回到宿主主机，建立刚才配置好ssh的镜像，取名test1

docker commit a8d4b8936b29 test1

运行镜像test1

docker run -d -p 10022:22 test1 /usr/sbin/sshd -D

4、安装jdk、hadoop

从宿主主机拷贝jdk、hadoop到镜像系统/opt目录

docker cp jdk-8u161-linux-x64.tar 0d3dc5424d22:/opt

docker cp hadoop-3.1.0.tar 0d3dc5424d22:/opt

解压拷贝到/usr/local/jdk8 /usr/local/hadoop

tar -xf hadoop-3.1.0.tar

tar -xf jdk-8u161-linux-x64.tar

mv hadoop-3.1.0 /usr/local/hadoop

mv jdk1.8.0_161 /usr/local/jdk8

配置jdk、hadoop环境变量

vi ~/.bashrc

source ~/.bashrc

export JAVA_HOME=/usr/local/jdk8

export HADOOP_HOME=/usr/local/hadoop

export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

修改hadoop配置文件

配置/usr/local/hadoop/etc/hadoop/hadoop-env.sh

vi hadoop-env.sh export JAVA_HOME=/usr/local/jdk8

配置/usr/local/hadoop/etc/hadoop/core-site.xml

vi core-site.xml

fs.defaultFS

hdfs://localhost:9000

配置/usr/local/hadoop/etc/hadoop/hdfs-site.xml

vi hdfs-site.xml

dfs.replication

1

配置/usr/local/hadoop/etc/hadoop/mapred-site.xml

vi mapred-site.xml

mapreduce.framework.name

yarn

mapreduce.admin.user.env

HADOOP_MAPRED_HOME=$HADOOP_COMMON_HOME

yarn.app.mapreduce.am.env

HADOOP_MAPRED_HOME=$HADOOP_COMMON_HOME

配置/usr/local/hadoop/etc/hadoop/yarn-site.xml

vi yarn-site.xml

yarn.nodemanager.aux-services

mapreduce_shuffle

yarn.nodemanager.env-whitelist

JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME

5、修改启动关闭脚本

vi start-dfs.sh 、 vi stop-dfs.sh 在开始部分加入以下配置

HDFS_DATANODE_USER=root

HADOOP_SECURE_DN_USER=hdfs

HDFS_NAMENODE_USER=root

HDFS_SECONDARYNAMENODE_USER=root

vi start-yarn.sh 、 vi stop-yarn.sh 在开始部分加入以下配置

YARN_RESOURCEMANAGER_USER=root

HADOOP_SECURE_DN_USER=yarn

YARN_NODEMANAGER_USER=root

5、ssh免密登录

ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

chmod 0600 ~/.ssh/authorized_keys

6、格式化namenode

hdfs namenode -format

7、启动dfs、yarn

start-dfs.sh启动dfs

start-yarn.sh启动yarn

8、试用

hdfs dfs -mkdir /input

hdfs dfs -put ../etc/hadoop/*.xml /input

hadoop jar ../share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.0.jar grep /input /output 'dfs[a-z.]+'

閱讀更多 Lucif墮落天使 的文章

關鍵字: Hadoop HDFS MapReduce