使用sqoop 導入MySQL數據到 HBASE

1. 安裝sqoop

下載

<code>wget https://mirror.bit.edu.cn/apache/sqoop/1.4.7/sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz/<code>

解壓到/usr/local/目錄，重命名為sqoop

修改/usr/local/sqoop/conf/sqoop-env.sh文件

<code># Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# included in all the hadoop/># should not be executable directly
# also should not be passed any arguments, since we need original $*

# Set Hadoop-specific environment variables here.

#Set path to where bin/hadoop is available
#export HADOOP_COMMON_HOME=
export HADOOP_COMMON_HOME=/usr/local/hadoop


#Set path to where hadoop-*-core.jar is available
#export HADOOP_MAPRED_HOME=
export HADOOP_MAPRED_HOME=/usr/local/hadoop


#set the path to where bin/hbase is available
#export HBASE_HOME=
export HBASE_HOME=/usr/local/hbase

#Set the path to where bin/hive is available
#export HIVE_HOME=

#Set the path for where zookeper config dir is
#export ZOOCFGDIR=
export ZOOCFGDIR=/usr/local/zookeeper/conf
/<code>

2. 開始使用sqoop import 導入數據表

編寫sqoop-import-hbase.sh腳本

<code>#!/bin/sh

${SQOOP_HOME}/bin/sqoop import \\
--connect jdbc:mysql://ip:port/db \\
--username dbuser \\
--password dbpwd \\
--table order_info \\
--null-string "" \\
--hbase-table db:order_info
--column-family order_info \\
--hase-row-key serial_num \\
--num-mappers 4/<code>

執行腳本，靜待導入完成

過程出現error: KeeperErrorCode = NoNode for /hbase/master 錯誤

檢查發現zookeeper中zoo.cfg 集群配置和myid不匹配

由於未作分區設置，HBASE中數據全部在同一個region

importing_data_into_hbase 手冊 https://sqoop.apache.org/docs/1.4.2/SqoopUserGuide.html#_importing_data_into_hbase

閱讀更多 花青蚨 的文章

關鍵字: Hadoop Hive 數據