搭建5节点hadoop+zookeeper+hbase+spark+kafka+storm(1):Zookeeper

本篇介绍Zookeeper集群安装。
版本:java 1.8,Hadoop 2.7.3,HBase 1.2.5,Zookeeper 3.4.10,Spark 2.1.1,Scala 2.12.2,Kafka 0.10.2.1,Storm 1.1.0。
以下操作都是以root用户进行,如需用户组和权限设定需自行配置。


1. 服务器信息及节点分配

服务器信息:

No Hostname 内网IP 外网IP OS
1 node01 192.168.5.11 192.168.205.50 Centos7
2 node02 192.168.5.12 192.168.205.62 Centos7
3 node03 192.168.5.13 192.168.205.63 Centos7
4 node04 192.168.5.14 192.168.205.70 Centos7
5 node05 192.168.5.15 192.168.205.102 Centos7

节点分配:

节点 node01 node02 node03 node04 node05
namenode YES YES NO NO NO
datanode NO NO YES YES YES
journalnode YES YES YES YES YES
zookeeper YES YES YES YES YES
hbase 主Master 备份Master RegionServer RegionServer RegionServer
spark YES YES YES YES YES
kafka YES YES YES YES YES
storm YES YES YES YES YES

2. 安装Zookeeper

2.1 下载安装包

1
wget http://mirrors.hust.edu.cn/apache/zookeeper/zookeeper-3.4.10/zookeeper-3.4.10.tar.gz

2.2 创建zoo.cfg

复制zoo_sample.cfg为zoo.cfg,修改dataDir和dataLogDir路径,并在最后添加节点信息:

zoo.cfg
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/usr/zookeeper/data
dataLogDir=/usr/zookeeper/logs
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1

server.1=node01:2888:3888
server.2=node02:2888:3888
server.3=node03:2888:3888
server.4=node04:2888:3888
server.5=node05:2888:3888

2.3 安装部署

写一个简易部署脚本:

zk.sh
1
2
3
4
5
6
7
8
9
# /bin/bash

tar -xvf ./zookeeper-3.4.10.tar.gz
mv ./zookeeper-3.4.10 /usr/zookeeper
mkdir /usr/zookeeper/data
mkdir /usr/zookeeper/logs
mv ./zoo.cfg /usr/zookeeper/conf/zoo.cfg
echo "export ZOOKEEPER_HOME=/usr/zookeeper" >> /root/.bash_profile
echo "export PATH=\$ZOOKEEPER_HOME/bin:\$PATH" >> /root/.bash_profile

修改权限并执行,执行时确保zk.sh, zoo.cfg和zookeeper-3.4.10.tar.gz在同一目录下:
1
2
3
chmod +x ./zk.sh
./zk.sh
source /root/.bash_profile

2.4 创建myid

在zookeeper的dataDir下添加myid,记录该节点的zk id,注意id要与zoo.cfg中该节点的server.x匹配,以server.1为例:

1
2
cd /usr/zookeeper/data
echo "1" > myid

2.5 部署所有节点

使用sftp将zk.sh, zoo.cfg, zookeeper-3.4.10.tar.gz传到其他所有节点上并按照以上步骤进行部署。


3. Zookeeper使用

3.1 启动Zookeeper集群

所有节点上执行:

1
zkServer.sh start

3.2 查看Zookeeper状态

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
[root@node01 ~]# zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /usr/zookeeper/bin/../conf/zoo.cfg
Mode: follower
[root@node01 ~]# ssh node02
[root@node02 ~]# zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /usr/zookeeper/bin/../conf/zoo.cfg
Mode: follower
[root@node02 ~]# ssh node03
[root@node03 ~]# zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /usr/zookeeper/bin/../conf/zoo.cfg
Mode: follower
[root@node03 ~]# ssh node04
[root@node04 ~]# zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /usr/zookeeper/bin/../conf/zoo.cfg
Mode: follower
[root@node04 ~]# ssh node05
[root@node05 ~]# zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /usr/zookeeper/bin/../conf/zoo.cfg
Mode: leader

3.3 停止Zookeeper

1
zkServer.sh stop

3.4 重启Zookeeper

1
zkServer.sh restart

参照资料:
hadoop、zookeeper、hbase、spark集群环境搭建
Zookeeper3.4.9分布式集群安装