安装GlusterFS
这里我们使用的是centos7版本。官方快速安装手册建议使用Fedora21以上的版本。
Step 1 – 使用虚拟机创建至少2个节点
- CentOS7
-
hostname :gluster1 gluster2
-
network: extranl network: 192.168.100.0/24 , storage network: 10.0.2.0/24
- 2块磁盘，一块系统盘，一块给GlusterFS存储。
Step 2 -格式化磁盘.
格式化磁盘，块大小512B
[root@glusterfs1 glusterfs]# mkfs.xfs -i size=512 /dev/vdb1
meta-data=/dev/vdb1 isize=512 agcount=4, agsize=327616 blks
= sectsz=512 attr=2, projid32bit=1
= crc=0 finobt=0
data = bsize=4096 blocks=1310464, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=0
log =internal log bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
创建挂载目录
[root@glusterfs1 glusterfs]# mkdir /mnt/glusterfs/
修改系统挂载文件实现自动挂载
[root@glusterfs1 glusterfs]# vim /etc/fstab
[root@glusterfs1 glusterfs]# cat /etc/fstab
/dev/vdb1 /mnt/glusterfs xfs defaults 1 2
#
# /etc/fstab
# Created by anaconda on Sat Aug 8 05:37:16 2015
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/centos-root / xfs defaults 1 1
UUID=36995c57-4284-4433-a0b7-fc571670e886 /boot xfs defaults 1 2
/dev/mapper/centos-swap swap swap defaults 0 0
然后挂载上来,大小5GB。
[root@glusterfs1 glusterfs]# mount -a
[root@glusterfs1 glusterfs]# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/centos-root 7022592 1409368 5613224 21% /
devtmpfs 933432 0 933432 0% /dev
tmpfs 942208 0 942208 0% /dev/shm
tmpfs 942208 8568 933640 1% /run
tmpfs 942208 0 942208 0% /sys/fs/cgroup
/dev/vda1 508588 139920 368668 28% /boot
/dev/vdb1 5231616 32944 5198672 1% /mnt/glusterfs
另一台作同样的操作
[root@gluster2 ~]# fdisk /dev/vdb
Welcome to fdisk (util-linux 2.23.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x70ba8ed9.
Command (m for help): n
Partition type:
p primary (0 primary, 0 extended, 4 free)
e extended
Select (default p): p
Partition number (1-4, default 1):
First sector (2048-10485759, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-10485759, default 10485759):
Using default value 10485759
Partition 1 of type Linux and of size 5 GiB is set
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
[root@gluster2 ~]# mkfs.xfs -i size=512 /dev/vdb1
meta-data=/dev/vdb1 isize=512 agcount=4, agsize=327616 blks
= sectsz=512 attr=2, projid32bit=1
= crc=0 finobt=0
data = bsize=4096 blocks=1310464, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=0
log =internal log bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
[root@gluster2 ~]# mkdir /mnt/glusterfs
[root@gluster2 ~]# echo '/dev/vdb1 /mnt/glusterfs xfs defaults 1 2' >> /etc/fstab
[root@gluster2 ~]# cat /etc/fstab
#
# /etc/fstab
# Created by anaconda on Sat Aug 8 05:37:16 2015
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/centos-root / xfs defaults 1 1
UUID=36995c57-4284-4433-a0b7-fc571670e886 /boot xfs defaults 1 2
/dev/mapper/centos-swap swap swap defaults 0 0
/dev/vdb1 /mnt/glusterfs xfs defaults 1 2
[root@gluster2 ~]# mount -a
[root@gluster2 ~]# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/centos-root 7022592 1409840 5612752 21% /
devtmpfs 933432 0 933432 0% /dev
tmpfs 942208 0 942208 0% /dev/shm
tmpfs 942208 8568 933640 1% /run
tmpfs 942208 0 942208 0% /sys/fs/cgroup
/dev/vda1 508588 139920 368668 28% /boot
/dev/vdb1 5231616 32944 5198672 1% /mnt/glusterfs
[root@gluster2 ~]# Step 3 - 安装 GlusterFS
首先安装官方软件源，
[root@glusterfs1 glusterfs]# cd /etc/yum.repos.d/
[root@glusterfs1 yum.repos.d]# wget />
--2015-09-18 17:13:27-- />
Resolving download.gluster.org (download.gluster.org)... 50.57.69.89
Connecting to download.gluster.org (download.gluster.org)|50.57.69.89|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1079 (1.1K) [text/plain]
Saving to: ‘glusterfs-epel.repo’
100%[==========================================================================================>] 1,079 --.-K/s in 0s
2015-09-18 17:13:27 (76.0 MB/s) - ‘glusterfs-epel.repo’ saved [1079/1079]
[root@glusterfs1 yum.repos.d]#
在所有节点上运行安装命令。
[root@glusterfs1 yum.repos.d]# yum install glusterfs-server -y
...
Dependencies Resolved
====================================================================================================================================
Package Arch Version Repository Size
====================================================================================================================================
Installing:
glusterfs-server x86_64 3.6.5-1.el7 glusterfs-epel 736 k
Installing for dependencies:
attr x86_64 2.4.46-12.el7 base 66 k
glusterfs x86_64 3.6.5-1.el7 glusterfs-epel 1.3 M
glusterfs-api x86_64 3.6.5-1.el7 glusterfs-epel 68 k
glusterfs-cli x86_64 3.6.5-1.el7 glusterfs-epel 156 k
glusterfs-fuse x86_64 3.6.5-1.el7 glusterfs-epel 96 k
glusterfs-libs x86_64 3.6.5-1.el7 glusterfs-epel 298 k
gssproxy x86_64 0.3.0-10.el7 base 81 k
keyutils x86_64 1.5.8-3.el7 base 54 k
libbasicobjects x86_64 0.1.1-24.el7 base 24 k
libcollection x86_64 0.6.2-24.el7 base 40 k
libevent x86_64 2.0.21-4.el7 base 214 k
libini_config x86_64 1.1.0-24.el7 base 50 k
libnfsidmap x86_64 0.25-11.el7 base 46 k
libpath_utils x86_64 0.2.1-24.el7 base 27 k
libref_array x86_64 0.1.4-24.el7 base 26 k
libtalloc x86_64 2.1.1-1.el7 base 30 k
libtevent x86_64 0.9.21-3.el7 base 31 k
libtirpc x86_64 0.2.4-0.5.el7 base 85 k
libverto-tevent x86_64 0.2.5-4.el7 base 9.0 k
nfs-utils x86_64 1:1.3.0-0.8.el7 base 362 k
quota x86_64 1:4.01-11.el7 base 176 k
quota-nls noarch 1:4.01-11.el7 base 89 k
rpcbind x86_64 0.2.0-26.el7 base 55 k
rsyslog-mmjsonparse x86_64 7.4.7-7.el7_0 base 27 k
tcp_wrappers x86_64 7.6-77.el7 base 78 k
Transaction Summary
====================================================================================================================================
Install 1 Package (+25 Dependent packages)
Total download size: 4.1 M
Installed size: 13 M
...
Complete!
启动管理进程
[root@glusterfs1 yum.repos.d]# ll /usr/lib/systemd/system/gluster*
-rwxr-xr-x. 1 root root 308 Aug 24 18:05 /usr/lib/systemd/system/glusterd.service
-rw-r--r--. 1 root root 544 Jun 19 19:59 /usr/lib/systemd/system/glusterfsd.service
[root@glusterfs1 yum.repos.d]# systemctl status glusterd.service
glusterd.service - GlusterFS, a clustered file-system server
Loaded: loaded (/usr/lib/systemd/system/glusterd.service; disabled)
Active: active (running) since Fri 2015-09-18 17:23:31 CST; 14s ago
Process: 2850 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid (code=exited, status=0/SUCCESS)
Main PID: 2851 (glusterd)
CGroup: /system.slice/glusterd.service
└─2851 /usr/sbin/glusterd -p /var/run/glusterd.pid
Sep 18 17:23:31 glusterfs1 systemd[1]: Starting GlusterFS, a clustered file-system server...
Sep 18 17:23:31 glusterfs1 systemd[1]: Started GlusterFS, a clustered file-system server.
Step 4 - 配置 trusted pool
互相探测一把。[root@glusterfs1 yum.repos.d]# sudo firewall-cmd --zone=public --add-port=24007/tcp --permanent
success
[root@glusterfs2 yum.repos.d]# sudo firewall-cmd --zone=public --add-port=24007/tcp --permanent
success
[root@glusterfs1 yum.repos.d]# cat /etc/hosts
10.0.2.35 gluster1
10.0.2.36 gluster2
[root@glusterfs2 yum.repos.d]# cat /etc/hosts
10.0.2.35 gluster1
10.0.2.36 gluster2
[root@glusterfs1 yum.repos.d]# ping glusterfs2 -c 1
PING glusterfs2 (10.0.2.36) 56(84) bytes of data.
64 bytes from glusterfs2 (10.0.2.36): icmp_seq=1 ttl=64 time=0.902 ms
--- glusterfs2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.902/0.902/0.902/0.000 ms
[root@glusterfs2 ~]# ping glusterfs1 -c 1
PING glusterfs1 (10.0.2.35) 56(84) bytes of data.
64 bytes from glusterfs1 (10.0.2.35): icmp_seq=1 ttl=64 time=1.03 ms
--- glusterfs1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.037/1.037/1.037/0.000 ms
[root@glusterfs1 yum.repos.d]# gluster peer probe glusterfs2
peer probe: failed: Probe returned with unknown errno 107
打开防火墙端口限制。
[root@glusterfs1 yum.repos.d]# firewall-cmd --zone=public --add-port=24007-24047/tcp
success
[root@glusterfs1 yum.repos.d]# firewall-cmd --list-all
public (default, active)
interfaces: ens3 ens8
sources:
services: dhcpv6-client ssh
ports: 6789/tcp 24007-24047/tcp 6800-7300/tcp
masquerade: no
forward-ports:
icmp-blocks:
rich rules:
[root@glusterfs1 yum.repos.d]# telnet glusterfs2 24007
Trying 10.0.2.36...
Connected to glusterfs2.
Escape character is '^]'.
^]
telnet> quit
Connection closed.
[root@glusterfs1 yum.repos.d]# gluster peer probe glusterfs2
peer probe: success.
[root@glusterfs2 glusterfs]# gluster peer probe glusterfs1
peer probe: success. Host glusterfs1 port 24007 already in peer list
两台节点都已经成功加入pool。
Step 5 - 启用GlusterFS 卷
在两台节点上执行相同的操作。
[root@glusterfs1 ~]# mkdir /mnt/glusterfs/gv0
在任何一个节点上执行下面的操作
[root@glusterfs1 ~]# mkdir /mnt/glusterfs/gv0
[root@glusterfs1 ~]# gluster
[root@glusterfs1 ~]# gluster volume create gv0 replica 2 glusterfs1:/mnt/glusterfs/gv0 glusterfs2:/mnt/glusterfs/gv0
volume create: gv0: success: please start the volume to access data
[root@glusterfs1 ~]# gluster volume start gv0
volume start: gv0: success
[root@glusterfs1 ~]# gluster volume info
Volume Name: gv0
Type: Replicate
Volume ID: f6d91e62-4f0c-471d-8f9f-634e9eac5993
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: glusterfs1:/mnt/glusterfs/gv0
Brick2: glusterfs2:/mnt/glusterfs/gv0
日志信息可以查看
[root@glusterfs2 glusterfs]# tail -f /var/log/glusterfs/etc-glusterfs-glusterd.vol.log
Step 6 - 测试GlusterFS 卷
在这一步，我将使用一个外网客户端节点挂载已经创建好的卷。
[root@ceph_client ~]# mkdir /mnt/gluster
[root@ceph_client ~]# mount -t glusterfs glusterfs1:/gv0 /mnt/gluster/
mount: unknown filesystem type 'glusterfs'
必须安装glusterfs官方端。
[root@ceph_client ~]# yum install glusterfs-fuse -y
for i in `seq -w 1 100`; do cp -rp /var/log/messages /mnt/copy-test-$i; done
First, check the mount point:
ls -lA /mnt | wc -l You should see 100 files returned. Next, check the GlusterFS mount points on each server:
ls -lA /data/brick1/gv0 You should see 100 per server using the method we listed here. Without replication, in a distribute only volume (not detailed here), you should see about 50 each.