glusterfs分布式文件系统配置

GlusterFS的安装配置

1.安装

Glusterfs使用了fuse，首先要安装fuse。yum即可

yum  install libibverbs-devel
yum  install fuse fuse-devel

下载  编译.

tar zxvf glusterfs-3.2.2.tar.gz 
cd glusterfs-3.2.2
 ./configure
make && make install

编译过程中需要python的ctypes支持。下载并编译即可

tar xzf ctypes-0.9.9.3.tar.gz
cd ctypes-0.9.9.3
python setup.py build
python setup.py test
python setup.py install

安装过程完成，很简单没有什么特殊的地方(python2.5之后自带ctypes，就不用安装了)

2.配置

i.server端：

vi /usr/local/etc/glusterfs/server.vol

volume brick
type storage/posix # POSIX FS translator
option directory /gfs # 共享空间的路径
end-volume

volume locker
type features/posix-locks
subvolumes brick
end-volume

### Add network serving capability to above brick.
volume server
type protocol/server
option transport-type tcp/server
option listen-port 6996 # 默认传输端口6996，如果服务器上有多个有剩余空间的分区，那么可以用另外的端口起多个gluster
subvolumes locker
option auth.addr.brick.allow * #  允许 brick 卷的访问
option auth.addr.locker.allow *
end-volume

现在可以启动server端了

 glusterfsd -f /usr/local/etc/glusterfs/server.vol

ps 后可以看到gluster进程

6271 ?        Ssl    0:00 glusterfsd -f /usr/local/etc/glusterfs/server.vol

ii.server端namespace的配置文件 namespace是类似hadoop里面namenode的应用。分出一个挂载点来实现

vi  /usr/local/etc/glusterfs/server.nc.vol

### Export volume "brick" with the contents of "/home/export" directory.
volume brick
type storage/posix # POSIX FS translator
option directory /home/nc # Export this directory
end-volume

### Add network serving capability to above brick.
volume server
type protocol/server
option transport-type tcp/server
option bind-address 10.11.156.204 
option listen-port 6999 
subvolumes brick
option auth.addr.brick.allow * 
end-volume

像启动server一样启动

glusterfsd -f /usr/local/etc/glusterfs/server.nc.vol

iii.客户端文件

整个过程中配置客户端的文件是相对最复杂的。

vi  /usr/local/etc/glusterfs/client.vol

volume client0
type protocol/client
option transport-type tcp/client
option remote-host 10.11.156.141 # IP address of the remote brick
option remote-port 6996 # default server port is 6996
option remote-subvolume locker # name of the remote volume
end-volume

volume client1
type protocol/client
option transport-type tcp/client
option remote-host 10.11.156.142
option remote-port 6996
option remote-subvolume locker
end-volume

volume client2
type protocol/client
option transport-type tcp/client
option remote-host 10.11.156.143 # IP address of the remote brick
option remote-port 6996 # default server port is 6996
option remote-subvolume locker # name of the remote volume
end-volume

.........

volume clientN
type protocol/client
option transport-type tcp/client
option remote-host 10.11.156.144 # IP address of the remote brick
option remote-port 6996 # default server port is 6996
option remote-subvolume locker # name of the remote volume
end-volume

volume namespacenode
type protocol/client
option transport-type tcp
option remote-host 10.11.156.204 #这里输入刚才指定的那个namespace server的IP 和端口
option remote-port 6999
option remote-subvolume brick
end-volume

volume distribute
type cluster/distribute #在3.* 以前的版本中distribute模块的名字叫做unify，目前很少有文档更新。需要注意下
subvolumes client1 client0 client2 ... clientN #有几个客户端就填写几个
option scheduler alu #alu是GlusterFS根据各个server的空间占用情况和负载情况自动分配文件的一种方式。除此以外还有常用的rr等。
option namespace namespcenode
end-volume

启动客户端

glusterfs -l /tmp/glusterfs.log -f /usr/local/etc/glusterfs/client.vol /home/glu/

把结果输出到 /tmp/glusterfs.log以便出现问题的时候查询

df -h 查看结果

[@tc_156_141 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1             3.9G  608M  3.1G  17% /
/dev/sda6             112G  462M  106G   1% /opt
/dev/sda3             3.9G  431M  3.3G  12% /var
/dev/sda5             4.9G  2.2G  2.5G  48% /usr
tmpfs                  12G   52K   12G   1% /dev/shm
/dev/sdb               11T  5.1M   11T   1% /gfs
glusterfs#/usr/local/etc/glusterfs/client.vol
                       41T   21M   41T   1% /home/glu

最后一行就是GlusterFS了。 glusterfs#/usr/local/etc/glusterfs/client.vol

41T 21M 41T 1% /home/glu

now，一个成功的分布式存储系统成功的上线了。