分类： 服务器与存储
2009-12-23 14:02:08
MFS 分布式存储系统文档
(for debian)
目录
MFS(moosefs)是一款分布式存储系统,可以将文件分布式存储到多台服务器上.
结构分为3部分:
元数据服务器:
主数据服务器master只能有1台(容易出现单点故障),负责管理文件系统.
数据存储服务器chunkserver:
数据存储服务器chunkserver,存储数据,多个chunkserver直接数据互相同步.
客户端:
使用MFS文件系统来存储和访问的主机称为MFS的客户端，成功挂接MFS文件系统以后，就可以像以前使用NFS一样共享这个虚拟性的存储了。(可以挂载到分区)
结构图如下(图1):
图1
下载:
#wget
服务器准备
建议使用3台服务器进行测试,2台存储,一台客户端
性能测试工具准备
读写性能测试工具:
iozone
bonnie++
网站访问测试:
ab
autobench
httperf
httpload
flood
webbench
客户端安装fuse支持包
#apt-get install fuse-utils libfuse-dev
添加用户:
#useradd mfs -s /bin/false
编译安装:
#./configure --prefix=/usr/local/mfs --with-default-user=mfs --with-default-group=mfs
#make&&make install;
配置元服务器:
#vi /usr/local/mfs/etc/mfsmaster.cfg
内容如下:
WORKING_USER = mfs //用户
WORKING_GROUP = mfs //用户组
LOCK_FILE = /var/run/mfs/mfsmaster.pid //lock文件路径,如果系统没有需要手工创建,并将属主改成mfs
DATA_PATH = /usr/local/mfs/var/mfs //元数据存放路径,如果没有需要手工创建,并将属主改成mfs
SYSLOG_IDENT = mfsmaster //日志级别
# BACK_LOGS = 50 //后台日志
REPLICATIONS_DELAY_INIT = 300 //同步延时时间
REPLICATIONS_DELAY_DISCONNECT = 3600 //同步超时时间
MATOCS_LISTEN_HOST = * //监听主机
MATOCS_LISTEN_PORT = 9420 //即元数据服务器使用9420这个监听端口来接受数据存储服务器chunkserver端的连接
MATOCU_LISTEN_HOST = * //监听主机
MATOCU_LISTEN_PORT = 9421 //元数据服务器在9421端口监听，用以接受客户端对MFS进行远程挂接（客户端以mfsmount挂接MFS）
# CHUNKS_LOOP_TIME = 300 //存储循环时间
# CHUNKS_DEL_LIMIT = 100 //删除时间
# CHUNKS_REP_LIMIT = 15 //重复时间
启动主服务器
#/usr/local/mfs/sbin//mfsmaster start
注意:
如果正常,系统日志会有记录,另外如果要关闭主服务器,使用/usr/local/mfs/sbin/mfsmaster –s 这种方式，如果直接使用kill杀死进程，将导致下次启动时出现找不到相关文件，而不能正常启动服务器。这个一定要谨慎。当然，如果发生了这个事情，还是可以通过 mfsmetastore 来恢复的。
添加用户组
# useradd mfs -s /bin/false (debian)
编译安装:
为了区分,加入--disable-mfsmaster参数
#./configure --prefix=/usr/local/mfs --with-default-user=mfs --with-default-group=mfs --disable-mfsmaster
#make &&make install ;
配置chunkserver:
#vi /usr/local/mfs/etc/mfschunkserver.cfg
配置文件内容:
# WORKING_USER = mfs //用户
# WORKING_GROUP = mfs //用户组
# DATA_PATH = /usr/local/mfs/var/mfs //数据存放路径
# LOCK_FILE = /var/run/mfs/mfschunkserver.pid //LOCK文件存放路径
# SYSLOG_IDENT = mfschunkserver //日志级别
# BACK_LOGS = 50
# MASTER_RECONNECTION_DELAY = 30 //重新连接延时
# MASTER_HOST = mfsmaster //MASTER服务器地址
# MASTER_PORT = 9420 //端口
# MASTER_TIMEOUT = 60 //超时时间
# CSSERV_LISTEN_HOST = * //数据复制服务器地址
# CSSERV_LISTEN_PORT = 9422 //这个监听端口用于与其它数据存储服务器间的连接，通常是数据复制
# CSSERV_TIMEOUT = 60 //接收超时时间
# CSTOCS_TIMEOUT = 60
# HDD_CONF_FILENAME = /usr/local/mfs/etc/mfshdd.cfg //分配给MFS使用的磁盘空间配置文件的位置
更改配置文件
#vi /usr/local/mfs/etc/mfshdd.cfg
加入:
/home/mfs
注意:
这里需要将要存放数据的分区或硬盘写进去,并将属主修改成mfs,还有如果测试,可以创建一个大文件来共享
我们这里可以用一个分区来做测试
#mkdir /home/mfs
#chown –R mfs.mfs /home/mfs
启动:
#/usr/local/mfs/sbin/mfschunkserver start;
注意:
跟主数据服务器master相似，执行命令 /usr/local/mfs/sbin/mfschunkserver –s , chunkserver服务就停下来了。为了使系统重启过程能自动启动chunkserver 服务，可以通过在/etc/rc.local文件追加行 /usr/local/mfs/sbin/mfschunkserver start 来达到这个目的（master的自动重启处理也可同样处理）。
3.3 客户端安装配置
#创建用户
#useradd mfs –s /bin/false
编译安装:
为了区别,客户端不用安装master和chunkserver
#modprobe fuse
#./configure --prefix=/usr/local/mfs --with-default-user=mfs --with-default-group=mfs --disable-mfsmaster --disable-mfschunkserver --enable-mfsmount
#make &&make install
挂载:
#mkdir /mnt/mfs
#/usr/local/mfs/bin/mfsmount –h xxx.xxx.xxx
常用MFSMOUNT 参数:
-w 指定mount路径,默认为/mnf/mfs
-h 指定MASTER服务器地址
-p 指定端口默认9421
测试/mnt/mfs目录的数据读写情况,并关闭一台存储服务器,测试数据读写的完整性
使用bonnie测试MFS
#bonnie -d /mnt/mfs/ -s 16000 -m mfs -uroot
结果:
Version 1.03 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
mfs
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 373 0 469 0 477 0 389 0 471 0 477 0
mfs,
说明:
测试数据为
使用bonnie测试本地磁盘读写
# bonnie -d /home/datatest/ -s 16000 -m mfs –uroot
结果:
Version 1.03 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
mfs
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
mfs,
说明:
测试数据
从上面2个数据来看,磁盘I/O性能是本地磁盘的1/6左右,本地磁盘为RAID5.
configure 说明
`configure' configures MFS
Usage: ./configure [OPTION]... [VAR=VALUE]...
To assign environment variables (e.g., CC, CFLAGS...), specify them as
VAR=VALUE. See below for descriptions of some of the useful variables.
Defaults for the options are specified in brackets.
Configuration:
-h, --help display this help and exit 显示帮助
--help=short display options specific to this package 显示这个包的特定选项
--help=recursive display the short help of all the included packages 显示短帮助
-V, --version display version information and exit
-q, --quiet, --silent do not print `checking...' messages
--cache-file=FILE cache test results in FILE [disabled]
-C, --config-cache alias for `--cache-file=config.cache'
-n, --no-create do not create output files
--srcdir=DIR find the sources in DIR [configure dir or `..']
Installation directories:
--prefix=PREFIX install architecture-independent files in PREFIX
[/usr/local]
--exec-prefix=EPREFIX install architecture-dependent files in EPREFIX
[PREFIX]
By default, `make install' will install all the files in
`/usr/local/bin', `/usr/local/lib' etc. You can specify
an installation prefix other than `/usr/local' using `--prefix',
for instance `--prefix=$HOME'.
For better control, use the options below.
Fine tuning of the installation directories:
--bindir=DIR user executables [EPREFIX/bin]
--sbindir=DIR system admin executables [EPREFIX/sbin]
--libexecdir=DIR program executables [EPREFIX/libexec]
--sysconfdir=DIR read-only single-machine data [PREFIX/etc]
--sharedstatedir=DIR modifiable architecture-independent data [PREFIX/com]
--localstatedir=DIR modifiable single-machine data [PREFIX/var]
--libdir=DIR object code libraries [EPREFIX/lib]
--includedir=DIR C header files [PREFIX/include]
--oldincludedir=DIR C header files for non-gcc [/usr/include]
--datarootdir=DIR read-only arch.-independent data root [PREFIX/share]
--datadir=DIR read-only architecture-independent data [DATAROOTDIR]
--infodir=DIR info documentation [DATAROOTDIR/info]
--localedir=DIR locale-dependent data [DATAROOTDIR/locale]
--mandir=DIR man documentation [DATAROOTDIR/man]
--docdir=DIR documentation root [DATAROOTDIR/doc/mfs]
--htmldir=DIR html documentation [DOCDIR]
--dvidir=DIR dvi documentation [DOCDIR]
--pdfdir=DIR pdf documentation [DOCDIR]
--psdir=DIR ps documentation [DOCDIR]
Program names:
--program-prefix=PREFIX prepend PREFIX to installed program names
--program-suffix=SUFFIX append SUFFIX to installed program names
--program-transform-name=PROGRAM run sed PROGRAM on installed program names
System types:
--build=BUILD configure for building on BUILD [guessed]
--host=HOST cross-compile to build programs to run on HOST [BUILD]
--target=TARGET configure for building compilers for TARGET [HOST]
Optional Features:
--disable-FEATURE do not include FEATURE (same as --enable-FEATURE=no)
--enable-FEATURE[=ARG] include FEATURE [ARG=yes]
--disable-dependency-tracking speeds up one-time build
--enable-dependency-tracking do not reject slow dependency extractors
--disable-largefile omit support for large files
--disable-mfsmaster Don't build mfsmaster
--disable-mfschunkserver
Don't build mfschunkserver
--disable-mfsmount Don't build mfsmount
Optional Packages:
--with-PACKAGE[=ARG] use PACKAGE [ARG=yes]
--without-PACKAGE do not use PACKAGE (same as --with-PACKAGE=no)
--with-default-user=USER
Choose default user to run daemons as
--with-default-group=GROUP
Choose default group to run daemons as
Some influential environment variables:
CC C compiler command
CFLAGS C compiler flags
LDFLAGS linker flags, e.g. -L
nonstandard directory
LIBS libraries to pass to the linker, e.g. -l
CPPFLAGS C/C++/Objective C preprocessor flags, e.g. -I
CPP C preprocessor
PKG_CONFIG path to pkg-config utility
FUSE_CFLAGS C compiler flags for FUSE, overriding pkg-config
FUSE_LIBS linker flags for FUSE, overriding pkg-config
Use these variables to override the choices made by `configure' or to help
it to find libraries and programs with nonstandard names/locations.
Report bugs to
