StorNext FS安装指南－１

一、环境描述：

硬件描述：

1. Brocade 3950

2. S2A 8000（本例中3个LUN，分别为MetaData（30GB）、Journal（100MB）、数据）

3. DELL 2650 (内存 2G，建议4G)

软件描述：

1. Red Hat Enterprise Linux/x86 AS 4 update 1 （2.6.9-11）

2. FS Server/Client 软件

二、 FS系统配置

1．准备工作

1.1配置网络

# vi /etc/hosts

127.0.0.1 localhost.localdomain localhost

10.0.5.25 mds1 //这里为两台MDS

10.0.5.26 mds2

1.2 安装Emulex LP8000/982驱动和应用程序

安装驱动程序 tar xzf lpfc-.tar.gz //解压安装包

如：#tar zxvf lpfc_2.6_driver_kit-8.0.16.17-2.tar.gz

#cd lpfc_2.6_driver_kit-8.0.16.17-2

#./lpfc-install //执行安装脚本

#reboot //重起服务器

注意：Red Hat Enterprise Linux/x86 4 updata 1 内核中已经加载Emulex LP8000/982驱动，可以不做这一步。

如果在安装驱动程序中，报IO和IRQ参数错，检查Firmware版本，升级为最新的版本。

安装emulex hbanyware应用程序

#tar xvf elxlinuxapps-2.1a25-8.0.16.17-1-1.tar

#./install //安装emulex hnanyware

#/usr/sbin/hbanyware/hbanyware //运行emulex hnanyware，查看WWN，硬件信息。

注意：需要使用emulex hnanyware时需要安装lpfc_2.6_driver_kit-8.0.16.17-2.tar.gz程序，而linux 系统自带的LPFC的驱动不能被emulex hnanyware所以识别。

1.3 效验卡加载。

通过cat /proc/scsi/scsi 命令查看卡检测到的磁盘信息

如：# cat /proc/scsi/scsi

Host: scsi0 Channel: 00 Id: 06 Lun: 00

Vendor: PE/PV Model: 1×5 SCSI BP Rev: 1.1

Type: Processor ANSI SCSI revision: 02

Host: scsi2 Channel: 00 Id: 00 Lun: 03

Vendor: DDN Model: S2A 8000 Rev: 5.02

Type: Direct-Access ANSI SCSI revision: 03

Host: scsi2 Channel: 00 Id: 00 Lun: 08

Vendor: DDN Model: S2A 8000 Rev: 5.02

Type: Direct-Access ANSI SCSI revision: 03

1.4 效验SCSI、RAID、卡加载顺序 ——–此步骤在redhat AS 4以后的版本中，不需要做。

下面的例子是redhat as 3 update4

如果SCSI、RAID卡的加载在卡后面。不然在本地SCSI系统盘的SWAP分区，不可用。如下

#cat /etc/module.conf

alias scsi_hostadapter lpfc // Emulex

alias scsi_hostadapter1 aic7xxx //scsi卡

alias usb-controller usb-ohci

options scsi_mod max_scsi_luns=256 scsi_allow_ghost_devices=1

需要将加载顺序改变，如

alias scsi_hostadapter1 aic7xxx //scsi卡

alias scsi_hostadapter2 lpfc // Emulex

alias usb-controller usb-ohci

options scsi_mod max_scsi_luns=256 scsi_allow_ghost_devices=1

重建内核

# cd /boot

# mv initrd-2.4.21-27.ELsmp.img initrd-2.4.21-27.ELsmp.img.ori

# mkinitrd initrd-2.4.21-27.ELsmp.img 2.4.21-27.ELsmp

# sync

# reboot

2．安装 FS

2.1 以root用户登录Linux，插入 FS CD

#cd /mnt/cdrom

如果CDROM没有被mounted，那么执行#mount /dev/cdrom /mnt

# cd /mnt/cdrom/StorNextFS/linux //此路径根据实际情况而定

#uname -a 查看当前系统内核的版本。

如：uname -a

Linux mds01 2.6.9-11.ELsmp #1 SMP Fri May 20 18:26:27 EDT 2005 i686 i686 i386 GNU/Linux

# rpm -ivh snfs-server_RH_AS_4_269-2.6.3-32.i386.rpm

# rpm –ivh snfs-client_RH_AS_4_269-2.6.3-32.i386.rpm

注意，在linux上服务器上必须安装Server和client两个软件。

2.2 修改环境变量

#vi /etc/profile

# Path manipulation

if [ `id -u` = 0 ]; then

pathmunge /usr/cvfs/bin

fi

ulimit -S -c 0 > /dev/null 2 > &1

ulimit -c unlimited > /dev/null 2 >&1

注销root用户，重新以root用户登陆系统，才能使环境变量起作用

2.3 磁盘Label

2.3.1 查看盘阵信息，执行fdisk -l

如： # fdisk –l

Disk /dev/sdc (Sun disk label): 0 heads, 0 sectors, 52221 cylinders

Units = cylinders of 1 * 512 bytes

Device Flag Start End Blocks Id System

/dev/sdc3 u 0 -872571904 -436285952 5 Whole disk

Note: sector size is 2048 (not 512)

Disk /dev/sdg (Sun disk label): 0 heads, 0 sectors, 959 cylinders

Units = cylinders of 1 * 512 bytes

Device Flag Start End Blocks Id System

/dev/sdg3 u 0 62902272 31451136 5 Whole disk

Disk /dev/sdl (Sun disk label): 0 heads, 0 sectors, 2 cylinders

Units = cylinders of 1 * 512 bytes

Device Flag Start End Blocks Id System

/dev/sdl3 u 0 192512 96256 5 Whole disk

2.1查看disk Label

#/usr/cvfs/bin/cvlabel –l

/dev/sdc [DDN S2A 8000 5.02] CVFS ” unknown ” Sectors: 3422394368.

SectorSize: 1024. Maximum sectors: 3422394368.

/dev/sdg [DDN S2A 8000 5.02] CVFS ” unknown ” Sectors: 62900224.

SectorSize: 512. Maximum sectors: 62900224.

/dev/sdl [DDN S2A 8000 5.02] CVFS ” unknown ” Sectors: 190464. S

ectorSize: 512. Maximum sectors: 190464.

2.2 编辑disk Label配置文件

# /usr/cvfs/bin/cvlabel -c > /usr/cvfs/config/cvlabels

//将cvlabel输出到cvlabels配置文件中

# vi /usr/cvfs/config/cvlabels

将unkown更改为H_Upload，MD_H_Upload， JR_H_Upload。。。等，如本例中：

H_Upload /dev/sdc # host 0 lun 0 sectors 3422394368 sector_size 1024 inquiry [DDN S2A 8000 5.02] serial 60001FF020021101030197FECD180000

MD_H_Upload /dev/sdg # host 0 lun 4 sectors 62900224 sector_size 512 inquiry [DDN S2A 8000 5.02] serial 60001FF0200211010301932703C00400

JR_H_Upload /dev/sdl # host 0 lun 9 sectors 190464 sector_size 512 inquiry [DDN S2A 8000 5.02] serial 60001FF020021101030194A700030900

2.3 disk Label

# /usr/cvfs/bin/cvlabel /usr/cvfs/config/cvlabels

*WARNING* This program will over-write volume labels on the

devices specified in the file “/usr/cvfs/config/cvlabels”.

After execution, the devices will only be usable by the

CentraVision File System. You will have to re-partition the

devices to use them on a different file system.

Do you want to proceed? (Y / N ) -> y

/dev/sdc [DN S2A 8000 5.02] CVFS ” unknown ” Sectors: 3422394368.

Do you want to label it SNFS - Name: H_Upload Sectors: 3422394368? (Y / N ) ->y

New Volume Label -Device: /dev/sda SNFS Label: H_Upload Sectors: 3422394368.

/dev/sdg [DN S2A 8000 5.02] CVFS ” unknown ” Sectors: 62900224.

Do you want to label it SNFS - Name: MD_H_Upload Sectors: 62900224? (Y / N ) ->y

New Volume Label -Device: /dev/sdb SNFS Label: MD_H_Upload Sectors: 62900224.

/dev/sdl [DN S2A 8000 5.02] CVFS ” unknown ” Sectors: 190464.

Do you want to label it SNFS - Name: JR_H_Upload Sectors: 190464? (Y / N ) ->y

New Volume Label -Device: /dev/sdb SNFS Label: JR_H_Upload Sectors: 190464.

Done. 3 source lines. 3 labels.

3. 配置 FS

3.1创建 FS文件系统的配置文件,

若查看参数信息，键入#man -M /usr/cvfs/man cvfs_config

# cd /usr/cvfs/config

#cp .. /examples/example.cfg ./H_Upload.cfg //例程中拷贝出文件系统的配置文件

# vi /usr/cvfs/config/H_Upload.cfg

GlobalSuperUser Yes ## Must be set to Yes for SNMS Managed File Systems ##

WindowsSecurity No

Quotas No

FileLocks No

DataMigration No ## SNMS Managed File Systems Only ##

InodeExpandMin 32K

InodeExpandInc 128K

InodeExpandMax 8M

FsBlockSize 4K

JournalSize 4M

AllocationStrategy Round

MaxConnections 78 //该数为一般设为客户数加+10

ForceStripeAlignment Yes

Debug 0×0

MaxLogSize 4M

MaxLogs 4

## Globals Defaulted

ThreadPoolSize 32 # default 16, 512 KB memory per thread

InodeCacheSize 32K # 800-1000 bytes each, default 8K

# BufferCacheSize 64M # default 32MB

# StripeAlignSize 2M # auto alignment, default MAX(StripeBreadth)

# MaxMBPerClientReserve 50 # in MBs, default 100 MB reserved per client

# OpHangLimitSecs 300 # default 180 secs

# DataMigrationThreadPoolSize 128 # Managed only, default 8

#************************************************************

# A disktype section for defining disk hardware parameters.

#************************************************************

[DiskType MD_H_Upload] ##1+1 Raid 1 Mirrored Pair##

Sectors 62900224 ## Sectors Per Disk From Command “cvlabel -l”

SectorSize 512

[DiskType JR_H_Upload] ##1+1 Raid 1 Mirrored Pair##

Sectors 190464

SectorSize 512

[DiskType H_Upload] ##4+1 Raid 5 Lun for Regular Data##

Sectors 3422394368

SectorSize 1024 //这里的1024与盘塔上该LUN的Block Size 相一致。

#************************************************************

# A disk section for defining disks in the hardware configuration.

#************************************************************

[Disk MD_H_Upload]

Status UP

Type MD_H_Upload

[Disk JR_H_Upload]

Status UP

Type JR_H_Upload

[Disk H_Upload]

Status UP

Type H_Upload

#************************************************************

# A stripe section for defining stripe groups.

*************************************************************

[StripeGroup MetaFiles]

Status UP

MetaData Yes

Journal No

Exclusive Yes

Read Enabled

Write Enabled

StripeBreadth 256K

MultiPathMethod Rotate

Node MD_H_Upload 0

[StripeGroup JournFiles]

Status UP

Journal Yes

MetaData No

Exclusive Yes

Read Enabled

Write Enabled

StripeBreadth 256K

MultiPathMethod Rotate

Node JR_H_Upload 0

[StripeGroup H_Upload_Files]

Status UP

Exclusive No ##Non-Exclusive StripeGroup for all Files##

Read Enabled

Write Enabled

StripeBreadth 1M //该参数根据应用可以调整

MultiPathMethod Rotate

Node H_Upload 0

若有多个文件系统依次创建文件系统配置文件

#vi /usr/cvfs/config/ file_system_name.cfg

# /etc/init.d/cvfs restart //重起cvfs服务。

Unmounting SNFS filesystems

Stopping SNFS Daemons

Stopping SNFS PortMapper

Waiting for FSMs to finish

SNFS Stop [ 确定 ]

Initializing Filesystem (SNFS)

No disk subsystem config for SNFS, disks not started

Loading SNFS modules

Starting /usr/cvfs/bin/fsmpm..

Starting /usr/cvfs/bin/cvfsd…

Mounting SNFS filesystems

SNFS Initialized [ 确定 ]

# /usr/cvfs/bin/cvmkfs H_Upload //初始化文件系统

File System File System Initializer.

** WARNING ** This will destroy all contents and data of the

— [H_Upload] —

file system. Re-initialization cannot be un-done!

Do you want to proceed? (Y/[N]) -> y

Re-initializing file system ‘ H_Upload ‘.

Shared Meta Data File System.

Meta Data Root is on “ MetaFiles “.

Resetting Arbitration Block.

Writing Configuration Information Block.

Writing Root Directory Block.

Writing Disk Information Block.

Writing Stripe Group Index “ JournFiles “.

Resetting File System Journal.

Creating Free List.

Writing Initial 1024 Inodes.

Writing Initialization Control Block.

Resetting Super Block.

Writing Reserved Allocations.

File system ‘ H_Upload ‘ successfully created.

同样方法依次初始化文件系统 # /usr/cvfs/bin/cvmkfs file_system_name

3.2 创建文件系统配置文件列表的文件

# cd /usr/cvfs/config

# cp ../examples/fsmlist.example ./fsmlist

//例程中拷贝出文件系统配置文件列表的配置文件

#vi /usr/cvfs/config/fsmlist

//将当前的 FS文件系统的配置文件H_Upload.cfg的文件名写入fsmlist

file_system_name . 0

如： H_Upload . 0

// 其中的 . 表示当前机器的ip地址,0是优先,可为一个0~7. 但当前默认的为0是activate

3.3 创建文件系统配置服务器列表的文件

# cd /usr/cvfs/config #cp ../examples/fsnameservers.example ./fsnameservers

//将当前的 FS文件系统的服务器的ip写入fsnameservers

如: 10.5.0.25

10.5.0.26

在做群集时,可以将每台服务器ip加入进来.并保持每一台的fsnameservers文件一致

3.2 手动启动文件系统配置

# /usr/cvfs/bin/cvadmin

File System Administrator

Enter command(s)

For command help, enter “help” or “?”.

List FSS

File System Services (* indicates service is in control of FS):

1>* H_Upload [0] located on mds1:32790 (pid 2949)

//我们可以看到系统已经自动将H_Upload启动并激活了.

* 表示激活

如果没有显示文件系统名称, 一般需要重新启动cvfs服务或检查fsmlist配置

或者手动启动,如下

Select FSM “none”

snadmin> start H_Upload

Start FSS ” H_Upload ”

FSS ‘ H_Upload ‘ start initiated.

FSS ‘ H_Upload ‘ started.

snadmin> activate H_Upload

Activate FSM ” H_Upload ”

Created : Fri Aug 19 10:02:50 2005

Active Connections: 24

Fs Block Size : 4K

Msg Buffer Size : 4K

Disk Devices : 3

Stripe Groups : 3

Mirror Groups : 0

Fs Blocks : 855598592 (3.19 TB)

Fs Blocks Free : 838919232 (3.13 TB) (98%)

snadmin (H_Upload) > show long

Show stripe groups (File System “H_Upload”)

Stripe Group 0 [MetaFiles] Status:Up,MetaData,Exclusive

Total Blocks:7862528 (29.99 GB) Free:7827253 (29.86 GB) (99%)

MultiPath Method:Rotate

Stripe Depth:1 Stripe Breadth:64 blocks (256.00 KB)

Realtime limit IO/sec:0 (~0 mb/sec) Non-Realtime reserve IO/sec:0

Committed RTIO/sec:0 Non-RTIO clients:0 Non-RTIO hint IO/sec:0

Disk stripes:

Primary Stripe 0 [MetaFiles] Read:Enabled Write:Enabled

Node 0 [MD_H_Upload]

Stripe Group 1 [JournFiles] Status:Up,Journal,Exclusive

Total Blocks:23808 (93.00 MB) Free:22784 (89.00 MB) (95%)

MultiPath Method:Rotate

Stripe Depth:1 Stripe Breadth:64 blocks (256.00 KB)

Realtime limit IO/sec:0 (~0 mb/sec) Non-Realtime reserve IO/sec:0

Committed RTIO/sec:0 Non-RTIO clients:0 Non-RTIO hint IO/sec:0

Disk stripes:

Primary Stripe 0 [JournFiles] Read:Enabled Write:Enabled

Node 0 [JR_H_Upload]

Stripe Group 2 [H_Upload_Files] Status:Up

Total Blocks:855598592 (3.19 TB) Free:838919232 (3.13 TB) (98%)

MultiPath Method:Rotate

Stripe Depth:1 Stripe Breadth:256 blocks (1024.00 KB)

Realtime limit IO/sec:0 (~0 mb/sec) Non-Realtime reserve IO/sec:0

Committed RTIO/sec:0 Non-RTIO clients:0 Non-RTIO hint IO/sec:0

Disk stripes:

Primary Stripe 0 [H_Upload_Files] Read:Enabled Write:Enabled

Node 0 [H_Upload]

snadmin (H_Upload) > who //查看系统用户信息

Who (File System “H_Upload”)

# SNFS I.D. Type Location Up Time License Expires

– ———- —- ————— ——— —————

0> FSM 1d 3h 55m N/A

1> F1F6B61DB CLI 10.5.0.12 0d 3h 6m Permanent

37> ADM mds1 0d 0h 0m N/A

snadmin (H_Upload) >exit

4. Mount FS //当linux作为client时，需要mount cvfs文件系统

# mkdir /H_Upload //创建mount点的目录

# chmod 777 / H_Upload //设置/usr/snfs目录的权限，当前权限为可读可写可执行

# vi /etc/fstab //当系统启动时，自动装载 FSH_Upload

H_Upload /H_Upload cvfs verbose=yes 0 0

//手动装载 FS文件系统，

# mount -t cvfs H_Upload /H_Upload

mount.cvfs: Filesystem H_Upload mounted on / H_Upload