分类： 服务器与存储
2016-12-12 10:43:32
某企业的后台存储(netapps)两块磁盘损坏，生产阵列为RAID-DP模式，处于double degraded状态。由于阵列已过保。客户不愿付出高额的维护费用，希望我们公司能提供支持和帮助。在我们操作netapps存储之前，客户已经把所有相关的信息都从存储中备份出来了。
为了体现我公司的技术实力，并解决这个问题，我们在网上下载了大量的文档资料，并安装、调试了netapps 存储的虚拟机。我们对虚拟机做了相关的模拟操作。
最后我们得到这样一个解决思路：先把netapps 中degraded 阵列干掉，然后重建来解决这个问题。
1、off掉降级的Lun
lun offline /vol/vol1/db/db.lun
2、把lun unmap
lun ummap /vol/vol1/db/db.lun
3、干掉lun
lun destroy /vol/vol1/db/db.lun
4、卸载降级的volume
vol offline /vol/vol0
vol offline /vol/vol1
5、干掉volume
vol destroy /vol/vol0
vol destroy /vol/vol1
6、新建volume
netapps> vol create vol0 -t raid_dp -d 0b.16 0b.32 0b.33 0b.18 ....
7、重建cifs
netapps> cifs setup
但是我们在做第四步的时候出错了，
netapps> vol offline /vol/vol0
报如下的错误
offlining root volume 'vol0' is not allowed
为什么不能把vol0 offline呢？我们查找了大量的文档，可以看到文档中对volume 0(root volume)的评价，基本上说的是所有配置信息都存储在volume 0上，而volume 0是阵列初始化的时候被建立的。那么要修改volume 0 ，是不是只有重建阵列呢？
About the root volume
Every filer has a root volume. It is the volume from which the filer reads
configuration files. During setup, the filer creates a default volume, named
vol0, and designates it as the root volume.
Initializing all disks erases all data
You might need to initialize all disks and create a single new file system in the following circumstances:You decide to redeploy an existing filer and need to completely reconfigure it.
Network Appliance Technical Support advises you that the only way to recover from an error is to initialize all disks.
Caution
Initializing all disks causes all existing data to be lost.
Procedure for initializing all disks
Complete the steps in the following table to initialize all disks.
Step Action
1 If the console is... Then...
Displaying the filer prompt, for example, toaster>.
Place the system boot diskette into the diskette drive of the filer
and enter reboot.Not displaying the filer prompt. Reboot from diskette as
described in “Booting from system boot diskette” on page 392.
2 When the boot menu appears, enter 4 to choose Initialize all disks.
Results: The filer initializes all the disks and creates a
single-volume file system.
我们咨询了netapps的工程师，它们给了我们这样的建议：
说这是因为所有系统信息都在root volume下，这个时候是无法把rootvg down下来的，
只有两个解决方案:
1>购买两个新盘，替换上去，让磁盘重新加入这个磁盘组
2>重建netapps的文件系统，把设备初始化，这样将重装netapps的 vol0。
由于没有新盘，我们只有选择 1 的方法，初始化阵列并重头配置阵列。
拟配置的过程：
1、初始化阵列
第一步：启动新安装的控制器，并在启动过程提示按Ctrl+C进入maintenance mode时，按Ctrl+C，进入维护模式。一定要看到提示后在操作，选择4A，(4a) Same as option 4, but create a flexible root volume.
如下所示:
2、进入配置信息
初始化阵列其实是这样一个过程，把阵列中的所有盘初始化，并把flash卡中的再写入磁盘阵列的前3块盘中，这三块盘组成了一个
RAID-DP的 RAID组。
我们等待了30分钟，看到阵列变成这个样子:图:
我们按ctrl+c 跳过搜寻dhcp服务器，并暂时不配置 setup的选项，进入下面的界面，先查看磁盘阵列的情况：
Sysconfig –r ,结果如下图所示：
可以看到，系统自动建立了一个 aggregate 卷， aggr0，属于RAID-DP格式，而且
包含了三块盘。所有的基本信息都写在 /aggr0/ 的rg0上。
而其他的盘都被初始化为 spare模式。
3、加入合适的license:
为了实现特定的功能，需要为netapps磁盘阵列加入合适的license:
配置 vol1，设置为RAID-DP格式
加完License之后，重启阵列
进入阵列之后，敲入？回车，可以看到阵列支持的一系列命令行，如下图所示：
选择license，回车，我们可以看到相关的license已经被加入了
4、加入IP地址以待管理
为了保证每次阵列重启之后，都会利用原来保存的配置继续对外提供服务，我们需要把机器的IP地址，阵列名，路由参数都写到配置文件中。
根据文档显示，可以有两种方式配置
1>是修改 /etc/rc 这个文件，
我们可以通过cifs文件系统的方式对etc下的这个文件访问并修改:
比如此次我们就采用这种方式: //192.168.240.1/etc$
然后修改下面的rc文件。
2>通过setup命令，来配置相关信息，如下所示
|||||||||||||||||||||||||||||||||||||||
hnfas270-1> setup
The setup command will rewrite the /etc/rc, /etc/exports,
/etc/hosts, /etc/hosts.equiv, /etc/dgateways, /etc/nsswitch.conf,
and /etc/resolv.conf files, saving the original contents of
these files in .bak files (e.g. /etc/exports.bak).
Are you sure you want to continue? [yes] yes
NetApp Release 7.1: Fri Dec 23 02:11:20 PST 2005
System ID: 0084212027 (hnfas270-1)
System Serial Number: 311202311 (hnfas270-1)
System Rev: E0
slot 0: System Board
Processors: 2
Processor revision: B2
Processor type: 1250
Memory Size: 1022 MB
slot 0: FC Host Adapter 0b
21 Disks: 2856.0GB
1 shelf with LRC, 1 shelf with EFH
slot 0: FC Host Adapter 0c
slot 0: Dual SB1250-Gigabit Ethernet Controller
e0a MAC Address: 00:a0:98:03:a0:ba (auto-1000t-fd-up)
e0b MAC Address: 00:a0:98:03:a0:bb (auto-1000t-fd-up)
slot 0: NetApp ATA/IDE Adapter 0a (0x00000000000001f0)
0a.0 245MB
Please enter the new hostname [hnfas270-1]:
Do you want to configure virtual network interfaces? [y]: y
Number of virtual interfaces to configure? [1] 1
Name of virtual interface #1 [vif0]:
Is vif0 a single [s] or multi [m] virtual interface? [m]
Number of links for vif0? [2]
Name of link #1 for vif0 [e0a]:
Name of link #2 for vif0 [e0b]:
Please enter the IP address for Network Interface vif0 [192.168.240.1]:
Please enter the netmask for Network Interface vif0 [255.255.255.224]:
Please enter media type for vif0 {100tx-fd, tp-fd, 100tx, tp, auto (10/100/1000)} [auto]:
Would you like to continue setup through the web interface? [n]: y
executing: ifconfig vif0 192.168.240.1 mediatype auto netmask 255.255.255.224
Point a web browser to
or
to complete setup. At any time, you can continue with the console setup.
Please enter the name or IP address of the default gateway: 192.168.240.30
The administration host is given root access to the filer's
/etc files for system administration. To allow /etc root access
to all NFS clients, enter 'all' below.
Please enter the name or IP address of the administration host [192.168.240.9]: Wed Jun 2 19:11:42 GMT [nbt.nbns.registrationComplete:info]: NBT: All CIFS name registrations have completed for the local server.
192.168.243.83
Where is the filer located? []:
What language will be used for multi-protocol files (Type ? for list)?:?
Supported language codes are:
/etc/lang/lang.cfg is missing
Use vol lang command to set filer language after files installed
What language will be used for multi-protocol files (Type ? for list)?:
language not set
Do you want to run DNS resolver? [n]:
Do you want to run NIS client? [n]:
This system will send event messages and weekly reports to Network Appliance Technical Support. To disable this feature, enter "options autosupport.support.enable off" within 24 hours. Enabling Autosupport can significantly speed problem determination and resolution should a problem occur on your sy
stem. For further information on Autosupport, please see:
Press the return key to continue.
The initial aggregate currently contains 2 disks; you may add more
disks to it later using the "aggr add" command.
Now type 'reboot' for changes to take effect.
hnfas270-1> reboot
关于setup文档 :
How the filer uses
the /etc/rc file
The filer executes the commands in the /etc/rc file on the root volume at boot
time to configure the filer.
If your filer is licensed to run the CIFS protocol, the /etc/rc file must be present at
boot time for the CIFS protocol to be enabled. No CIFS-specific information is
entered in /etc/rc as a result of the setup procedure.
All the commands in the /etc/rc file are executable from the command line—
there are no commands that are restricted to being executed from within the
/etc/rc file.
rc
NAME
na_rc - system initialization command script
SYNOPSIS
/etc/rc
DESCRIPTION
The command script /etc/rc is invoked automatically during system initialization. Since the filer has no
local editor, /etc/rc must be edited from an NFS client with root access to /etc. Alternately, you can use
the setup command to generate a new /etc/rc file without using NFS.
EXAMPLE
This is a sample /etc/rc file as generated by setup:
#Auto-generated by setup Tue Jun 2 21:23:52 GMT 1994
hostname toaster.netapp.com
ifconfig e0 ‘hostname‘-0
ifconfig e1a ‘hostname‘-1
route add default MyRouterBox 1
routed on
timezone Atlantic/Bermuda
savecore
FILES
/etc/rc
5、在剩余的15块spare盘上，用13块盘新建一个卷组，卷组名vol1,剩下两块用做spare盘
命令如下所示：
Vol create vol1 –t raid_dp –d 0b.19 0b.21 0b.23 0b.24 0b.25 0b.26 0b.27 0b.28 0b.29 0b.33 0b.34 0b.36
如下图所示：
关于创建volume的建议
RAID volume creation
The current recommendation from NetApp is to create large 14-drive volumes using RAID-DP. Also recommended is letting Data ONTAP pick which drives to use in a volume creation. With the defaults in DataONTAP (option raid.default_raidtype is raid_dp), this makes volume/raid group creation fairly straightforward.
For a fully configured (48TB raw) R200:
To create up to the first 11 volumes, just run vol create vol<#> 14, substituting the vol<#> for its sequential number, and with 14 being the number of disks to be used in the volume (including 2 for dual-parity raid). When you get to create the last volume, run vol create vol12 10. This will leave 2 drives available as a spare.
For any other size, follow the same instructions as above (14-drive volumes), and create the last volume with the remaining drives, leaving 2 drives for spares
6、在vol1上配置qtree ，名字叫做db
Qtree create /vol/vol1/db
关于qtree的概念：
A qtree is a special subdirectory of the root directory of a volume. Qtree parameters: You can set the following parameters on a qtree: u security style: NTFS (Windows NT file system), UNIX, or mixed u oplocks setting: On or Off u disk space and file limits, as described in Chapter 12, “Managing Quotas and Maximum Number of Files.” Note Qtrees used to be called quota trees. Any quota trees created in previous releases of Data ONTAP are now qtrees.
Volumes and qtrees:
A volume has all the properties of a qtree except，It can have qtrees under it.
It has different defaults than a qtree.
Unless expressly mentioned otherwise, whatever applies to qtrees also applies to volumes.
建立一个qtree：
Creating a qtree in a volume other than the root volume
If you want to create a qtree (for example, news) in a particular volume (for
example, users), use the following command:
qtree create /vol/users/news
改变qtree的权限:
Example with a volume: To change the security model of the root directory
of the users volume to mixed so that, outside of a qtree in the volume, one file can have NTFS security and another UNIX security, use qtree security /vol/users/ mixed
Uses of qtrees:
You use qtrees to group files that have similar characteristics, much in the way that you use volumes. However, qtrees are much more flexible than volumes. For details, see “Using qtrees” on page 283.
7、增加一个cifs用户,目的是让客户端能用这个用户访问存储上的信息：
查看用户信息,用下面的命令：
Hnfas270-1>useradmin user list
结果如下图所示:
添加一个用户scan，属于Administrators组
useradmin user add scan –g Administrator
8、配置CIFS服务器
敲入 cifs setup ,配置相关信息，如下图所示：
使用cifs shares 查看cifs的共享情况：
9、让qtree 为cifs 文件系统共享
建立一个cifs文件共享，共享服务名为db1 ,对应的qtree路径为 /vol/vol1/db
Cifs shares -add db1 /vol/vol1/db
命令结束后，再检查cifs共享情况，可以见到 刚才的配置信息已经起作用了
到此成功的配置了netapps阵列，并在阵列上建立了cifs服务。
