Chinaunix首页 | 论坛 | 博客
  • 博客访问: 2397205
  • 博文数量: 328
  • 博客积分: 4302
  • 博客等级: 上校
  • 技术积分: 5486
  • 用 户 组: 普通用户
  • 注册时间: 2010-07-01 11:14
个人简介

悲剧,绝对的悲剧,悲剧中的悲剧。

文章分类

全部博文(328)

文章存档

2017年(6)

2016年(18)

2015年(28)

2014年(73)

2013年(62)

2012年(58)

2011年(55)

2010年(28)

分类: SOLARIS

2014-10-10 16:34:10

A system administrator must know some of the basic concepts of storage systems. In this post I will try to explain some of the concepts and useful commands related to identifying HBA cards, WWN numbers and fiber channel paths. Before we start with the commands let us get acquainted with some of the storage related terms.

Fiber channel intiator and target

Initiator : Initiator is typically a server with HBA card installed. The initiatorinitiates the connection over the fiber channel to the storage system.

Target : Target is typically the storage system itself. Target servers the requests from the initiators for the data.
WWN : WWN stands for the world wide name. Similar to a network card having a unique MAC address, every HBA port will have a unique WWN number.

Solaris 10 and Solaris 11

We can get a list of HBA ports using the luxadm command. Remember that luxadm command only work on hba’s that are bound to the Oracle drivers. So if native (non oracle) Emulex driver is in use then the luxadm command will not work. In that case use the fcinfo command.

# luxadm -e port 

/devices/ssm@0,0/pci@18,600000/SUNW,emlxs@1/fp@0,0:devctl          CONNECTED
/devices/ssm@0,0/pci@18,600000/SUNW,emlxs@1,1/fp@0,0:devctl        CONNECTED
/devices/ssm@0,0/pci@1c,600000/SUNW,emlxs@1/fp@0,0:devctl          CONNECTED
/devices/ssm@0,0/pci@1c,600000/SUNW,emlxs@1,1/fp@0,0:devctl        CONNECTED
"CONNECTED" --> HBA has established connection with either target or initiator. 
"NOT CONNECTED" --> HBA has not established a connection with some other node or it is connected to a switch that has no target.

The fcinfo command give us more info on the HBA ports (model, manufactures etc) and the WWN numbers of the HBA ports.

# fcinfo hba-port
HBA Port WWN: 23000800xxxxxxx
        OS Device Name: /dev/cfg/c1
        Manufacturer: QLogic Corp.
        Model: 2200
        Firmware Version: 2.01.145
        FCode/BIOS Version: ISP2200 FC-AL Host Adapter Driver: 1.14 01/11/20
        Serial Number: not available
        Driver Name: qlc
        Driver Version: 20090415-2.30
        Type: L-port
        State: online
        Supported Speeds: 1Gb
        Current Speed: 1Gb
        Node WWN: 22000800xxxxxxx
HBA Port WWN: 10000000cxxxxxxx
        OS Device Name: /dev/cfg/c5
        Manufacturer: Emulex
        Model: LP10000DC-S
        Firmware Version: 1.92a1 (T2D1.92A1)
        FCode/BIOS Version: Boot:5.01a4 Fcode:1.50a4
        Serial Number: BG52130223
        Driver Name: emlxs
        Driver Version: 2.31t (2009.04.05.14.10)
        Type: N-port
        State: online
        Supported Speeds: 1Gb 2Gb
        Current Speed: 2Gb
        Node WWN: 20000000cxxxxxxx
HBA Port WWN: 10000000cxxxxxxx
        OS Device Name: /dev/cfg/c6
        Manufacturer: Emulex
        Model: LP10000DC-S
        Firmware Version: 1.92a1 (T2D1.92A1)
        FCode/BIOS Version: Boot:5.01a4 Fcode:1.50a4
        Serial Number: BG52130223
        Driver Name: emlxs
        Driver Version: 2.31t (2009.04.05.14.10)
        Type: N-port
        State: online
        Supported Speeds: 1Gb 2Gb
        Current Speed: 2Gb
        Node WWN: 20000000cxxxxxxx

Check the fields “Model:” and “Manufacturer:” in the output above. You can identify various manufactures of the HBA cards from these fields.

Node WWN - WWN of HBA itself 
Port WWN - WWN on a specific port of the HBA
LPxxxxxx-S are Oracle Emulex HBA Cards (ie it has a “-S” for Sun at end).
LPxxxxxx-E are NON-Oracle HBAs usually resold by EMC.
LPxxxxxx-H are NON-Oracle HBAs usually resold by Hitachi.
LPxxxxxx-M are NON-Oracle HBAs usually resold by
375-xxxx-xx are Oracle Qlogic HBA Cards.
QLAxxxx may or may not be Oracle Qlogic HBA Cards so you would need to use “prtpicl Command:” to determine.
Model: 2200 with Manufacturer: QLogic Corp is a old Oracle Fibre Channel Arbitrated Loop (FC-AL) HBA Cards.

For Solaris 8 and 9

The fcinfo command may not work in Solaris 8 and 9. So we’ll have to use the below command. but the command would also show other devices such as network cards. look for the driver-name to identify the HBA cards.

# prtpicl -v | egrep "subsystem-id|model|devfs-path|driver-name|binding-name" | awk '/subsystem-id/,/binding-name/' | sed '/subsystem-id/{x;p;x;}' | nawk -F: '{ print $2 }'

subsystem-id     0x13a1
model    LSI,1064
devfs-path       /pci@0,600000/pci@0/pci@8/pci@0/scsi@1
driver-name      mpt
binding-name     pci1000,50

subsystem-id     0x1648
devfs-path       /pci@0,600000/pci@0/pci@8/pci@0/network@2
driver-name      bge
binding-name     pci14e4,1648

subsystem-id     0x1648
devfs-path       /pci@0,600000/pci@0/pci@8/pci@0/network@2,1
driver-name      bge
binding-name     pci14e4,1648 subsystem-id     0xfe00
model    LPe11000-M4
devfs-path       /pci@0,600000/pci@0/pci@9/fibre-channel@0
driver-name      emlxs
binding-name     pciex10df,fe00 .........

Some of the old HBAs may not display the “subsystem-id” in the above comman output. In that case use the below commands :

# prtpicl -v | egrep -i "qla|lpfc"
# prtpicl -v | grep -i fibre-channel

For Example

# prtpicl -v | grep -i fibre-channel
           fibre-channel (obp-device, 4b300000160e)
           fibre-channel (obp-device, 4b30000016be)
           fibre-channel (obp-device, 4b3000001ca2)
           fibre-channel (obp-device, 4b3000001d52)

Each listing in the above output is a HBA port. To list the paths grep for the“devfs-path” string.

# prtpicl -v | grep -i devfs-path | grep -i fibre-channel | grep -v sd | nawk -F: '{ print $2 }
devfs-path        /pci@8,700000/fibre-channel@2
devfs-path        /pci@8,700000/fibre-channel@3
devfs-path        /pci@9,700000/fibre-channel@2
devfs-path        /pci@9,700000/fibre-channel@3

Using the prtdiag command
In case if the prtpicl command also doee not work, use the prtdiag command to identify the HBA cards. prtdiag may be very helpful in case of a Solaris x86/x64 system.

# prtdiag -v
......
========================= IO Cards =========================
                         Bus  Max
     IO   Port Bus       Freq Bus  Dev,
Brd  Type  ID  Side Slot MHz  Freq Func State Name                              Model
---- ---- ---- ---- ---- ---- ---- ---- ----- --------------------------------  ----------------------
I/O   PCI   8    B    4    33   33  1,0  ok    pci-pci1011,26.5/SUNW,qlc (scsi-+ PCI-BRIDGE
I/O   PCI   8    B    4    33   33  4,0  ok    SUNW,qlc-pci1077,2200.1077.4083.+ device on pci-bridge
I/O   PCI   8    A    1    66   66  1,0  ok    SUNW,emlxs-pci10df,fc00/fp (fp)   LP10000DC-S
I/O   PCI   8    A    1    66   66  1,1  ok    SUNW,emlxs-pci10df,fc00/fp (fp)   LP10000DC-S
.........

Gathering information on target port WWN

To get information on target port WWN use the “remote-port -p” option with fcinfo. The syntax for the same is :

# fcinfo remote-port -p [HBA Port WWN at server end (initiator)]

For example

# fcinfo remote-port -p 10000000c993b528
Remote Port WWN: 5006016239a01db6
        Active FC4 Types: SCSI
        SCSI Target: yes
        Node WWN: 50060160b9a01db6
Remote Port WWN: 5006016b3b205594
        Active FC4 Types: SCSI
        SCSI Target: yes
        Node WWN: 50060160bb205594

Another way to get the target port WWNs is using the luxadm command with the dump_map option :

# luxadm -e dump_map /devices/pci@8,700000/SUNW,qlc@3/fp@0,0:devctl
Pos  Port_ID Hard_Addr Port WWN         Node WWN         Type
0    664f13  0         50060e800428a551 50060e800428a551 0x0 (Disk device)
1    610b13  0         210000e08b1c4dd6 200000e08b1c4dd6 0x0 (Disk device)

Port_ID ending with “00” or “13” indicates a FC switch attached storage.

Getting LUN list from a specific storage port

We can also get a list LUNs coming from a specific storage port (i.e. target port). The syntax for getting this details :

# fcinfo remote-port -s -p 10000000c993b528
Remote Port WWN: 5006016239a01db6
        Active FC4 Types: SCSI
        SCSI Target: yes
        Node WWN: 50060160b9a01db6
        LUN: 0
          Vendor: DGC
          Product: LUNZ
          OS Device Name: /dev/rdsk/c2t5006016239A01DB6d0s2
        LUN: 1
          Vendor: DGC
          Product: RAID 5
          OS Device Name: /dev/rdsk/c3t60060160545D1C00C0369BCD3361DC11d0s2
        LUN: 33
          Vendor: DGC
          Product: RAID 5
          OS Device Name: /dev/rdsk/c3t60060160545D1C00B87CDAF33B44DC11d0s2


Administering LUNs

Many modern Oracle Solaris systems are attached directly to an external Storage Area Network (SAN) device containing several disk drives. The drives in this storage device are configured as virtual drives, and each is referred to as a logical unit. Each logical unit is identified by a number, the logical unit number or LUN. The LUN is attached to the server through a Fibre Channel connection. These LUNs appear as disk drives, just like an internal disk, but the device name is different in that it contains the device’s World Wide Name (WWN). Here’s an example:

# format Searching for disks ... done


AVAILABLE DISK SELECTIONS:
       0. c0t0d0 
       /pci@0,600000/pci@0/pci@8/pci@0/scsi@1/sd@0,0
       1. c0t1d0 
       /pci@0,600000/pci@0/pci@8/pci@0/scsi@1/sd@1,0
       2. c3t60A98000572D577465346D3936706348d0 
       /scsi_vhci/ssd@g60a98000572d577465346d3936706348
       3. c3t60A9800043346B74635A4F4371304A66d0 
       /scsi_vhci/ssd@g60a9800043346b74635a4f4371304a66
       4. c3t60A9800043346B74635A4F4370767969d0 
       /scsi_vhci/ssd@g60a9800043346b74635a4f4370767969
       5. c3t60A9800043346B7448344F4371543378d0 
       /scsi_vhci/ssd@g60a9800043346b7448344f4371543378
       6. c3t60A9800043346B7448344F4374437666d0 
       /scsi_vhci/ssd@g60a9800043346b7448344f4374437666
Specify disk (enter its number):

For disks 2–6, the WWN is in place of the target ID field. On a fiber-attached device, the WWN is a unique identifier used to uniquely identify each LUN in a Fibre Channel network. In the previous example, the server is connected to a NetApp data storage system connected via Fibre Channel. The LUNs were identified and dynamically configured during the boot process. They can be displayed using the format command as shown in the previous example.

Use the luxadm command to scan the devices and present a list of all LUNs and their logical names as follows:

# luxadm probe No Network Array enclosures found in /dev/es

Found Fibre Channel device(s):
  Node WWN:500a0980894b97b6 Device Type:Disk device
    Logical Path:/dev/rdsk/c3t60A98000572D577465346D3936706348d0s2
  Node WWN:500a0980894b97b6 Device Type:Disk device
    Logical Path:/dev/rdsk/c3t60A9800043346B74635A4F4371304A66d0s2
  Node WWN:500a0980894b97b6 Device Type:Disk device
    Logical Path:/dev/rdsk/c3t60A9800043346B74635A4F4370767969d0s2
  Node WWN:500a0980894b97b6 Device Type:Disk device
    Logical Path:/dev/rdsk/c3t60A9800043346B7448344F4371543378d0s2
  Node WWN:500a0980894b97b6 Device Type:Disk device
    Logical Path:/dev/rdsk/c3t60A9800043346B7448344F4374437666d0s2

Choose a logical name and display information about each individual LUN as follows:

# luxadm display /dev/rdsk/c3t60A98000572D577465346D3936706348d0s2 DEVICE PROPERTIES for disk: /dev/rdsk/c3t60A98000572D577465346D3936706348d0s2
 Vendor:          NETAPP
 Product ID:       LUN
 Revision:       7330
 Serial Num:      W-Wte4m96pcH
 Unformatted capacity: 204800.000 MBytes
 Read Cache:       Enabled
  Minimum prefetch:  0x0
  Maximum prefetch:  0x0
 Device Type:      Disk device
 Path(s):

 /dev/rdsk/c3t60A98000572D577465346D3936706348d0s2
 /devices/scsi_vhci/ssd@g60a98000572d577465346d3936706348:c,raw
  Controller      /devices/pci@1,700000/SUNW,emlxs@0/fp@0,0
   Device Address        500a0981894b97b6,5
   Host controller port WWN        10000000c9729bb9
   Class                 primary
   State                 ONLINE
  Controller
/devices/pci@0,600000/pci@0/pci@9/SUNW,emlxs@0/fp@0,0
  Device Address         500a0981994b97b6,5
  Host controller port WWN      10000000c972a675
  Class                   secondary
  State                     ONLINE

Provide the information obtained from the luxadm display command to your storage administrator so that you create the file system on the correct LUN. It’s very important that you select the correct LUN when more than one LUN is available.

The next step is to configure a file system on the LUN. I’ll create a ZFS file system on the LUN as follows:

# zpool create pool1 c3t60A98000572D577465346D3936706348d0

转载自: 和 

阅读(2679) | 评论(0) | 转发(0) |
给主人留下些什么吧!~~