[prev in list] [next in list] [prev in thread] [next in thread]
List: sun-managers
Subject: SUMMARY - Configuring StorageTek 2510 iSCSI for best ZFS
From: Chris Hoogendyk <hoogendyk () bio ! umass ! edu>
Date: 2009-09-22 17:11:19
Message-ID: 4AB90537.7030006 () bio ! umass ! edu
[Download RAW message or body]
hmm. I thought I owed a summary, but I can't seem to find that I ever
posted the question. I guess I just hammered my head against the wall
for enough days that I was sure I must have posted a question. Anyway, I
think it's worth posting a summary, because there were missing links in
the documentation, and, finally, a critical blog I found that helped me
figure out what to do.
It's hard to do a useful micro summary; but, basically, I was able to
work with command line only using Common Array Manager (CAM) to
configure the 2510 so that each drive was exported as a LUN. Then I
could assign drives to individual hosts (I have 2 directly connected to
the array), see them in `format`, and use zpool to group them into raidz.
-- *Physical setup* --
This configuration is a StorageTek 2510 with SAS drives and 2 T5220's.
The 2510 has one controller, and each T5220 is connected directly to one
of the iSCSI ports on that controller. The controller on the 2510 has a
management port and two data ports. There are stickers on the back near
the ports with the Mac addresses printed on them.
-- *Management port *--
Originally, I wanted to have command line only management directly over
the data lines (in line management). Since I was having trouble figuring
out how to set this thing up at all, I decided to just go ahead and
expose the management port on the 2510. I set up the Mac address of the
management port in our DHCP server, plugged the management port into a
switch in our private network, and tried to ping it. No ping and no
evidence from the DHCP server that it had requested or been given an
address. hmm. Turns out it does it using bootp. So, I added an
appropriate entry in /etc/bootptab on our server that does it. Power
cycled the 2510. Ping. Then I scanned it with nmap. nmap failed to
identify the device and said all ports were closed. OK. Cool. on to the
next step.
-- *Install CAM* --
Now I needed to install CAM. I thought I should look for the latest.
Turns out 6.2 and more recent say that they need a firmware upgrade of
the 2510 before they can talk to it. The software for the firmware
upgrade seemed to be GUI only. My server has no GUI. Really. Minimal
network install with whatever packages I needed beyond that individually
installed. No GUI. So. CAM 6.1.1.10 seemed to be the most recent before
6.2. I downloaded that and checked the readme. It had a command line
only install option. `./RunMe.bin -c`. I did that. Selected a custom
install, command line only with firmware. Somehow that claimed it was
installing Java 2 standard edition.
There were some differences in where the documentation said I would find
the software and where I actually found it. The critical application is
sscs and was found at /opt/SUNWstkcam/bin/. Since I was on the machine
that was to manage and access the array, I didn't need to do the `sscs
login ....`, and I could get basic information on the fly from `sscs
--help`.
-- *Documentation* --
Ultimately, I found three sources that gave me what I needed.
(1) The Sun StorageTek 2500 Series Array Installation Guide gave me the
overview of how to set up the array. In particular, the section on
Configuring iSCSI Tasks, the Table A-2, iSCSI Configuration Steps, and
the section with the Example for Solaris. It fell short in that at a
crucial step it simply says, "Follow CAM Documentation to . . . ." But,
the CAM documentation is very cursory on iSCSI, assuming, I guess, that
you will get it from the GUI. There is a table listing iSCSI commands
with no explanation, overview, or examples of how to use them.
(2) The man page for sscs was a key reference. However, it is 123 pages
long and doesn't really give you an overview of what you need to do or
how the different subcommands interact. It's also a bit hard to go
scrolling back and forth through it. So, I printed it out to a pdf
preview. Then I could search the pdf for "NAME", which was the section
title preceding every command. That gave me a sort of table of contents
into the commands that I could scan through. I still encountered at
least one command whose syntax turned out to be very different from what
the man page listed.
(3) I found a blog that gave me some critical conceptual overview and
details for what I was trying to do. That is referenced further down
when I get to that.
-- *Registering & Configuring the 2510* --
Following page 95 and forward in the Installation Guide.
[Note: I've slightly obscured the IP address in the following]
# cd /opt/SUNWstkcam/bin
# ./sscs add -i 172.xx.xx.xx registeredarray
Name Type Network Address Serial Number
--------- ---- --------------- --------------------------
unlabeled 2510 172.xx.xx.xx SUN.540-7198-01.0812BE3D2D
# ./sscs modify -N edsel array unlabeled
[After setting up a new J4200, this seemed an appropriate name for the 2510]
# ./sscs list firmware
Analyzing array edsel,(172.xx.xx.xx),200400a0b8490699
Controller: Some FRUs not at baseline.
Name Model Current Baseline
Tray.85.Controller.A LCSM100_I 06.70.42.10 06.70.54.10
Disk: Some FRUs not at baseline.
Name Model Current Baseline
Tray.85.Drive.01 HUS1530SBSUN300G SA02 SA04
Tray.85.Drive.02 HUS1530SBSUN300G SA02 SA04
Tray.85.Drive.03 HUS1530SBSUN300G SA02 SA04
Tray.85.Drive.04 HUS1530SBSUN300G SA02 SA04
Tray.85.Drive.05 HUS1530SBSUN300G SA02 SA04
System/NVSRAM: All FRUs at baseline
Name Model Current Baseline
edsel 2510 N1532-670843-901 N1532-670843-901
# ./sscs modify -a edsel firmware
WARNING: This command will load new firmware if needed and may impact
array management and data availability.
Do you wish to continue? [y,n] : y
Started firmware upgrade job Install:task28
# ./sscs list -a edsel jobs
Job ID: Install:task28 Status: Done
# ./sscs list firmware
Analyzing array edsel,(172.28.55.33),200400a0b8490699
Controller: All FRUs at baseline
Name Model Current Baseline
Tray.85.Controller.A LCSM100_I 06.70.54.10 06.70.54.10
Disk: Some FRUs not at baseline.
Name Model Current Baseline
Tray.85.Drive.01 HUS1530SBSUN300G SA02 SA04
Tray.85.Drive.02 HUS1530SBSUN300G SA02 SA04
Tray.85.Drive.03 HUS1530SBSUN300G SA02 SA04
Tray.85.Drive.04 HUS1530SBSUN300G SA02 SA04
Tray.85.Drive.05 HUS1530SBSUN300G SA02 SA04
System/NVSRAM: All FRUs at baseline
Name Model Current Baseline
edsel 2510 N1532-670843-901 N1532-670843-901
[So, it updated the firmware on the controller, but not on the drives.
It took a bit of fiddling around to find the correct syntax and details
to do that.]
# ./sscs modify -a edsel -t disk firmware
WARNING: This command will load new firmware if needed and may impact
array management and data availability.
Do you wish to continue? [y,n] : y
Started firmware upgrade job Install:task32
# ./sscs list -a edsel jobs
Job ID: Install:task32 Status: Done
Job ID: Install:task28 Status: Done
# ./sscs list firmware
Analyzing array edsel,(172.28.55.33),200400a0b8490699
Controller: All FRUs at baseline
Name Model Current Baseline
Tray.85.Controller.A LCSM100_I 06.70.54.10 06.70.54.10
Disk: All FRUs at baseline
Name Model Current Baseline
Tray.85.Drive.01 HUS1530SBSUN300G SA04 SA04
Tray.85.Drive.02 HUS1530SBSUN300G SA04 SA04
Tray.85.Drive.03 HUS1530SBSUN300G SA04 SA04
Tray.85.Drive.04 HUS1530SBSUN300G SA04 SA04
Tray.85.Drive.05 HUS1530SBSUN300G SA04 SA04
System/NVSRAM: All FRUs at baseline
Name Model Current Baseline
edsel 2510 N1532-670843-901 N1532-670843-901
[All set]
-- *Configure data ports on server* --
OK, so now I need to set up data connections between the T5220 and the
2510. They are connected directly with a plain ethernet cable (not a
crossover). I get the IP addresses for the 2510 data ports from sscs,
and then configure the T5220 with ifconfig.
# ./sscs list -a edsel iscsi-port
[output truncated]
Controller: A
iSCSI Port: A/1
Listening Port: 3260
IP Address: 192.168.130.101
Gateway: 0.0.0.0
Netmask: 255.255.255.0
iSCSI Port: A/2
Listening Port: 3260
IP Address: 192.168.131.101
Gateway: 0.0.0.0
Netmask: 255.255.255.0
# ifconfig -a
[just to check]
# ifconfig e1000g2 plumb
# ifconfig e1000g2 192.168.130.88 netmask 255.255.255.0 up
# ping 192.168.130.101
192.168.130.101 is alive!
# cat > /etc/hostname.e1000g2
192.168.130.101
# cat >> /etc/hosts
192.168.130.88 imladris.scsi.mor.nsm
192.168.130.101 edsel.scsi.mor.nsm
# cat >> /etc/netmasks
192.168.130.0 255.255.255.0
In the above, the ifconfig got it running; but, the additions to the
/etc files will make sure it comes up if/when the system is ever
rebooted. The fully qualified network names are made up to match, in
essence, the way we do our private networks. It won't go out on any
network, even private, since they are direct connected, but, anyway.
Morrill (mor) is our building, and NSM is our College within the University.
-- *Set up iSCSI initiators* --
This is basically following the steps in the Installation Guide, in
particular the Solaris example on p. 101 in my hard copy. Be sure to
check software versions and patches as instructed. I didn't have to mess
with any of that, because I was on Solaris 10 u7 (5/09) with the latest
recommended and security patches. Anyway. Need to find the IQN (iSCSI
Qualified Name) for the initiator associated with the T5220 and the data
ports on the 2510, and set up connections between them.
# iscsiadm list initiator-node
Initiator node name: iqn.1986-03.com.sun:01:00212813a10c.49188968
[rest of output cut]
# iscsiadm modify discovery --static enable
# iscsiadm add discovery-address 192.168.130.101:3260
# iscsiadm list discovery-address
Discovery Address: 192.168.130.101:3260
Target name:
iqn.1986-03.com.sun:2510.600a0b80004906990000000048c13342
Target address: 192.168.130.101:3260, 1
Target name:
iqn.1986-03.com.sun:2510.600a0b80004906990000000048c13342
Target address: 192.168.131.101:3260, 1
# iscsiadm add static-config
iqn.1986-03.com.sun:2510.600a0b80004906990000000048c13342,192.168.130.101
# iscsiadm list static-config
Static Configuration Target:
iqn.1986-03.com.sun:2510.600a0b80004906990000000048c13342,192.168.130.101:3260
# format
[intention is to check what is there now, before setting up drives on
the 2510]
Searching for disks...done
c6t8d0: configured with capacity of 16.00MB
AVAILABLE DISK SELECTIONS:
0. c1t0d0 <LSILOGIC-LogicalVolume-3000 cyl 65533 alt 2 hd 16 sec
273> raidboot
/pci@0/pci@0/pci@2/scsi@0/sd@0,0
1. c1t3d0 <SUN146G cyl 14087 alt 2 hd 24 sec 848>
/pci@0/pci@0/pci@2/scsi@0/sd@3,0
2. c6t8d0 <SUN-UniversalXport-0670 cyl 8 alt 2 hd 64 sec 64>
/iscsi/disk@0000iqn.1986-03.com.sun%3A2510.600a0b80004906990000000048c13342FFFF,31
Specify disk (enter its number): 2
selecting c6t8d0
[disk formatted]
Disk not labeled. Label it now? no
partition> p
Current partition table (default):
Total disk cylinders available: 8 + 2 (reserved cylinders)
Part Tag Flag Cylinders Size Blocks
0 root wm 0 0 (0/0/0) 0
1 swap wu 0 0 (0/0/0) 0
2 backup wu 0 - 7 16.00MB (8/0/0) 32768
3 unassigned wm 0 0 (0/0/0) 0
4 unassigned wm 0 0 (0/0/0) 0
5 unassigned wm 0 0 (0/0/0) 0
6 usr wm 0 - 7 16.00MB (8/0/0) 32768
7 unassigned wm 0 0 (0/0/0) 0
partition> q
[hmm. ??? What is that 16MB Thingy ???]
-- *Set up hosts & disks* --
OK. Getting there. Now I have to configure the hosts and disks on the
2510 so that the disks will be linked to the host where I want to mount
and format them (the T5220). This is where the Installation Guide drops
the ball, just saying, "Follow CAM documentation to: ...." The CAM
documentation, in turn, doesn't give a conceptual overview, assumes you
are using the GUI (which will automate a bunch of stuff), and doesn't
say much about iSCSI. This is where I found "Sun StorageTek 2540 / ZFS
Performance Summary" by Bob Friesenhahn
(http://www.simplesystems.org/users/bfriesen/zfs-discuss/2540-zfs-performance.pdf),
which describes setting up the 2540 to present each drive as a separate
unit. That gave me enough conceptual overview and detail to then mine my
pdf print of the man page for sscs and work out the commands to do it.
The basic overview is to create a separate storage pool for each disk.
Create a separate volume for each disk within its own separate storage
pool. Then link those to a host and set it up in zfs. He was dealing
with fiber channel and multipathing, but that doesn't matter. The CAM
concepts are the same.
First, just some housekeeping. Mining the pdf man page, (& some trial
and error) . . .
# ./sscs list -a edsel date
Date: Tue Sep 15 12:29:58 2009
# ./sscs modify -a edsel date 1755
# ./sscs list -a edsel date
Date: Tue Sep 15 17:55:16 2009
# ./sscs list -a edsel os-type
SOLARIS_MPXIO - Solaris (with Traffic Manager)
[rest of output clipped]
# ./sscs create -a edsel host marlin
# ./sscs create -a edsel host snapper
# ./sscs create -a edsel -h marlin -o solaris -i
iqn.1986-03.com.sun:01:00212813a10c.49188968 initiator marlin
[creating a couple of hosts and an initiator for the one host, naming
the host and the initiator with the same name]
# ./sscs list -a edsel disk
Tray: 85 Disk: t85d05
Capacity: 279.396 GB
Type: SAS
Speed (RPM): 15000
Status: Optimal
State: Enabled
Role: Unassigned
Virtual Disk: -
Firmware: SA04
Serial number: 000809C69Y7C J8W69Y7C
WWN: 50:00:CC:A0:05:43:DC:38
[clipping a bunch of output]
Tray: 85 Disk: t85d01
Capacity: 279.396 GB
Type: SAS
Speed (RPM): 15000
Status: Optimal
State: Enabled
Role: Unassigned
Virtual Disk: -
Firmware: SA04
Serial number: 000809C090LC J8W090LC
WWN: 50:00:CC:A0:05:38:E5:70
# ./sscs list -a edsel profile
Profile: Default
# ./sscs list -a edsel profile Default
Profile: Default
Profile In Use: yes
Factory Profile: yes
Description: Pre-configured Default profile
RAID Level: 5
Segment Size: 512 KB
Read Ahead: on
Optimal Number of Drives: variable
Disk Type: Fibre Channel
Dedicated Hot Spare: no
Pool: Default
Pool: disk1
[hmm. That just won't do. I wonder why they put that on the 2510 as the
default profile? So, create a new profile.]
# ./sscs create -a edsel -r 0 -s 128K -h on -n 1 -k SAS -H no -d "One
SAS disk" profile RAW_SAS
[That took a fair bit of trial and error, but it worked. Now create a
bunch of pools using that profile.]
# ./sscs create -a edsel -p RAW_SAS pool Disk-01
# ./sscs create -a edsel -p RAW_SAS pool Disk-02
# ./sscs create -a edsel -p RAW_SAS pool Disk-03
[etc.]
# ./sscs list -a edsel pool
Pool: Disk-01 Profile: RAW_SAS Configured Capacity: 0.000 MB
Pool: Disk-02 Profile: RAW_SAS Configured Capacity: 0.000 MB
Pool: Disk-03 Profile: RAW_SAS Configured Capacity: 0.000 MB
[clipped]
# ./sscs list -a edsel pool Disk-01
Pool: Disk-01
Description: null
Profile: RAW_SAS
Total Capacity: 0.000 MB
Configured Capacity: 0.000 MB
Available Capacity: 278.896 GB
[cool. Now create the corresponding volumes.]
# ./sscs create -a edsel -p Disk-01 -s 278.896gb -d t85d01 volume Disk-01
# ./sscs list -a edsel pool Disk-01
Pool: Disk-01
Description: null
Profile: RAW_SAS
Total Capacity: 278.895 GB
Configured Capacity: 278.895 GB
Available Capacity: 278.896 GB
Volume: Disk-01
# ./sscs list -a edsel volume Disk-01
Volume: Disk-01
Type: Standard
WWN:
60:0A:0B:80:00:49:06:99:00:00:02:26:4A:B0:EF:52
Pool: Disk-01
Profile: RAW_SAS
Virtual Disk: 1
Size: 278.895 GB
State: Free
Status: Online
Action: Initializing...
Condition: Optimal
Read Only: No
Controller: A
Preferred Controller: A
Modification Priority: High
RAID Level: 0
Segment Size: 128 KB
Read Cache: Enabled
Write Cache: Enabled
Write Cache with Replication: Disabled
Write Cache without Batteries: Disabled
Write Cache Active: True
Flush Cache After: 10 s
Disk Scrubbing: Enabled
Disk Scrubbing with Redundancy: Disabled
# ./sscs create -a edsel -p Disk-02 -s 278.896gb -d t85d02 volume Disk-02
# ./sscs create -a edsel -p Disk-03 -s 278.896gb -d t85d03 volume Disk-03
[and so on]
[Finally, map the volumes to the host.]
# ./sscs map -a edsel -P readwrite -h marlin volume Disk-01
# ./sscs map -a edsel -P readwrite -h marlin volume Disk-02
[etc.]
-- *Create iSCSI device link & format drives* --
That's basically it for setting up the 2510. Now, from the T5220's
perspective, I have to set up the iSCSI device links. This is straight
from the Installation Guide.
# devfsadm -i iscsi
# format
Searching for disks...done
c6t8d0: configured with capacity of 16.00MB
c7t600A0B8000490699000002264AB0EF52d0: configured with capacity of 275.99GB
c7t600A0B8000490699000002284AB0EFB6d0: configured with capacity of 275.99GB
[clipped]
AVAILABLE DISK SELECTIONS:
0. c1t0d0 <LSILOGIC-LogicalVolume-3000 cyl 65533 alt 2 hd 16 sec
273> raidboot
/pci@0/pci@0/pci@2/scsi@0/sd@0,0
1. c1t3d0 <SUN146G cyl 14087 alt 2 hd 24 sec 848>
/pci@0/pci@0/pci@2/scsi@0/sd@3,0
2. c6t8d0 <SUN-UniversalXport-0670 cyl 8 alt 2 hd 64 sec 64>
/iscsi/disk@0000iqn.1986-03.com.sun%3A2510.600a0b80004906990000000048c13342FFFF,31
3. c7t600A0B8000490699000002264AB0EF52d0 <SUN-LCSM100_I-0670 cyl
65533 alt 2 hd 128 sec 69>
/scsi_vhci/ssd@g600a0b8000490699000002264ab0ef52
4. c7t600A0B8000490699000002284AB0EFB6d0 <SUN-LCSM100_I-0670 cyl
65533 alt 2 hd 128 sec 69>
/scsi_vhci/ssd@g600a0b8000490699000002284ab0efb6
[clipped]
Specify disk (enter its number): 3
selecting c7t600A0B8000490699000002264AB0EF52d0
[disk formatted]
Disk not labeled. Label it now? no
> quit
-- *ZFS* --
So, I'm basically home free now, and I can configure ZFS using the
information from the format command. Something like, . . .
# zpool create mypool raidz c7t600A0B8000490699000002264AB0EF52d0 [and
so on]
I also tried setting up email notification, but I don't know if it works.
# ./sscs add -e hoogendyk@marlin.mor.net.nsm -m minor notification
local_email
[which doesn't match the man page, but rather is based on the error
messages I got when I entered it according to the man page]
-- *in Closing* --
I hope this extended howto turns out to be helpful to someone else. I
had a fair bit of trouble finding the information that made this
possible and working out the details. It should have been more
elementary than that. The Sun sales people said, "sure, you can do
that." Then, since we aren't doing things according to their book, it's
hard to actually get support and guidance. It is sort of like when you
want to do a Solaris install and minimize the OS for security purposes.
They actually have a security white paper (somewhat dated now)
recommending that. However, it isn't SOP. So you can't get support when
you try to do that. Even in a room full of Solaris people and several
Sun engineering types (at an OpenSolaris Users Group meeting), they just
shrug and say, "do a standard install."
Anyway, if your purpose is to use ZFS, and you really just want a JBOD,
I would recommend the J4200 and related products over the StorageTek
25xx. I think my multipathed dual SAS J4200 is much faster than my
non-multipathed single GigE connection 2510, and it was much easier to
set up. I haven't actually done I/O tests yet. We had originally thought
we were going to share drives off the 2510 between a couple of servers
and maybe even mount them read-only to our backup server, but that's not
really possible. Even though it can sit on a network, the drives are
basically owned and formatted by a single server. It's probably easier
and more featurefull to just have a SAS JBOD, maybe with expansion
trays, hanging on a single T5xxx server, using ZFS, and then share that
out if other servers want access. Sort of build your own 7000 series.
Comments appreciated.
If I get comments, and it seems warranted, I'll post a summary-2.
--
---------------
Chris Hoogendyk
-
O__ ---- Systems Administrator
c/ /'_ --- Biology & Geology Departments
(*) \(*) -- 140 Morrill Science Center
~~~~~~~~~~ - University of Massachusetts, Amherst
<hoogendyk@bio.umass.edu>
---------------
Erdvs 4
_______________________________________________
sunmanagers mailing list
sunmanagers@sunmanagers.org
http://www.sunmanagers.org/mailman/listinfo/sunmanagers
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic