Hard Question: Write Performance seems low (Sun StorageTek 6540)
 

This is a tough one to describe and ask help for, I know. I
apologize.

I have a relatively new Sun StorageTek 6540 RAID. When purchased we
only bought four trays of storage, 300GB 15,000RPM disks. We set up
all the hosts via a director class Brocade SAN switch (5300 I
think). Ever since I conducted my first performance tests (and I'm
not an expert) I was questioning the write performance compared to
other RAIDs and JBODs I have in-house. After having Sun Services do a
once over on it, they "thought" it was okay. So I let it go, that was
months ago.

Well now, we've fully populated the rack with new trays (600GB disks)
and all seemed well. But I had not performed any major tests or
production on it. Well now I'm trying to provision for production
and I'm still questioning the performance of writes. I realized I
still had a problem when testing our new 10GbE NICs in the hosts.

Turns out I can still FTP faster to the single platter root disk
(mirrored) or to RAM-FS (/tmp) than I can the ZFS or QFS based pool of
storage I've provisioned. I constantly FTP at around 54MB/s to the
6540 RAID. I've tried different LUN layouts, but nothing seems to
help. I've tried large & small, RAID-5 segments: 64, 128 & even
default 512. Nothing really changes. I've tried stripped & mirrored
volumes within the RAID and straight up RAID 5 6-disk virtual disks
for the volumes. I've used small volumes and even up to 9 x 2.7TB
volumes to create the large pool I'll need for production. But,
nearly all other storage I have is faster in wrights. Even on a T5220
with local RAID storage (ZFS JBOD).

I did squeeze out a wee bit speed using QFS file system, but not much,
and that was expected. I wrote faster to a directly attached Sun 6140
RAID with ZFS. I don't have the numbers in front of me, but the
concept of write times should be in order.

Using Solaris 10 Sparc on a T2 processor (Sun Blade 6320, and other
servers)

Write 5 GB file across new 10GbE network: (Even tried mkfile
command) Same basic results)

Write to any volume configuration on 6540 RAID: 54MB/s
Write to root disk (single platter): 62MB/s
Write to Sun 6140 RAID-5: 68MB/s
Write to RAM-disk (/tmp): 98MB/s

I've tried many different ways to write and validate the tests.

I've checked HBAs on all test hosts: Configured at 4Gbit, no errors,
no CRCs.
I've checked Brocade ports (both brocades for each HBA port) (HA'd).
Brocades indicate minimal port % usage.
Used stmsboot -e on hosts (Solaris 10u7, fully patched)

I know there are 50 things I'm mentioning. disk layouts etc, etc.
I'm not used to asking for deep questions like this. Sorry!

Any suggestions on where to look would be helpful. Again, I find no
errors anywhere in logs, or HBA stats, etc.

Thanks for any suggestions!