Low-end SAN performance: EMC AX100

We are needing to orient our data storage towards a SAN and we are
thinking of going with two 3 TB AX100's, and a 16 port switch. We have
6 Linux compute servers that currently need to access the two storage
units, however not all servers need access to both AX100's. We plan to
add at least two more Linux servers in the very near future.

I've seen a report (
http://groups.google.com/groups?hl=e...r4 %404ax.com
) about poor performance with the CX400's and was wondering if the same
sort of problem would be encountered with the AX100 since they both use
ATA (SATA in the case of the AX100) drives. We would be using large
volumes on each device that multiple servers would mount to.

I'm just starting to think about SAN's. I don't know a whole lot about
them. Would other devices like the Apple Xserve RAID or the NexSAN
ATABoy/Beast be similar to the AX100? We use a combination of large
files (between 80 MB and 1.5 GB) and small files.

I also have a couple of basic questions: What protocol is used to mount
these devices to our Linux machines? Is it NFS? SMB? Something else?
NFS in Linux is better than it was but it still has a lot to be
desired. Should I expect a big performance boost compared to what I'm
getting now with NFS over Gigabit ethernet connected to a 2TB 3Ware RAID
system that gets 200MB/s reads and 150MB/s writes (locally). That is,
will I not get any improvement since NFS might be the bottleneck? I'm
hoping that the Fibrechannel connection will help a lot.

We also have some machines that don't have PCI-X slots. Are there
straight PCI HBA's that can be used that still give good performance?
I've seen some HBA's in the $300 range. Are the $900 ones dramatically
better?

Thanks for your help. If anyone has any other ideas about how we might
set up our storage I'd be interested in hearing it.

Steve
--
__________________________________________________ ____________________
Steve Cousins, Ocean Modeling Group Email:
Marine Sciences, 208 Libby Hall http://rocky.umeoce.maine.edu
Univ. of Maine, Orono, ME 04469 Phone: (207) 581-4302

Steve Cousins wrote:

I've seen a report (
http://groups.google.com/groups?hl=e...r4 %404ax.com
) about poor performance with the CX400's and was wondering if the
same sort of problem would be encountered with the AX100 since they
both use ATA (SATA in the case of the AX100) drives.

The AX-100 and the CX-400 are completely different animals, so there is no
reason to expect the AX-100 to have similar performance characteristics.
Whether that is good or bad news is tough to tell since EMC hasn't published
any comparitive information :-)

..

I'm just starting to think about SAN's. I don't know a whole lot
about them. Would other devices like the Apple Xserve RAID or the
NexSAN ATABoy/Beast be similar to the AX100? We use a combination of
large files (between 80 MB and 1.5 GB) and small files.

Mostly read or mostly write or somewhere in between?

Do the servers need to share a single filesystem?


I also have a couple of basic questions: What protocol is used to
mount these devices to our Linux machines?

Today, you need a Fibre Channel host bus adapter in each LINUX machine and
the drives will the same as locally attached SCSI disk, i.e. they will have
/dev/hd entries a will need to formatted & mounted locally.


Is it NFS? SMB?
Something else? NFS in Linux is better than it was but it still has a
lot to be desired. Should I expect a big performance boost compared
to what I'm getting now with NFS over Gigabit ethernet connected to a
2TB 3Ware RAID system that gets 200MB/s reads and 150MB/s writes
(locally). That is, will I not get any improvement since NFS might be
the bottleneck? I'm hoping that the Fibrechannel connection will
help a lot.

If what you need is a shared filesystem, then if you go the SAN route you'll
need to add some sort of filesystem clustering software to each of your
servers, a SAN doesn't directly support anything similar to a shared
filesystem.


We also have some machines that don't have PCI-X slots. Are there
straight PCI HBA's that can be used that still give good performance?
I've seen some HBA's in the $300 range. Are the $900 ones
dramatically better?

Most PCI-X HBAs are smart enough to dumb down their operation when placed in
a PCI slot.

Thanks for your help. If anyone has any other ideas about how we
might set up our storage I'd be interested in hearing it.


You need to provide some addtional information about your requirements.


--
Nik Simpson

Nik Simpson wrote:

I'm just starting to think about SAN's. I don't know a whole lot
about them. Would other devices like the Apple Xserve RAID or the
NexSAN ATABoy/Beast be similar to the AX100? We use a combination of
large files (between 80 MB and 1.5 GB) and small files.

Mostly read or mostly write or somewhere in between?

Somewhere in between. When we run a model, it reads in about 50 MB of data per day and outputs about 80 MB per
day. By "per day" I mean "per model day". If we are re-running our model for 2003 then it needs to read in about
18 GB and output about 30 GB. However, other programs just read through these model output files and then create a
single 100 MB file.

Do the servers need to share a single filesystem?

Yes. Since all of these compute servers are Linux is there a prefered file system? GFS? Do these work well? I've
heard of CXFS from SGI and we do have an SGI machine (Origin 3000) but I've heard this is quite expensive. I
haven't priced any of these file systems out yet so maybe they are all expensive.

I also have a couple of basic questions: What protocol is used to
mount these devices to our Linux machines?

Today, you need a Fibre Channel host bus adapter in each LINUX machine and
the drives will the same as locally attached SCSI disk, i.e. they will have
/dev/hd entries a will need to formatted & mounted locally.


Ah. I see. That helps a lot.

If what you need is a shared filesystem, then if you go the SAN route you'll
need to add some sort of filesystem clustering software to each of your
servers, a SAN doesn't directly support anything similar to a shared
filesystem.


OK. See above.

We also have some machines that don't have PCI-X slots. Are there
straight PCI HBA's that can be used that still give good performance?
I've seen some HBA's in the $300 range. Are the $900 ones
dramatically better?

Most PCI-X HBAs are smart enough to dumb down their operation when placed in
a PCI slot.

Thanks for your help. If anyone has any other ideas about how we
might set up our storage I'd be interested in hearing it.

You need to provide some addtional information about your requirements.


What other information do you need? Basically, if NFS with Linux worked better I'd be happy with that. Since it
doesn't work as well as I'd like, I want to see what it would take to do the same thing with a SAN (or if there is
some other way please let me know; I use SMB and am mixed about it for our purposes) . We want a bunch of machines
to share the same data and be able to read/write to the same volume. By doing this I hope to increase performance
and centralize the storage to make it easier to manage the data.

Cheers,

Steve
--
__________________________________________________ ____________________
Steve Cousins, Ocean Modeling Group Email:
Marine Sciences, 208 Libby Hall http://rocky.umeoce.maine.edu
Univ. of Maine, Orono, ME 04469 Phone: (207) 581-4302

Steve Cousins wrote:
Nik Simpson wrote:

I'm just starting to think about SAN's. I don't know a whole lot
about them. Would other devices like the Apple Xserve RAID or the
NexSAN ATABoy/Beast be similar to the AX100? We use a combination of
large files (between 80 MB and 1.5 GB) and small files.

Mostly read or mostly write or somewhere in between?

Somewhere in between. When we run a model, it reads in about 50 MB
of data per day and outputs about 80 MB per day. By "per day" I mean
"per model day". If we are re-running our model for 2003 then it
needs to read in about 18 GB and output about 30 GB. However, other
programs just read through these model output files and then create a
single 100 MB file.


Any idea what sort of sustained rate you need, i.e. MB/s that you like to
make available to the client.

Is it a workflow type operation, i.e. server "a" reads the file does some
processing creates an output file, server "b" reads outhput, does some
processing an so on. Or do all the servers open the same file at the same
time & read it.

Do the servers need to share a single filesystem?

Yes. Since all of these compute servers are Linux is there a prefered
file system? GFS? Do these work well? I've heard of CXFS from SGI
and we do have an SGI machine (Origin 3000) but I've heard this is
quite expensive. I haven't priced any of these file systems out yet
so maybe they are all expensive.

You might want to ask about SAN filesystems in one of the LINUX groups as
well, I'll be homest, other than knowing the names of the various products I
know little about their pros & cons.


What other information do you need? Basically, if NFS with Linux
worked better I'd be happy with that.

It might be useful to tell us what you are unhappy with vis NFS,
performance? If it's performance, then how much performance do you need,
there might be NAS solutions that would be worth investigating. It would
also help to have some idea of the budget, i.e. is this a project where
everything has to be done for the absolute minimum bucks, or is this a
problem that you can spend money to solve?


--
Nik Simpson

Nik Simpson wrote:

Any idea what sort of sustained rate you need, i.e. MB/s that you like to
make available to the client.

It doesn't have to be blazing but it would be nice to have it as fast as
possible. If each client could sustain 40 to 50 MB/sec with large files that
would be great.

Is it a workflow type operation, i.e. server "a" reads the file does some
processing creates an output file, server "b" reads outhput, does some
processing an so on. Or do all the servers open the same file at the same
time & read it.

The programs are independent of one another. There is no work flow like what
you outline. Generally we have a bunch of researchers running different ocean
models who need access to the same initialization files and we have some compute
servers that they can run these models on. I want all of the compute servers to
look pretty much the same so the researchers don't need to worry about which
machine they are using.

What other information do you need? Basically, if NFS with Linux
worked better I'd be happy with that.

It might be useful to tell us what you are unhappy with vis NFS,
performance?

The main problem has been server load. Currently one of our compute nodes (dual
Opteron) also has the 2 TB RAID array on it. If a bunch of the compute servers
are hitting it at the same time that it is also doing some work then the load
can get quite high which reduces performance on everything. For instance I just
ran a process that would read 1.2 GB (thirty 40MB files), process it and output
about 11 MB. The file server is also currently running a long term model and
another machine is also running a long term model. The load on the file server
went from a little over 2 to above 11 and everything slowed down quite a bit.
Even if this server is not running a model, the load still can go up pretty high
if one machine is continually reading through the output files like this.

I believe it is a Linux NFS client issue. With the need to bring more servers
on-line in the future, I see NFS as being even more of a bottleneck (at least
with Linux clients). I don't want to increase the problem when a compute server
is added.

If it's performance, then how much performance do you need,
there might be NAS solutions that would be worth investigating. It would
also help to have some idea of the budget, i.e. is this a project where
everything has to be done for the absolute minimum bucks, or is this a
problem that you can spend money to solve?

A NAS solution might work and be more affordable, although the Linux client
issue might be a problem no matter what filer we use. In any case, we do have
some money to add storage and make things more efficient.

Thanks for your help with this Nik.

Steve
--
__________________________________________________ ____________________
Steve Cousins, Ocean Modeling Group Email:
Marine Sciences, 208 Libby Hall http://rocky.umeoce.maine.edu
Univ. of Maine, Orono, ME 04469 Phone: (207) 581-4302

Hi,

"Nik Simpson" wrote in message ...
Steve Cousins wrote:
Nik Simpson wrote:

I'm just starting to think about SAN's. I don't know a whole lot
about them. Would other devices like the Apple Xserve RAID or the
NexSAN ATABoy/Beast be similar to the AX100? We use a combination of
large files (between 80 MB and 1.5 GB) and small files.

Mostly read or mostly write or somewhere in between?

Somewhere in between. When we run a model, it reads in about 50 MB
of data per day and outputs about 80 MB per day. By "per day" I mean
"per model day". If we are re-running our model for 2003 then it
needs to read in about 18 GB and output about 30 GB. However, other
programs just read through these model output files and then create a
single 100 MB file.


Any idea what sort of sustained rate you need, i.e. MB/s that you like to
make available to the client.

Is it a workflow type operation, i.e. server "a" reads the file does some
processing creates an output file, server "b" reads outhput, does some
processing an so on. Or do all the servers open the same file at the same
time & read it.

Do the servers need to share a single filesystem?

Yes. Since all of these compute servers are Linux is there a prefered
file system? GFS? Do these work well? I've heard of CXFS from SGI
and we do have an SGI machine (Origin 3000) but I've heard this is
quite expensive. I haven't priced any of these file systems out yet
so maybe they are all expensive.

You might want to ask about SAN filesystems in one of the LINUX groups as
well, I'll be homest, other than knowing the names of the various products I
know little about their pros & cons.


What other information do you need? Basically, if NFS with Linux
worked better I'd be happy with that.

It might be useful to tell us what you are unhappy with vis NFS,
performance? If it's performance, then how much performance do you need,
there might be NAS solutions that would be worth investigating. It would
also help to have some idea of the budget, i.e. is this a project where
everything has to be done for the absolute minimum bucks, or is this a
problem that you can spend money to solve?


I would say that your requirements are pretty lightweight for cxfs :-)
It becomes cost effective for large i/o rates. But i have a simple
solution to your problem.

You have a Origin 3000 which is quite an i/o monster. You could put
the latest version of irix on it and turn it into an NFS server free
of cost.
The features supported by IRIX NFS server are :

(a) ~1 GB/s i/o rates !!! Not many can touch this. These scalability
fixes
were added to 6.5.22f and later versions of IRIX.

(b) Support for some nice direct i/o on the clients and some buffer
copy avoidance techniques.

Please dont be fooled by the high iops rate glorified by other vendors
because looking at your workload,it looks like you will benefit from
I/O throughput more
than IOPS.

Just my 2 cents.

Cheers,
Arun