Storage across multiple servers?

I am trying to find out how to do something that does not make much
sense to me. I have spoken to a few people who say they have taken
multiple servers (1u in this case) and striped the drives on those 1u
servers so that all servers in the group were seeing all data on all
servers.

Does this make sense to anyone?

To put it in context, this came up during a conversation over the
merits of NAS(nfs) vs SAN.

The above idea was given to me as a 'cheaper' solution.

I'm not really sure the 'cheaper' solution is what I want, but I was
intruqued about how this is possible.

Does anyone know how to do this or what this is called? It would seem
it might be a 'cluster file system' but I see nothing like that when
performing the usual Google searches.

And...for the record, this was on Linux and possibly a *bsd.

Thanks!

-- JB

Look for Google File System.


JB Orca wrote:
I am trying to find out how to do something that does not make much
sense to me. I have spoken to a few people who say they have taken
multiple servers (1u in this case) and striped the drives on those 1u
servers so that all servers in the group were seeing all data on all
servers.

Does this make sense to anyone?

To put it in context, this came up during a conversation over the
merits of NAS(nfs) vs SAN.

The above idea was given to me as a 'cheaper' solution.

I'm not really sure the 'cheaper' solution is what I want, but I was
intruqued about how this is possible.

Does anyone know how to do this or what this is called? It would seem
it might be a 'cluster file system' but I see nothing like that when
performing the usual Google searches.

And...for the record, this was on Linux and possibly a *bsd.

Thanks!

-- JB

Yes, I have seen that, however, the Google File System is not something
someone other than Google can use, correct?

I am looking for something that could be used by anyone.

Many thanks.


On 2004-11-22 09:57:26 -0500, Yura Pismerov said:


Look for Google File System.


JB Orca wrote:
I am trying to find out how to do something that does not make much
sense to me. I have spoken to a few people who say they have taken
multiple servers (1u in this case) and striped the drives on those 1u
servers so that all servers in the group were seeing all data on all
servers.

Does this make sense to anyone?

To put it in context, this came up during a conversation over the
merits of NAS(nfs) vs SAN.

The above idea was given to me as a 'cheaper' solution.

I'm not really sure the 'cheaper' solution is what I want, but I was
intruqued about how this is possible.

Does anyone know how to do this or what this is called? It would seem
it might be a 'cluster file system' but I see nothing like that when
performing the usual Google searches.

And...for the record, this was on Linux and possibly a *bsd.

Thanks!

-- JB

On 21 Nov 2004 22:03:53 -0800, (JB Orca) wrote:

I am trying to find out how to do something that does not make much
sense to me. I have spoken to a few people who say they have taken
multiple servers (1u in this case) and striped the drives on those 1u
servers so that all servers in the group were seeing all data on all
servers.

Does this make sense to anyone?

To put it in context, this came up during a conversation over the
merits of NAS(nfs) vs SAN.

The above idea was given to me as a 'cheaper' solution.

I'm not really sure the 'cheaper' solution is what I want, but I was
intruqued about how this is possible.

Does anyone know how to do this or what this is called? It would seem
it might be a 'cluster file system' but I see nothing like that when
performing the usual Google searches.

And...for the record, this was on Linux and possibly a *bsd.

Thanks!

-- JB

Look at Polyserve, Ibrix, GFS, GPFS, etc. These are all software
solutions for a High Performance File System. How the data is
accessed is up to you, either directly from the node servers or
re-shared as NFS.
A couple of hardware based solutions in this space are Panasas and
Acopia Networks.

~F

On 2004-11-22 17:54:33 -0500, Faeandar said:

On 21 Nov 2004 22:03:53 -0800, (JB Orca) wrote:

I am trying to find out how to do something that does not make much
sense to me. I have spoken to a few people who say they have taken
multiple servers (1u in this case) and striped the drives on those 1u
servers so that all servers in the group were seeing all data on all
servers.

Does this make sense to anyone?

To put it in context, this came up during a conversation over the
merits of NAS(nfs) vs SAN.

The above idea was given to me as a 'cheaper' solution.

I'm not really sure the 'cheaper' solution is what I want, but I was
intruqued about how this is possible.

Does anyone know how to do this or what this is called? It would seem
it might be a 'cluster file system' but I see nothing like that when
performing the usual Google searches.

And...for the record, this was on Linux and possibly a *bsd.

Thanks!

-- JB

Look at Polyserve, Ibrix, GFS, GPFS, etc. These are all software
solutions for a High Performance File System. How the data is
accessed is up to you, either directly from the node servers or
re-shared as NFS.
A couple of hardware based solutions in this space are Panasas and
Acopia Networks.

~F

Excellent. Thanks much.

Can I assume that AFS would also fit in here as well?

Thanks!

On Mon, 22 Nov 2004 19:54:42 -0500, JB Orca wrote:

On 2004-11-22 17:54:33 -0500, Faeandar said:

On 21 Nov 2004 22:03:53 -0800, (JB Orca) wrote:

I am trying to find out how to do something that does not make much
sense to me. I have spoken to a few people who say they have taken
multiple servers (1u in this case) and striped the drives on those 1u
servers so that all servers in the group were seeing all data on all
servers.

Does this make sense to anyone?

To put it in context, this came up during a conversation over the
merits of NAS(nfs) vs SAN.

The above idea was given to me as a 'cheaper' solution.

I'm not really sure the 'cheaper' solution is what I want, but I was
intruqued about how this is possible.

Does anyone know how to do this or what this is called? It would seem
it might be a 'cluster file system' but I see nothing like that when
performing the usual Google searches.

And...for the record, this was on Linux and possibly a *bsd.

Thanks!

-- JB

Look at Polyserve, Ibrix, GFS, GPFS, etc. These are all software
solutions for a High Performance File System. How the data is
accessed is up to you, either directly from the node servers or
re-shared as NFS.
A couple of hardware based solutions in this space are Panasas and
Acopia Networks.

~F

Excellent. Thanks much.

Can I assume that AFS would also fit in here as well?

Thanks!

No. AFS is a read-many replica file system but still only has a single
writeable volume, and that resides on a single host. It's a great
file system for traditional file IO and program reads in that it
allows client side caching, multi-location replicas (for network
performance), and transparent read failover to another replica in the
event the current volume is unavailable. But it is in no way a
performance file system like the ones I mentioned.

Also, AFS in non-trivial and generally requires an almost-dedicated if
not completely dedicated admin.

~F

On 2004-11-22 20:43:52 -0500, Faeandar said:

......snip....

No. AFS is a read-many replica file system but still only has a single
writeable volume, and that resides on a single host. It's a great
file system for traditional file IO and program reads in that it
allows client side caching, multi-location replicas (for network
performance), and transparent read failover to another replica in the
event the current volume is unavailable. But it is in no way a
performance file system like the ones I mentioned.

Also, AFS in non-trivial and generally requires an almost-dedicated if
not completely dedicated admin.

~F

Ok. Excellent. I appreciate the info very much.

What I need to accomplish is this:

I have a system that will need to start with roughly 5 terabytes of
storage space. It will very quickly grow to needing anywhere from
50-100 terabytes.

The problem we are attempting to solve is this: what is the best option
for the storage in this system? The original thought, before we
realized how big it was going to get, was just a large RAID direct
attach system. Then we thought about NAS or SAN, however, when I heard
the talk of spanning storage space across multiple servers this seemed
as though it might also be a good option.

We are much more interested in the data being safe than we are in the
raw speed of the devices. We can't have something _slow_ per se,
however, if I have to sacrifice some transfer speed in order to have
more safety for the files, that is acceptable.

I am still reading about the systems mentioned and trying to figure out
what would suit our needs.

The idea of a 'RAID' of servers seems fantastic. If I can use the
storage on 5 servers and stripe the data across them that would be
great, however, I have noticed with some of the options that in order
to add a new server the entire system needs to be taken down and
re-configured and brought back up.

That would not be possible for us as we really need to have as little
downtime as humanly possible. (Don't we all!)

The conversation about this is great and I really appreciate any input
that can be given.

Thanks much.

JB

On Tue, 23 Nov 2004 12:09:09 -0500, JB Orca wrote:

On 2004-11-22 20:43:52 -0500, Faeandar said:

.....snip....

No. AFS is a read-many replica file system but still only has a single
writeable volume, and that resides on a single host. It's a great
file system for traditional file IO and program reads in that it
allows client side caching, multi-location replicas (for network
performance), and transparent read failover to another replica in the
event the current volume is unavailable. But it is in no way a
performance file system like the ones I mentioned.

Also, AFS in non-trivial and generally requires an almost-dedicated if
not completely dedicated admin.

~F

Ok. Excellent. I appreciate the info very much.

What I need to accomplish is this:

I have a system that will need to start with roughly 5 terabytes of
storage space. It will very quickly grow to needing anywhere from
50-100 terabytes.

The problem we are attempting to solve is this: what is the best option
for the storage in this system? The original thought, before we
realized how big it was going to get, was just a large RAID direct
attach system. Then we thought about NAS or SAN, however, when I heard
the talk of spanning storage space across multiple servers this seemed
as though it might also be a good option.

We are much more interested in the data being safe than we are in the
raw speed of the devices. We can't have something _slow_ per se,
however, if I have to sacrifice some transfer speed in order to have
more safety for the files, that is acceptable.

I am still reading about the systems mentioned and trying to figure out
what would suit our needs.

The idea of a 'RAID' of servers seems fantastic. If I can use the
storage on 5 servers and stripe the data across them that would be
great, however, I have noticed with some of the options that in order
to add a new server the entire system needs to be taken down and
re-configured and brought back up.

That would not be possible for us as we really need to have as little
downtime as humanly possible. (Don't we all!)

The conversation about this is great and I really appreciate any input
that can be given.

Thanks much.

JB

First thing then forget AFS. There is a backup limitation of 8gb max
per volume, this is across all backup software platforms that I am
aware of. It's not an AFS limit, just fyi.

You never mention what you're going to be doing with this data. Is it
for a single server? multiple servers? Mult host access? Multi host
write access?

The technology you use really depends on the requirements of your data
and users. Post a little more info on what you are trying to
accomplish and we might be able to help. Storage is simply a means to
an end, not the end itself.

~F

JB Orca wrote:
....
I have a system that will need to start with roughly 5 terabytes of
storage space. It will very quickly grow to needing anywhere from
50-100 terabytes.

With these kinds of numbers, you'll have lot of drives and thus drive
failures will become quite common. Are you looking at using some RAID
configuration to overcome this ?

The problem we are attempting to solve is this: what is the best option
for the storage in this system? The original thought, before we
realized how big it was going to get, was just a large RAID direct
attach system. Then we thought about NAS or SAN, however, when I heard
the talk of spanning storage space across multiple servers this seemed
as though it might also be a good option.

This would be using your LAN to move data unless the server doing the
I/O happens to have the target disk locally available, right ? I guess
with gigabit ethernet this might not be such a problem anymore, except
for processor overhead.
A SAN will allow every server to use Fibre Channel to move the data,
your LAN and server cpus won't be loaded nearly as much. Depending on
your application load, you could save a lot on LAN switches and servers
by spending more on a SAN.

....
The idea of a 'RAID' of servers seems fantastic. If I can use the
storage on 5 servers and stripe the data across them that would be
great, however, I have noticed with some of the options that in order
to add a new server the entire system needs to be taken down and
re-configured and brought back up.

The only reason to go this way instead of a regular SAN would be cost I
guess; by using plain old scsi drives you'll cut down the cost
significantly. But again my first question, do you plan on using some
form of RAID (software, raid controller, raid enclosure,...) ?

If you just plug in a bunch of scsi drives into a bunch of servers and
start storing data on then, at a hundred terabytes worth of disks, you'll
be running around shutting down hosts in order to swap out disks all
day in my opinion.

Arne Joris

On 2004-11-23 12:30:35 -0500, Faeandar said:

On Tue, 23 Nov 2004 12:09:09 -0500, JB Orca wrote:

On 2004-11-22 20:43:52 -0500, Faeandar said:

.....snip....

No. AFS is a read-many replica file system but still only has a single
writeable volume, and that resides on a single host. It's a great
file system for traditional file IO and program reads in that it
allows client side caching, multi-location replicas (for network
performance), and transparent read failover to another replica in the
event the current volume is unavailable. But it is in no way a
performance file system like the ones I mentioned.

Also, AFS in non-trivial and generally requires an almost-dedicated if
not completely dedicated admin.

~F

Ok. Excellent. I appreciate the info very much.

What I need to accomplish is this:

I have a system that will need to start with roughly 5 terabytes of
storage space. It will very quickly grow to needing anywhere from
50-100 terabytes.

The problem we are attempting to solve is this: what is the best option
for the storage in this system? The original thought, before we
realized how big it was going to get, was just a large RAID direct
attach system. Then we thought about NAS or SAN, however, when I heard
the talk of spanning storage space across multiple servers this seemed
as though it might also be a good option.

We are much more interested in the data being safe than we are in the
raw speed of the devices. We can't have something _slow_ per se,
however, if I have to sacrifice some transfer speed in order to have
more safety for the files, that is acceptable.

I am still reading about the systems mentioned and trying to figure out
what would suit our needs.

The idea of a 'RAID' of servers seems fantastic. If I can use the
storage on 5 servers and stripe the data across them that would be
great, however, I have noticed with some of the options that in order
to add a new server the entire system needs to be taken down and
re-configured and brought back up.

That would not be possible for us as we really need to have as little
downtime as humanly possible. (Don't we all!)

The conversation about this is great and I really appreciate any input
that can be given.

Thanks much.

JB

First thing then forget AFS. There is a backup limitation of 8gb max
per volume, this is across all backup software platforms that I am
aware of. It's not an AFS limit, just fyi.

You never mention what you're going to be doing with this data. Is it
for a single server? multiple servers? Mult host access? Multi host
write access?
The technology you use really depends on the requirements of your data
and users. Post a little more info on what you are trying to
accomplish and we might be able to help. Storage is simply a means to
an end, not the end itself.

~F

Good point...here's some additional info:

The plan is to have multiple 'user-facing' servers that the users will
interact with. Placing files, pulling files, etc.

All of these front-end servers should use a shared storage system. The
idea being this: if we have n number of front-end servers we can
balance any load across them, as long as our shared storage system is
robust enough we should be in decent shape.

The majority of files being stored will be in the 5-40 meg range. We do
not expect to have many over 40 megs.

There will be a lot of files.

So the idea is multi-host access, both read and write.

"Storage is simply a means to an end, not the end itself."

That is really well said. Perhaps I'm thinking about this too much....

I'm basically trying to be as thorough as possible and make sure I try
to think of any possible solution before committing the time it will
take to learn some new stuff and get a dev system up and running.

The idea of the mutli-server file system seemed good to me for this one
reason: it seemed like creating a RAID but using cheaper hardware with
an easier path to adding storage.

The idea of needing this much storage is a bit new to me, so I'm trying
to learn as I go here.

If I did a direct attached raid device, say a 2 terabyte raid, that
would be great. But, when it comes time to expand the storage it seems
like it would be a mess to add additional storage to that type of
system, no?

As I mentioned about, I'm kind of learning some of this as I go, so I
appreciate that help and input.

Many thanks.

JB