New to NAS, have idea, looking for software to do it.

I am a smalltime consultant with several clients having similar
scenarios. Typically they have less than 100 Windows workstations and
anywhere from 1 to 10 servers. Some servers are virtual hosts - just
getting started in the virtual world.

My idea is this: If I have up to 100 PC's, often each one has many GB
of free disk space on the local disk drives. Is there any software
that would allow me to designate some portion of this unused disk
space on all those PC's to a central server and combine them as one
logical drive? I often think of it as the Borg episode on Star Trek.
It seems like a waste to have all that free storage on workstations
and no way to manage it as one resource. It would seem someone has
thought of this and developed software to do that. Maybe iSCSI is the
answer?

Input appreciated.

On Nov 16, 5:45*pm, Bob Meyers wrote:
I am a smalltime consultant with several clients having similar
scenarios. Typically they have less than 100 Windows workstations and
anywhere from 1 to 10 servers. Some servers are virtual hosts - just
getting started in the virtual world.

My idea is this: If I have up to 100 PC's, often each one has many GB
of free disk space on the local disk drives. Is there any software
that would allow me to designate some portion of this unused disk
space on all those PC's to a central server and combine them as one
logical drive? I often think of it as the Borg episode on Star Trek.
It seems like a waste to have all that free storage on workstations
and no way to manage it as one resource. It would seem someone has
thought of this and developed software to do that. Maybe iSCSI is the
answer?


Certainly you can put an iSCSI host on each workstation with spare
disk space, and then mount each of those LUNs on a server, and then
put a volume on top of those. Basically this would be pretty trivial
(if a bit tedious) to set up, the main limit you're going to run into
is the maximum number of partitions you can put in a single volume.

But, the big problem is the availability and reliability of the
"disks" on the workstations. Not only are those workstation class
disks, you're at the mercy of the workstation being up and running,
and the users not doing anything bad to the slice of space dedicated
to the common storage area. Some stuff like mirroring and RAID may
give you the ability to survive missing workstations.

And given the cost of disk space, this seems like far more hassle than
its worth.

More formal systems of this type are usually called "distributed
storage" and there are several efforts on different OSs to implement
that. But the focus is usually more on collecting the storage from a
bunch of servers, rather than workstations.

Bob Meyers wrote:
I am a smalltime consultant with several clients having similar
scenarios. Typically they have less than 100 Windows workstations and
anywhere from 1 to 10 servers. Some servers are virtual hosts - just
getting started in the virtual world.

My idea is this: If I have up to 100 PC's, often each one has many GB
of free disk space on the local disk drives. Is there any software
that would allow me to designate some portion of this unused disk
space on all those PC's to a central server and combine them as one
logical drive? I often think of it as the Borg episode on Star Trek.
It seems like a waste to have all that free storage on workstations
and no way to manage it as one resource. It would seem someone has
thought of this and developed software to do that. Maybe iSCSI is the
answer?

Input appreciated.

I don't know if doing something like this would be /practical/, but it
might be fun!

When you say "iSCSI", I guess you are thinking of having each machine
host an iSCSI target serving out a section of the local hard disk. The
severer would make use of these distributed chunks as volumes on a big
raid drive. This is going to have a number of issues. Since you have
workstations being switched on and off, or rebooted (they are windows
workstations...), the iSCSI disks will go off-line. That means that you
need significant redundancy in your raid setup, and are likely to spend
a great deal of time doing re-builds. To avoid re-building each "drive"
as they come on line again, you would want to use a raid system that
supports write-intent bitmaps such as Linux md raid. Of course, more
redundancy means more traffic, so this would be pretty hard on the network.

Another idea would be to use some sort of distributed file system.
Examples include GFS and Lustre. However, they are not designed for
this sort of environment, so they will not be ideal.

Bob Meyers wrote:

....

My idea is this: If I have up to 100 PC's, often each one has many GB
of free disk space on the local disk drives. Is there any software
that would allow me to designate some portion of this unused disk
space on all those PC's to a central server and combine them as one
logical drive? I often think of it as the Borg episode on Star Trek.
It seems like a waste to have all that free storage on workstations
and no way to manage it as one resource. It would seem someone has
thought of this and developed software to do that.

Well, yes and no. In the '90s there was a Berkeley project called
'Network of Workstations' that focused on networking commodity PCs into
a high-performance compute cluster, and it included mechanisms (such as
the Zebra file system, IIRC) for sharing their storage in the manner
which you describe. And some more recent distributed file systems could
probably be configured to use workstation storage in the manner that you
describe, simply by using a portion of each workstation as one of their
clustered 'servers'.

But, as others have pointed out, storage is extremely inexpensive these
days and part of what makes a PC 'personal' is that the owner controls
it exclusively (i.e., has access to all its hardware if desired,
including the option to turn it off). While there's still a reasonable
case to be made for including a local disk in each PC there may not be
much of a case for sharing this local storage externally: rather, a
central server (or, for large installations, a central cluster of
servers) can provide shared space for everyone at modest cost while
reducing the load on the network significantly (since most needs will
still be met by each local disk).

There's plenty of software available to ensure that the central storage
gets backed up consistently, which might be more of a problem with the
data as distributed as you suggest. And you can use higher-quality
components (e.g., ECC RAM, perhaps even 'end-to-end' checking and
redundancy using something like the ZFS file system - or a more
expensive commercial alternative from NetApp) to help ensure that the
shared data maintains reasonable integrity. All in all, your users will
likely have far fewer headaches - and this reduction in headaches will
likely save more actual cash than the cost of the additional hardware.

- bill

On 2009-11-16 18:45:11 -0500, Bob Meyers said:

I am a smalltime consultant with several clients having similar
scenarios. Typically they have less than 100 Windows workstations and
anywhere from 1 to 10 servers. Some servers are virtual hosts - just
getting started in the virtual world.

My idea is this: If I have up to 100 PC's, often each one has many GB
of free disk space on the local disk drives. Is there any software
that would allow me to designate some portion of this unused disk
space on all those PC's to a central server and combine them as one
logical drive? I often think of it as the Borg episode on Star Trek.
It seems like a waste to have all that free storage on workstations
and no way to manage it as one resource. It would seem someone has
thought of this and developed software to do that. Maybe iSCSI is the
answer?

Input appreciated.


Stuff like this is relatively common in the virtualization world; think
of the LeftHand VSA or similar. Unfortunately, most of the software
that might allow you to do this in a physical world are really more
oriented toward server infrastructure and not desktop infrastructure.

So, is it possible? Probably. Is it recommended? Not necessarily.

--
Scott

on the 17/11/2009 02:06 lid wrote the following:
Bob kenjka:
I am a smalltime consultant with several clients having similar
scenarios. Typically they have less than 100 Windows workstations and
anywhere from 1 to 10 servers. Some servers are virtual hosts - just
getting started in the virtual world.

My idea is this: If I have up to 100 PC's, often each one has many GB
of free disk space on the local disk drives. Is there any software
that would allow me to designate some portion of this unused disk
space on all those PC's to a central server and combine them as one
logical drive? I often think of it as the Borg episode on Star Trek.
It seems like a waste to have all that free storage on workstations
and no way to manage it as one resource. It would seem someone has
thought of this and developed software to do that. Maybe iSCSI is the
answer?

DFS (Distributed File System), but I don't know how it will work with your
scenario... Someone before me pointed that these drives in workstations are
desktop class, so reliability is very low...

There is also DataCore software you could try to investigate, but it's much
better used on server infrastructure...

Not 100% certain but I believe Active Circle does what the OP is asking for,
i.e. consolidating local storage from various servers into a global storage
pool. Never used it, don't know whether it's worth the money etc. And I
don't have time to look at their web site to see whether my memory is
accurate to be honest.

--
Fx

On 2009³â11¿ù17ÀÏ, ¿ÀÀü8½Ã45ºÐ, Bob Meyers wrote:
I am a smalltime consultant with several clients having similar
scenarios. Typically they have less than 100 Windows workstations and
anywhere from 1 to 10 servers. Some servers are virtual hosts - just
getting started in the virtual world.

My idea is this: If I have up to 100 PC's, often each one has many GB
of free disk space on the local disk drives. Is there any software
that would allow me to designate some portion of this unused disk
space on all those PC's to a central server and combine them as one
logical drive? I often think of it as the Borg episode on Star Trek.
It seems like a waste to have all that free storage on workstations
and no way to manage it as one resource. It would seem someone has
thought of this and developed software to do that. Maybe iSCSI is the
answer?

Input appreciated.

Please refer to "LifeBoat" paper published in LISA'04. (http://
http://www.usenix.org/event/lisa04/t...bonkenburg.pdf)
It proposes p2p-backup/restore architecture to utilize free resource
of each client.
Actually, it works in decentralized fashion, not centralized one you
imagined, but I think it is the right choice when considering
scalability problem.