Metadata Server ?

Hi,
does anybody know if it's possible to have a kind of metadata server ?

I explain, with 25Tb of data and many directories of 100 000 files,
when i try "ls -all"
it's too long .....
I'd enough time to take a coffee

So i would like to find a kind of metadata server synchronize with the
nas file system.

I think use a virtual distributed file system like Lustre or Gfarm.


thanks in advance,

Séb.

On Fri, 19 Oct 2007 16:19:20 +0200, Sebastien Campion
wrote:

Hi,
does anybody know if it's possible to have a kind of metadata server ?

I explain, with 25Tb of data and many directories of 100 000 files,
when i try "ls -all"
it's too long .....
I'd enough time to take a coffee

So i would like to find a kind of metadata server synchronize with the
nas file system.

I think use a virtual distributed file system like Lustre or Gfarm.


thanks in advance,

Séb.

If the file system supports a seperation of data and metadata, like
SAMFs or Lustre, then you actually *have* to have a metadata server.
If the fs does not support it then you can't have one, at least not
without doing some very unsupported things.

~F

Thanks, i learned more about the filesystem, it's WAFL by NetApp.
In documentation, i see "The WAFL file system store metadata in files,
three files :
- inode file
- block-map file
- inode map"

But i don't know where i can find it.

I'd only NFS (or CIFS) protocol enable (it's a NAS NetApp) so i don't
know if it's feasible to connect Lustre on NFS.
If it's ok, i will install Lustre.

Séb.


But i don't know where

Faeandar wrote:
On Fri, 19 Oct 2007 16:19:20 +0200, Sebastien Campion
wrote:


Hi,
does anybody know if it's possible to have a kind of metadata server ?

I explain, with 25Tb of data and many directories of 100 000 files,
when i try "ls -all"
it's too long .....
I'd enough time to take a coffee

So i would like to find a kind of metadata server synchronize with the
nas file system.

I think use a virtual distributed file system like Lustre or Gfarm.


thanks in advance,

Séb.


If the file system supports a seperation of data and metadata, like
SAMFs or Lustre, then you actually *have* to have a metadata server.
If the fs does not support it then you can't have one, at least not
without doing some very unsupported things.

~F

On 2007-10-22, Sebastien Campion wrote:

I'd only NFS (or CIFS) protocol enable (it's a NAS NetApp) so i don't
know if it's feasible to connect Lustre on NFS.
If it's ok, i will install Lustre.

No, lustre is a completely different file system. It can't help your NetApp.

100.000 files in each directory is going to be painfull to list,
no matter what file system you're using. Things that might help:

- run a name service cache daemon (nscd) to cache uid/gid lookups
- don't resolve uid/gid by using "ls -ln"
- use "locate" as your "metadata server", if it contains enough
metadata for your needs.
- see if an fscache/cachefs can help
http://people.redhat.com/~dhowells/fscache/FS-Cache.pdf
- tune your NFS client http://www.netapp.com/library/tr/3183.pdf


-jf