If this is your first visit, be sure to check out the FAQ by clicking the link above. You may have to register before you can post: click the register link above to proceed. To start viewing messages, select the forum that you want to visit from the selection below. |
|
|
Thread Tools | Display Modes |
#11
|
|||
|
|||
Starting with XP, Windows uses this prefetch for both app binaries and for
kernel modules - the latter speeds up booting a lot. There's lots of file systems that prefetch blocks and objects... UNIX has prefetched blocks for sequentially accessed files since the late '70s at least. Prefetch for sequentially accessed files is in NT since version 3.1, and is governed by a flag to CreateFile - you can turn it off if you want. You can also turn off the cache usage for a file at all, which makes the IO to this file being the same as IO to the raw partition. Thus, in NT, there is no need ever in software using raw partitions for data storage - use the noncached file instead. If such a file is not fragmented - which is usually true, since it is created as one huge chunk - then the FS overhead is around 20 machine instructrions (or so) per read/write operation. This simplifies the database administration greatly. No need in repartitioning for this purpose. That doesn't sound like what Bill was talking about... could you go Prefetch by reading the contents (or the metadata) of several files in a single IO operation. Prefetching pages of the executable image by formerly gathered stats on how it used to be executed. -- Maxim Shatskih, Windows DDK MVP StorageCraft Corporation http://www.storagecraft.com |
#12
|
|||
|
|||
"Faeandar" wrote in message ... I am concerned about the following: 1) Fast read and write performance for consistant stream of small files 2) Ability to store LARGE ( 10million per 200GB) amount of small files For both 1 and 2 you need to understand that there is *no* server or filesystem that can speedily do what your talking about. That's kind of a subjective matter, isn't it? Unless you expect miracles, that is. IIUC, NetApp's file system should do a pretty good job of handling bulk small-file writes: it certainly captures relevant meta-data updates in stable RAM, and may capture the data itself there as well - such that it can be written to disk in large, efficient chunks. And fast access to large numbers of small files is first a matter of having the directory tree mostly in RAM to make look-ups fast (at least assuming a competent directory structure that doesn't require linear searches to find files within a directory): one would expect a NetApp server to do this if it was configured with sufficient memory, and you might pick up a few cache hits on recently-accessed file contents as well. Beyond that, experimental file systems have tried shuffling files on disk based on access patterns to allow a single physical disk access to pick up the file(s) that are likely to be needed next, but I don't know of any commercial file system that does this (though Windows - gasp! - reportedly has some such mechanism to expedite application loading). Let me clarify. For the majority of data access patterns WAFL handles small files very well, better than most in fact. The reason is that the clients generally are not accessing all bazillion files at the same time. My sentiment was primarily in regards to backups, which I did not mention so how would anyone know? Sorry about that. For general small file access for clients/apps WAFL is great. For full file system scans (like backup) nothing handles it well, except something like FlashBackup which converts the inode map into a bitmap. Kinda cool actually but I digress. Not all that off-topic for this forum, though. Full-disk (non-file-structured) backups can perform pretty well, especially if the file system pays some attention to consolidating used space on the disk rather than letting it sprawl promiscuously across the entire capacity (when the disk isn't nearly full, anyway) and the backup mechanism can recognize this. Of course, unless the backup is on a random-access device this makes anything short of a full-disk restore a rather lengthy process. A file system that consolidated smallish files on the disk if they were in the same directory could significant expeditefull-system file-structured backups - but I don't know of any that do. - bill |
#13
|
|||
|
|||
"Peter da Silva" wrote in message ... In article , Maxim S. Shatskih wrote: to be needed next, but I don't know of any commercial file system that does this (though Windows - gasp! - reportedly has some such mechanism to expedite application loading). Starting with XP, Windows uses this prefetch for both app binaries and for kernel modules - the latter speeds up booting a lot. There's lots of file systems that prefetch blocks and objects... UNIX has prefetched blocks for sequentially accessed files since the late '70s at least. That doesn't sound like what Bill was talking about... could you go into more detail, perhaps? The mechanism that I was referring to involves analyzing access patterns at run-time and reorganizing the files on the disk to allow the disk's (or possibly OS's) prefetch behavior, if enabled, to fetch what is likely to be the next data requested. It's only slightly analogous to the prefetch mechanisms (sequential, strided, etc.) traditionally employed within a single file (with no attempt to reorganize the on-disk placement based on run-time analysis of access patterns). IIRC John Wilkes' group at HP did some work in this area, possibly in connection with the research that later became the AutoRAID product (and might even have productized the mechanism). And, as noted, Windows supports it in more limited (boot-time and/or application-load) contexts where repetitive behavior is the norm and access-pattern-capture is a one-time rather than on-going overhead. - bill |
#14
|
|||
|
|||
There's lots of file systems that prefetch blocks and objects... UNIX has
prefetched blocks for sequentially accessed files since the late '70s at least. That doesn't sound like what Bill was talking about... could you go into more detail, perhaps? This MSDN article by Solomon/Russinovich describes XP's prefetch in details: "Windows XP: Kernel Improvements Create a More Robust, Powerful, and Scalable OS" -- Maxim Shatskih, Windows DDK MVP StorageCraft Corporation http://www.storagecraft.com |
#15
|
|||
|
|||
In article ,
Bill Todd wrote: It's only slightly analogous to the prefetch mechanisms (sequential, strided, etc.) traditionally employed within a single file (with no attempt to reorganize the on-disk placement based on run-time analysis of access patterns). No? I see it as an elaboration of the same kind of caching strategies that have been going on for decades. First you have contiguous files (RSX, etc) and pre-allocation of executables (RTE, RSX to some extent). Then you have prefetching and 'sticky' executables (V6) and cylinder groups (BSD), and no doubt more other things I haven't heard of than I have. Anyway, do you have a URL for this Windows technique? -- I've seen things you people can't imagine. Chimneysweeps on fire over the roofs of London. I've watched kite-strings glitter in the sun at Hyde Park Gate. All these things will be lost in time, like chalk-paintings in the rain. `-_-' Time for your nap. | Peter da Silva | Har du kramat din varg, idag? 'U` |
#16
|
|||
|
|||
Anyway, do you have a URL for this Windows technique?
I have posted the MSDN article chapter title to another message to the same forum, you can search http://msdn.microsoft.com for it. In two words: cache/VM manager gather page faults stats on the driver and user-mode binaries, then prefetch service ask the VM for this stats and saves it to some .PF file. Then, when the binary (or the OS) is loaded, the prefetch file shows what pages to preload. All drivers share the same .PF file, so, they are preloaded in several IO requests and not on per-file basis. -- Maxim Shatskih, Windows DDK MVP StorageCraft Corporation http://www.storagecraft.com |
#17
|
|||
|
|||
In article ,
Maxim S. Shatskih wrote: Prefetch for sequentially accessed files is in NT since version 3.1, and is governed by a flag to CreateFile - you can turn it off if you want. Not surprised. NT seems to have a fairly decent OS buried somewhere beneath the horror that is Win32. Prefetching pages of the executable image by formerly gathered stats on how it used to be executed. [... etc...] I'm sorry, none of this seems to involve actually moving data around on disk based on usage, which is what I was asking about. -- I've seen things you people can't imagine. Chimneysweeps on fire over the roofs of London. I've watched kite-strings glitter in the sun at Hyde Park Gate. All these things will be lost in time, like chalk-paintings in the rain. `-_-' Time for your nap. | Peter da Silva | Har du kramat din varg, idag? 'U` |
#18
|
|||
|
|||
In article ,
Maxim S. Shatskih wrote: This MSDN article by Solomon/Russinovich describes XP's prefetch in details: "Windows XP: Kernel Improvements Create a More Robust, Powerful, and Scalable OS" I don't suppose you could actually provide the URL...? -- I've seen things you people can't imagine. Chimneysweeps on fire over the roofs of London. I've watched kite-strings glitter in the sun at Hyde Park Gate. All these things will be lost in time, like chalk-paintings in the rain. `-_-' Time for your nap. | Peter da Silva | Har du kramat din varg, idag? 'U` |
#19
|
|||
|
|||
Prefetch for sequentially accessed files is in NT since version 3.1, and is
governed by a flag to CreateFile - you can turn it off if you want. Not surprised. NT seems to have a fairly decent OS buried somewhere beneath the horror that is Win32. USER in particular is horror, though yes, there are some UNIX "toolkits" which are even more horror-some. As about GDI and the NT kernel - decent stuff, though there are drawbacks which make the script-based and command-line administration more complex. I'm sorry, none of this seems to involve actually moving data around on disk based on usage, which is what I was asking about. Never heard on XP doing automatic data moving to speed up things. You must defragment the disk manually. -- Maxim Shatskih, Windows DDK MVP StorageCraft Corporation http://www.storagecraft.com |
#20
|
|||
|
|||
In article ,
Maxim S. Shatskih wrote: Then, when the binary (or the OS) is loaded, the prefetch file shows what pages to preload. All drivers share the same .PF file, so, they are preloaded in several IO requests and not on per-file basis. OK, it's not the same kind of thing, then. It's prefetching, not disk layout optimization. -- I've seen things you people can't imagine. Chimneysweeps on fire over the roofs of London. I've watched kite-strings glitter in the sun at Hyde Park Gate. All these things will be lost in time, like chalk-paintings in the rain. `-_-' Time for your nap. | Peter da Silva | Har du kramat din varg, idag? 'U` |
Thread Tools | |
Display Modes | |
|
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
WL 400 Tuning and fix ( Solution customer care N° 4203 ) | Free | Compaq Computers | 0 | November 12th 04 01:01 AM |
Howto install the drivers for mobility Radeon 9200 on Compaq X1000 with Windows Server 2003 !!! (solution) | Razvan | Ati Videocards | 1 | August 12th 04 07:13 AM |
Headphone Solution for High End Rig | Gamer | Ati Videocards | 2 | December 18th 03 02:29 AM |
Soyo and ATI Compatiblity Problem in XP ----- SOLUTION FOUND!!!! | Hoss.N.Pfeffer | Ati Videocards | 1 | October 9th 03 09:43 PM |
Artec or Memorex 48U Scanner Solution | BMac | Scanners | 0 | September 25th 03 07:06 PM |