Using Storage Spaces with win 10

On Thu, 10 Mar 2016 17:23:53 -0500, Paul wrote:

That's a multi-threaded performance test.

Which is suited to testing "servers", not so much
sustained transfer rate that a user may favor.

My guess is, you're back to crafting a "file copy"
test of some sort.

Paul

What I'm really interested in is performance in searching (fairly)
large database files within Access or FileMaker Pro. I deal with data
searches in Access files of 300 to 600 MB, and FileMaker Pro database
files of up to 3 GB. I don't know if performance in that is comparable
to performance in copying files. I'm also forced to deal with huge MS
Word files, but I think Word is the limitation there, not the
hardware.

I'm beginning to think that I'm going into too much analysis on this,
ie, it's not really worth the effort for whatever I might eventually
learn.

On Thu, 10 Mar 2016 16:37:46 -0600, Charlie Hoffpauir
wrote:

I'm also forced to deal with huge MS
Word files, but I think Word is the limitation there, not the
hardware.

Indices on better databases are keys to instantaneous retrievals. I've
used IFY extensively, though old and primarily for text;- depends on
your needs and pattern retrievals, e.g. bolean, fuzzy logic, &etc for
usefulness. Given once IFY's indices are built, though, they're fast.
Of course and over an extent of material massive enough to be
unspeakably tedious and slow slugging, say with *NIX ports, Grep, a
likes to DOS/Command variants, were the same searches conducted
per-file-basis or a limited extent of hardware storage capacity.

I'm not up on what the newer versions about, past a notice of its
availability of recent. In any event the technology behind indices,
as far as I'm aware, hasn't yet been exceeded: once being what
empowered telephony conglomerates to automate instantaneous
identification to virtually everyone connected to their grids (copper
POTS), when at a golden age of automated computer-generated voice
retrievals. Before everyone migrated to competitive satellite
carriers due to excessive cost and technological factors in
advancement of communication to third-world constraints.

Shareware Version Discontinued. Now Freeware!

IYF - X ! 4.0 (Alfa08) to (Alpha16)
* Fix - Operator Boolean (Seach Files)

IYF - X ! 4.0 (Alfa03) to (Alpha05)
+ Add Sort Databases by Name,Total Items,Date
Suggested by kiwichick
+ Add Select All - None Databases
+ Add method of search with Command-Line.
Sintaxis; iyf-x.exe --databases:database1,database2
--files:files1,files2
Examples: (% is space)
iyf-x.exe --databases:Network --files:*.txt
iyf-x.exe --databasesisk%C,Disk%D,Network
--files:*.txt,*.jpg,*.pdf
Suggested by deyavi.
* Fix Filter Tree
* Fix Unicode - Search Text Inside Files
* Fix and Improved Internally IYF (GUI and Speed)
* New Demo Online (Index & Search)
* Special Thanks to William Athens for your donation.

IYF - X ! 4.0 (Alfa02) to (Alpha03)
+ Add Tree for Results
+ Add Search for ISO (Mode Exif)
* Special Thanks to Fred Speck for your donation.

IYF - X ! 4.0 (Alfa01) to (Alfa02)
+ Add Search for Make and Model (Mode Exif)
+ Add Search for Flash Fired - Not Fired - Any

IYF - X ! 4.0 (WIP 07.06.07) to (WIP 08.06.07)
* Fix Unicode - Search Text Inside Files

IYF - X ! 4.0 (WIP 30.05.07) to (WIP 07.06.07)
+ Unicode - Search Text Inside Files
* Fix Time Index. Report by Miguel Nobau
* Fix Filter Folder. Report by Kees van den Dries

IYF Home! 3.1 to IYF X ! 4.0 (WIP 30.05.07)
+ Unicode - Search Filenames
+ Thumbnails
+ Update Graphics (viewer jpg, tif, png, etc)
+ Search Inside Files (Search Text - Hexadecimal)
+ Shared Index Database - One Computer Index and All computer Search
+ Import - Export Databases
+ Run from CD
+ Freeware

Warning
This version change Databases to Unicode.
Old versions of IYF Home! and Revolution! no work with this new
Unicode databases.

Enjoy

/Rafael Castro/

Charlie Hoffpauir wrote:
On Thu, 10 Mar 2016 17:23:53 -0500, Paul wrote:

That's a multi-threaded performance test.

Which is suited to testing "servers", not so much
sustained transfer rate that a user may favor.

My guess is, you're back to crafting a "file copy"
test of some sort.

Paul

What I'm really interested in is performance in searching (fairly)
large database files within Access or FileMaker Pro. I deal with data
searches in Access files of 300 to 600 MB, and FileMaker Pro database
files of up to 3 GB. I don't know if performance in that is comparable
to performance in copying files. I'm also forced to deal with huge MS
Word files, but I think Word is the limitation there, not the
hardware.

I'm beginning to think that I'm going into too much analysis on this,
ie, it's not really worth the effort for whatever I might eventually
learn.

I am frequently disappointed when using what should be
fast setups. A stream benchmark on my test computer, gives
around 17-18GB/sec. I have a RAMDisk, and some sort of "erase"
command I used on it, it hit 10GB/sec (might have been
Diskpart clean all). On Win7 x32, the RAMdisk hits 7GB/sec,
because the memory is in PAE space and not coming out of the
regular memory pool. I cannot seem to get Large Pages working
on any desktop OS on the Windows side (there's no evidence it
is working, even though a particular registry entry was set).
Win10, the RAMDisk is all over the place (depends on the day
of the week, and the air temperature, and what the computer
ate for lunch). I can only be assured of 1GB/sec (out of that
17GB/sec number), and there can be spikes down to the 300MB/sec
speed range.

Then, when it comes to fast random access, where we know a
RAMDisk could drop as low as 1usec with good software, I
seem to be seeing the file system as the bottleneck. You
just can't access more than a couple thousand files a second.

My conclusion after a lot of different tests, is an SSD
is better performance per dollar, than sticking RAM into
the computer.

If you really wanted to use Storage Spaces for some reason,
you could use a pair of SSD drives. Maybe that's about the
best you could do. I don't really see a reason why Storage
Spaces couldn't use DMA like any other storage operation,
so the overhead should be low, and the performance about as
good as the SSD can manage. The file system overhead
still prevents the full performance of SSD drives
from being realized.

The other option, would be to find a RAID card with
cache DIMM on it, and stick the largest cache DIMM it
can take, on board. In the hopes that the OS cannot
mess with it, and degrade its performance. Then don't
use Storage Spaces (eliminating one layer of software),
use RAID 1, and have one SSD protect the other SSD.

When I tested a Linux LiveCD (where /tmp is mounted
on RAM), I got around 7GB/sec on that. Which is
respectable. I haven't really carried out any
seek tests, or small file tests there, to see
whether it flies in terms of bottlenecks or not.

But for all the experiments I've carried out, I've
been mostly disappointed. The capper was the following
one. I set up VirtualBox, put the .vhd file on the
RAMDisk, installed an OS. Then, something I've always
wanted to try, is "defragmenting a file system which
is sitting on RAM". Well, guess how fast that
ended up doing I/O ? One megabyte/sec. The emulation
of OS behavior was so complete, it *sucked* like a
regular hard drive sucks. In years past, I'd dreamed
of a day where the computer didn't use any slow
storage devices, and it *still* sucked :-( You
should have seen the look on my face.

*******

I saw a reference recently, to a third-party cache
software for the computer. But I don't think I bookmarked
it or anything. If I happen to remember the name of
it, and what hardware it might be tied to, I'll
post back.

Paul

On Thu, 10 Mar 2016 17:23:53 -0500, Paul wrote:

That's a multi-threaded performance test.

Which is suited to testing "servers", not so much
sustained transfer rate that a user may favor.

My guess is, you're back to crafting a "file copy"
test of some sort.

Couldn't he just change his -t parameter to 1 and get a single thread?
I find multiple threads in my Paragon backup break up the 4gig backup
files into several thousand fragments and that is not quick :-(

--
Faster, cheaper, quieter than HS2
and built in 5 years;
UKUltraspeed http://www.500kmh.com/

On Fri, 11 Mar 2016 06:46:42 +0000 (GMT), "Rodney Pont"
wrote:

On Thu, 10 Mar 2016 17:23:53 -0500, Paul wrote:

That's a multi-threaded performance test.

Which is suited to testing "servers", not so much
sustained transfer rate that a user may favor.

My guess is, you're back to crafting a "file copy"
test of some sort.

Couldn't he just change his -t parameter to 1 and get a single thread?
I find multiple threads in my Paragon backup break up the 4gig backup
files into several thousand fragments and that is not quick :-(

Good question! And since the system is 2-way mirror, wouldn't 2
threads be appropriate? Maybe I'll play around with this some more
over the weekend.

On Fri, 11 Mar 2016 07:51:45 -0600, Charlie Hoffpauir wrote:

Good question! And since the system is 2-way mirror, wouldn't 2
threads be appropriate? Maybe I'll play around with this some more
over the weekend.

I wouldn't have thought that 2 threads would be helpful. I expect the
driver to copy the data to the two mirror drives without the
applications help. Results for both would be interesting however.

--
Faster, cheaper, quieter than HS2
and built in 5 years;
UKUltraspeed http://www.500kmh.com/

Charlie Hoffpauir wrote:
When I set up Storage Spaces in my Win 10 system for data, I chose
two-way mirror, using 2 physical drives, because I wanted "resiliency"
(I wanted data redundancy in case of a drive failure). That seems to
work fine. But now that I've learned a bit more about storage spaces,
I'm wondering if maybe "read" performance would be improved if more
than 2 physical drives were used. The thinking is that if the data
were spread across 3 instead of 2, reads might be 50% faster? Is this
the case? Is it by default, or must one configure the storage spaces
to use 3 columns instead of two at initial installation to get the
higher peformance? (I'm not referring to using 3 drives to get a
parity effect..... I don't need that big a hit on write performance)
I'm guessing that if I simply add another drive to my existing setup,
I woudln't see any increased read performance...... is that also
correct? One last question... since the Storage Spaces interface
doesn't seem to allow for specifying the number of columns and since
the articles I've read state that this must be done with powershell,
can anyone tell me the Powershell command that would do this?

I've not mentioned Simple Storage spaces, but the MS docs I've read
states that adding more drives does directly increase read
performance... but I'm not willing to give up the data redundancy.
Also, I'm not really needing 3-way mirror.

I did a test tonight with two disks and four disks.

In 2-way mirror, four disks does not double the bandwidth.
It isn;t RAID 10.

It seems to operate like this.

Disk 0 + Disk 1 --- span
|
Mirror
|
Disk 2 + Disk 3 --- span

On my disks, the write with two disk and with four
disks, operated at the same speed. 135MB/sec or so.

Also, in my testing the "fsutil... setvaliddata" method
didn't work. So I had to resort to an old-fashioned technique
to eliminate the cache. Create an extra-large file on a
storage device not associated with the test, read it,
and make sure the extra-large file is larger than
the system file cache. (I used a 16GB file on an 8GB machine.)
This ensures that all memory of the file you just wrote
to the "Storage Space" NTFS partition is forgotten. Then,
when you copy the file off and do your read test case,
you get a pure hardware speed.

I use a RAMDisk as the second storage device, so it will
have minimal impact on the results.

I don't see a point in testing seek time, as if the seeks
hit in the cache, they might be very fast. And if the
seeks miss in the cache, the seek should be as fast as
the disk. The only thing I didn't attempt to measure,
is whether the first of the two sides of the mirror
to deliver data, gets to deliver it right away or not
(the way a traditional hardware RAID 1 would work).

Anyway, I had fun, and I won't be using Storage Spaces
for any real work. I don't have enough big disks to make
it worthwhile. I was using 4 x 500GB for this test. And
I will be restoring them from backup, to put the original
data back in place.

To delete the Storage Space, you go to Disk Management
first and delete the disk letter in there. Then, when you
use the Storage Spaces interface, do Change Settings and
click Delete, it runs without error. The only thing it
doesn't do, is it doesn't "release" the disks to disk
management, as ordinary disks. So now I have to figure out
how to fix that.

Paul

On Sat, 12 Mar 2016 01:02:52 -0500, Paul wrote:

Charlie Hoffpauir wrote:
When I set up Storage Spaces in my Win 10 system for data, I chose
two-way mirror, using 2 physical drives, because I wanted "resiliency"
(I wanted data redundancy in case of a drive failure). That seems to
work fine. But now that I've learned a bit more about storage spaces,
I'm wondering if maybe "read" performance would be improved if more
than 2 physical drives were used. The thinking is that if the data
were spread across 3 instead of 2, reads might be 50% faster? Is this
the case? Is it by default, or must one configure the storage spaces
to use 3 columns instead of two at initial installation to get the
higher peformance? (I'm not referring to using 3 drives to get a
parity effect..... I don't need that big a hit on write performance)
I'm guessing that if I simply add another drive to my existing setup,
I woudln't see any increased read performance...... is that also
correct? One last question... since the Storage Spaces interface
doesn't seem to allow for specifying the number of columns and since
the articles I've read state that this must be done with powershell,
can anyone tell me the Powershell command that would do this?

I've not mentioned Simple Storage spaces, but the MS docs I've read
states that adding more drives does directly increase read
performance... but I'm not willing to give up the data redundancy.
Also, I'm not really needing 3-way mirror.

I did a test tonight with two disks and four disks.

In 2-way mirror, four disks does not double the bandwidth.
It isn;t RAID 10.

It seems to operate like this.

Disk 0 + Disk 1 --- span
|
Mirror
|
Disk 2 + Disk 3 --- span

On my disks, the write with two disk and with four
disks, operated at the same speed. 135MB/sec or so.

Also, in my testing the "fsutil... setvaliddata" method
didn't work. So I had to resort to an old-fashioned technique
to eliminate the cache. Create an extra-large file on a
storage device not associated with the test, read it,
and make sure the extra-large file is larger than
the system file cache. (I used a 16GB file on an 8GB machine.)
This ensures that all memory of the file you just wrote
to the "Storage Space" NTFS partition is forgotten. Then,
when you copy the file off and do your read test case,
you get a pure hardware speed.

I use a RAMDisk as the second storage device, so it will
have minimal impact on the results.

I don't see a point in testing seek time, as if the seeks
hit in the cache, they might be very fast. And if the
seeks miss in the cache, the seek should be as fast as
the disk. The only thing I didn't attempt to measure,
is whether the first of the two sides of the mirror
to deliver data, gets to deliver it right away or not
(the way a traditional hardware RAID 1 would work).

Anyway, I had fun, and I won't be using Storage Spaces
for any real work. I don't have enough big disks to make
it worthwhile. I was using 4 x 500GB for this test. And
I will be restoring them from backup, to put the original
data back in place.

To delete the Storage Space, you go to Disk Management
first and delete the disk letter in there. Then, when you
use the Storage Spaces interface, do Change Settings and
click Delete, it runs without error. The only thing it
doesn't do, is it doesn't "release" the disks to disk
management, as ordinary disks. So now I have to figure out
how to fix that.

Paul

Many thanks Paul.... that really answers my question.

I'm now only curious why your disks aren't released after you removed
them. When I completed tests, I simply went into Storage Spaces
management and went through the removal steps there..... no
eliminating the letters at all (in fact the letters are still there
because I left two drives in place).

Charlie Hoffpauir wrote:
On Sat, 12 Mar 2016 01:02:52 -0500, Paul wrote:

Charlie Hoffpauir wrote:
When I set up Storage Spaces in my Win 10 system for data, I chose
two-way mirror, using 2 physical drives, because I wanted "resiliency"
(I wanted data redundancy in case of a drive failure). That seems to
work fine. But now that I've learned a bit more about storage spaces,
I'm wondering if maybe "read" performance would be improved if more
than 2 physical drives were used. The thinking is that if the data
were spread across 3 instead of 2, reads might be 50% faster? Is this
the case? Is it by default, or must one configure the storage spaces
to use 3 columns instead of two at initial installation to get the
higher peformance? (I'm not referring to using 3 drives to get a
parity effect..... I don't need that big a hit on write performance)
I'm guessing that if I simply add another drive to my existing setup,
I woudln't see any increased read performance...... is that also
correct? One last question... since the Storage Spaces interface
doesn't seem to allow for specifying the number of columns and since
the articles I've read state that this must be done with powershell,
can anyone tell me the Powershell command that would do this?

I've not mentioned Simple Storage spaces, but the MS docs I've read
states that adding more drives does directly increase read
performance... but I'm not willing to give up the data redundancy.
Also, I'm not really needing 3-way mirror.
I did a test tonight with two disks and four disks.

In 2-way mirror, four disks does not double the bandwidth.
It isn;t RAID 10.

It seems to operate like this.

Disk 0 + Disk 1 --- span
|
Mirror
|
Disk 2 + Disk 3 --- span

On my disks, the write with two disk and with four
disks, operated at the same speed. 135MB/sec or so.

Also, in my testing the "fsutil... setvaliddata" method
didn't work. So I had to resort to an old-fashioned technique
to eliminate the cache. Create an extra-large file on a
storage device not associated with the test, read it,
and make sure the extra-large file is larger than
the system file cache. (I used a 16GB file on an 8GB machine.)
This ensures that all memory of the file you just wrote
to the "Storage Space" NTFS partition is forgotten. Then,
when you copy the file off and do your read test case,
you get a pure hardware speed.

I use a RAMDisk as the second storage device, so it will
have minimal impact on the results.

I don't see a point in testing seek time, as if the seeks
hit in the cache, they might be very fast. And if the
seeks miss in the cache, the seek should be as fast as
the disk. The only thing I didn't attempt to measure,
is whether the first of the two sides of the mirror
to deliver data, gets to deliver it right away or not
(the way a traditional hardware RAID 1 would work).

Anyway, I had fun, and I won't be using Storage Spaces
for any real work. I don't have enough big disks to make
it worthwhile. I was using 4 x 500GB for this test. And
I will be restoring them from backup, to put the original
data back in place.

To delete the Storage Space, you go to Disk Management
first and delete the disk letter in there. Then, when you
use the Storage Spaces interface, do Change Settings and
click Delete, it runs without error. The only thing it
doesn't do, is it doesn't "release" the disks to disk
management, as ordinary disks. So now I have to figure out
how to fix that.

Paul

Many thanks Paul.... that really answers my question.

I'm now only curious why your disks aren't released after you removed
them. When I completed tests, I simply went into Storage Spaces
management and went through the removal steps there..... no
eliminating the letters at all (in fact the letters are still there
because I left two drives in place).

I used PTEDIT32 to fix them. It's no longer offered for
download, so you'd already need to have a copy.

The partition on a Storage Spaces disk is marked "0xEE",
which as far as I know, is the "GPT marker". I changed
the partition type to "0x00", rebooted, and Disk Management
then began to show the pool disks, as ordinary disks. So
it wasn't a big deal to fix.

Initially, I wanted to use the "official" way, use
Microsoft DiskPart. But the physical disks in question,
would not show up in "list disks", so I couldn't swat at
them from there.

But an actual partition table editor, made the job easy.

After the reboot and review in Disk Management, I could
then use the disks again for other purposes.

As of this moment, all the disks have their original
content on them again. So my house-cleaning process
is done.

And now I know, if I need to do any RAID testing
some day, I *do* have the materials to do it.
Just takes a little backup and restore to free
up the resources needed.

Paul