Yousuf Khan wrote:
On 1/10/2018 1:08 AM, Yousuf Khan wrote:
On 1/9/2018 2:15 AM, Paul wrote:
To establish baseline performance, you can use Robocopy to copy the
files to a RAMdisk. That should expose any source HDD performance
anomalies.
And in the "not a fair fight" category, you can also try 7ZIP
and use "Store Mode", which does not compress a file tree when
creating an archive on another storage device. That should
return a much better result than your Macrium result.
Hmm, those are interesting ideas.
Okay, so I took your advice, and ran a Robocopy to a RAMdisk to see how
fast that goes. It still took over 2.5 hours just to copy to RAMdisk!
Robocopy's own output showed:
Total Copied Skipped Mismatch FAILED Extras
Dirs : 21 20 1 0 0 0
Files : 652802 652802 0 0 0 0
Bytes : 1.580 g 1.580 g 0 0 0 0
Times : 2:38:31 2:35:58 0:00:00 0:02:32
Speed : 181312 Bytes/sec.
Speed : 10.374 MegaBytes/min.
Strange observations: The total space of the files is 1.6 GB, while the
total allocated space is 3.2 GB. This makes sense, in that NTFS blocks
are probably not being fully utilized from file to file. However, the
strange thing I ran into on the RAMdisk is that I constantly kept
running out of space, even though I allocated more than 3.2 GB
everytime. I started out allocating 3.5 GB, then 5 GB, and then finally
8 GB, and then it finally worked! Right now, my 8 GB RAMdisk is showing
Used space of 5.9 GB, while Free space is 2.1 GB. So if the files are
only taking up 3.2 GB, then 2.7 GB of space is being taken up by
something else? NTFS metadata maybe?
Then I tried an archiving operation from the RAMdisk back to the HDD,
using the zip-store method, and that took 51:49 minutes. I assume it
will take Macrium about as long to archive off of the RAMdisk as a zip
archiver would, so I could conceivably save a ton of time just by
copying the folder to a RAMdisk and then archiving off of the RAMdisk.
But then I'd need to keep an 8GB RAMdisk in memory for those few times
that I need to archive this folder?!?
Yousuf Khan
Did you use a tool to check which, if any, of the source files have
alternate data streams (ADS)? The size of those alternate streams are
not reported in normal tools.
When you download a file from a web server, an ADS is added to the file
to flag the "zone" for that file was the Internet. That's why you can
get some security issues regarding files you downloaded versus files you
copied from storage media. The 26-byte ADS is named Zone.Identifier.
The ZoneID attribute just has a numeric value (3 = Internet).
An easy way to strip any ADS from files is to copy the files to a FAT
drive. ADS is only supported in NTFS, not in FAT. There are other
tools or tricks to strip ADS(es) from a file but that's the one that
comes to mind right now.
I've used Stream Armor to find files that have one, or more, ADS
attached to them; see
http://securityxploded.com/streamarmor.php. It
knows some ADS types: favicon, Zone.Identifier, and text file (although
the content of the stream could be huge). If multiple ADSes, especially
unknown types, are attached to a file, it flags them as suspicious.
For example, I could send you a copy of notepad.exe which is 189KB in
size but it could have an ADS attached to it that is gigabytes in size.
Of course, however you get the file means the ADS has to be preserved.
https://www.owasp.org/index.php/Wind...te_data_stream