A computer components & hardware forum. HardwareBanter

If this is your first visit, be sure to check out the FAQ by clicking the link above. You may have to register before you can post: click the register link above to proceed. To start viewing messages, select the forum that you want to visit from the selection below.

Go Back   Home » HardwareBanter forum » General Hardware & Peripherals » Homebuilt PC's
Site Map Home Register Authors List Search Today's Posts Mark Forums Read Web Partners

Copieing 60.000 items from Windows Live Mail extreme performance degradation



 
 
Thread Tools Display Modes
  #11  
Old December 5th 18, 07:01 PM posted to alt.comp.hardware.pc-homebuilt
[email protected]
external usenet poster
 
Posts: 533
Default Copieing 60.000 items from Windows Live Mail extreme performance degradation

OPERATION SUCCCESSFULL.

MAIL ARCHIVE TRANSITIONED TO NEW DRIVE W: =D

All is working again.

The shutdown trick of P drive and terminate and restart windows live mail worked... no need to further waste time on it's stupid recovery bull****

Pfew...

Options-mail-advanced-maintenance-storage folder

This tiny little program called Windows Live Mail... still has some fighting power inside of it ! =D

Now it can enjoy another 4 GIGAWATTS/BYTES OF DARTH VADER FIRE POWER YEAH =D LOL.

This calls for a tie defender = ppzzzzzzzwieee..

Ok and now I really have to go read my newly retrieved e-mails =D

Bye for now,
Skyvader over and out ! =D
  #12  
Old December 5th 18, 07:53 PM posted to alt.comp.hardware.pc-homebuilt
Paul[_28_]
external usenet poster
 
Posts: 1,467
Default Copieing 60.000 items from Windows Live Mail extreme performancedegradation

wrote:
Disabling the mouse is done by clicking on the title bar of cmd.exe window and going to properties and disabling "quick edit mode".

Quick edit mode is insane.

Anyway I will try the following command:

robocopy w: r: /MIR /ETA

This is the last damn thing I am going to try and if I don't like it, this will be the end of it, though I did already noticed "lagging in sound playback" so this could indicate it's somewhat faster perhaps.

Here goes:

Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation. All rights reserved.

C:\Users\Skybuckr:

R:\dir
Volume in drive R is TestTestTest
Volume Serial Number is D4F6-722F

Directory of R:\

File Not Found

R:\w:

W:\dir
Volume in drive W is Windows Live Mail
Volume Serial Number is 103F-69D4

Directory of W:\

05/12/2018 17:23 DIR Windows Live Mail
0 File(s) 0 bytes
1 Dir(s) 4,536,350,720 bytes free

W:\robocopy w: r: /MIR /ETA


It's a folder copying tool, and I was hoping you'd just
copy a folder, not a whole hard drive partition.

To copy the C: drive, requires at least one other special
parameter to avoid problems with Junction Points.

To measure drive performance, on an older OS...

Start : Run

perfmon.msc

In there, right click to add new performance counters.
Add the "Physical Disk" one, which has "Disk write bytes per second"
or similar. As well as "Disk read bytes per second".

The perfmon.msc display will show the transfer rate.

On Win8/Win10, the Task Manager performance display, and
the particular disk, you click on that and a transfer
graph is shown.

On Win7/Win8/Win10, there is Resource Monitor, a button
you click from the Performance screen of Task Manager.
it shows transfer rate versus "program", such as robocopy.

There are plenty of ways to monitor transfers, without
resorting to "/ETA".

Robocopy is capable of async transfer, where reads and writes
can overlap on two drives. It's unlike a conventional
copy where "read" then "write" happens, as the tool alternates
between drives for each phase. The extent to which overlap
happens on the async (non-blocking) transfer, has varied
a lot over the years. But with some luck, you might see
it overlapping the reads and writes (on two different hard
drives). If you transfer files between two partitions
on the same hard drive, the operation will be serialized
by the hardware itself (the need to move the heads back
and forth between partitions).

Paul
  #14  
Old December 6th 18, 04:01 PM posted to alt.comp.hardware.pc-homebuilt
[email protected]
external usenet poster
 
Posts: 533
Default Copieing 60.000 items from Windows Live Mail extreme performance degradation

On Wednesday, December 5, 2018 at 7:53:49 PM UTC+1, Paul wrote:
wrote:
Disabling the mouse is done by clicking on the title bar of cmd.exe window and going to properties and disabling "quick edit mode".

Quick edit mode is insane.

Anyway I will try the following command:

robocopy w: r: /MIR /ETA

This is the last damn thing I am going to try and if I don't like it, this will be the end of it, though I did already noticed "lagging in sound playback" so this could indicate it's somewhat faster perhaps.

Here goes:

Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation. All rights reserved.

C:\Users\Skybuckr:

R:\dir
Volume in drive R is TestTestTest
Volume Serial Number is D4F6-722F

Directory of R:\

File Not Found

R:\w:

W:\dir
Volume in drive W is Windows Live Mail
Volume Serial Number is 103F-69D4

Directory of W:\

05/12/2018 17:23 DIR Windows Live Mail
0 File(s) 0 bytes
1 Dir(s) 4,536,350,720 bytes free

W:\robocopy w: r: /MIR /ETA


It's a folder copying tool, and I was hoping you'd just
copy a folder, not a whole hard drive partition.

To copy the C: drive, requires at least one other special
parameter to avoid problems with Junction Points.

To measure drive performance, on an older OS...

Start : Run

perfmon.msc

In there, right click to add new performance counters.
Add the "Physical Disk" one, which has "Disk write bytes per second"
or similar. As well as "Disk read bytes per second".

The perfmon.msc display will show the transfer rate.

On Win8/Win10, the Task Manager performance display, and
the particular disk, you click on that and a transfer
graph is shown.

On Win7/Win8/Win10, there is Resource Monitor, a button
you click from the Performance screen of Task Manager.
it shows transfer rate versus "program", such as robocopy.

There are plenty of ways to monitor transfers, without
resorting to "/ETA".

Robocopy is capable of async transfer, where reads and writes
can overlap on two drives. It's unlike a conventional
copy where "read" then "write" happens, as the tool alternates
between drives for each phase. The extent to which overlap
happens on the async (non-blocking) transfer, has varied
a lot over the years. But with some luck, you might see
it overlapping the reads and writes (on two different hard
drives). If you transfer files between two partitions
on the same hard drive, the operation will be serialized
by the hardware itself (the need to move the heads back
and forth between partitions).

Paul


I tried this, it does work... it's a bit cumbersome to setup... especially the scale is somewhat weird.

I setup it up into "report mode" for the graph...

I will have to do more experimentation with seconds for measuring the average and such... it was set to 100.

This is a bit long... 10 or 15 secs might be more appriorate.

The robocopy copy speed fluctuates a lot between reading and writing.

Eventually it did go way down to just 1.7 megabytes/sec.

So this more or less does prove NTFS has a problem as far as I am concerned.

A new file system could be designed which stores many small files into 1 single file instead of many to reduce seeks on disk, and instead seek in memory.

Anyway the VCL media player wasn't even playing this time.

I am not convinced that robocopy is any faster, it may actually be slower though, though the console i/o test might have something to do with that though it didn't seem to be too bad.

Overall not much difference speed wise... maybe at the start though... but those were large files... maybe for larger files robocopy might be a bit faster, not sure about that.

It's interesting to see with what file it's busy though... though for long folders the text display becomes a bit short.

Also now estimated time left which is a big short coming in robocopy + perfmon

But thanks for your tips, they were a bit interesting ! =D

Especially the perfmon... it does show "total disk performance"... something which resource monitor struggles with somewhat... I think... not sure... there is total i/o... but I am not sure it's seperated into read vs write speeds for totals.

Bye,
Skybuck =D
  #15  
Old December 6th 18, 06:24 PM posted to alt.comp.hardware.pc-homebuilt
Paul[_28_]
external usenet poster
 
Posts: 1,467
Default Copieing 60.000 items from Windows Live Mail extreme performancedegradation

wrote:

But thanks for your tips, they were a bit interesting ! =D

Especially the perfmon... it does show "total disk performance"...
something which resource monitor struggles with somewhat...
I think... not sure... there is total i/o... but I am not sure
it's seperated into read vs write speeds for totals.

Bye,
Skybuck =D


The perfmon.msc has separate graphs for read and write,
if you select the appropriate "physical disk" entries.

Once you find a "disk write bytes per second" type entry, look
down as it has additional selectors such as "Total I/O",
"Drive X", "Drive Y" and so on. I usually leave mine
set to "Total" as I'm only doing single source, single
destination experiments, so "Total" still tells me about
a particular drive.

The graph scale is a bit baffling. I always modify that,
and make it "20000" for modern hard drives, as they
top out at 200MB/sec or so. The value "40000" might be
good for a cheap SATA SSD for a full scale (400MB/sec
on some cheap drives). For example, the Intel 545S I
just got, has a read at 383MB/sec. The Samsung gives
around 440MB/sec on a good day.

*******

I would have expected trouble with NTFS eventually.
You can safely do around one million files in a
single folder. I have seen bugs in File Explorer,
at just the 60,000 file level (File Explorer goes into
a loop and doesn't come back).

You can cheat, and use a backup utility that sequentially
copies the clusters while doing a partition backup. That
will maintain the speed, because the head movement pattern
is quite different.

If a file is fragmented, the transfer slows right down.
For example, this is one of my old SATA scratch drives.
Normally it reads at 100MB/sec near the beginning.

In these pictures, a 20GB file has been fragmented on purpose
with a utility intended for the job. The transfer rate when
reading the file, drops to 8MB/sec. The tool cannot set
a high enough fragmentation level to hit 1MB/sec (which
I've seen before in practice). The fragmentation picture
in this example, has a 20GB frag.bin file and a 10GB zero.bin
file. The zero.bin is not fragmented (only 3 fragments in it).
The frag.bin has 100,000 fragments. I do a read of the zero.bin
10GB file, to flush the system read cache (so there won't
be any cheating during the benchmark run). Then, I read
the frag.bin file and see how fast it'll go.

https://i.postimg.cc/zG5tz1sz/hard-d...ss-fragged.gif

I created the 20GB file first, then applied fragmentation to
it with this tool. It you set the slider for too-small fragments,
the process of creating fragmentation is dog-slow. You have
to be *really* patient to use this tool, and once you
know it's running OK, just walk away...

https://www.passmark.com/products/fragger.htm

I prepare "thrashing" test cases on my RAMdisk and
then clone over to a real hard drive, when I know
the thrashing during preparation, would be hard on the
movements of the disk arm. Using the fragger program
on the RAMdisk is *still* slow. Dog-slow. 300KB/sec slow.
One CPU core railed. But at least with that tool, I'm
getting better quality fragmentation than with a
little C program I wrote. If you want good
fragmentation, you really need to use the Microsoft API
for moving clusters around.

https://docs.microsoft.com/en-us/win...gmenting-files

The API was intended to help people write defragmenter
programs, but in the Passmark case, they use it to
make files fragmented again. The purpose of doing that,
is for preparing disks which you intend to benchmark
with commercial defragmenter programs.

Paul

 




Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Ms-Dos and Windows 7 have a major weakness related to date/times when copieing files. Skybuck Flying[_4_] Homebuilt PC's 12 November 15th 15 06:51 PM
Mcafee + windows mail species8350 Dell Computers 0 April 14th 07 05:28 PM
performance degradation backing up small files alan Storage & Hardrives 2 April 27th 04 05:47 AM
Found culprit in performance degradation... Carl Fenley Nvidia Videocards 1 February 29th 04 09:44 PM
'Swen-mail' and the elapsed time between a Usenet newsgroup post with a valid e-mail addres and the arrival of the first infected message in the mail box Phil Weldon Overclocking 12 October 7th 03 08:14 AM


All times are GMT +1. The time now is 11:17 AM.


Powered by vBulletin® Version 3.6.4
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright ©2004-2024 HardwareBanter.
The comments are property of their posters.