Why is this folder so slow?

On 2020-05-01 00:24, Yousuf Khan wrote:
On 4/27/2020 6:15 PM, T wrote:
Hi Yousuf,

When I see things like this, it is usually a failing
drive, especially when the index on teh offending
directory never finishes.

This will show up like a soar thumb if yo run your
drive through gsmartcontrol: check the error logs and
run the self tests

Brand new drive, less than a month old, hasn't had a chance to get old yet.

Â*Â*Â*Â*Yousuf Khan

means nothing. test it

Yousuf Khan wrote:
On 5/1/2020 1:52 PM, Frank Slootweg wrote:
I can - sort of - understand that, but because these 580,000 were
giving you so much hardship, I would have expected you to look at a
few of them, see that they were just News articles and take it from
there, i.e. set/lower the News retention settings in Thunderbird.

No, I knew those were the message files, considering that there were so
many of them, what else could they have been? But often there are other
files interspersed among them, that can often go overlooked because it's
overwhelmed by the mass of all of the main files. Just let the backup
software handle backing all of it up.

Yes, but "Just let the backup software handle backing all of it up."
was the *problem* which made you start this thread! You can't have it
both ways, it either was a problem/annoyance/whatever, or it wasn't!

As to "But often there are other files interspersed among them, that
can often go overlooked because it's overwhelmed by the mass of all of
the main files.", I didn't say to bluntly clobber all the *files*/
*folders*, but to set/lower the News retention settings in Thunderbird.
I.e. let *Thunderbird* do it *safely*, instead of you doing it
(possibly) unsafely.

You *do* know how to set/lower the News retention settings in
Thunderbird, don't you!? (You snipped my other comments about that, so I
don't know if you've set/lowered them now.)

On 5/2/2020 9:45 AM, Frank Slootweg wrote:
Yes, but "Just let the backup software handle backing all of it up."
was the*problem* which made you start this thread! You can't have it
both ways, it either was a problem/annoyance/whatever, or it wasn't!

Have what both ways?? I wanted the data backed up, and it was doing
that, but slowly. That's what the problem was that I wanted fixed.

As to "But often there are other files interspersed among them, that
can often go overlooked because it's overwhelmed by the mass of all of
the main files.", I didn't say to bluntly clobber all the*files*/
*folders*, but to set/lower the News retention settings in Thunderbird.
I.e. let*Thunderbird* do it*safely*, instead of you doing it
(possibly) unsafely.

You*do* know how to set/lower the News retention settings in
Thunderbird, don't you!? (You snipped my other comments about that, so I
don't know if you've set/lowered them now.)

Hard to say what happened in the deep past. At some point perhaps
Thunderbird was taking so long to delete old messages, so it may have
been locking up or fail-exiting constantly, especially at a time when
this may have been running on slow HDD's rather than SSD's. So a bandaid
solution may have come up to prevent it from doing any further
deletions, which got implemented, it stabilized the system, and then
forgotten about. The problem with having run with something for so many
decades, previous problems are not even in memory anymore. I don't even
know if this is what happened, that's just my guess at this point.

Yousuf Khan

On 4/26/2020 9:24 PM, Yousuf Khan wrote:
I have a folder on one of my SSD drives that takes 8 to 10 hours to back
up. It is only about 1.4 GB, but it is allocated 2.4 GB of space
altogether, and there are 580,000 files here. Indicates that per file
it's using up a little bit over half of a cluster on average. File
system is NTFS.

Meanwhile, this same drive can backup the remainder of the drive in
under 2 hours, and the remainder of the drive is 390 GB! Is NTFS this
inefficient for small files like this?

Â*Â*Â*Â*Yousuf Khan

Okay, so after fixing the problem with my News folder, I kept
researching what these millions of little files were, that were clogging
up my News folder. The files had an extension of WDSEML. Later I found
out that these same files are also there in Email folders, hundreds of
thousands of them too.

Initially, I thought that these must be the bodies of the messages that
Thunderbird uses to store emails and newsgroup messages. But after a bit
of research, I found out that Thunderbird itself has no use for these
files. Thunderbird does generate them, but it doesn't use them itself.
Instead it is generated only for the benefit of Windows' Search and
Indexing application. Windows Search uses it to be able to let you
search messages through the Windows Search box. So once Thunderbird
generates these files for Windows Search, it no longer has any use for
them anymore, as it stores its own internal data in a different set of
files. In fact, these WDSEML files are saved copies of individual
messages out of Thunderbird's own database. So Thunderbird maintains it
own database, but it never cleans up these copies ever in its life.
WDSEML means "Windows Desktop Search Email", in fact. I also think this
is only a specific problem with Thunderbird under Windows, it probably
isn't an issue in Thunderbird under other OS'es like Linux.

You can easily delete all of these messages, but of course Thunderbird
will regenerate them again as they come in. So what you have to do is
tell Thunderbird not to generate these files for Windows anymore. You go
into Thunderbird's options menu and turn it off (Tools → Options, then
select Advanced → General → System Integration → Allow Windows search to
search messages).

https://fileinfo.com/extension/wdseml

You can also delete them more easily by searching for and deleting just
the folders in which they reside, rather than the individual files.
These folders have an extension called *.MOZMSGS.

Yousuf Khan

Yousuf Khan wrote:
On 4/26/2020 9:24 PM, Yousuf Khan wrote:
I have a folder on one of my SSD drives that takes 8 to 10 hours to
back up. It is only about 1.4 GB, but it is allocated 2.4 GB of space
altogether, and there are 580,000 files here. Indicates that per file
it's using up a little bit over half of a cluster on average. File
system is NTFS.

Meanwhile, this same drive can backup the remainder of the drive in
under 2 hours, and the remainder of the drive is 390 GB! Is NTFS this
inefficient for small files like this?

Yousuf Khan

Okay, so after fixing the problem with my News folder, I kept
researching what these millions of little files were, that were clogging
up my News folder. The files had an extension of WDSEML. Later I found
out that these same files are also there in Email folders, hundreds of
thousands of them too.

Initially, I thought that these must be the bodies of the messages that
Thunderbird uses to store emails and newsgroup messages. But after a bit
of research, I found out that Thunderbird itself has no use for these
files. Thunderbird does generate them, but it doesn't use them itself.
Instead it is generated only for the benefit of Windows' Search and
Indexing application. Windows Search uses it to be able to let you
search messages through the Windows Search box. So once Thunderbird
generates these files for Windows Search, it no longer has any use for
them anymore, as it stores its own internal data in a different set of
files. In fact, these WDSEML files are saved copies of individual
messages out of Thunderbird's own database. So Thunderbird maintains it
own database, but it never cleans up these copies ever in its life.
WDSEML means "Windows Desktop Search Email", in fact. I also think this
is only a specific problem with Thunderbird under Windows, it probably
isn't an issue in Thunderbird under other OS'es like Linux.

You can easily delete all of these messages, but of course Thunderbird
will regenerate them again as they come in. So what you have to do is
tell Thunderbird not to generate these files for Windows anymore. You go
into Thunderbird's options menu and turn it off (Tools → Options, then
select Advanced → General → System Integration → Allow Windows search to
search messages).

https://fileinfo.com/extension/wdseml

You can also delete them more easily by searching for and deleting just
the folders in which they reside, rather than the individual files.
These folders have an extension called *.MOZMSGS.

Yousuf Khan

In the business that would be called a "lazy implementation".

All they would have to do, is write a "search provider" and Windows
could use that to pump the files in an OLE fashion. It could have
been done by making no temporary files at all (flow from MORK file
or MBOX or whatever, right into the Windows.edb, in terms of writes).

But that would also put too much Windows-ecosystem code into
the tool, which is a no-no in cross platform tool design. You
have to keep your "philosophical purity" at all costs. Which means
using OpenGL for graphics (cross platform), instead of DirectX and X11
as separate platform interfaces.

I guess there's some benefit to federated search that includes
your email, but to my way of thinking this would only clutter up
a search result later.

Then you'd find yourself typing this in the File Explorer search box:

file:mytaxes.xlsx

instead of

mytaxes

because in the latter one, 500K of your emails are
going to get searched too. Using the file: keyword
would help staunch the mess inside the federated
database. The second search, the results would likely
scroll off the screen, obscuring the thing you really
wanted.

You might also discover the Windows.edb file is bloated
beyond recognition, because of that file set. It might
range around 1GB for a vanilla install, but after that
Thunderbird thing got indexed, it would likely double
at the very least.

You can rebuild the Windows.edb index file, using
the Indexing Options control panel in Windows 10.
I would give that a whirl after the TB folder has
had all the cruft removed. It'll take about three
hours to index the regular C: files (but this assumes
you've customized the searched folders to include
most of C: , versus the very shallow folder set used
by default).

Even finding Windows.edb is hard :-) The File Explorer
search won't allow you to find it. You'll need Agent
Ransack or Everything.exe to find that file, just so
you can see the current size, and decide whether it
needs a rebuild or not.

Aren't computers wonderful ? Such labor saving. "It
slices, it dices, it makes Julienne Fries." I don't
think I've ever made Julienne Fries, but I bet
Windows 10 has done all the pre-work for that,
over and over and over again...

Paul

Yousuf Khan wrote:

Okay, so after fixing the problem with my News folder, I kept
researching what these millions of little files were, that were
clogging up my News folder. The files had an extension of WDSEML.
Later I found out that these same files are also there in Email
folders, hundreds of thousands of them too.

Initially, I thought that these must be the bodies of the messages
that Thunderbird uses to store emails and newsgroup messages. But
after a bit of research, I found out that Thunderbird itself has no
use for these files. Thunderbird does generate them, but it doesn't
use them itself. Instead it is generated only for the benefit of
Windows' Search and Indexing application. Windows Search uses it to
be able to let you search messages through the Windows Search box. So
once Thunderbird generates these files for Windows Search, it no
longer has any use for them anymore, as it stores its own internal
data in a different set of files. In fact, these WDSEML files are
saved copies of individual messages out of Thunderbird's own
database. So Thunderbird maintains it own database, but it never
cleans up these copies ever in its life. WDSEML means "Windows
Desktop Search Email", in fact. I also think this is only a specific
problem with Thunderbird under Windows, it probably isn't an issue in
Thunderbird under other OS'es like Linux.

You can easily delete all of these messages, but of course
Thunderbird will regenerate them again as they come in. So what you
have to do is tell Thunderbird not to generate these files for
Windows anymore. You go into Thunderbird's options menu and turn it
off (Tools → Options, then select Advanced → General → System
Integration → Allow Windows search to search messages).

https://fileinfo.com/extension/wdseml

You can also delete them more easily by searching for and deleting
just the folders in which they reside, rather than the individual
files. These folders have an extension called *.MOZMSGS.

Interesting find. I don't remember looking at this option when I
previously trialed Thunderbird. Is this option enabled by default? If
so, a very bad choice my Mozilla.

If I had not known about this option (and I was still using
Thunderbird), and after finding the superfluous and unwanted wdseml
files (since I do *not* want Windows search looking into my e-mails to
confuse those hits with those of files where I want to find by name or
content), I probably would've added them to the Include option in
CCleaner which I sometimes run manually but is also a daily scheduled
event in Task Scheduler to run before the daily backup. I have other
programs that leave **** behind that I want purged, so I go into
CCleaner's options, Include section, and define a template of what to
include in CCleaner's cleanup. Some programs, for example, will save
files for a 'resume' function, like a downloader, to continue the
operation when I next load the program. Nope, if I killed/exited the
program then I do not want it wasting time when I next load the program.
I don't even let my web browsers resume a prior session, and configure
them to purge all local data upon their exit.

Back when I used MS Outlook, it was configured by default to allow
Windows Search to look inside my e-mails. No thanks. I disabled that.
If I want to search my e-mails, I'll do that search from inside the
e-mail program. I don't want e-mails mixed in with other file results
in a global search. As I recall, Outlook's search would bitch with an
info insert at the top of the search results that I had Windows Search
disabled for Outlook, but that's exactly how I wanted it to work.

In addition, I used auto-archiving in Outlook not only to move old
e-mails into an archive store, but also to expire and delete very old
e-mails. When they get over 5 years old, I don't need them anymore. I
had auto-archive move messages older than 1 year into the archive, and
had auto-archive delete messages older than 5 years in the archive.
Actually I chained archives together for different expirations: archive
messages older than 1 year into archive1year, archive messages older
than 2 years from archive1year into archive2year, and so on. Eventually
I decided I didn't need that level of granularity for storing old
messages, and just went with a single archive for anything older than a
year but purged anything older than 5 years from the archive. I
certainly would not want those old and deleted e-mails still lingering
in a search database or, in your case with Thunderbird, lingering around
in wdseml files.

To me, having Windows Search dig around inside everything is for those
boobs that are slobs. They haven't a clue how to organize their data,
or are too lazy to do it. They pile thousands of e-mails into the Inbox
folder instead of organize the old e-mails into separate pending or
archive folders, and God forbid they delete old e-mails. They'll pile
thousands of image files into a single folder instead of use folders to
organize them. Foldering is an organizational feature that some users
just seem incapable or unwilling to use. As disorganized is their data
is probably the same for how disorderly is their home.

I'm a bit surprized Mozilla, in developing a cross-platform product,
whould give a gnat's fart about kowtowing to Microsoft's search feature
in Windows. Hell, Mozilla doesn't even use the global certificate store
in Windows (use certmgr.msc to see) within Firefox, and instead uses a
private cert store inside of Firefox (and why some programs have to do a
double cert install: once into the Windows global cert store and again
into Firefox's private cert store). If users are going to search their
old e-mails, why would they not do that from inside of Thunderbird?
They're searching on e-mails, not on some pic they stored from their
camera or a copy of their tax form. Overreaching got even worse in
Windows 10 with Cortana (which I disabled). All this forensic-like
searching to cater to data slobs.

On 5/11/2020 4:23 PM, VanguardLH wrote:
Interesting find. I don't remember looking at this option when I
previously trialed Thunderbird. Is this option enabled by default? If
so, a very bad choice my Mozilla.

I have no idea, when it came in, nor when it became an option. That's
the problem with using a program for so many years and decades, you stop
looking at its configuration, and accept it doing things by default.

One thing I did find out about this option is that it can be set in two
separate places within the Thunderbird options menu, and that they are
not synchronized with each other, for some reason! In one subsection of
the options, it was shown as not selected, but in a different submenu it
shows up again, and it was selected! So I just unselected in both
places, I don't have time to figure out what the differences are, or why
it's in two places. I just hope it's not in 3 places! I assume that
there was some kind of a redesign of the options interface, and so
somebody decided to move the location of this option, but may have
forgotten to remove it from the original place. This may be one of the
problems you run into due to this being an amateur collaborative design
effort, and there's not a unified design goal. I won't say this is only
a problem with amateur projects, Microsoft itself does plenty of these
things, you have to relearn Office or Windows everytime you upgrade it.

If I had not known about this option (and I was still using
Thunderbird), and after finding the superfluous and unwanted wdseml
files (since I do*not* want Windows search looking into my e-mails to
confuse those hits with those of files where I want to find by name or
content), I probably would've added them to the Include option in
CCleaner which I sometimes run manually but is also a daily scheduled
event in Task Scheduler to run before the daily backup. I have other
programs that leave **** behind that I want purged, so I go into
CCleaner's options, Include section, and define a template of what to
include in CCleaner's cleanup. Some programs, for example, will save
files for a 'resume' function, like a downloader, to continue the
operation when I next load the program. Nope, if I killed/exited the
program then I do not want it wasting time when I next load the program.
I don't even let my web browsers resume a prior session, and configure
them to purge all local data upon their exit.

That's interesting, I did not realize that CCleaner can be custom
configured to get rid of whatever files you like?

But actually regarding getting rid of these files themselves. For years
I was fooled into thinking that they were actually important files that
Thunderbird uses. You take one of these files and open it in a text
editor, and you see right away that it looks like an email or a
newsgroup message, so you easily think that this is how those messages
are stored in Thunderbird. So I didn't dare to delete them.

To me, having Windows Search dig around inside everything is for those
boobs that are slobs. They haven't a clue how to organize their data,
or are too lazy to do it. They pile thousands of e-mails into the Inbox
folder instead of organize the old e-mails into separate pending or
archive folders, and God forbid they delete old e-mails. They'll pile
thousands of image files into a single folder instead of use folders to
organize them. Foldering is an organizational feature that some users
just seem incapable or unwilling to use. As disorganized is their data
is probably the same for how disorderly is their home.

Microsoft does a ton of intrusive or esoteric things that it thinks are
stuff users want, but nobody does, and nobody ends up using them in the
end. Then Microsoft removes them, much to the consternation of the
couple of tenths of a percent of people who did use them and found them
useful. For example, goodbye Homegroups in Windows networking, or the
ignoring of folder Libraries nowadays. Both of these were features that
came in with Windows Vista or 7, and are going away already. I liked
both of those features, and it annoys me that they are going away.

I'm a bit surprized Mozilla, in developing a cross-platform product,
whould give a gnat's fart about kowtowing to Microsoft's search feature
in Windows. Hell, Mozilla doesn't even use the global certificate store
in Windows (use certmgr.msc to see) within Firefox, and instead uses a
private cert store inside of Firefox (and why some programs have to do a
double cert install: once into the Windows global cert store and again
into Firefox's private cert store). If users are going to search their
old e-mails, why would they not do that from inside of Thunderbird?
They're searching on e-mails, not on some pic they stored from their
camera or a copy of their tax form. Overreaching got even worse in
Windows 10 with Cortana (which I disabled). All this forensic-like
searching to cater to data slobs.

Well, as Paul mentioned, Mozilla actually didn't really do much to
integrate it into Thunderbird. All they did was create tons of little
text files which Windows Search can then look through, instead of
directly integrating the Windows search API so that Windows can look
directly into the Thunderbird database itself. Lazy programming. Maybe
it was their attempt at competing against Microsoft Outlook, which did
integrate the Windows search API, so that it can say that Windows search
also works across Thunderbird?

In the meantime, I was sitting here completely unaware of all of these
useless features that were clogging up my backups. Just as I was typing
this, one of my backups started in the background and it is already
finished, in about a half-hour. Previously this backup used to take
between 1.5 to 2.5 hours! In fact, I can probably reintegrate the News
backups into my main scheduled backup again, which I had to separate out
years ago, due to how long it was taking. If I had done nothing, I would
have eventually had to remove my Emails folder from the main backup too,
because that folder was starting to buildup with this crud too.

Yousuf Khan

On 5/11/2020 1:54 PM, Paul wrote:
In the business that would be called a "lazy implementation".

Absolutely, when I used to program, this is the sort of hacky
programming I'd see other programmers implementing, and it would be my
job to clean this up. I did not know that Mozilla would be doing exactly
the same sort of thing as you'd see a local office programmer doing.

All they would have to do, is write a "search provider" and Windows
could use that to pump the files in an OLE fashion. It could have
been done by making no temporary files at all (flow from MORK file
or MBOX or whatever, right into the Windows.edb, in terms of writes).

Yeah, I don't blame them for not getting too integrated into the Windows
ecosystem. If it's something that all operating systems provide in some
form or another, then it can be generalized through a standard
C-library, and they wouldn't have to do special implementations for each
OS. Are there similar functions available in other competing OSes, like
the Microsoft OLE?

I guess there's some benefit to federated search that includes
your email, but to my way of thinking this would only clutter up
a search result later.

I'm betting that they created this hacky implementation, just to say
that they can integrate into Windows Search, just like their competitor
Microsoft Outlook does. The WDSEML file is nothing more than their
standard EML file, which is their way of exporting individual messages
to a text format that you can transfer around easily.

You might also discover the Windows.edb file is bloated
beyond recognition, because of that file set. It might
range around 1GB for a vanilla install, but after that
Thunderbird thing got indexed, it would likely double
at the very least.

Yeah, I often see Windows.edb getting hammered when looking through the
Windows Resource Monitor app.

You can rebuild the Windows.edb index file, using
the Indexing Options control panel in Windows 10.
I would give that a whirl after the TB folder has
had all the cruft removed. It'll take about three
hours to index the regular C: files (but this assumes
you've customized the searched folders to include
most of C: , versus the very shallow folder set used
by default).

No, I think I'll look into optimizing Windows.edb sometime in the
distant future, at this point in time, I'm done with optimizing my
filesystem. Until another problem arises. :-)

Even finding Windows.edb is hard :-) The File Explorer
search won't allow you to find it. You'll need Agent
Ransack or Everything.exe to find that file, just so
you can see the current size, and decide whether it
needs a rebuild or not.

Well, even Agent Ransack didn't see it, but I assume I'll need to run it
as admin.

The Windows Search is so braindead. You can tell it to search for
something like "data", but it won't find words like "database" even
though data is in the name. You'd have to directly search for "database"
to find anything in the Windows search, no wonder people keep using
Agent Ransack.

Yousuf Khan

VanguardLH wrote:
Yousuf Khan wrote:
[...]

[About .wdseml (Windows Desktop Search Email) files/messages:]

You can easily delete all of these messages, but of course
Thunderbird will regenerate them again as they come in. So what you
have to do is tell Thunderbird not to generate these files for
Windows anymore. You go into Thunderbird's options menu and turn it
off (Tools ? Options, then select Advanced ? General ? System
Integration ? Allow Windows search to search messages).

https://fileinfo.com/extension/wdseml

You can also delete them more easily by searching for and deleting
just the folders in which they reside, rather than the individual
files. These folders have an extension called *.MOZMSGS.

Interesting find. I don't remember looking at this option when I
previously trialed Thunderbird. Is this option enabled by default? If
so, a very bad choice my Mozilla.

No, the option is off by default.

Frank Slootweg wrote:

VanguardLH wrote:
Yousuf Khan wrote:
[...]

[About .wdseml (Windows Desktop Search Email) files/messages:]

You can easily delete all of these messages, but of course
Thunderbird will regenerate them again as they come in. So what you
have to do is tell Thunderbird not to generate these files for
Windows anymore. You go into Thunderbird's options menu and turn it
off (Tools ? Options, then select Advanced ? General ? System
Integration ? Allow Windows search to search messages).

https://fileinfo.com/extension/wdseml

You can also delete them more easily by searching for and deleting
just the folders in which they reside, rather than the individual
files. These folders have an extension called *.MOZMSGS.

Interesting find. I don't remember looking at this option when I
previously trialed Thunderbird. Is this option enabled by default? If
so, a very bad choice my Mozilla.

No, the option is off by default.

https://bugzilla.mozilla.org/show_bug.cgi?id=552769

I see someone requested all those .wdseml files (under the .mozmsgs
folders) get deleted if the "Allow Windows Search" option gets disabled.
Opened on 10 YEARS AGO! Status is still New. Geezus.

https://bugzilla.mozilla.org/show_bug.cgi?id=1249056

Tbird will hang at times when moving a folder with IMAP. But then
Windows Search seems to have problems when IMAP items are moved, as
noted at:

https://bugzilla.mozilla.org/show_bug.cgi?id=567212

Then at:

https://bugzilla.mozilla.org/show_bug.cgi?id=553048

users try to disable the option but it immediately reenables itself.
Rude! Provide the option but do not honor the user's choice. Yousuf
needs to check if the option: (1) remains disabled across multiple
restarts of Thunderbird; and, (2) if the option remains enabled if the
..wdseml files that he deleted are not replace with newly generated
..wdseml files.

The more I have dig into Thunderbird and its bugs whether reported or
not, the more I get the feeling that the "developers" are CSCI
undergraduates, and over the years the turn over of volunteers resulted
in no old farts left that are intimate with the entire product. When
Mozilla declared it was considering dumping Thunderbird onto other
open-source organizations (like how OpenOffice got dumped at the Apache
Software Foundation) was when I decided to terminate my trial of
Thunderbird. Well, that and my exasperation with Thunderbird that
pushed me to also dump it after a 6-month trial.

https://www.zdnet.com/article/mozill...en-to-firefox/

"pull the plug with six months' notice if the Thunderbird project does
not make "meaningful progress in short order" in creating technical
infrastructure that's independent of Mozilla Corporation's."

That article is dated back in 2017. So, what magical evolution in
development resources has occurred for Thunderbird in the meantime?

"Mozilla stopped throwing resources at the project in 2012"

Somewhat explains why a vast number of big tickets have never been
addressed, but there are tickets still listed as New dating back to
2004. From https://en.wikipedia.org/wiki/Mozill...erbird#History,
lots of wavering on what to do with this lead balloon.