RAM Slack?

I have been reading some stuff about digital forensics and have seen several comments about "RAM slack" where supposedly some old OS versions were so idiotic that patches of RAM larger than the desired file to be saved, were written to disk. To me this sounds utterly absurd.

Davej wrote:
I have been reading some stuff about digital forensics
and have seen several comments about "RAM slack" where
supposedly some old OS versions were so idiotic that
patches of RAM larger than the desired file to be saved,
were written to disk. To me this sounds utterly absurd.

The disk drive writes sectors, not bytes. Some utilities
"do the right thing" within those boundaries, by supporting
read-modify-write on sub-sector sized chunks.

The file system deals in clusters (two files cannot occupy the
same cluster, and the usage of clusters wastes space on the
end of the cluster). If a file is 2KB and a cluster is 32KB,
that's 30KB "wasted" at the end, with somewhat random
information potentially in that end-bit.

The details are hidden, in the sense that the file system
keeps length information in units of bytes, and as long
as the file system doesn't read past the logical end of
a file, everything is OK.

But if you're looking at a partition with a hex editor,
you can see everything, including the bits you're not
supposed to be able to see. When reading at the
physical layer (the layer below the file system),
you see it all.

In one case, Microsoft Office may have made things worse,
by rounding something at the program level to sector-sized
chunks, and that left an undefined area at the end of
the file. A file reading utility could have seen the leftovers
at the end, stuff that Office would be ignoring the next
time it read the file in. That's application level mischief
and not as common. If the RAM buffer Microsoft had been
using, was zeroed before usage, you would not have noticed
anything was amiss. (I think when this issue was announced
many years ago, I checked at work, and could see garbage at
the end of a file I examined.) Perhaps this was also an issue
if you sent an MSWD as an attachment in an email - the person
at the other end would get that garbage at the end too, a
potential information leakage. So yeah, that was very very bad.

Paul

On Wed, 14 Feb 2018 09:37:01 -0800 (PST), Davej
wrote:

I have been reading some stuff about digital forensics and have seen several comments about "RAM slack" where supposedly some old OS versions were so idiotic that patches of RAM larger than the desired file to be saved, were written to disk. To me this sounds utterly absurd.

Disks **are** written in the sector size. You can't write less.

All you can do is ensure that the extra space is wiped.

On Wed, 14 Feb 2018 13:20:02 -0500, Paul
wrote:

In one case, Microsoft Office may have made things worse,
by rounding something at the program level to sector-sized
chunks, and that left an undefined area at the end of
the file. A file reading utility could have seen the leftovers
at the end, stuff that Office would be ignoring the next
time it read the file in. That's application level mischief
and not as common. If the RAM buffer Microsoft had been
using, was zeroed before usage, you would not have noticed
anything was amiss. (I think when this issue was announced
many years ago, I checked at work, and could see garbage at
the end of a file I examined.) Perhaps this was also an issue
if you sent an MSWD as an attachment in an email - the person
at the other end would get that garbage at the end too, a
potential information leakage. So yeah, that was very very bad.

Unless the file is to be distributed it doesn't even strike me as
mischief. I've got a program that I have written that very well might
have this "bug" in it. One file consists of millions of records that
will be accesed semi-randomly. All I/O is done in chunks of records
to fill IIRC 4k pages--and the next chunk of records starts at the
next page, a bit of slack. (This is actually more to keep writes from
spilling across pages.)

I can't see it being a leak, though--the file is useless on any
machine other than the one that created it so nobody would ever send
it to someone.

Loren Pechtel wrote:
On Wed, 14 Feb 2018 13:20:02 -0500, Paul
wrote:

In one case, Microsoft Office may have made things worse,
by rounding something at the program level to sector-sized
chunks, and that left an undefined area at the end of
the file. A file reading utility could have seen the leftovers
at the end, stuff that Office would be ignoring the next
time it read the file in. That's application level mischief
and not as common. If the RAM buffer Microsoft had been
using, was zeroed before usage, you would not have noticed
anything was amiss. (I think when this issue was announced
many years ago, I checked at work, and could see garbage at
the end of a file I examined.) Perhaps this was also an issue
if you sent an MSWD as an attachment in an email - the person
at the other end would get that garbage at the end too, a
potential information leakage. So yeah, that was very very bad.

Unless the file is to be distributed it doesn't even strike me as
mischief. I've got a program that I have written that very well might
have this "bug" in it. One file consists of millions of records that
will be accesed semi-randomly. All I/O is done in chunks of records
to fill IIRC 4k pages--and the next chunk of records starts at the
next page, a bit of slack. (This is actually more to keep writes from
spilling across pages.)

I can't see it being a leak, though--the file is useless on any
machine other than the one that created it so nobody would ever send
it to someone.

I would fix that if I were you.

What I do, is I review my handiwork with a hex editor,
testing as I go. To verify that I'm not making a mess.

https://mh-nexus.de/en/hxd/

Initializing a 4K buffer should not be expensive (at
least, compared to the speed of modern disk drives).

Or fill the tail of the buffer, just before you do the write.

One reason for "keeping output clean", is actually to make
inspection with the hex editor easier.

Also, if you ever have to do data recovery on a damaged
image, you'll appreciate not having to "swim through crap".
The cleaner you keep things... there will be downstream
benefits.

Paul

On Thu, 15 Feb 2018 02:28:56 -0500, Paul
wrote:

I would fix that if I were you.

What I do, is I review my handiwork with a hex editor,
testing as I go. To verify that I'm not making a mess.

https://mh-nexus.de/en/hxd/

Initializing a 4K buffer should not be expensive (at
least, compared to the speed of modern disk drives).

Or fill the tail of the buffer, just before you do the write.

One reason for "keeping output clean", is actually to make
inspection with the hex editor easier.

Also, if you ever have to do data recovery on a damaged
image, you'll appreciate not having to "swim through crap".
The cleaner you keep things... there will be downstream
benefits.

The thing is that the whole thing is a cache--without the file system
that created it it is pure crap. And since the real data is tens of
megabytes of effectively random hex data nobody's going to be
inspecting it.

Loren Pechtel wrote:
On Thu, 15 Feb 2018 02:28:56 -0500, Paul
wrote:

I would fix that if I were you.

What I do, is I review my handiwork with a hex editor,
testing as I go. To verify that I'm not making a mess.

https://mh-nexus.de/en/hxd/

Initializing a 4K buffer should not be expensive (at
least, compared to the speed of modern disk drives).

Or fill the tail of the buffer, just before you do the write.

One reason for "keeping output clean", is actually to make
inspection with the hex editor easier.

Also, if you ever have to do data recovery on a damaged
image, you'll appreciate not having to "swim through crap".
The cleaner you keep things... there will be downstream
benefits.

The thing is that the whole thing is a cache--without the file system
that created it it is pure crap. And since the real data is tens of
megabytes of effectively random hex data nobody's going to be
inspecting it.

Humans like patterns.

If you delineate items with a tail-of-zeros, it'll be
easier to spot the pattern. Your file will also be
easier to compress, if the file ever needs to be
sent over the network to you for analysis. That's a
side benefit of "whitening" those areas.

I expect you're doing some kind of memory mapped I/O and
that's why you like the 4K value. On a lot of disk drives,
the physical "quanta" there is 512 bytes (512e or 512n drive).
So you could use a slightly smaller quanta, if only the hard drive
side of things mattered. I don't use memory mapped I/O,
and so I don't know whether the page size is a big win
there or not. All I know of memory mapped I/O, is it
can misbehave with really large files (I've seen some slow
I/O because of it). This may have something to do with
how the system deals with garbage collection or something,
but because I don't understand the mechanism, I expect
you know all these details and can explain it.

Paul