the "I can't see all 4 GiB RAM" problem - what are the reserved addresses?

There are thousands of posts in forums, where some novice whines:

I increased my RAM to 4 gigs but Windoze/Ubuntu only sees XXXX!

And the answer is that a significant percentage of the 32-bit
address space is reserved for devices (even if there is no
onboard VGA) so upgrade to a 64-bit OS and add even more RAM.

I checked a few PCs with 4 GiB RAM, and I get about 3.25 GiB in
one and another even less than 3 GiB. It seems a lot to lose
a gigabyte. Is there a detailed account somewhere of what these
reserved addresses are, and why so much is needed?

"Orson Cart" wrote in message
...

There are thousands of posts in forums, where some novice whines:

I increased my RAM to 4 gigs but Windoze/Ubuntu only sees XXXX!

And the answer is that a significant percentage of the 32-bit
address space is reserved for devices (even if there is no
onboard VGA) so upgrade to a 64-bit OS and add even more RAM.

I checked a few PCs with 4 GiB RAM, and I get about 3.25 GiB in
one and another even less than 3 GiB. It seems a lot to lose
a gigabyte. Is there a detailed account somewhere of what these
reserved addresses are, and why so much is needed?

The biggest chunk is taken by the video card. The video
card's on-board VRAM is mapped into the system's address
space, which is within the 4GB address space for a 32 bit OS
The result is that if you have a 1GB video card, 1GB is lost
to the OS and apps. Unless they're large address aware,
most 32 bit apps only use 2GB of address space.

Most PCs now come with 64 bit Windows as default, so it's
becoming a moot point. That's good, as a lot of video cards
are now have 1 to 2 GB of VRAM.

Orson Cart wrote:
There are thousands of posts in forums, where some novice whines:

I increased my RAM to 4 gigs but Windoze/Ubuntu only sees XXXX!

And the answer is that a significant percentage of the 32-bit
address space is reserved for devices (even if there is no
onboard VGA) so upgrade to a 64-bit OS and add even more RAM.

I checked a few PCs with 4 GiB RAM, and I get about 3.25 GiB in
one and another even less than 3 GiB. It seems a lot to lose
a gigabyte. Is there a detailed account somewhere of what these
reserved addresses are, and why so much is needed?

First, some concepts.

OSes have supported PAE (physical address extension) for a while.

The processor and chipset have a 36 bit address bus. The addresses are
physical. The memory mapper, maps a 4GB chunk of virtual space, to
some area of the 64GB physical space.

What that means is, if you had a computer with 64GB of memory
installed, you started 16 different applications each using
4GB of memory, they could use the entire 64GB, and all from the
comfort of a 32 bit OS. One program cannot use the entire 64GB
on its own. To use all the memory, takes 16 or more programs.
(If the OS was 64 bit, and the application was "pure 64 bit" type,
then you could use the entire 64GB in one program.)

Now, Microsoft disabled PAE, after WinXP SP2. The virtual to physical
mapping process, needed a bit for NX (no execute protection), and while
on the one hand, the necessary space in the mapper occurs if you use
PAE mode, at the same time, Microsoft set the "memory license" to 4GB.
It means PAE is enabled by default, but you're prevented from
using the entire 64GB by virtue of WinXP "not being a server OS".
We wouldn't want to cut in the lucrative server market, with
desktop OSes.

Someone here, tested out these concepts, in Vista 32, and managed to
change the OS such that it would run with 8GB (on a 32 bit OS).
That's where I got this info. This demonstrates that in fact, there
is no real reason to not being doing it this way.

http://www.geoffchappell.com/notes/w...nse/memory.htm

*******

I have one other experience to share with you.

I run WinXP SP3 x32. For an experiment, I installed 6GB of memory
(all the DDR2 I own). I installed this RAM Disk software and used it to define
a 2GB RAM disk.

http://memory.dataram.com/products-a...tware/ramdisk?

This is a benchmark of the RAMDisk when installed. (My RAM isn't
very fast.)

http://img196.imageshack.us/img196/8...am2gbabove.gif

Next, I put a 2GB pagefile on that disk. I made the pagefile, the
only pagefile on the system.

Then, I started loading up applications.

Normally, at around 2.6GB usage or so (of 3.25 "free"), the system
becomes "grumpy". There is paging out to disk. IF I launch another application,
the general slowdown might be too much for me to continue to run that way.

When the RAMDisk is used for paging, I can sail right past 3.25GB with hardly
any effect. No slowdown. And the reason for that, is the 4GB/sec I/O rate and
zero seek time, makes paging virtually invisible. I was even able to
go past the 4GB mark (I stopped at 4.5GB, because there wasn't any point
going further). Now, of that 4GB, some is always stored on disk, so
it's not all paged in at the same time. The 3.25GB limit still exists.
One of the applications will be paged out. So it's not entirely effects free.
But for a lot of situations, it gives the impression I've got a 4GB (or more)
system. It's because paging is super-fast with that RAMDisk.

In terms of the license:

1) You can't have more than 4GB of addresses. These are
addresses that user programs are using. These are
addresses in "user space".

2) The RAMDisk is a driver, and lives in driver space with the kernel.
(If the driver dies, the kernel dies.) That appears to let it have
access to the entire PAE space. (Note that on AMD systems, like S939
or later, PAE space on AMD is 40 bits.)

3) The addresses used for PCI and PCI Express busses, are real. They represent
"memory like" addresses, which are kept below the 4GB mark purportedly so
that only 32 bit addressing will be needed on PCI cards. The bus usage
subtracts from the 4GB limit.

4) The sum total of the remaining, accessible physical memory, plus the
pagefile, is available for program usage. On a regular system, the hard
drive is painfully slow, if you go over 3.25GB.

5) In terms of address allocations, they're done by the BIOS. The BIOS plans
the address map, before the OS starts. Addresses are assigned in wasteful
256MB chunks. If you have no PCI cards, but there is physical evidence of a
PCI bus being present, 256MB is assigned for it. If you have a PCI Express
bus, it's treated like PCI. If you need 256MB + 1 byte for a PCI Express video
card, then the BIOS allocates 512MB. I have a 512MB video card on PCI Express,
and my "free" is 3.25GB. Someone using two 512MB video cards in SLI, would
find the "free" value was 2.75GB.

Now, is that RAMDisk bulletproof ? Not entirely. I had one event, where I
started a program, and there was no GUI on the screen. The program was running,
but there was no interface. On another occasion, I started a 3D game, and "kaboom"
it exited, just like that. So while the RAMDisk makes a great demo of what is
possible with PAE, and loopholes in the memory license, it isn't a good enough
config to keep for everyday usage.

I consider that Geoff Chappell article, to be a good reference. At least,
to get above the usual statement of limits.

HTH,
Paul

In message someone claiming to be Paul
typed:

Now, Microsoft disabled PAE, after WinXP SP2.

That's not quite true, PAE is supported and used by default in WinXP
SP3, Vista and Windows 7, just not 4GB access.

The 4GB address space limit was enforced for other reasons, mainly
because every single kernel mode component (which includes many drivers)
needs to be 4GB aware in order to reliably use more than 4GB of space.
This wasn't a big deal for OEM Server configurations, but in practice
x64 was less work to roll out than to start certifying 4GB 32-bit
workstation environments.

The 4GB ramdrive setup was a unique one in that only the ramdrive
itself would exist above 4GB, so you could generally get away with
drivers that couldn't handle 4GB address spaces since they'd rarely (if
ever) see such addresses.

--
It's always darkest before dawn. So if you're going to
steal your neighbor's newspaper, that's the time to do it.

I have checked a few of my PCs, and one has 1.2 GiB missing in action,
but only has a 256 MiB graphics card (I prefer the low-power no-fan
models). I can only think this is due to a third-party chip for the
legacy I/O (i.e. IDE). So BIOS must reserve another 256 MiB for that?

On 1/9/2012 7:45 AM, Orson Cart wrote:
I have checked a few of my PCs, and one has 1.2 GiB missing in action,
but only has a 256 MiB graphics card (I prefer the low-power no-fan
models). I can only think this is due to a third-party chip for the
legacy I/O (i.e. IDE). So BIOS must reserve another 256 MiB for that?

Almost every I/O device has a segment or two of address space, used by
device drivers for communications with the device. That set of address
segments must be mapped into the 4GB total address space.

Remember that there are bunch of I/O widgets built into any modern MoBo,
and they use as much address space as the add-in cards.

Device designers have not always been careful about conservation of
address space. And, since many devices were designed in an era when PCs
shipped with a few MBs of RAM and had a few GBs of address space,
conservation of address space just did not matter. To paraphrase
Senator Dirkson, "A MB here, a MB there; pretty soon you're talking
about serious space."
--
Cheers, Bob

Orson Cart wrote:
I have checked a few of my PCs, and one has 1.2 GiB missing in action,
but only has a 256 MiB graphics card (I prefer the low-power no-fan
models). I can only think this is due to a third-party chip for the
legacy I/O (i.e. IDE). So BIOS must reserve another 256 MiB for that?

Start by checking for built-in graphics, even if you're not using them.
Some built-in graphics have a static and a dynamic allocation. The static one
may be defined in the BIOS, and can be cranked down (but not
always to zero). That might help a bit. The dynamic allocation doesn't
hurt, because it's only called on when needed (like, gaming with built-in
graphics).

There was one Dell desktop, that "hoists" memory at the 2GB mark, instead
of using a more granular approach (i.e. even with a ****ty PCI video card
with small memory, it still hacks off all that space). To see all the memory
in that case, you need a 64 bit OS (or, for someone to fix the Dell BIOS
code). If you install 4GB, you see 2GB free in a 32 bit OS, as the rest
is hoisted above the 4GB mark.

You can try Device Manager, View : Resource by type, then use the
Memory entry. On mine, it almost looks like both PCI and PCI Express
are treated as a "PCI bus". My video card spans C0000000-DFFFFFFF
or 512MB (which is correct). From E0000000-EFFFFFFF is listed as
motherboard resources (256MB). The rest of the map (0xF+) is sparsely filled
and not well used. So mine doesn't make much sense, except to conclude
maybe it hoists on gigabyte boundaries, instead of 256MB boundaries.
Implying I'm wasting 256MB address space for no good reason. Maybe the
actual PCI bus hides in the "motherboard resources" ?

The datasheet for the Intel chipset, sometimes has a memory map in it,
which shows the potential mappings. Again, not easy to interpret, and
correlate with what you're seeing on the actual machine. Since I
can't get Imageshack to work right now, I'm trying out this one instead.
This is from the X48 datasheet (page 35?) for my chipset.

http://i.imgur.com/DxXg2.gif

Paul