Re: Kernel tweaks for AMD 64 -- programs and memory models

["Bryan J. Smith" <b.j.smith@ieee.org>]

Someone wrote:  
> That may be the case. I don't know what switch actually
> turns the kernel over to be fully 64 bit (if there is
> such a switch)

Yes, there _is_!

It's all about "memory models."  Typically _any_ OS that runs
a kernel can provide different memory models to _any_
program, as long as its not bigger than its own.

E.g.,
 16-bit DOS (16-bit offset) could provide 64KiB (COM)
 20-bit DOS (16-bit segment + 16-bit offset) could provide
newer 640KiB/1MiB (EXE)
 32-bit DOS (16-bit segment + 32-bit offset) Protected Memory
Interface (DPMI) could provide 16-64+MiB of memory, as well
as legacy COM and EXE compatibility.

Early OS/2, Novell NetWare (DOS-based) as well as all MS-DOS
7.x (Windows 95/98/Me) versions _use_ DPMI.  NT and Linux can
provide DOS/Win16 Emulation via DPMI (that can run COM and
EXE as well).  In NT, this is the NT Virtual DOS Machine
(NTVDM).  In Linux, this is typically DOSEmu, DOSBox, etc...
(although some software emulate a full 8086/8088, etc...).

[ NOTE:  There were a few 24-bit (286/16MiB, 24-bit offset)
and 32-bit (386/4GiB, 32-bit offset) Extenders that were not
so compatible -- long story. ]

Programs and libraries of different memory models can_not_
use each other though.  E.g., Win16 could not use Win32
libraries and vice-versa.  Microsoft did, however, come up
with Win16-on-Win32 so some shared objects, etc... could be
used and run on the same OS.

Now let's talk the 32-bit space and higher.

In Intel i386, there is a 16-bit segment and a 32-bit offset
register.  The offset register is capable of addressing
32-bit (4GiB) "flat."  The segment register is often used to
designate a location in memory as the "start".  Adding the
16-bit segment to the 32-bit off-set is known as
"normalization."

In the Intel i686 (Pentium Pro on-ward) processors can now
map above 32-bit.  In coordination with the 16-bit segment
register with Processor Address Extensions (PAE), which is a
bit on the CPU.  They use a 3-level Translation Table, hence
36-bit PAE mode.  The offset still allows programs to use the
same 32-bit flat space and work together, the OS is doing the
translation and paging above 4GiB.  Some programs can be
written to PAE36, although interacting with 32-bit libraries
may or may not be an issue.

Before I can continue, I have to explain kernel (including
I/O, memory mapped I/O and other reservations) and userspace.

In NT (including NT4.0, 5.0/2000 and 5.1/XP/2003) use a split
2GiB/2GiB kernel/user model.  The consumer and even entry
Server versions (even XP Pro???) only allow an absolute of
4GiB -- possibly only 2GiB of "user" memory (yeah, forget
more memory for running Battlefield 2).  Advanced Server
versions still use the same model, but they support PAE and
can map in 512MiB pages in the 2GiB user portion, up to
64GiB.

Linux kernels can use a variety of modes.  In fact, if you
have 1GiB or less of memory, it is recommend you use the
32-bit 3GiB/1GiB kernel/user model, which gives you 960MiB
(just under 1GiB) usable, but uses _no_ page translations
(10-20% faster).  The other mode is to use "HIMEM" or the
3-level PAE mode.  The most popular is now the 4GiB/4GiB
model.  The kernel is always in memory, but is virtualized to
a 4GiB area.  It then maps in different 4GiB user pages as
necessary.

But in all cases, there is still only a 32-bit offset
register, and the 16-bit segment register is used alongside
page tables.

The *NEW* 52-bit PAE model is provided in the initial x86-64
processors.  It actually uses an ingenius way to address up
to 48-bit (256TiB) of memory.  It's called "Flat Mode" -- the
16-bit segment and 32-bit offset registers now become a
non-segmented, 48-bit address, hence the 48-bit limitation. 
Using a 4-level, 52-bit PAE address, this can provide _full_
32-bit and 36-bit PAE compatibility.

In other words, a 52-bit PAE kernel can also provide 36-bit
PAE and 32-bit memory model compatibility (probably DPMI as
well, to a point).

FYI, there is a 40-bit (1TiB) _physical_ platform limitation
to current AMD64 designs.  It's due to the Athlon64/Opteron's
core design being shared with the older EV6 Althon.


NZG <ngustavson@emacinc.com> wrote:
> O I C
> I have to patch it.

How?

> Hmm, if I do this will all my applications have to be
> recompiled as 64 bit programs?
> Or will it still be able to run 32 bit programs?

A PAE52 _kernel_ can run 32-bit and PAE36 applications and
libraries _unchanged_.  *However*, if you upgrade to PAE52
libraries, 32-bit/PAE36 applications can_not_ use them.

What Microsoft does is exactly that, they ship an OS with a
PAE52 kernel, but still the Win32 "OS" with 32-bit/PAE36
libraries and applications.  Vendors who want to release
their own PAE52 applications for Win64 have to largely come
up with their own, *OR* use Win32-on-Win64 (WoW) function
calls which dratically _reduce_ the performance of the 64-bit
app.

BTW, I thought we've been through this before (I posted this
URL on this list before):  
http://lists.leap-cf.org/pipermail/leaplist/2005-September/003473.html

I'm sorry if my explanations are "techie" but these are
programmer/memory model concepts.

If you want to get really "heavy duty," we can talk about how
the processor works internally.  ;->


-- 
Bryan J. Smith                | Sent from Yahoo Mail
mailto:b.j.smith@ieee.org     |  (please excuse any
http://thebs413.blogspot.com/ |   missing headers)

-
To unsubscribe, send email to majordomo@silug.org with
"unsubscribe silug-discuss" in the body.