[prev in list] [next in list] [prev in thread] [next in thread] 

List:       eros-arch
Subject:    small spaces and large memories
From:       Jonathan S. Shapiro shap () eros ! cis ! upenn ! edu
Date:       1998-01-09 19:42:02
[Download RAW message or body]

As part of the performance work for my thesis, I have been preparing
to implement small address spaces (per Liedtke), and have run into a
puzzle.  Norm did not know of any particularly elegant solution.  I
have come up with a satisfactory partial solution, and wanted to share
the idea.

In the EROS small spaces design, the virtual address space is divided
as follows:

	Address Range
	Base		Bound		Size	Use
	0x00000000	0xbfffffff	3G	large space (unique)
	0xc0000000	0xdfffffff	0.5G	small (128k) spaces (shared)
	0xe0000000	0xffffffff	0.5G	kernel (shared)

In EROS, the small spaces area is divided into 4096 spaces, each of
which is 128 Kbytes.

Thinking about this ran me smack into a problem: we have machines at
Penn that have 0.5 Gb of physical memory in them, which memory cannot
be mapped into kernel virtual memory in the above design.  More
generally, the PPro machines currently support up to 64 Gbytes of
physical memory, which will not fit in the whole of the address space.

Page space can be taken out of the kernel memory map without a lot of
difficulty.  I wrote up a design note, which you can find at

	http://www.cis.upenn.edu/~eros/design-notes/KernelMemory.html

Once this is done, two ugly issues remain: user PTE management and
kernel mapping management.

The user PTE management issue is just plain ugly.  Options appear to
be dynamic mapping of page tables on the fly, maintaining an inverse
mapping, or pre-reserving page tables and associated mapping space.
Issues are layed out in the design note.  I'm probably going to do the
last option.

The messier problem is managing the kernel mapping tables.  On the one
hand you don't want to pre-allocate the physical pages corresponding
to the kernel page tables because the size of the necessary mapping
varies widely according to the amount of available physical memory.
Also, you would like to take advantage of variable page sizes in
kernel mappings.

On the other hand, if you build the kernel mapping dynamically there
is then a problem of finding a place to stand when you need to build a
new (and therefore as yet unmapped) kernel page table.  This is the
piece that Norm did not see a simple solution to.

In writing my way through the option space, I found a solution to the
kernel memory map problem that seems to get out of most of the mess
and offers some other advantages as well: 

Build a dedicated address space (which won't ordinarily be used) that
maps the available physical memory in the large space portion of the
map. If the physical memory is too big, don't lose sleep over the
stuff that won't fit.  Activate this mapping when manipulating the
kernel map, and/or when the debugger needs to examine physical memory.

In practice, I don't expect to use this for anything other than
manipulating the kernel map and examining physical memory from within
the debugger.  The point is that this map is very easy to build
minimally, and the kernel map manager can switch to it and back when
necessary, which eliminates the need for a really knotty piece of
self-reflexive code in the kernel map manager.  

Given the presence of large pages, the map isn't even all that big.
If you're prepared to lose some pointer error catching around the
edges in this map, you can build the whole map out of large pages, and
can even just map the whole of the physical address space whether
there is memory behind it or not (which is probably what I shall do).


Other thoughts:

My expectation is that 4096 small spaces is too many, because it is
significantly larger than the context cache and the proposed design
does not record space hints within the domain (though perhaps it
should).

The real needs of the system might better be served by using the space
a bit differently:

	2048 small spaces, each 128k
	256  mid-range spaces, each 1M

or even as:

	2048 small spaces, each 128k
	16   mid-range spaces, each 4M
	
The space bank, in particular, would benefit from the availability of
a mid-size space on most platforms.

This is an experiment that I intend to try after the basic small space
design is completed.

[prev in list] [next in list] [prev in thread] [next in thread]