tru64 unix internals iiprocess address space1 - 1 representing a process’s address space chapter...

Tru64 UNIX Internals II Process Address Space1 - 1

Representing a Process’s Address Space

Chapter One


While Compaq believes the information included in this presentation is correct as of the date produced, it is subject to change without notice. All rights reserved.

Compaq, the Compaq logo, DIGITAL and the DIGITAL logo are trademarks of Compaq Computer Corporation. IBM is a registered trademark of International Business Machines. Microsoft is a registered trademark of Microsoft Corporation.


Topics Introduction

What a Process ExpectsDifferent Regions in an Address Space

Process Virtual Memory Maps Operations on Process Virtual Memory

Maps


What a process sees

STACK

TEXT

DATA

HEAP

Shared Libraries

3 - 8K Pages

Virtual Address Space

2^42 bytes

0

TEXT

DATA

MMAP and SYSVSegments


What the VM System is Doing

Virtual Address Space

STACK

TEXT

DATA

HEAP

TEXT

DATA

2^42 bytes

0

Physical Memory Swap Partition

File System Partition

ZFOD

TEXT pages

MMAP SHARED pages

INITIALIZED DATA andMMAP PRIVATE pages

STACK, UNINITIALIZED DATA, and HEAPpages


Task VAS Map

vm_map Memory Objectsvm_objects

Virtual AddressSpace (VAS)

Stack

Text

Data

Heap

III

AAA

BBB

CCC

DDD

EEE

FFF

GGG

HHHMappedData

Virtual Memory Components

Swap Partition

File System

000AAA rwxBBB

HHH rwxIII

CCC r-xDDD

DDD rwxEEE

FFF rwxGGG

vm_

ma

p_e

ntr

y st

ruct

ure

s

pmap


Objectifying VM - Operations Vectors

1 1fff 80001 2000 0000

rwx

1 2000 00001 2000 4000

r-x

1 4000 00001 4000 2000

rwx

1 4000 20001 4000 c000

rwx

3ff 8003 c0003ff 8080 c000

r-x

10 000 -400 0000 0000

pmap

Task

vm_map

.map

vm_pmap

stack

text init data

bss sharedlibrary

pagelist

pagelist

pagelist

pagelist

vm_objects

vm_map_entries

generic part

type specific partSwap info

vnodeinfo

Swapinfo

vnodeinfo

type specific operations vector




VM Skiplists (1) Prior to V5.0

vm_map_entries were an ordered circular link list where the beginning and end elements were set to it’s vm_map

in order by the start of the vm address the map entry represented

the vm_map and each vm_map_entry contained the vm_map_links data structure:

struct vm_map_links { struct vm_map_entry *prev; /* previous entry */ struct vm_map_entry *next; /* next entry */ vm_offset_t start; /* start address */ vm_offset_t end; /* end address */ };


VM Skiplists (2) Skip lists were introduced in V5.0

Optimize vm_map_entry management in large address spaces

Each vm_map_entry contains the vm_map_links data structure:

struct vm_map_links { struct vm_map_entry *vml_prev; /* previous entry */ vm_offset_t vml_start; /* start address */ vm_offset_t vml_end; /* end address */ struct vm_map_entry *vml_sl_next[MESL_NLEVELS];};

vm_map_entries are still an ordered circular link list along vml_sl_next[0]

Other levels (vml_sl_next[1-3]) are randomly assigned when the number of vm_map_entries reach or exceed MESL_THRESHOLD (16)


VM Skiplists (3) the vm_map

Also contains the vm_map_links data structure: struct vm_map_links { struct vm_map_entry *vml_prev; /* previous entry */ vm_offset_t vml_start; /* start address */ vm_offset_t vml_end; /* end address */ struct vm_map_entry *vml_sl_next[MESL_NLEVELS];};

Where vml_prev points to the last vm_map_entry vml_start is the value of the smallest vm address in this task vml_end is the value of the greatest allowable vm address for this task vm_sl_next[0] points to the first vm_map_entry vm_sl_next[1-3] are only randomly used if the number of vm_map_entries is greater

than MESL_THRESHOLD (16)

vm_mesl_hilevel is set to the maximum active level used in the skip lists (0 if not used)


VM Skiplists - an illustration

vml_prevvml_sl_next[0]vml_sl_next[1]vml_sl_next[2]vml_sl_next[3]


vm_mesl_hilevel=2

Task

vm_map.map

vm_map_entries

...vml_prev

vml_sl_next[0]vml_sl_next[1]vml_sl_next[2]vml_sl_next[3]



vml_start 1 1fff 8000

vml_end 1 2000 0000

vml_start 1 2000 0000vml_end 1 2000 4000

vml_start 1 4000 0000vml_end 1 4000 2000

vml_start 3ff 8003 c000vml_end 3ff 8080 c000

vml_start 10 000 -vml_end 400 0000 0000


Special Case: Kernel Map Submaps

pmap

Kernel Task

vm_map_entrys

vm_map

vm_object.submap

vm_map_entrys

vme_uobject.vm_object

pagelists

vm_objects

pagelists

vm_objects


vm_links.next vm_links.prev

"kernel map"vm_map

vm_pmap


Memory Objects can be Shared & Backed by Objects

vm_map

pmap

task A

vm_pmap

vm_links.nextvm_links.prev

vm_map_entrys

vm_object

ref_cnt=2

vm_objects

ob_memq

pagelists

vm_map

pmap

task B

vm_map_entrys

ob_ops

backing object

vm_anon_object


VM Data Structures (1)Structure Function

vm_map Acts as head of a doubly linkedcircular list of vm_map_entrystructures

vm_map_ops Contains pointers to specific functions that perform operations on address maps

pmap Machine-dependent structuredescribing VA to PA mapping

vm_map_entry Describes a contiguous region ofvirtual address space within the taskaddress space with common attributes

vm_map_entry_ops Pointers to specific functions that perform operations on memory


VM Data Structures (2)

Structure Function

vm_object Maintains information about residentand non-resident memory pages toresolve requests for them. There are

several types of objects:

- File system shared vnode

- mmapped vnode

- Anonymous memory

- Swap object

- Memory managed device

vm_object_ops Contains pointers to specific functions that perform operations on memory

objects


struct vm_map (1)struct vm_map {

struct vm_map_links vm_links; /*Links to vm_map_entrys */unsigned int vm_nentries; /*Number of vme entries */unsigned int

vm_is_mainmap :1, /*Main or submap */ vm_copy_map:1, /*Kernel copy submap */

vm_entries_pageable:1, /*Kernel map entries */ vm_wait_for_space:1, /*Wait for space */vm_umap:1, /*User space map */vm_lazy_map:1, /*Map swapped lazily? */:20, /*Filler */vm_mesl_hilevel:6; /*Skip list, max active level*/

struct mesl_path vm_mesl_path;/*mesl search path (default 4 levels)*/

vm_map_entry_t vm_hint; /*Hint for quick lookups*/decl_simple_lock_data(, vm_hint_lock)/*lock for vm_hint*/struct vm_map_ops *vm_ops; /*Operations on address

space*/lock_data_t vm_lock; /*Lock for map data */

...


struct vm_map (2)…

vm_size_t vm_size; /*Task's virtual size*/ struct pmap *vm_pmap; /*Pointer to physical map

(page tables) */decl_simple_lock_data(,vm_ref_lock) /*vm_ref_count lock*/vm_map_entry_t vm_first_free; /*First free space hint*/vm_offset_t vm_private; /*Map private information*/int vm_ref_count; /*Reference count*/int vm_res_count; /*Map resident count*/unsigned int vm_fault_rate; /*Pagefaults over time*/ int vm_pagefaults; /*Accumulated pagefault*/unsigned int vm_faultrate_time;/*Last time updated fault

rate*/unsigned short vm_color[2]; /*Color buckets*/

};


Operations on Mapsvm_xxx_map() calls are actually macros

to invoke map operation instances, e.g.:#define vm_deallocate_map vm_ops->mo_deallocate

Three instances of map ops are used:User Maps vm_umap.cKernel Maps vm_kmap.cCopy Maps vm_cmap.c

represents memory copied from an address map (copy on write)

used for inter-map copy operations to support copyin() and copy out() functions

act like a subset of the fork() operation


Operations on Address Maps (1) Operation Function

mo_deallocate Dereferences a map, deleting it if noreferences remain

mo_fault Handles a page fault

mo_wire Wires pages in a range

mo_allocate Allocates an address space

mo_map_enter Allocates a range in the specified map

mo_protect Sets protection attribute on a rangemo_inherit Sets inheritance attribute on a

range: inheritance affects how the map will be shared with child maps

mo_keep_on_exec Sets keep-on-exec state of address range

mo_exec Supports exec() system call; deallocates range not marked keep-on-exec


Operations on Address Maps (2)Operation Function

mo_delete Deletes address space

mo_check_protection Checks protection attribute against specified protection

mo_copy_overwrite Copies and overwrites an address space

mo_copyout Copies an address space (copies map)

mo_copyin Creates a copy map

mo_fork Supports fork () system call; return a duplicate of the given map


struct vm_map_entry (1)struct vm_map_entry {

struct vm_map_links vme_links; /*Links to other entries*/

struct vm_map *vme_map; /*Map which owns us*/

union vm_map_object vme_uobject; /*Object or submap*/

/* typedef union vm_map_object {struct vm_object *vm_object;struct vm_map *sub_map;

} vm_map_object_t; */

union {vm_offset_t tvme_offset; /*Offset into object*/struct vm_seg *tvme_seg; /*Points to segment map*/

} vmet;

struct vm_map_entry_ops *vme_ops; /*Map entry operations*/

struct vpage vme_vpage; /*Virtual page reference used during faults*/

decl_simple_lock_data(, vme_faultlock) /*Fault locking*/

unsigned int vme_faults; /*Active faults*/

… /* continued on next page*/


struct vm_map_entry (2)… /*struct vm_map_entry continued from previous page*/

union {struct { unsigned int /*User map entry info*/

uvme_faultwait : 1,uvme_keep_on_exec: 1,uvme_inheritance : 2,uvme_maxprot : 3;

} uvme;

struct { unsigned int /*Kernel map entry info*/kvme_faultwait: 1,kvme_is_submap: 1,kvme_copymap : 1;

} kvme;} vmeu;

vm_offset_t vme_private; /*Private data for map storage specific to map

entry type*/

};


VM Map Entry Operations (1) Kernel Map Entries

Currently invalid regions k_mape_inv.c IO space mappings k_mape_io.c Normal kernel virtual

k_mape_mem.c

User Map Entries Anonymous memory

u_mape_anon.c Memory mapped device u_mape_dev.c Segment address space u_mape_seg.c Shared segment space u_mape_ssm.c System V shared memory u_mape_shm.c File System vnode u_mape_vp.c


VM Map Entry Operations (2)Operation Function

me_fault Satisfies fault request

me_dup Duplicates a map entry

me_unmap Deletes virtual space within the map entry

me_msync Supports msync() system call for mapped character devices (only on some platforms)

me_lockup Locks virtual space

me_swap Swaps out map entry state (not supported)

me_core Handles sparse user core files

me_control Manages various functions

me_protect Sets protection of virtual space

me_check_protect Checks protection settings

me_kluster Callback for object managers to support bringing

in multiple pages on a page fault

me_copy Callback mechanism for Mach copy release

me_grow Grow map entry memory up or down


struct vm_objectstruct vm_object {

struct vm_page *ob_memq; /* Resident memory */

decl_simple_lock_data (,ob_lock)

struct vm_object_ops *ob_ops; /* Object Operations*/

struct vm_object *ob_aux_obj; /* Auxiliary object */

int ob_ref_count; /* Number of references */

int ob_res_count; /* Object resident count*/

vm_size_t ob_size; /* Size of object */

int ob_resident_pages; /* Pages in ob_memq list*/

ushort ob_flags; /* Flags */

ushort ob_type; /* Object type */

};Implementation-specific Object Information


VM Object Types (ob_type) (1)

ob_type Structure Where Used

0 OT_NULL nkernel_object Kernel memory allocators

and buffer cache

1 OT_UBC vm_ubc_object File system shared vnode used by UBC

2 OT_ANON vm_anon_object Anonymous memory

3 OT_SWAP vm_swap_object Swap object

4 OT_DEVMAP u_dev_object Memory mapped device

5 OT_KERNEL kernel_object Non-paged kernel memory

vm_submap_object Placeholder for submap operations

6 OT_PKERNEL pkernel_object Pageable kernel memory

7 OT_SHM vm_shm_object System V shared memory

8 OT_SEG vm_seg_object Text segment shared address


VM Object Types (ob_type) (2)

ob_type Structure Where Used

0x09 OT_RM_K_RCV -layered product- Reflective memory, kernel receive

0x0a OT_RM_K_TRANS -layered product- RM kernel transmit

0x0b OT_RM_U_RCV -layered product- RM user receive

0x0c OT_RM_U_TRANS -layered product- RM user transmit

0x0d OT_RM_U_DUAL -layered product- RM user dual window

0x0e OT_RM_K_DUAL -layered product- RM kernel dual window

0x0f OT_SSM vm_ssm_object System V sharedmemory, segmented

0x10 OT_STACK vm_stack_object Thread Stack region

0x11 OT_ZKERNEL zkernel_object Kernel ZFOD memory

0x12 OT_LAST Maximum


VM Object Flags (vm_flags)ob_flags Mnemonic Description

0x0001 OB_SWAPON Object swapping enabled

0x0002 OB_SWAP Object is being swapped

0x0004 OB_SWAPWAIT Waiting for swap to Complete

0x0008 OB_CHANGE Object change taking place

0x0010 OB_CHANGEWAIT Waiting for change to complete

0x0020 OB_SEMWAIT Semaphore wait

0x0040 OB_SEGALLOC A segment is being allocated

0x0080 OB_SEGWAIT Waiting for seg allocation to complete

0x0100 OB_DO_NOT_COALSES Don't coalesce this object

0x0200 OB_UBCWAIT Wait for write completion in UBC

0x0400 OB_RM_EXISTING A reflective memory object

0x0800 OB_GRANHINT A granularity hint object

0x8000 OB_FUNNEL Object is funneled


VM Object Operations (1)Operation Function

ops_lock_try Tries to lock an object

ops_unlock Unlocks an object

ops_reference References the object; increments the reference and resident counts

ops_deallocateDeallocates the object and decrements the reference and resident counts; if not referenced, frees the object

ops_pagein Pages in one or more pages

ops_pageout Pages out one or more pages

ops_swap Swaps out the object

ops_control Performs various control functions

ops_pagectl Performs various page control functions

ops_pagesteal Steals a page

ops_pagewrite Writes a page


VM Object Operations (2)Wired kernel memory kernel_object_oops k_mape_mem.c

Pageable kernel memory pkernel_object_oops k_mape_mem.c

Anonymous memory u_anon_oops u_mape_anon.c

Mmap device u_dev_oops u_mape_dev.c

Segment memory u_seg_oops u_mape_seg.c

SysV shared memory u_shm_oops u_mape_shm.c

Shared segment memory u_ssm_oops u_mape_ssm.c

Mapped file memory u_vp_oops u_mape_vp.c


Kernel Memory Objects nkernel_object: OT_NULL

A kernel object for funny mappings like kernel memory allocators and the traditional buffer cache

kernel_object: OT_KERNELAll wired-down kernel memory belongs to a single virtual

memory object (kernel_object) to avoid wasting data structures

pkernel_object: OT_PKERNELAll pageable kernel memory (thread stack_layout

structures) that transition from a non-pageable object to a pageable one because of a unwiring belongs to a single object ("pkernel_object")

zkernel_object: OT_ZKERNELKernel ZFOD memory


Anonymous Memory Object (OT_ANON)

void *ao_pshared

struct vm_object ao_object

unsigned int ao_flags

unsigned int ao_ranon

vm_offset_t ao_rbase

struct vm_object *ao_bobject

vm_offset_t ao_boffset

struct vm_anon **ao_anons

simple lock ao_lbitslock

unsigned long *ao_albits

unsigned long ao_wanted

struct vm_page *ob_memqsimple lock ob_lockstruct vm_object_ops *ob_opsstruct vm_object *ob_aux_objint ob_ref_countint ob_res_countvm_size_t ob_sizeint ob_resident_pagesushort ob_flagsushort ob_type

stru

ct v

m_o

bje

ct

struct vm_anon_object


Clustering with struct vm_anon

struct vm_anon_objectpage list

optionalbacking object

struct vm_anon

to struct vm_swap

ob_memq

ao_bobject

an_un.an_page

an_un.an_swap

ao_object

page list

ao_anons


Shared Memory Object (OT_SHM)

struct vm_shm_objectstruct vm_anon_object

struct vm_anon_object so_anon_objectstruct shmid_internal *so_spunsigned int so_wire_count


struct shmid_internal

struct shmid_ds { struct ipc_perm shm_perm time_t shm_atime time_t shm_dtime time_t shm_ctime size_t shm_segsz pid_t shm_lpid pid_t shm_cpid shmatt_t shm_nattch secinfo_t *shm_secattr } s;struct vm_object *shm_object


Thread Stack Object (OT_STACK)

struct vm_stack_object struct vm_anon_object

struct vm_anon_object stk_aobjvm_size_t stk_rsizevm_size_t stk_ysizevm_size_t stk_incrunsigned long stk_node

struct vm_anon_object stk_aobjvm_size_t stk_rsizevm_size_t stk_ysizevm_size_t stk_incrunsigned long stk_node


UBC Memory Object (OT_UBC)

struct vm_objectstruct vm_object vu_objectstruct vfs_ubcops *vu_opsunion vfs_private { struct vnode *vp struct bfAccess *ap } vu_vfp;struct vm_page *vu_cleanplstruct vm_page *vu_cleanwplstruct vm_page *vu_dirtywplint vu_wirecntint vu_nsequentialvm_offset_t vu_loffsetunsigned int vu_stampunsigned int vu_txtrefsimple lock vu_seglockstruct vm_seg *vu_seglistvoid *vu_psharedvm_page_t cached_free_pages

struct vm_ubc_object


Swap Memory Object (OT_SWAP)struct vm_swap_object

ob_ref_countob_res_count

ob_sizeob_resident_pages

ob_flags

ob_memq<lock>ob_ops vm_object_ops

ob_type

vm_page

sw_sp

sw_object

struct vm_object

to struct vm_swap


Segment Memory Object (OT_SEG)

struct vm_seg_objectstruct vm_object



struct shmid_internal

struct shmid_ds { struct ipc_perm shm_perm time_t shm_atime time_t shm_dtime time_t shm_ctime size_t shm_segsz pid_t shm_lpid pid_t shm_cpid shmatt_t shm_nattch secinfo_t *shm_secattr } s;struct vm_object *shm_object


Segment Memory Object (OT_SEG)struct vm_seg_object

ob_ref_count

ob_res_count

ob_size

ob_resident_pages

ob_flags

ob_memq

<lock>

ob_ops = u_anon_oopvm_object_ops

ob_type

vm_page

sego_segbase

sego_pmap

sego_flags

sego_cfl

sego_cbl

sego_nseg

sego_object

sego_seglist


Process Actions That Affect VM System Interfaces, e.g.:

fork() exec() brk(), sbrk(), mmap()mlock() shmget(), shmctl() getrlimit()/setrlimit()

Exceptions, e.g.:Page fault Invalid Reference Invalid Access


Initial Process Virtual Memory(fork())

1 1fff 80001 2000 0000

rwx

1 2000 00001 2000 4000

r-x

1 4000 00001 4000 2000

rwx

1 4000 20001 4000 c000

rwx

3ff 8003 c0003ff 8080 c000

r-x

10 000 -400 0000 0000

pmap

Task

vm_map

.map

vm_pmap

stack text init data bss sharedlibrary

pagelist

pagelist

pagelist

pagelist

vm_objects

vm_map_entries

Child

Parent

COW r--


Expanding Process Memory (brk(),sbrk())

1 1fff 80001 2000 0000

rwx

1 2000 00001 2000 4000

r-x

1 4000 00001 4000 2000

rwx

1 4000 20001 4002 c000

rwx

3ff 8003 c0003ff 8080 c000

r-x

10 000 -400 0000 0000

pmap

Task

vm_map

.map

vm_pmap

stack text init data bss sharedlibrary

pagelist

pagelist

pagelist

pagelist

vm_objects

vm_map_entries


Memory Management FaultsRegister Values

Fault Condition MMCSR-Value in a1

Translation not valid 0

Access violation 1

Fault on READ 2

Fault on EXECUTE 3

Fault on WRITE 4

Fault Condition Cause Register-Value in a2

Instruction fetch -1

Load instruction 0

Store instruction 1


I-Stream Trap Code Flow in PAL

Program Hardware PALcode Entry to Kernel

IstreamAccessViolation

Switch stack to kernel *Allocate 6 quads on stackSave gp,pc,ps,a0,a1,a2 on kernel stackexc_addr stack pckgp r29Bad VA a0MMCSR a1cause code a2XentMM exec_addrhw_rel

_XentMM : (locore.s)Allocate 27 more quads on stackSave all remaining registers

if (pmap_fault_on() == FAIL) then try trap()

Restore registers from sackcall_pal PAL_rtl

Disable interruptsEnter pal modeps is unchangedI-stream mapping offD-stream mapping onpc exc_addrpc pallbase + 7E0

* - only from user mode


Locating PagesTask X

vm_map_entrys


vm_links.next vm_links.prev

vm_map

0 -1 ffff

x2 0000 - 24 ffff

rw25 0000 - 47 ffff

rw7e00 0000 - 7fff ffff

rw

pmapvm_pmap

pagetable

text stack

access violation20170

pagelists

vm_ops

executionflow

(*mo_fault)(...);


Handling a Page Fault Handling a Page Fault

If access is not allowed a signal is posted to the offending thread.

If access is allowed, the page-fault handler follows the pointer to the vm_object. Associated with the object is list of virtual pages belonging to the object:

If the desired page belongs to the object, its contents are fetched from the associated memory object.


Source References (1 of 3) src/kernel/sys/vm_map.h

definitions of vm_map, vm_map_entry and their ops src/kernel/vm: implementations of Map Ops

vm_map.c generic map operations vm_umap.c user map operations vm_kmap.c kernel map operations vm_cmap.c copy map operations

src/kernel/vm Implementation of Kernel Map Entry Ops k_mape_inv.c invalid kernel map entry operations k_mape_io.c kernel io space map entry operations k_mape_mem.c kernel memory map operations


Source References (2 of 3) src/kernel/vm Implementations of User Map Entry Ops

u_mape_anon.cuser anonymous map entry operations u_mape_dev.c user mapped device map entry operations u_mape_seg.c user map text segment entry operations u_mape_ssm.c user map shared text segment entry operations u_mape_shm.c user shared memory map entry operations u_mape_vp.c user map file map entry operations

src/kernel/vm/vm_object.h vm_object and vm_object_ops definitions


Source References (3 of 3) src/kernel/vm Definitions of Object Types

vm_object.c definition of nkernel, kernel and pkernel objects vm_anon.h definition of vm_anon_object and

vm_shm_objects vm_swap.h definition of vm_swap_object u_mape_seg.h definition of vm_seg_object vm_ubc.h definition of vm_ubc_object

src/kernel/vm Implementations of Object Ops k_mape_mem.cWired kernel memory kernel_object_oops k_mape_mem.cPageable kernel memory pkernel_object_oops u_mape_anon.cAnonymous memory u_anon_oops u_mape_dev.c Mmap device u_dev_oops u_mape_seg.c Segment memory u_seg_oops u_mape_shm.c SysV shared memory u_shm_oops u_mape_ssm.c Shared segment memory u_ssm_oops u_mape_vp.c Mapped file memory u_vp_oops

tru64 unix internals iiprocess address space1 - 1 representing a process’s address space chapter...

Documents