Chapter 7UNIX and LINUX

2

Outline

• Overview

• Processes in UNIX

• Memory management in UNIX

• I/O in UNIX

• UNIX file system

3

Overview• Why some programmers like UNIX better than

Windows?– Although GUI may be easy for beginners, they provide

little flexibility and no insight into how the system works!– UNIX has GUI, too!

• History of UNIX– UNICS in Bell Labs– PDP-11 UNIX– Portable UNIX, portable C compiler– Berkeley UNIX (BSD)– UNIX standards: POSIX from IEEE– MINIX and Linux

4

UNIX Goals (Philosophy)

• Designed by programmers, for programmers

• Simple, elegant and consistent

• Power and flexibility– A small number of basic elements– One program should do just one thing– An infinite combination to suit the applications

• No useless redundancy– Simple interface

5

Interfaces to UNIX

Hardware(CPU, memory, disks, terminals, etc)

UNIX operating system(process management, memory management,

the file system, I/O, etc)

Standard library(e.g. open, close, read, write, fork, etc)

Standards utility programs(shell, editors, compilers, etc)

Users

User interface

Library interface

System call interface

User mode

Kernel mode

6

The UNIX Shell: A Command Line Interface• Wait for a command line• Extract the first word from the command line, use

it as program name, run the program• Pass arguments• Flags: argument controlling the operation or

specify an optional value• Wild cards and magic characters• Standard input/output• Filter and pipe symbol

– grep ter *.t | sort | head –20 | tail –5 > foo

• Background execution and shell scripts

7

UNIX Utility Programs

• File and directory manipulation commands: cp, mv

• Filters: grep, head, sort, tail

• Program development tools: cc

• Text processing: vi, edit

• System administration: ps

• Miscellaneous: time

8

Structure of UNIX KernelSystem calls Interrupts and traps

Terminal handling SocketsFile

namingMapping

Page faults Signal

handling

Process creation

and termination

Raw tty

Cooked tyy

Network protocols

File systems

Virtual memory

Line disciplines

RoutingBuffer cache

Page cache Process scheduling

Character devicesNetwork device drivers

Disk device drivers Process dispatching

Hardware

9

Outline

• Overview

• Processes in UNIX

• Memory management in UNIX

• I/O in UNIX

• UNIX file system

10

Basic Concepts

• UNIX is a multiprogramming system– User processes– Daemons

• Create new processes by system call fork()– Parent process and child process– PID, parent process gets child PID from fork()

• Pipes between two processes– Shell pipelines are implemented using pipes

• Signals: soft interrupt

11

Process Management System Calls in UNIX pid=fork() Create a child process identical to the parent

pid=waitpid(pid, &statloc, opts) Wait for a child to terminate

s=execve(name, argv, envp) Replace a process’ core image

exit(status) Terminate process execution and return status

s=sigaction(sig, &act, &oldact) Define action to take on signals

s=sigreturn(&context) Return from a signal

s=sigprocmask(how, &set, &old) Examine or change the signal mask

s=sigpending(set) Get the set of blocked signals

s=sigsuspend(sigmask) Replace the signal mask and suspend the process

s=kil(pid, sig) Send a signal to a process

residual=alarm(seconds) Sent the alarm clock

s=pause() Suspend the caller until the next signal

12

Thread Management

• Not every UNIX has threads package– Threads package becomes popular

• Threads can be implemented in either user space or kernel– POSIX does not specify

13

Thread Management System CallsThread Call Description

pthread_create Create a new thread in the caller’s address space

pthread_exit Terminate the calling thread

pthread_join Wait for a thread to terminate

pthread_mutex_init Create a new mutex

pthread_mutex_destroy Destroy a mutex

pthread_mutex_lock Lock a mutex

pthread_mutex_unlock Unlock a mutex

pthread_cond_init Create a condition variable

pthread_cond_destroy Destroy a condition variable

pthread_cond_wait Wait on a condition variable

pthread_cond_signal Release one thread waiting on a condition variable

14

Implementation of Processes• Two key data structures for processes• Process table: always in main memory

– Scheduling parameters, memory image, signals, miscellaneous

– A process has one entry in the table

• User structure: in memory only when the process is in memory– Machine registers, system call state, file

descriptor table, accounting info, kernel stack– A block per process, adjacent to the stack

segment

15

Executing “ls” Typed to the Shell

sh sh sh

PID=501 PID=748 PID=748

Fork code Exec code

1. Fork call

2. New sh created

3. Exec call

4. Sh overlaid with ls

New process Same process

1. Allocate child’s process table entry2. Fill child’s entry from parent3. Allocate child’s stack and user area4. Fill child’s user area from parent5. Allocate PID for child6. Set up child to share parent’s text7. Copy page tables for data and stack8. Set up sharing of open files9. Copy parent’s registers to child

1. Find the executable program2. Verify the execute permission3. Read and verify the header4. Copy arguments, environ to kernel5. Free the old address space6. Allocate new address space7. Copy arguments, environ to stack8. Reset signals9. Initialize registers

16

Threads in UNIX

• Threads in user space: a user space library

• Threads in kernel: may lead to many problems– How to maintain the correct traditional UNIX

semantics?– File I/O– Signal handling– All solutions to these problems cause

something to break somewhere

17

Threads in Linux

• System call clone– pid=clone(func, stack_ptr, sharing_flgs, arg)

• Sharing_flgs determine whether the thread is in current process or in a new process– In the same process: changes visible to other

threads

• The new thread executes func

• The new thread has its own stack

18

Scheduling in UNIX

• A two level algorithm– Low-level algorithm: pick the process to run– High-level algorithm: move processes between

memory and disk

• Low-level algorithm uses multiple queues– Priority=CPU_usage+nice+base

19

Multilevel Queue Structure

Highest …

Process waiting in

kernel mode

-4 Waiting for disk I/O

-3 Waiting for disk buffer

-2 Waiting for terminal input

-1 Waiting for terminal output

0 Waiting for child to exist

0 User priority 0 Process waiting in

user mode1 User priority 1

2 User priority 2

3 User priority 3

Lowest …

20

Scheduling in Linux

• Scheduling is based on threads

• Three classes of threads– Real-item FIFO, non-preemptive, highest

priority– Real-time round robin, preemptive, high priority– Timesharing

• Scheduling based on priority and quantum

21

Booting UNIX

• Read & run first sector of the boot disk– MBR has a small program (up to 512 bytes)– Load program boot

• Boot reads the root directory of boot device• Boot reads OS kernel, boot ends• Kernel starts, set parameters & environment• Initialization, build kernel data structure• System auto-configuration, detect devices• Load device drivers• Start process 0

22

Process 0

• Continue initialization

• Program the real-time clock

• Mount the root file system

• Create init (process 1) and page daemon (process 2)

23

Process 1

• Check flags of single-user/multi-user mode

• Single user mode– Fork off a process running shell– Wait for that process to exit

• Multi-user mode– Fork off a process

• Run system initialization shell script /etc/rc

– Read /etc/ttys, fork off processes for terminals• Run gtty

24

Sequence of Processes in Booting

Process 0

Page

daemo

n

Process 2ini

t

Process 1

gett

ylogin sh

cp

Login:

Terminal 0

Password:

Terminal 1 Terminal 2

% cp f1 f2

25

Loading Drivers

• Dynamically loading– One binary code can be used everywhere– Drivers are loaded dynamically, even through a

network

• Non-dynamically loading– Only the system administrator can make kernel

binary code– No one can insert any component into the

kernel

26

Outline

• Overview

• Processes in UNIX

• Memory management in UNIX

• I/O in UNIX

• UNIX file system

27

Three Segments in UNIX Processes• Text segment: program’s executable code

– Read-only, size fixed after process is created

• Data segment: program’s variables, strings, and other data– Initialized data and un-initialized data (BSS)– Un-initialized global variables are in BSS, save

space– Size can change

• Stack segment

28

Sharing Between Processes

• Text segment & mapped file can be shared

29

Memory Management System Calls

• POSIX does not specify

• Common system calls

System call description

s=brk(addr) Change data segment size

a=mmap(addr, len, prot, flags, fd, offset) Map a file in

s=unmap(addr, len) Unmap a file

30

Swapping

• High level scheduling: swapper

• What cause swapping?– Fork() needs memory for a child process– Brk() needs to expand a data segment– Stack needs to be expanded

• Which process will be swapped out?– The ones blocked/waiting for I/O– The ones with high (priority + residence time)

• The ones consuming much CPU/staying long in mem

31

Swapping Back Processes

• Check processes on disk every few seconds– Find processes are ready– Select the one staying on disk longest

• If have enough free memory (easy swap), bring that process in

• Otherwise, swap out some processes in main memory

• No process is swapped out until it stays in memory for 2 seconds

32

Paging in UNIX• Do not use the working set model• Done by kernel & page daemon (process 2)• Never page out kernel and core map

– Core map records info about contents of the page frames

33

Page Replacement Algorithm

• Run every 250 msec by page daemon

• If insufficient free page frames, page daemon transfers pages to disk

• A modified version of the clock algorithm

34

Memory Management in Linux

• A three-level paging scheme– Directory, middle, page + offset

• Buddy algorithm

64

32

32

32

16

16

32

16

8

8

32

16

8

8

32

8

8

8

8

32

8448

8

32

8448

8

32

844

16

35

Outline

• Overview

• Processes in UNIX

• Memory management in UNIX

• I/O in UNIX

• UNIX file system

36

Input/Output in UNIX

• Integrate devices into file system as special files– Each I/O device is assigned a path name

• Block special files and character special files

37

Networking and Sockets• Socket: interface to network• Types of networking supported by sockets

– Reliable connection-oriented byte stream– Reliable connection-oriented packet stream– Unreliable packet transmission

Sending process

Socket

Receiving process

User space

kernel space

38

I/O System Calls in UNIX

• POSIX specifies function calls for terminal

• Function call ioctl is used in many UNIXFunction call Description

s=cfsetospeed(&termios, speed) Set the output speed

s=cfsetispeed(&termios, speed) Set the input speed

s=cfgetospeed(&termios, speed) Get the output speed

s=cfgetispeed(&termios, speed) Get the input speed

s=tcsetattr(fd, opt, &termios) Set the attributes

s=tcgetattr(fd, &termios) Get the attributes

39

Outline

• Overview

• Processes in UNIX

• Memory management in UNIX

• I/O in UNIX

• UNIX file system

40

UNIX File• A sequence of bytes

– No distinction between ASCII, binary, or any other kinds of files

• Directories are stored as files

• Important directories in UNIX systemsDirectory Contents

bin Binary (executable) programs

dev Special files for I/O devices

etc Miscellaneous system files

lib Libraries

usr User directories

41

Mounting and Locking

• Only one file system (tree)– One disk is mounted in another disk’s file tree

• UNIX allows user to lock files– From single byte to an entire file– Shared locks and exclusive locks– A process specifies whether it wants to block if

the lock cannot be placed– Shared locked regions may overlap

42

System Calls About FilesSystem call Description

fd=create(name, mode) One way to create a new file

fd=open(file, how, …)Open a file for reading, writing or both

s=close(fd) Close an open file

n=read(fd, buffer, nbytes) Read data from a file into a buffer

n=write(fd, buffer, nbytes) Write data from a buffer into a file

position=lseek(fd, offset, whence) Move the file pointer

s=stat(name, &buf) Get a file’s status information

s=fstat(fd, &buf) Get a file’s status information

s=pipe(&fd[0]) Create a pipe

s=fcntl(fd, cmd, …) File locking and other operations

43

System Calls About DirectoriesSystem call Description

s=mkdir(path, mode) Create a new directory

s=rmdir(path) Remove a directory

s=link(oldpath, newpath) Create a link to an existing file

s=unlink(path) Unlink a file

s=chdir(path) Change the working directory

dir=opendir(path) Open a directory for reading

s=closedir(dir) Close a directory

dirent=readdir(dir) Read one directory entry

Rewinddir(dir)Rewind a directory so it can be reread

44

Disk Layout of Classical UNIX• Boot block: contain code to boot computer

• Superblock: contain critical info about the layout of the file system– # of i-nodes, # of disk blocks, the start of the list

of free disk blocks, …

• I-nodes

• Data blocksBoot block Super blk I-nodes Data blocks

45

Reading A File• System call n=read(fd, buffer, nbytes)• Start with file descriptor fd, find i-node

– Put pointer to the i-node in file descriptor?• Only one current position for one file, recorded in i-

node• Two processes open one file with different current

position

– Put current position info in file descriptor?• One file can have multiple current positions• P1 and P2 open f at the same time, P2 want to write

after P1 finishes• P2 cannot get the correct current position

46

Open File description Table

47

Berkeley Fast File System

• Allow file names up to 255 characters

• Divide the disk up into cylinder groups– Each with its own superblock, i-nodes and data

blocks– Whenever possible, blocks are allocated in the

cylinder group containing the i-node

• Two block sizes– Large files use large block size

48

Summary

• Three interfaces to UNIX: – Shell, C library, and system calls

• Key concepts in UNIX– Process, memory model, I/O, and file system

• Process management in UNIX– Process table and user structure

• Memory model: text, data and stack

• I/O and file system