11

Operating Systems and Operating Systems and SecuritySecurity

22

OS SecurityOS Security

OSs are large, complex programsOSs are large, complex programs– Many bugs in any such programMany bugs in any such program– We have seen that bugs can be security We have seen that bugs can be security

threatsthreats

Here we are concerned with security Here we are concerned with security provided by OSprovided by OS– Not concerned with threat of bad OS softwareNot concerned with threat of bad OS software

Concerned with OS as security Concerned with OS as security enforcerenforcer

33

IntroductionIntroduction Operating systems provide the lowest layer Operating systems provide the lowest layer

of software visible to usersof software visible to users

Operating systems are close to the hardwareOperating systems are close to the hardware – – Often have complete hardware accessOften have complete hardware access

If the operating system isn’t protected, the If the operating system isn’t protected, the machine isn’t protectedmachine isn’t protected

Flaws in the OS generally compromise all Flaws in the OS generally compromise all security at higher levelssecurity at higher levels

44

Why Is OS Security So Important?Why Is OS Security So Important?

The OS controls access to application memoryThe OS controls access to application memory

The OS controls scheduling of the processorThe OS controls scheduling of the processor

The OS ensures that users receive the resources The OS ensures that users receive the resources they ask forthey ask for

If the OS isn’t doing these things securely, If the OS isn’t doing these things securely, practically anything can go wrongpractically anything can go wrong

So almost all other security systems must assume So almost all other security systems must assume a secure OS at the bottoma secure OS at the bottom

55

OS Security ChallengesOS Security Challenges Modern OS isModern OS is multi-user multi-user and and multi-multi-

tasking tasking OS must deal withOS must deal with

– MemoryMemory– I/O devices (disk, printer, etc.)I/O devices (disk, printer, etc.)– Programs, threadsPrograms, threads– Network issuesNetwork issues– Data, etc.Data, etc.

OS must protect processes from other OS must protect processes from other processes and users from other usersprocesses and users from other users– Whether accidental or maliciousWhether accidental or malicious

66

Single User Vs. Multiple UserSingle User Vs. Multiple UserMachinesMachines

The majority of today’s computers usually The majority of today’s computers usually support a single usersupport a single user

– – Sometimes one at a time, sometimes only one Sometimes one at a time, sometimes only one everever

Some computers are still multi-userSome computers are still multi-user – – MainframesMainframes – – ServersServers – – Network-of-workstation machinesNetwork-of-workstation machines

Single user machines often run multiple Single user machines often run multiple processes, thoughprocesses, though

77

Server Machines Vs. GeneralServer Machines Vs. GeneralPurpose MachinesPurpose Machines

Most server machines provide only Most server machines provide only limited serviceslimited services

– – Web page accessWeb page access

– – File accessFile access

– – DNS lookupDNS lookup

Security problems are simpler for Security problems are simpler for them them

88

Downloadable Code and SingleDownloadable Code and SingleUser MachinesUser Machines

Applets and other downloaded code Applets and other downloaded code should run in a constrained modeshould run in a constrained mode

Using access control on a finer Using access control on a finer granularity than the usergranularity than the user

Essentially the same protection Essentially the same protection problem as multiple usersproblem as multiple users

99

Mechanisms for SecureMechanisms for SecureOperating SystemsOperating Systems

Most operating system security is Most operating system security is based on separationbased on separation

– – Keep the bad guys away from the Keep the bad guys away from the good stuffgood stuff

– – Since you don’t know who’s bad, Since you don’t know who’s bad, separate most thingsseparate most things

1010

Separation MethodsSeparation Methods

Physical separationPhysical separation

– – Different machinesDifferent machines Temporal separationTemporal separation

– – Same machine, different timesSame machine, different times Logical separationLogical separation

– – HW/software enforcementHW/software enforcement Cryptographic separationCryptographic separation

1111

The Problem of SharingThe Problem of Sharing Separating stuff is actually pretty easySeparating stuff is actually pretty easy

The hard problem is allowing The hard problem is allowing controlled sharingcontrolled sharing

How can the OS allow users to share How can the OS allow users to share exactly what they intend to share?exactly what they intend to share?

– – In exactly the ways they intendIn exactly the ways they intend

1212

OS Security FunctionsOS Security Functions

Memory protectionMemory protection– Protect memory from users/processesProtect memory from users/processes

File protectionFile protection– Protect user and system resourcesProtect user and system resources

AuthenticationAuthentication– Determines and enforce authentication resultsDetermines and enforce authentication results

AuthorizationAuthorization– Determine and enforces access controlDetermine and enforces access control

1313

Memory ProtectionMemory Protection

Fundamental problemFundamental problem– How to keep users/processes separate?How to keep users/processes separate?

SeparationSeparation– Physical separation Physical separation separate devices separate devices– Temporal separation Temporal separation one at a time one at a time– Logical separation Logical separation with with

hardware/softwarehardware/software– Cryptographic separation Cryptographic separation make make

information unintelligible to outsiderinformation unintelligible to outsider– Or any combination of the aboveOr any combination of the above

1414

Memory ProtectionMemory Protection

Base/bounds register Base/bounds register lower and lower and upper address limitupper address limit

Assumes contiguous spaceAssumes contiguous space

Like a FenceLike a Fence users cannot cross a users cannot cross a specified addressspecified address

1515

Memory ProtectionMemory Protection

Tagging Tagging specify protection of each specify protection of each addressaddress++ Extremely fine-grained protection Extremely fine-grained protection-- High overhead High overhead can be reduced by tagging can be reduced by tagging

sections instead of individual addressessections instead of individual addresses

More common is segmentation and/or More common is segmentation and/or pagingpaging– Protection is not as flexibleProtection is not as flexible– But much more efficientBut much more efficient

1616

SegmentationSegmentation Divide memory into logical units, such asDivide memory into logical units, such as

– Single procedureSingle procedure– Data in one array, etc.Data in one array, etc.

Can enforce different access restrictions on Can enforce different access restrictions on different segmentsdifferent segments

Any segment can be placed in any memory Any segment can be placed in any memory location (if location is large enough)location (if location is large enough)

OS keeps track of actual locationsOS keeps track of actual locations

1717

SegmentationSegmentation

program

memory

1818

SegmentationSegmentation

OS can place segments anywhereOS can place segments anywhere OS keeps track of segment locations OS keeps track of segment locations

as as <segment,offset><segment,offset> Segments can be moved in memorySegments can be moved in memory Segments can move out of memorySegments can move out of memory All address references go thru OSAll address references go thru OS

1919

Segmentation AdvantagesSegmentation Advantages

Every address reference can be Every address reference can be checkedchecked– Possible to achieve Possible to achieve complete complete

mediationmediation Different protection can be applied to Different protection can be applied to

different segmentsdifferent segments Users can share access to segmentsUsers can share access to segments Specific users can be restricted to Specific users can be restricted to

specific segmentsspecific segments

2020

Segmentation DisadvantagesSegmentation Disadvantages How to reference How to reference <segment,offset><segment,offset> ? ?

– OS must know OS must know segmentsegment sizesize to verify access is to verify access is within within segmentsegment

– But some segments can grow during execution (for But some segments can grow during execution (for example, dynamic memory allocation)example, dynamic memory allocation)

– OS must keep track of OS must keep track of variablevariable segment sizessegment sizes

Memory fragmentation is also a problemMemory fragmentation is also a problem– Compacting memory changes tablesCompacting memory changes tables

A lot of work for the OSA lot of work for the OS

More complex More complex more chance for mistakes more chance for mistakes

2121

PagingPaging

Like segmentation, but fixed-size Like segmentation, but fixed-size segmentssegments

Access via Access via <page,offset><page,offset> Plusses and minusesPlusses and minuses

++ Avoids fragmentation, improved efficiency Avoids fragmentation, improved efficiency

++ OS need not keep track of variable segment OS need not keep track of variable segment sizessizes

-- No logical unity to pages No logical unity to pages

-- What protection to apply to a given page? What protection to apply to a given page?

2222

PagingPaging

program

memory

Page 1

Page 0

Page 2

Page 3

Page 4

Page 2

Page 1

Page 0

Page 3

Page 4

2323

Protecting InterprocessProtecting InterprocessCommunicationsCommunications

Operating systems provide various kinds Operating systems provide various kinds of interprocess communicationsof interprocess communications

– – MessagesMessages

– – SemaphoresSemaphores

– – Shared memoryShared memory

– – SocketsSockets

How can we be sure they’re used How can we be sure they’re used properly?properly?

2424

IPC Protection IssuesIPC Protection Issues

How hard it is depends on what you’re How hard it is depends on what you’re worried aboutworried about

For the moment, let’s say we’re worried For the moment, let’s say we’re worried about one process improperly using IPC to about one process improperly using IPC to get info from anotherget info from another

– – Process A wants to steal information from Process A wants to steal information from process Bprocess B

How would process A do that?How would process A do that?

2525

Message SecurityMessage Security

Can process B use message based IPC to steal the secret?

2626

How Can B Get the Secret?How Can B Get the Secret?

He can convince the system he’s AHe can convince the system he’s A

– – A problem for authenticationA problem for authentication He can break into A’s memoryHe can break into A’s memory

– – That doesn’t use message IPCThat doesn’t use message IPC

– – And is handled by page tablesAnd is handled by page tables He can forge a message from someone He can forge a message from someone

else to get the secretelse to get the secret He can “eavesdrop” on someone else who He can “eavesdrop” on someone else who

gets the secretgets the secret

2727

Forging An IdentityForging An Identity

2828

Operating System ProtectionsOperating System Protections The operating system knows who The operating system knows who

each process belongs toeach process belongs to

It can tag the message with the It can tag the message with the identity of the senderidentity of the sender

If the receiver cares, he can know the If the receiver cares, he can know the identityidentity

2929

How About Eavesdropping?How About Eavesdropping?

3030

What’s Really Going on Here?What’s Really Going on Here? On a single machine, what is a message send, On a single machine, what is a message send,

really?really?

A message is copied from a process buffer to A message is copied from a process buffer to an OS bufferan OS buffer

– – Then from the OS buffer to another process’ Then from the OS buffer to another process’ bufferbuffer

• • If attacker can’t get at processes’ internal If attacker can’t get at processes’ internal buffers and can’t get at OS buffers, he can’t buffers and can’t get at OS buffers, he can’t “eavesdrop”“eavesdrop”

3131

File ProtectionFile Protection How do we apply these access protection How do we apply these access protection

mechanisms to a real system resource?mechanisms to a real system resource?

Files are a common example of a typically Files are a common example of a typically shared resourceshared resource

If an OS supports multiple users, it needs to If an OS supports multiple users, it needs to address the question of file protectionaddress the question of file protection

3232

Unix File ProtectionUnix File Protection

A model for protecting files A model for protecting files developed in the 1970sdeveloped in the 1970s

Still in very wide use todayStill in very wide use today

– – With relatively few modificationsWith relatively few modifications

But not very flexibleBut not very flexible

3333

Unix File Protection Unix File Protection PhilosophyPhilosophy

Essentially, Unix uses a limited ACLEssentially, Unix uses a limited ACL

Only three subjects per fileOnly three subjects per file – – OwnerOwner – – GroupGroup – – OtherOther

Limited set of rights specifiableLimited set of rights specifiable – – Read, write, executeRead, write, execute – – Special meanings for some file typesSpecial meanings for some file types

3434

Unix GroupsUnix Groups A set of Unix users can be joined into a A set of Unix users can be joined into a

groupgroup

All users in that group receive common All users in that group receive common privilegesprivileges

– – Except file owners always get the owner Except file owners always get the owner privilegesprivileges

A user can be in multiple groupsA user can be in multiple groups

But a file has only one groupBut a file has only one group

3535

Setuid and SetgidSetuid and Setgid Unix mechanisms for changing your user Unix mechanisms for changing your user

identity and group identityidentity and group identity

Either indefinitely or for the run of a single Either indefinitely or for the run of a single programprogram

Created to deal with inflexibilities of the Created to deal with inflexibilities of the Unix access control modelUnix access control model

But the source of endless security problemsBut the source of endless security problems

3636

Unix File Access Control andUnix File Access Control andComplete MediationComplete Mediation

Unix doesn’t offer complete mediationUnix doesn’t offer complete mediation

File access is checked on open to a fileFile access is checked on open to a file – – For the requested modes of accessFor the requested modes of access

Opening program can use the file in the Opening program can use the file in the open mode for as long as it wantsopen mode for as long as it wants

– – Even if the file’s access permissions Even if the file’s access permissions changechange

Substantially cheaper in performanceSubstantially cheaper in performance

3737

Pros and Cons of Unix FilePros and Cons of Unix FileProtection ModelProtection Model

+ Low cost+ Low cost + Simple and easy to understand+ Simple and easy to understand + Time tested+ Time tested

– – Lacking in flexibilityLacking in flexibility • • In granularity of controlIn granularity of control – – Subject and objectSubject and object • • In what controls are possibleIn what controls are possible – – No complete mediationNo complete mediation

3838

Other OS Security FunctionsOther OS Security Functions OS must enforce access controlOS must enforce access control AuthenticationAuthentication

– Passwords, biometricsPasswords, biometrics– Single sign-on, etc.Single sign-on, etc.

AuthorizationAuthorization– ACLACL– CapabilitiesCapabilities

OS is an attractive target for attack!OS is an attractive target for attack!

3939

Desired Security Features of aDesired Security Features of aNormal OSNormal OS

Authentication of usersAuthentication of users Memory protectionMemory protection File and I/O access controlFile and I/O access control General object access controlGeneral object access control Enforcement of sharingEnforcement of sharing Fairness guaranteesFairness guarantees Secure IPC and synchronizationSecure IPC and synchronization Security of OS protection mechanismsSecurity of OS protection mechanisms

4040

Extra Features for a Trusted OSExtra Features for a Trusted OS

Mandatory and discretionary access Mandatory and discretionary access controlcontrol

Object reuse protectionObject reuse protection Complete mediationComplete mediation Audit capabilitiesAudit capabilities Intruder detection capabilitiesIntruder detection capabilities

4141

Trusted Operating SystemTrusted Operating System

An OS is An OS is trustedtrusted if we rely on it for if we rely on it for– Memory protectionMemory protection– File protectionFile protection– AuthenticationAuthentication– AuthorizationAuthorization

Every OS does these thingsEvery OS does these things But if a trusted OS fails to provide these, But if a trusted OS fails to provide these,

our security failsour security fails

4242

Trust vs SecurityTrust vs Security SecuritySecurity is a judgment is a judgment

of effectivenessof effectiveness Judged based on Judged based on

specified policyspecified policy Security depends on Security depends on

trust relationshipstrust relationships

Trust Trust implies implies reliancereliance

Trust is binaryTrust is binary Ideally, only trust Ideally, only trust

secure systemssecure systems All trust All trust

relationships should relationships should be explicitbe explicit

4343

Trusted Operating SystemsTrusted Operating Systems

Trust Trust implies relianceimplies reliance A trusted system is relied on for securityA trusted system is relied on for security An untrusted system is not relied on for An untrusted system is not relied on for

securitysecurity If all untrusted systems are If all untrusted systems are

compromised, your security is unaffectedcompromised, your security is unaffected Ironically,Ironically, only a trusted system can only a trusted system can

break your security!break your security!

4444

Trusted OSTrusted OS

OS mediates interactions between OS mediates interactions between subjects (users) and objects subjects (users) and objects (resources)(resources)

Trusted OS must decideTrusted OS must decide– Which objects to protect and howWhich objects to protect and how– Which subjects are allowed to do whatWhich subjects are allowed to do what

4545

General Security PrinciplesGeneral Security Principles Least privilege Least privilege like “low watermark” like “low watermark” SimplicitySimplicity Open design (Kerchoffs Principle)Open design (Kerchoffs Principle) Complete mediationComplete mediation White listing (preferable to black White listing (preferable to black

listing)listing) SeparationSeparation Ease of useEase of use But commercial OSs emphasize But commercial OSs emphasize

featuresfeatures– Results in complexity and poor securityResults in complexity and poor security

4646

MAC and DACMAC and DAC

Mandatory Access Control (MAC)Mandatory Access Control (MAC)– Access not controlled by owner of objectAccess not controlled by owner of object– Example: User does not decide who holds Example: User does not decide who holds

a a TOP SECRET TOP SECRET clearanceclearance Discretionary Access Control (DAC)Discretionary Access Control (DAC)

– Owner of object determines accessOwner of object determines access– Example: UNIX/Windows file protectionExample: UNIX/Windows file protection

If DAC and MAC both apply, MAC wins If DAC and MAC both apply, MAC wins

4747

Object Reuse ProtectionObject Reuse Protection

OS must prevent leaking of infoOS must prevent leaking of info

ExampleExample– User creates a fileUser creates a file– Space allocated on diskSpace allocated on disk– But same space previously usedBut same space previously used– ““Leftover” bits could leak informationLeftover” bits could leak information– Magnetic remanence is a related issueMagnetic remanence is a related issue

4848

How To Achieve OS SecurityHow To Achieve OS Security

Kernelized designKernelized design

Separation and isolation mechanismsSeparation and isolation mechanisms

VirtualizationVirtualization

Layered designLayered design

4949

Advantages of KernelizationAdvantages of Kernelization

Smaller amount of trusted codeSmaller amount of trusted code

Easier to check every accessEasier to check every access

Separation from other complex pieces of Separation from other complex pieces of the systemthe system

Easier to maintain and modify security Easier to maintain and modify security featuresfeatures

5050

Security KernelSecurity Kernel KernelKernel is the lowest-level part of the OS is the lowest-level part of the OS Kernel is responsible forKernel is responsible for

– SynchronizationSynchronization– Inter-process communicationInter-process communication– Message passingMessage passing– Interrupt handlingInterrupt handling

The The security kernelsecurity kernel is the part of the is the part of the kernel that deals with securitykernel that deals with security

Security kernel contained within the Security kernel contained within the kernelkernel

5151

Security KernelSecurity Kernel

Why have a security kernel?Why have a security kernel? All accesses go thru kernelAll accesses go thru kernel

– Ideal place for access controlIdeal place for access control Security-critical functions in one Security-critical functions in one

locationlocation– Easier to analyze and test Easier to analyze and test – Easier to modifyEasier to modify

More difficult for attacker to get in More difficult for attacker to get in “below” security functions“below” security functions

5252

Reference MonitorReference Monitor An important security concept for OS designAn important security concept for OS design

A A reference monitor reference monitor is a subsystem that controls is a subsystem that controls access to objectsaccess to objects

– – It provides (potentially) complete mediationIt provides (potentially) complete mediation

Must be tamper-resistant Must be tamper-resistant

Must be analyzableMust be analyzable– SmallSmall– Simple, etc.Simple, etc.

5353

Trusted Computing BaseTrusted Computing Base

TCBTCB everything in the OS that we everything in the OS that we rely on to enforce securityrely on to enforce security

If everything outside TCB is subverted, If everything outside TCB is subverted, trusted OS would still be trustedtrusted OS would still be trusted

TCB protects users from each otherTCB protects users from each other– Context switching between usersContext switching between users– Shared processesShared processes– Memory protection for usersMemory protection for users– I/O operations, etc.I/O operations, etc.

5454

TCB ImplementationTCB Implementation

Security may occur many places within Security may occur many places within OSOS

Ideally, design security kernel first, and Ideally, design security kernel first, and build the OS around itbuild the OS around it– Reality is usually the other way aroundReality is usually the other way around

Example of a trusted OS: Example of a trusted OS: SCOMPSCOMP– Developed by HoneywellDeveloped by Honeywell– Less than 10,000 LOC in SCOMP security Less than 10,000 LOC in SCOMP security

kernelkernel– Win XP has 40,000,000 lines of code! Win XP has 40,000,000 lines of code!

5555

TCB DesignTCB Design

HardwareSecurity kernelOperating systemUser space

Security kernel is the security layer

5656

Assurance of Trusted Operating Assurance of Trusted Operating SystemsSystems

How do I know that I should trust How do I know that I should trust someone’s operating system?someone’s operating system?

What methods can I use to achieve What methods can I use to achieve the level of trust I require?the level of trust I require?

5757

Assurance MethodsAssurance Methods

TestingTesting

Formal verificationFormal verification

ValidationValidation

5858

Secure Operating SystemSecure Operating SystemStandardsStandards

If I want to buy a secure operating If I want to buy a secure operating system, how do I compare options?system, how do I compare options?

Use established standards for OS Use established standards for OS securitysecurity

Several standards existSeveral standards exist

5959

Some Security StandardsSome Security Standards

U.S. Orange BookU.S. Orange Book

European ITSECEuropean ITSEC

U.S. Combined Federal CriteriaU.S. Combined Federal Criteria

Common Criteria for Information Common Criteria for Information Technology Security EvaluationTechnology Security Evaluation

6060

The U.S. Orange BookThe U.S. Orange Book

The earliest evaluation standard for The earliest evaluation standard for trusted operating systemstrusted operating systems

Defined by the Department of Defined by the Department of Defense in the late 1970sDefense in the late 1970s

Now largely a historical artifactNow largely a historical artifact

6161

Purpose of the Orange BookPurpose of the Orange Book

To set standards by which OS security To set standards by which OS security could be evaluatedcould be evaluated

Fairly strong definitions of what features Fairly strong definitions of what features and capabilities an OS had to have to and capabilities an OS had to have to achieve certain levelsachieve certain levels

Allowing “head-to-head” evaluation of Allowing “head-to-head” evaluation of security of systemssecurity of systems

– – And specification of requirementsAnd specification of requirements

6262

The Common CriteriaThe Common Criteria

Modern international standards for Modern international standards for computer systems securitycomputer systems security

Covers more than just operating systemsCovers more than just operating systems

Design based on lessons learned from Design based on lessons learned from earlier security standardsearlier security standards

Lengthy documents describe the Common Lengthy documents describe the Common CriteriaCriteria

6363

Basics of Common CriteriaBasics of Common CriteriaApproachApproach

Highly detailed methodology for Highly detailed methodology for specifying specifying

1. What security goals a system has1. What security goals a system has

2. What environment it operates in2. What environment it operates in

3. What mechanisms it uses to 3. What mechanisms it uses to

achieve its security goalsachieve its security goals

4. Why anyone should believe it does 4. Why anyone should believe it does soso

6464

Logging and AuditingLogging and Auditing

An important part of a complete An important part of a complete security solutionsecurity solution

Practical security depends on knowing Practical security depends on knowing what is happening in your systemwhat is happening in your system

Logging and auditing is required for Logging and auditing is required for that purposethat purpose

6565

LoggingLogging No security system will stop all No security system will stop all

attacksattacks – – Logging what has happened is vitalLogging what has happened is vital to dealing with the holesto dealing with the holes

Logging also tells you when someone Logging also tells you when someone is trying to break inis trying to break in

– – Perhaps giving you a chance to closePerhaps giving you a chance to close possible holespossible holes

6666

AuditingAuditing

Security mechanisms are greatSecurity mechanisms are great

– – If you have proper policies to use themIf you have proper policies to use them

Security policies are greatSecurity policies are great

– – If you follow themIf you follow them

For practical systems, proper policies and For practical systems, proper policies and consistent use are a major security consistent use are a major security problemproblem

6767

AuditingAuditing A formal (or semi-formal) process of A formal (or semi-formal) process of

verifying system securityverifying system security

A requirement if you really want your A requirement if you really want your systems to run securelysystems to run securely

Auditing should be done Auditing should be done – – Periodically Periodically – – After making major system changesAfter making major system changes – – When problems ariseWhen problems arise

6868

What Does an Audit Cover?What Does an Audit Cover?

Conformance to policyConformance to policy Review of control structuresReview of control structures Examination of audit trail (logs)Examination of audit trail (logs) User awareness of securityUser awareness of security Physical controlsPhysical controls Software licensing and intellectual Software licensing and intellectual

property issuesproperty issues