Why Security Testing Why Security Testing Is HardIs Hard

Herbert H. ThompsonHerbert H. Thompson

Presenter: Alicia YoungPresenter: Alicia Young

IntroductionIntroduction

Software Testing good at verifying requirementsSoftware Testing good at verifying requirements UML helps move from specification to test UML helps move from specification to test

casescases Several bugs routinely escape testingSeveral bugs routinely escape testing

Not specification ViolationsNot specification Violations Would escape most automated testingWould escape most automated testing

Examine Security bugs to discover why testing Examine Security bugs to discover why testing can be difficultcan be difficult

Side-Effect BehaviorSide-Effect Behavior

Input A -> result BInput A -> result B What if Input A also resulted in C?What if Input A also resulted in C?

Overt – unexpected dialog box appearsOvert – unexpected dialog box appears Subtle – writing a file or opening a network portSubtle – writing a file or opening a network port

RDISK utility for WindowsRDISK utility for Windows Creates an emergency Repair DiskCreates an emergency Repair Disk Temporary file created with Universal PermissionsTemporary file created with Universal Permissions During testing, product responds as specifiedDuring testing, product responds as specified

Intended vs. ImplementedIntended vs. Implemented

The State of Security TestingThe State of Security Testing

Exploit Libraries (Librarian Method)Exploit Libraries (Librarian Method) New Products tested with only this libraryNew Products tested with only this library Finds old vulnerabilities with no hope of finding Finds old vulnerabilities with no hope of finding

anything newanything new

Problem is…this strategy actually works!Problem is…this strategy actually works! Developers repeatedly make the same mistakesDevelopers repeatedly make the same mistakes Current software is really buggyCurrent software is really buggy

Applications will eventually become immune to Applications will eventually become immune to these test casesthese test cases

The Need for TechniquesThe Need for Techniques

Test like detectivesTest like detectives Past bugs teach us how vulnerabilities get into Past bugs teach us how vulnerabilities get into

our applicationsour applications The key is to learn new techniques of finding bugsThe key is to learn new techniques of finding bugs

Four General Classes of testing techniquesFour General Classes of testing techniques DependenciesDependencies Unanticipated user inputUnanticipated user input Techniques to expose Design VulnerabilitiesTechniques to expose Design Vulnerabilities Techniques to expose implementation vulnerabilitiesTechniques to expose implementation vulnerabilities

Dependency Insecurities and Dependency Insecurities and FailuresFailures

Software resides in co-dependent environmentSoftware resides in co-dependent environment Two Security ConcernsTwo Security Concerns

Application may inherit insecuritiesApplication may inherit insecurities External security service resource may failExternal security service resource may fail

Internet Explorer’s Content AdvisorInternet Explorer’s Content Advisor Content advisor password protects classes of sitesContent advisor password protects classes of sites If the library fails to load, Internet explorer permits If the library fails to load, Internet explorer permits

access to any previously blocked siteaccess to any previously blocked site

Cause of Dependency FailuresCause of Dependency Failures

Severely under-applied inputs to softwareSeverely under-applied inputs to software Error handling code gets little testing scrutinyError handling code gets little testing scrutiny

These types of failures need to be examinedThese types of failures need to be examined

Unanticipated User InputUnanticipated User Input

Inputs that cause undesirable side effects and Inputs that cause undesirable side effects and require special testingrequire special testing Reserved wordsReserved words Escape charactersEscape characters Long stringsLong strings Boundary valuesBoundary values

Most well known side-effect: Buffer OverflowMost well known side-effect: Buffer Overflow Input that can be interpreted as commandsInput that can be interpreted as commands

Design InsecuritiesDesign Insecurities

Many Security Vulnerabilities designed into Many Security Vulnerabilities designed into applicationapplication Seeing high-level impact on an application or host is Seeing high-level impact on an application or host is

difficultdifficult

Test InstrumentationTest Instrumentation Many applications shipped with itMany applications shipped with it Bypassing security controls for ease of testingBypassing security controls for ease of testing

Ports left openPorts left open Insecure default values and configurationsInsecure default values and configurations

Implementation InsecuritiesImplementation Insecurities

Perfect design means nothing if Implementation Perfect design means nothing if Implementation is flawedis flawed

Man-in-the-middle attackMan-in-the-middle attack Attacker gets between time application checks Attacker gets between time application checks

security and when the application uses informationsecurity and when the application uses information Xterm – can be exploited to allow a restricted user Xterm – can be exploited to allow a restricted user

to append data to the password fileto append data to the password file

Standard Bug-Severity RankingsStandard Bug-Severity Rankings UrgentUrgent

System crash, Unrecoverable data loss, jeopardizes personnelSystem crash, Unrecoverable data loss, jeopardizes personnel HighHigh

Impairment of critical system functions and no work-around Impairment of critical system functions and no work-around existsexists

MediumMedium Impairment of critical system functions and work-around Impairment of critical system functions and work-around

existsexists LowLow

Inconvenience, annoyanceInconvenience, annoyance NoneNone

None of the above or an enhancementNone of the above or an enhancement

The Need For ToolsThe Need For Tools

Testers generally rewarded for both quantity and Testers generally rewarded for both quantity and severity of bugsseverity of bugs

Side-effect bugs may not get noticed or even Side-effect bugs may not get noticed or even dismissed by managersdismissed by managers

Equipped with proper tools testers would notice Equipped with proper tools testers would notice odd behaviorodd behavior Writing of a temporary fileWriting of a temporary file Sending of extra network packetsSending of extra network packets

New ToolsNew Tools

Regmon and Filemon – monitor application Regmon and Filemon – monitor application interactions with registry and file systeminteractions with registry and file system www.sysinternals.comwww.sysinternals.com

App-Sight – monitors environmental App-Sight – monitors environmental interactionsinteractions www.identify.comwww.identify.com

Holodeck – Fine grain control over interactions Holodeck – Fine grain control over interactions between application and environmentbetween application and environment www.sisecure.comwww.sisecure.com

Paper AnalysisPaper Analysis

Quality Software is Secure SoftwareQuality Software is Secure Software

Important points madeImportant points made Better testing techniquesBetter testing techniques Better testing toolsBetter testing tools Design concernsDesign concerns

Questions?Questions?