Recovering Configuration Information for Configurable Systems via Symbolic

Execution

Justin Gorham, Xusheng Xiao, Tao XieNorth Carolina State Universityjwgorham,xxiao,txie@ncsu.edu

Modern Software Systems

• Designed to be highly customizable– Extensibility– Flexibility– Portability

Configurable Systems

• A software system with mechanisms in place for implementing preplanned variations in system behavior

• Characterized by configuration options• Each configuration option has a configuration

domain

Formal Definition

• Configuration option – A label that represents a dimension of a system that can be set

• Configuration domain – The set of possible values that can be selected for a configuration option

• ‘configuration’: - a mapping of configuration options to a subset of their respective domains to create an instantiation of a configurable system

Example: Microsoft Word

option

Example: Microsoft Word

option domain

Configuration Binding Time

• Early Feature Binding– Software Product Lines• Boeing: Bold Stroke Avionics Software Family• Nokia: Mobile Phones• Toshiba: Software Product Line for Electric Power

Generation Plant Monitoring and Control

• Dynamically Reconfigurable Systems– NASA’s Deep Space 1 Remote Agent software

• User Configurable Systems

Runtime User Configurable System

• Configuration options are assigned at runtime• Obtain values typically through configuration

files and command lines• Options typically bind at startup time or at an

early stage of runtime

Example: (win)grep

Command Line Options•/h|/H - printing a usage help explanatory text;•/c - print a count of matching lines for each input file;•/i - ignore case in pattern;•/l - print just files (scanning will stop on first match);•/n - prefix each line of output with line number;•/r - recursive search in subdirectories;•/E:reg_exp - the Regular Expression used as search pattern. •/F:files - the list of input files. The files can be separated by commas as in

Example: JPF

• Uses a dictionary object to parse a hierarchical set of property files– Site Properties– Project Properties– Application Properties

• Primarily uses a <key> = <value> style of configuration

JPF: Site Properties

• site.properties file is machine specific and not part of any JPF project

• Each project is listed as a <name>=<directory> pair

JPF: Site Properties

• site.properties file is machine specific and not part of any JPF project

• Each project is listed as a <name>=<directory> pair

jpf-core = ${user.home}/projects/jpf/jpf-corejpf-shell = ${user.home}/projects/jpf/jpf-shell jpf-awt = ${user.home}/projects/jpf/jpf-awt ... extensions=${jpf-core},${jpf-shell}

JPF: Project Properties

• Each JPF project contains a jpf.properties file in its root directory

• defines the paths that need to be set for the component to work properly

JPF: Project Properties

1. <project-name>.native_classpath2. <project-name>.classpath3. <project-name>.test_classpath 4. <project-name>.sourcepath

JPF: Project Properties

1. <project-name>.native_classpath2. <project-name>.classpath3. <project-name>.test_classpath 4. <project-name>.sourcepath

jpf-aprop = ${config_path}

#--- path specifications jpf-aprop.native_classpath = build/jpf-aprop.jar;lib/antlr-runtime-3.1.3.jarjpf-aprop.classpath = build/examples jpf-aprop.test_classpath = build/tests jpf-aprop.sourcepath = src/examples

#--- other project specific settingslistener.autoload=${listener.autoload},javax.annotation.Nonnull,...listener.javax.annotation.Nonnull=gov.nasa.jpf.aprop.listener.NonnullChecker ...

JPF: Project Properties

1. <project-name>.native_classpath2. <project-name>.classpath3. <project-name>.test_classpath 4. <project-name>.sourcepath

jpf-aprop = ${config_path}

#--- path specifications jpf-aprop.native_classpath = build/jpf-aprop.jar;lib/antlr-runtime-3.1.3.jarjpf-aprop.classpath = build/examples jpf-aprop.test_classpath = build/tests jpf-aprop.sourcepath = src/examples

#--- other project specific settingslistener.autoload=${listener.autoload},javax.annotation.Nonnull,...listener.javax.annotation.Nonnull=gov.nasa.jpf.aprop.listener.NonnullChecker ...

JPF: Application Properties

• Used to tell JPF what main class it should start to execute

• Stored in *.jpf properties files that are part of the test projects

• target setting defines the main class of SUT• Also define target_args along with other JPF

properties that define how the application is to be checked

JPF: Application Properties

• Used to tell JPF what main class it should start to execute

• Stored in *.jpf properties files that are part of the test projects

• target setting defines the main class of SUT• Also define target_args along with other JPF

properties that define how the application is to be checked

#--- dependencies on other JPF projects@using = jpf-awt@using = jpf-shell

#--- what JPF should runtarget = RobotManager

#--- other stuff that defines how to run JPFlistener+=,.listener.OverlappingMethodAnalyzer

shell=.shell.basicshell.BasicShellawt.script=${config_path}/RobotManager-thread.escg.enumerate_random=true...

Scenario

• Developer/Testers– QA to validate configurations– Software Configuration Space Explosion

• End Users

Software Configuration Space Explosion

• Example– 20 configuration options– The domain for each option is either ‘true’ or

‘false’– 220 = 1,048,576 possible configurations– Add another option, configuration space doubles

• Infeasible to test a large configuration space

Motivation

• Extract configuration information for a runtime user configurable system– How to extract available configuration options– How to extract constraints among options and

their values• domain of a configuration option• relationships among options and/or their values

– How to extract runtime behavior of configurations• Fault due to interaction

Extracting Configuration Information

• Manually search through source code– Expensive, error prone…

• Static analysis– Cannot capture runtime behavior of configurtions

• Combinatorial Testing– Black Box, Scalability Issues

• Symbolic Execution

Thanks to Dr. Tao Xie for reusing his slide

Symbolic Execution

Code to generate inputs for:

Constraints to solve

a!=null a!=null &&a.Length>0 a!=null &&a.Length>0 &&a[0]==1234567890

void CoverMe(int[] a){ if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug");}

Observed constraints

a==nulla!=null &&!(a.Length>0)a!=null &&a.Length>0 &&a[0]!=1234567890

a!=null &&a.Length>0 &&a[0]==1234567890

Data

null

{}

{0}

{123…}a==null

a.Length>0

a[0]==123…T

TF

T

F

F

Execute&MonitorSolve

Choose next path

Done: There is no path left.

Negated condition

SE with Configurable Systems

• Using Symbolic Evaluation to Understand Behavior in Configurable Software Systems

• Reisner, E; Song, C; Ma, K; Foster, J; Porter A

Figures on this page from “Using Symbolic Evaluation to Understand Behavior in

Configurable Software Systems“

SE with Configurable Systems

Figures on this page from “Using Symbolic Evaluation to Understand Behavior in

Configurable Software Systems“

SE with Configurable Systems

Figures on this page from “Using Symbolic Evaluation to Understand Behavior in

Configurable Software Systems“

SE with Configurable Systems

• Required to know configuration options in advance

• Program inputs may interact with configuration options

Parameterized Unit Test

Parameterized Configuration Mechanism

Proposed Solution

Unit TestsCode Under

Test

Configuration Mechanism

Inputs

Option Value

Output: Test Suite with concrete

inputs

Proposed Solution

PUTsCode Under

Test

ParameterizedConfiguration Mechanism

Output: Test Suite with concrete inputs and

assumptions on configuration options

Our Approach

Implementation

Challenges

• Object Creation and Data Transformation• Static Configuration Options

Future Directions

END

• Questions