isolated parameterized unit testing with pex and moles

Isolated Parameterized Unit Testingwith Pex and Moles

Nikolai Tillmann, Jonathan “Peli” de HalleuxMicrosoft Research

Learning objectives After I attend this class I will be able to...

Write Unit Tests Coverage, assertions, isolation

Use Moles to Isolate Unit Tests Test legacy code

Write Pex Parameterized Unit Tests Achieve high code coverage

Preparation

We will use Pex for all exercises Pex includes Moles Visual Studio 2010 Power Tools http://research.microsoft.com/Pex http://www.pexforfun.com

Works with .NET 2, 3.5, 4, x86 and x64 Visual Studio 2008, 2010, Command line

(Alpha) Silverlight support

http://research.microsoft.com/Pex



http://www.pexforfun.com/

Preparation

Install latest public versionpex.powertool.x86.msi

Unit Testing

Quiz: Unit testing What is a unit test?

A unit test is a small program with assertions

Test a single (small) unit of code

Unit Testing

void AddAndCount() { // Arrange int item = 3; // Act var list = new List(); list.Add(item); // Assert Assert.AreEqual(1, list.Count);}

static string ReadFooValue() { string[] lines = File.ReadAllLines(@"t:\myapp.ini"); foreach (var line in lines) { int index = line.IndexOf('='); string name = line.Substring(0, index); if (name == "Foo") { string value = line.Substring(index); return value; } } return null;}

A=BFoo=CC=D

t:\myapp.ini

Quiz: Code Coverage

How much block coverage do we need?1. 50%2. 80%3. 100%4. Block coverage alone is not enough

Quiz: Coverage

How much block coverage do we need?1. 50%2. 80%3. 100%4. Block coverage alone is not

enough▪ Research: no correlation between high code

coverage and quality

Quiz: White box testing

Do we need more tests to get 100% cov.?[TestMethod]

void MissingFoo() { File.WriteAllLines(@"t:\myapp.ini", new string[0]); Reader.ReadFooValue();}

[TestMethod]void ExistingFoo() { File.WriteAllLines(@"t:\myapp.ini",new string[]{“Foo=b”}); Reader.ReadFooValue();}

Quiz: Assertions

Why write Assertions (or not)?1. Documentation2. Double check your program3. Please your manager4. Prevent future bugs5. Validate user inputs6. Catch errors early

Quiz: Assertions

Which Assertions should you write?1. Assert.IsTrue(value == “b”);2. Assert.IsTrue(value == null);3. Assert.IsTrue(String.IsNullOrEmpty(value)

)4. Assert.IsTrue(true);5. No assertions

[TestMethod]void MissingFoo() { File.WriteAllLines(@"t:\myapp.ini",new string[]{“a=b”}); string value = Reader.ReadFooValue(); Assert.IsTrue(????);}

Quiz: Coverage + Assertions What gives you confidence in the

code?1. High coverage, few assertions2. Low coverage, many assertions3. High coverage, many assertions4. Low coverage, no assertions5. I wrote it

Quiz: Coverage + Assertions What gives you confidence in the

code?1. High coverage, few assertions2. Low coverage, many assertions3. High coverage, many assertions▪ Research: Experienced developers write

good assertions, junior developers write ‘debugging’ assertions

4. Low coverage, no assertions5. I wrote it

string ReadFooValue() { string[] lines = File.ReadAllLines(@"t:\myapp.ini"); foreach (var line in lines) { int index = line.IndexOf('='); string name = line.Substring(0, index); if (name == "Foo") { string value = line.Substring(index + 1); return value; } } return null;}

A=BFoo=CC=D

t:\myapp.ini

Quiz: Isolation

In the example, what are the external dependencies?1. Network Share2. Local Disk3. No file system, all in memory

string ReadFooValue() { string[] lines = File.ReadAllLines(@"t:\myapp.ini"); ...

Quiz: Isolation

What is the problem with a Local Disk?1. Mapping already exists2. Cannot run tests concurrently3. Disk full4. Access rights

Unit TestingExercise

Map local directory:> mkdir c:\foo> net use t: \\[machinename]\c$\foo

Create C# class library, copy in Reader snippet

Create test project, write unit tests Run unit tests Optional: Measure code coverage

Definition of Unit Test

What is a good Unit Test? A Unit Test is a program

that runs fast the code under test, without environment dependencies, with assertions

What is a good Unit Test Suite? A set of Unit Tests which achieves

high code coverage

Break10 Minutes

Isolation with Moles

Code under test should not depend on hard-coded environment dependencies:

How do you mitigate the Local Disk issues?1. Always run on the same machine, same

hardware, same credentials, same time, same temperature, same solar system configuration

2. Refactoring: use Streams3. Refactoring: introduce IFileSystem4. Refactoring: pass the lines as parameter5. Change implementation of File.ReadAllLines

Dependency hell

var lines = File.ReadAllLines(@"t:\myapp.ini");





Dependency hell

var lines = File.ReadAllLines(@"t:\myapp.ini"); Reality

checkRefactoring

not always an option





Dependency hell



How do you change File.ReadAllLines?1. Override static method2. Changing the CLR (and recompiling it)3. Rewrite application in JScript4. Code instrumentation

Dependency hell



How do you change File.ReadAllLines?1. Override static method2. Changing the CLR (and recompiling it)3. Rewrite application in JScript4. Code instrumentation – the Moles

framework

Dependency hell


Motivation for Moles

Why another isolation framework? Specifically designed to enable Pex

Simple, Well-defined semantics▪ “Replace any .NET method”

Type safe

Moles = Replace any .NET with a delegate


File.ReadAllLines = delegate(string fn) MFile.ReadAllLinesString = delegate(string fn){ return new string[0];};

What if we could replace File.ReadAllLines?

Moles

Mole Types Code Generation

// System.IOpublic static class File { public static string[] ReadAllLines(string fn);}

// System.IO.Molespublic class MFile { public static Func<string, string[]> ReadAllLinesString { set; }}// delegate R Func<T, R>(T t);

Injecting Detours at Runtime

// System.IOpublic static class File { public static string[] ReadAllLines(string fn) { if (MFile.ReadAllLinesString != null) return MFile.ReadAllLines(fn); … original code }}

Automatically injected

at runtime

Quiz: Func<T>

Match the delegates with the methods?

1. Func<string>2. Action3. Action<string>4. Func<bool,string>5. Func<string, bool>6. Action<int>7. Action<List<T>, int>8. Func<string,string[]>

a) bool File.Exists(string)b) Console.WriteLine(string)c) void Flush()d) String.Empty {get;}e) List<T>.Capacity {set;}f) string[]

File.ReadAllLines(string)

C# 3.0 Lambdas

MFile.ReadAllLinesString = delegate(string fileName) { return new string[]{“a=b”}; }

C# 3.0 Lambdas

MFile.ReadAllLinesString = (fileName) => { return new string[]{“a=b”}; }

C# 3.0 Lambdas

MFile.ReadAllLinesString = (fileName) => new string[]{“a=b”};

C# 3.0 Lambdas

MFile.ReadAllLinesString = fileName => new string[]{“a=b”};

Quiz: Lambdas

Match the Lambdas with the methods1. () => “”2. () => {}3. s => {}4. (s) => “”5. (s) => false

a) bool File.Exists(string)b) Console.WriteLine(string)c) void Flush();d) String.Empty {get;}e) string ToString();

MolesExercise I

Add moles for mscorlib to the test project Add new Item Moles and Stubs for Testing mscorlib.moles

Write test using moles Run test Debug test

Exercise II

static string ReadFooValue() { if (!File.Exists(@"t:\myapp.ini")) return null; string[] lines = File.ReadAllLines(@"t:\myapp.ini"); ...

Exercise II

[TestMethod, ...]void ReadFooValueTest() { MFile.BehavedAsNotImplemented(); ...

Constructors and Instance methodsstatic string ReadFooValue() { var reader = new StreamReader(“t:\myapp.ini”); string content = reader.ReadToEnd();

void ReadFooValueTest(string content) { MStreamReader.ConstructorString = delegate(StreamReader me, string file) => { var mole = new MStreamReader(me); mole.ReadToEnd = () => content; };

Exercise IIstatic string ReadFooValue() { var reader = new StreamReader(@"t:\myapp.ini")) var lines = reader.ReadToEnd().Split(‘\n’); foreach (var line in lines) { int index = line.IndexOf('='); string name = line.Substring(0, index); if (name == "Foo") { string value = line.Substring(index + 1); return value; } } return null;}} Exercise III

Quiz: Moles Usage When should you use Moles (and not)?

1. Always use Moles to solve isolation issues2. With Moles, one does not need to use interfaces

anymore3. Moles only should be used for 3rd party API, use

interfaces for isolation in your APIs4. Moles can be used in production code5. Moles lets you get away with untestable APIs6. Moles make test cases more robust7. With Moles, you do not need integration tests anymore8. Moles make tests easier to understand9. Moles is for poor programmers, real programmers rely

on interfaces

Exercise (optional) Step-by-Step Tutorials “Getting Started with Moles”

http://research.microsoft.com/pex/molestutorial.pdf

“Unit Testing SharePoint Services with Pex”http://research.microsoft.com/pex/pexsharepoint.pdf

Tip: Check if the external website matches the version of Pex you installed;the latest tutorials also ship with the installer.




http://research.microsoft.com/pex/pexsharepoint.pdf

What you learned so far

The Definition of Unit Testing

Unit Test Isolation through Moles

Break10 Minutes

Parameterized Unit Testing

The Recipe of Unit Testing

var list = new List(); // Act list.Add(item); var count = list.Count;

Assert.AreEqual(1, count); // Assert}

void AddAndCount() { int item = 3; // Arrange

Quiz: list.Add(???)

Which value matters?1. 02. -1, 13. int.MaxValue, int.MinValue4. it does not matter5. I don’t know until I read the code

list.Add(???);

Separation of Specification of behavior Data to achieve coverage

Parameterized Unit Testingvoid AddAndCount(List list, int item) { var count = list.Count; list.Add(item); Assert.AreEqual(count + 1, list.Count);}

for any list,

for any item,

… adding 1 item increases Count by

1

Problem

Where does the data come from? Random data generator Real customer data Ranges Some values hand-picked by

dev/testerDynamic Symbolic Execution

Data Generationby Dynamic Symbolic Execution

60

Goal: Given a program with a set of input parameters, automatically generate a set of input values that, upon execution, will exercise as many statements as possible

How would you do it?

Data Generation Challenge

Code to generate inputs for:

Constraints to solve

a!=null a!=null &&a.Length>0

a!=null &&a.Length>0 &&a[0]==1234567890

void CoverMe(int[] a){ if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug");}

Observed constraints

a==nulla!=null &&!(a.Length>0)a!=null &&a.Length>0 &&a[0]!=1234567890

a!=null &&a.Length>0 &&a[0]==1234567890

Datanull

{}

{0}

{123…}a==null

a.Length>0

a[0]==123…T

TF

T

F

F

Execute&MonitorSolveChoose next path

Done: There is no path left.

Dynamic Symbolic ExecutionWhat tests will Pex generate?



Data

T

TF

T

F

F



What tests will Pex generate?

void CoverMe2(int[] a, int i) { if (a[i] == a.Length) throw new Exception("bug");}

Hint: a[i]...

if (a === null) throw new NullReferenceException();If (i < 0 | i >= a.Length) throw new ArgumentOutOfRangeException();*(a+i)



Data

T

TF

T

F

F




void CoverMe2(int[] a, int i) { a[0] = -1; if (a[i] == a.Length) throw new Exception("bug");}

Hint: a[i]...

if (a === null) throw new NullReferenceException();if (i < 0 | i >= a.Length) throw new ArgumentOutOfRangeException();*(a+i)



Data

T

TF

T

F

F




void Assert(bool value) { if (false == value) throw new Exception("bug");}



Data

T

TF

T

F

F



void Foo(int value) { Assert(value == 123);}void Assert(bool value) { if (false == value) throw new Exception("bug");}

Create new project in Visual Studio. Insert CoverMe method below. Right-click on CoverMe method, and select “Run Pex”. Inspect results in tablepublic static void CoverMe(int[] a) { if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug");}

Dynamic Symbolic ExecutionDemo

ldtoken Point::GetXcall __Monitor::EnterMethodbrfalse L0ldarg.0call __Monitor::Argument<Point>

L0: .try { .try { call __Monitor::LDARG_0 ldarg.0 call __Monitor::LDNULL ldnull call __Monitor::CEQ ceq call __Monitor::BRTRUE brtrue L1 call __Monitor::BranchFallthrough call __Monitor::LDARG_0 ldarg.0 …

ldtoken Point::X call __Monitor::LDFLD_REFERENCE ldfld Point::X call__Monitor::AtDereferenceFallthrough br L2

L1: call __Monitor::AtBranchTarget call __Monitor::LDC_I4_M1 ldc.i4.m1

L2: call __Monitor::RET stloc.0 leave L4 } catch NullReferenceException {

‘ call__Monitor::AtNullReferenceException rethrow }

L4: leave L5} finally { call __Monitor::LeaveMethod endfinally }

L5: ldloc.0ret

class Point { int x; static int GetX(Point p) { if (p != null) return p.X; else return -1; } }

How does Pex monitor code?

PexMoles

(Pex / Moles Architecture)

ExtendedReflectionDynamic Runtime Analysis

Z3SMT solver

Exercise (Optional / homework)Code Digging with Pex

Open digger.pdfhttp://research.microsoft.com/pex/digger.pdf

Open VS2008 Follow PDF tutorial

You can pick up initial project from digger\digger.sln

Stop at section 6: A Glimpse of Parameterized Unit Tests

http://research.microsoft.com/pex/digger.pdf




Code Digging Exercise

1. Refactor logic in helper method (as usual)

2. Make helper method “public”3. Right-click “Run Pex” in helper

method4. Fix bugs 5. Make helper method “private”

(note: future versions of Pex will support private methods)

DiggerExercisestatic string ReadFooValue() { string[] lines = File.ReadAllLines(@"t:\myapp.ini"); return ReadFooValue(lines);}public static string ReadFooValue(string[] lines) { foreach (var line in lines) { int index = line.IndexOf('='); string name = line.Substring(0, index); if (name == "Foo") { string value = line.Substring(index + 1); return value; } } return null;}

Break10 Minutes

PexParameterized Unit Testing

Test Project

Test Generation Work-Flow

Code-Under-Test

Project

Parameterized Unit Tests Pex Generated

Tests

Add a reference to Microsoft.Pex.Framework.dll Add the following parameterized unit test “Run Pex Explorations”

Parameterized Unit TestingExercise I – Crash Test

using Microsoft.Pex.Framework;[TestClass, PexClass(typeof(Reader))]public partial class ReaderTest { [PexMethod] public void Crash(string[] lines) { MFile.ReadAllLines = () => lines; Reader.ReadFooValue(); }}

Press “1 Uninstrumented method” Select row Click on “Instrument Assembly” (lower right) Run Pex Explorations again

Execute All Tests in the Solution

Parameterized Unit TestingExercise II – Crash Test

Add assumptions and run Pex again

Parameterized Unit Testing Assumptions

[PexMethod]public void Crash(string[] lines) { PexAssume.IsNotNull(lines);

MFile.ReadAllLines = () => lines; Reader.ReadFooValue();}

Parameterized Unit Testing Pex Asserts Observed Results

[PexMethod]public string Regression(string[] lines) { PexAssume.IsNotNull(lines); MFile.ReadAllLines = () => lines; // Observe return Reader.ReadFooValue();}

Manually review outputs Pex inserts Assertions automatically

Parameterized Unit Testing Assert something

[PexMethod]void FooExist([PexAssumeNotNull]string value) { string[] lines = { “Foo=“ + value }; MFile.ReadAllLines = () => lines; var result = Reader.ReadFooValue(); PexAssert.AreEqual(value, result);}

“Ensure that we parse a Foo entry correctly”

Pattern4A – Assume, Arrange, Act, Assert

Assume, Arrange, Act, Assert[PexMethod]void Add(List<T> target, T value) { // Assume PexAssume.IsNotNull(target); // Arrange var count = target.Count; // Act target.Add(value); // Assert Assert(target.Count == count + 1); }

Break10 Minutes

Pex from theCommand linepex.exe

pex.exe test.dll

HTML Reports

Rich information, used by Pex developers Full event log history Parameter

table Code

coverage etc…

Limitationsand other DetailsIt’s called Parameterized Unit Testing

The yellow event bar notifies about important events, including certain limitations

Event Bar

Click on issue kind for more information

Events View

You should act on these events: Refactor your code, or tell Pex to ignore it in the future, let Pex analyze (“instrument”) more code,

if possible.

Instrumenting more code If Pex reports that some code was

uninstrumented, you may tell Pex to instrument and analyze it(if possible)

Instrumenting more code Code instrumentation on Demand

Instrumentation has high performance overhead

Some parts of the code better ignored Use PexInstrument… attributes

Pex will often suggest and insert those attributes for you

[assembly: PexInstrumentAssembly(“Foo”)]

Pex understand managed .NET code only Pex does not understand native code.

Problem if branching over values obtained from the environment Pex may not automatically detect all such

cases.

Testability

if (!File.Exists(f)) throw ...

File System?

Hidden Complexity

Pex analyzes every executed .NET instruction

Some used libraries may be surprisingly expensive to analyze XML parsing repeatedly converting data between different

representationsvoid Sum(string[] A) { var sum = “0”; foreach(var a in A) sum = (int.Parse(a) + int.Parse(sum)).ToString(); if(sum == “123”) throw new Exception(); }

Don’t do this.

Exploration Boundaries

Configurable bounds include: TimeOut MaxBranches MaxCalls MaxConditions▪ Number of conditions that depend on test

inputs MaxRuns ConstraintSolverTimeOut ConstraintSolverMemoryLimit

Exercise LimitationsGoal: Understand limitations.

Apply Pex to

public static void Y2kParser(string s) { if (DateTime.Parse(s).Year == 2000) throw new Exception(“found it”);}

Patternsfor Parameterized Unit Tests

PatternRoundtrip For an API f(x), f-1(f(x)) = x for all x

void ToStringParseRoundtrip(int value) { string s = value.ToString(); int parsed = int.Parse(s); Assert.AreEqual(parsed, value);}

PatternReachability Indicate which portions of a PUT should be

reachable.[PexAssertReachEventually]public void Constructor(object input){ new Foo(input); PexAssert.ReachEventually(); }

PatternSame Observable Behavior Given two methods f(x) and g(x), and a

method b(y) that observes the result or the exception behavior of a method, assert that f(x) and g(x) have same observable behavior under b, i.e. b(f(x)) = b(g(x)) for all x.

public void ConcatsBehaveTheSame( string left, string right) { PexAssert.AreBehaviorsEqual( () => StringFormatter.ConcatV1(left, right), () => StringFormatter.ConcatV2(left, right));}

PatternsHow to create objectsTips and tricks

Pex needs an instance of Reader but… Reader is abstract… Reader has no public constructors… Reader has many constructors… …

Pex has heuristics but will call for help

Creating objects is hard[PexMethod]void ReaderTest(Reader reader) { if (reader.Value == “foo”) ...}

Hand-written factory method:public static ReaderFactory { [PexFactoryMethod(Reader)] public static Reader Create(string value)

{ return Reader.HiddenCreate(value);

}}

Pex will explore both factory method and PUT!

Factory methodsReader

Object CreationExercisepublic class Reader { string fileName; private Reader(string fileName) { this.fileName = fileName; } public static Reader Create(string fileName) { return new Reader(fileName); } string ReadFooValue() { var lines = File.ReadAllLines(this.fileName); }}

HomeworkPatternsRead patterns paper:

patterns.pdf http://research.microsoft.com/Pex/patterns

.pdf

http://research.microsoft.com/Pex/patterns.pdf

http://research.microsoft.com/Pex/patterns.pdf

Break10 Minutes

Pex and Moles TogetherExercise

public class IniReader { string section; public IniReader(string section) { this.section = section; } public string ParseIni(string key) { using (var reader = new StreamReader(@"t:\myapp.ini")) { bool inSection = false; string line; while((line = reader.ReadLine()) != null) { if (line[0] == '[' && line[line.Length - 1] == ']') { string currentSection = line.Substring(1, line.Length - 2); inSection = currentSection == section; } else { int index = line.IndexOf('='); if (index > -1){ string currentKey = line.Substring(0, index); if (key == currentKey) { return line.Substring(index); } } } } return null;}}

Wrapping up

What you learned today

The Definition of Unit Testing

Unit Test Isolation through Moles

Parameterized Unit Tests with Pex

Where to go from here

Web: http://research.microsoft.com/pex/ Links to blogs, more documentation,

stubs, moles

General questions: http://social.msdn.microsoft.com/Forums/en-US/pex/

http://research.microsoft.com/pex/



http://social.msdn.microsoft.com/Forums/en-US/pex/




Thank you

http://research.microsoft.com/pexBecome a fan on Facebook

isolated parameterized unit testing with pex and moles

Documents