property-based testing and generators (lua)

Your systems. Working as one.

Property-based Testing and Generators

Sumant Tambe7/29/2015

Agenda

• Property-based Testing

• The need for generators

• Design/Implementation of the generator library

• Using mathematics for API design

• Value constraints in type definitions

• Reactive generators

• Generators at compile-time (type generators)

8/25/2015 © 2015 RTI 2

Why should you care?

• If you write software for living and if you are serious about any of the following– Bug-free code

– Productivity• Modularity

• Reusability

• Extensibility

• Maintainability

– Performance/latency• Multi-core scalability

– and having fun while doing that!

8/25/2015 © 2015 RTI 3

What’s a Property

• Reversing a linked-list twice has no effect– reverse(reverse(list)) == list

• Compression can make things smaller– compress(input).size <= input.size

• Round-trip serialization/deserialization has no effect– deserialize(serialize(input)) == input

8/25/2015 © 2015 RTI 5

Property-based Testing

• Complementary to traditional example-based testing

• Specify post-conditions that must hold no matter what

• Encourages the programmer to think harder about the code

• More declarative

• Need data generators to produce inputs

• Free reproducers! – Good property-based testing frameworks shrink inputs to

minimal automatically

• Famous: Haskell’s QuickCheck, Scala’s ScalaTest

8/25/2015 © 2015 RTI 6

A property-test in RefleX

8/25/2015 © 2015 RTI 7

template <class T>

void test_roundtrip_property(const T & in)

{

T out; // empty

DDS::DynamicData dd(...); // empty

reflex::update_dynamicdata(dd, in);

reflex::read_dynamicdata(out, dd)

assert(in == out);

}

• What’s the input data?

• What are the input types?

Data Generators

8/25/2015 © 2015 RTI 8

Lua has less syntactic noise than C++. So, code examples are in Lua

Data Generators

8/25/2015 © 2015 RTI 9

• Simplest data generator = math.random

math.randomseed(os.time())

local r = math.random(6)

print(r) -- one of 1, 2, 3, 4, 5, 6

r = math.random(20, 25)

print(r) -- one of 20, 21, 22, 23, 24, 25

Data Generators

8/25/2015 © 2015 RTI 10

• A boolean generatorfunction boolGen()

return math.random(2) > 1

end

• An uppercase character generator

function upperCaseGen() -- A=65, Z=90

return string.char(math.random(65, 90))

end

• An int16 generatorfunction int16Gen()

return math.random(MIN_INT16, MAX_INT16)

end

Data Generators

8/25/2015 © 2015 RTI 11

• A range generatorfunction rangeGen(lo, hi)

return math.random(lo, hi)

end

• Inconvenient because args must be passed every time

Use state!

Closure Recap

8/25/2015 © 2015 RTI 12

• A simple closurelocal f = function ()

return boolGen()

end

print(f()) -- true or false

• Another closurelocal a = 20

local f = function ()

print(a)

end

f() -- 20

a = 30

f() -- 30

Closures capture state

8/25/2015 © 2015 RTI 13

• Back to the range generator

function rangeGen(lo, hi)

return function()


end

end

local f = rangeGen(20, 25)

print(f()) -- one of 20, 21, 22, 23, 24, 25

Closures capture state

8/25/2015 © 2015 RTI 14

• The oneOf generator

function oneOfGen(array)

local len = #array –- get length of the array

return function()

return array[math.random(1, len)]

end

end

local days = { “Sun”, “Mon”, “Tue”, “Wed”, “Thu”, “Fri”, “Sat” }

local f = oneOfGen(days)

print(f()) -- one of Sun, Mon, Tue, Wed, Thu, Fri, Sat

8/25/2015 © 2015 RTI 15

“Closures are poor man's objects; Objects are poor

man's closure”

-- The venerable master Qc Na Source Google images

Closure vs Object

• Is function a sufficient abstraction for anygenerator?

function () {

return blah;

}

• Answer depends upon your language– No. In Lua, C++, …

– Yes. In Haskell

8/25/2015 © 2015 RTI 16

The Generator Library

8/25/2015 17

local Gen = require(“generator”)

local rangeGen = Gen.rangeGen(20,25)

for i=1, 5 do

print(rangeGen:generate()) –- 5 values in range(20, 25)

end

Github: https://github.com/rticommunity/rticonnextdds-ddsl

local stepperGen = Gen.stepperGen(1, 10)

for i=1, 5 do

print(stepperGen:generate()) –- 1, 2, 3, 4, 5

end

local infiniteGen = Gen.stepperGen(1, 10, 1, true)

while true do

print(infiniteGen:generate()) –- 1, 2, 3, 4, ...

end

https://github.com/rticommunity/rticonnextdds-ddsl

The Generator Library

8/25/2015 18

local Gen = require(“generator”)

local charGen = Gen.uppercaseGen()

local maxLen = 64

local strGen = Gen.stringGen(maxLen, charGen)

for i=1, 5 do

print(strGen:generate()) -- 5 uppercase strings upto size 64

end

Github: https://github.com/rticommunity/rticonnextdds-ddsl

sequenceGen is quite similar

https://github.com/rticommunity/rticonnextdds-ddsl

The Generator library (batteries included)

8/25/2015 © 2015 RTI 19

boolGen posFloatGen emptyGen

charGen posDoubleGen singleGen

wcharGen floatGen constantGen

octetGen doubleGen oneOfGen

shortGen lowercaseGen inOrderGen

int16Gen uppercaseGen stepperGen

int32Gen alphaGen rangeGen

int64Gen alphaNumGen seqGen

uint16Gen printableGen arrayGen

uint32Gen stringGen aggregateGen

uint64Gen nonEmptyStringGen enumGen

The Generator “Class”• Lua does not have “classes” but that’s not important

8/25/2015 © 2015 RTI 20

local Generator = {} –- an object that will serve as a class

function Generator:new(generateImpl)

local o = {}

o.genImpl = generateImpl -- A method assignment

-- some additional metatable manipulation here

return o

end

function Generator:generate()

return self.genImpl()

end

function rangeGen(lo, hi)

return Generator:new(function ()


end)

end

Composing Generators

• Producing more complex Generators from one or more simpler Generators

8/25/2015 © 2015 RTI 22

Generator

map

reduce

append

where

zipMany

concatMap

take

scan

Mapping over a generator

8/25/2015 © 2015 RTI 23

local days = { “Sun”, “Mon”, “Tue”, “Wed”, “Thu”, “Fri”, “Sat” }

local fiboGen = Gen.fibonacciGen()

-- 0 1 1 2 3 5 8 ...

local plus1Gen = fiboGen:map(function (i) return i+1 end)

-- 1 2 2 3 4 6 9 ...

local dayGen = plus1Gen:map(function (j) return days[j] end)

-- Sun, Mon, Mon, Tue, Wed, Fri, nil, ...

for i=1, 6 do

print(dayGen:generate()) -- Sun, Mon, Mon, Tue, Wed, Fri

end

Zipping generators

8/25/2015 © 2015 RTI 24

local xGen = Gen.stepperGen(0,100)

local yGen = Gen.stepperGen(0,200,2)

local pointGen =

Gen.zipMany(xGen, yGen, function (x, y)

return { x, y }

end)

for i=1, 5 do

print(pointGen:generate()) –- {0,0}, {1,2}, {3,4}, ...

end

Zipping generators

8/25/2015 © 2015 RTI 25

local xGen = Gen.stepperGen(0,100)

local yGen = Gen.stepperGen(0,200,2)

local colorGen = Gen.oneOfGen({ “RED”, “GREEN”, “BLUE” })

local sizeGen = Gen.constantGen(30)

local shapeGen =

Gen.zipMany(xGen, yGen, colorGen, sizeGen,

function (x, y, color, size)

return { x = x,

y = y,

color = color,

shapesize = size }

end)

• The mandatory Shapes example

Using Data Domain Specific Language (DDSL)

8/25/2015 © 2015 RTI 26

local ShapeType = xtypes.struct{

ShapeType = {

{ x = { xtypes.long } },

{ y = { xtypes.long } },

{ shapesize = { xtypes.long } },

{ color = { xtypes.string(128), xtypes.Key } },

}

}

local shapeGen = Gen.aggregateGen(ShapeType)

for i=1, 5 do

print(shapeGen:generate()) –- output next slide

end

• Use Lua DDSL to define types

Five random shapes

8/25/2015 © 2015 RTI 27

COLOR?!

Use a Collection of Generators

8/25/2015 © 2015 RTI 28


ShapeType = {

{ x = { xtypes.long } },

{ y = { xtypes.long } },

{ shapesize = { xtypes.long } },

{ color = { xtypes.string(128), xtypes.Key } },

}

}

local shapeGenLib = {}

shapeGenLib.x = Gen.stepperGen(0, 100)

shapeGenLib.y = Gen.stepperGen(0, 200, 2)

shapeGenLib.color = Gen.oneOf({ "RED", "GREEN", "BLUE" })

shapeGenLib.shapesize = Gen.rangeGen(20, 30)

local shapeGen = Gen.aggregateGen(ShapeType, shapeGenLib)

for i=1, 5 do

print(shapeGen:generate()) –- output next slide

end

Five sensible shapes

8/25/2015 © 2015 RTI 29

In not so distant future…

8/25/2015 © 2015 RTI 30


ShapeType = {

{ x = { xtypes.long,

xtypes.constraint{ “[ x | x <- stepperGen(0,100) ]” }}},

{ y = { xtypes.long,

xtypes.constraint{ “[ x*x | x <- stepperGen(0,200,2) ]” }}},

{ size = { xtypes.long,

xtypes.constraint{ “[ x | x <- rangeGen(20,30) ]” }}},

{ color= { xtypes.string(128),

xtypes.constraint{ “[ x | x <- constantGen('BLUE') ]” }}}

}

}

• Direct applicability in • MEP Message Emulation Processes (i.e., generate data)• MCT Generator expressions may be specified directly in the UI

8/25/2015 © 2015 RTI 31

The Mathematics of the Generator API

Generators

• General design principal

– Create a language to solve a problem

– API is the language

– Reuse parsing/compilation of the host language (duh!)

• The “language” of Generators

– Primitive values are basic generators

– Complex values are composite generators

– All APIs result in some generator• Especially, map, zip, reduce, etc.

– I.e., generators form an algebra

8/25/2015 © 2015 RTI 32

Algebra

A first-class abstraction

+Compositional* API

8/25/2015 © 2015 RTI 33

*Composition based on mathematical concepts

8/25/2015 © 2015 RTI 34

What Math Concepts?

• Algebraic Structures from Category Theory–Monoid

–Monad

– Functor

–Applicative

–Group

– Endofunctor

• Where to begin?–Haskell

– But innocent should start with Scala perhaps

8/25/2015 © 2015 RTI 35

Does it have anything to do with DDS?

Yes!

Composable API for DDS exists. It’s Rx4DDSAlso, RPC-over-DDS (futures). Both are humble beginnings

And remember, sky is the limit

http://rticommunity.github.io/rticonnextdds-reflex/

Next Items

• Value constraints in type definitions

• Reactive generators

• Generators at compile-time (type generators)

• (Back to) Testing RefleX API properties

8/25/2015 © 2015 RTI 36

Constraints in Data Models

• Database Schemas

• XSD Restrictions

8/25/2015 © 2015 RTI 37

CREATE TABLE products (

product_no integer UNIQUE NOT NULL,

name text,

price numeric CHECK (price > 0)

);

<xs:element name="age">

<xs:simpleType>

<xs:restriction base="xs:integer">

<xs:minInclusive value="0"/>

<xs:maxInclusive value="120"/>

</xs:restriction>

</xs:simpleType>

</xs:element>

<xs:element name="prodid">

<xs:simpleType>

<xs:restriction base="xs:integer">

<xs:pattern value="[0-9][0-9][0-9]"/>

</xs:restriction>

</xs:simpleType>

</xs:element>

Constraints in Data Models

• STANAG 4607– Ground Moving Target Indicator

• Object Constraint Language– Defined by OMG: part of the UML standard

– Huge!

8/25/2015 © 2015 RTI 38

In not so distant future…

8/25/2015 © 2015 RTI 39


ShapeType = {

{ x = { xtypes.long,

xtypes.constraint{ “[ x | x <- stepperGen(0,100) ]” }}},

{ y = { xtypes.long,

xtypes.constraint{ “[ x*x | x <- stepperGen(0,200,2) ]” }}},

{ size = { xtypes.long,

xtypes.constraint{ “[ x | x <- rangeGen(20,30) ]” }}},

{ color= { xtypes.string(128),

xtypes.constraint{ “[ x | x <- constantGen('BLUE') ]” }}}

}

}

• Direct applicability in • MEP Message Emulation Processes (i.e., generate data)• MCT Generator expressions may be specified directly in the UI

General Syntax for Data Generation

• Mathematical Set Builder Notation

• Suggested Generator Syntax

– [ x | x <- stepperGen() ]

– [ 2*x | x <- fibonacciGen() ]

– [ “ID” .. i | i <- stepperGen(1, 256) ]

• Translated mechanically to the Generator API– map, flatMap, etc.

– List Comprehension in disguise• Python, Haskell has it. Lua does not.

8/25/2015 © 2015 RTI 40

Value Dependencies

• List Comprehension alone is not sufficient

• What if data members have value dependencies– o.y = o.x * o.x

– o.len = length(o.somearray)

– o.min = minimum(o.somearray)

– o.max = maximum(o.somearray)

8/25/2015 © 2015 RTI 41

Syntax for Value Dependencies

• Generator Syntax seems to work for value dependencies too

• Just refer to the generator of the parent attribute

– [ a*a | a <- $(x) ]

– [ len(a) | a <- $(somearray) ]

– [ min(point.x) | point <- t,

t <- $(trajectory) ]

(Assuming trajectory is an array of points )

8/25/2015 © 2015 RTI 42

Pull Generators aren’t enough

• Pull Generators don’t handle dependencies well– At least not without state

– Requires multiple passes through state for every generated sample • Seems expensive

– Dependencies must be “interpreted” for every sample• I.e., needs to refer type description to do its job

• Dependencies can’t be “compiled” for performance

– A relatively simple and powerful alternative exists

8/25/2015 © 2015 RTI 43

How dependencies break

• The ShapeType generator asks leaf generators to produce a new value.

• Dependencies are immediately lost

8/25/2015 © 2015 RTI 44

X

Y

ShapeType

Y = X2

Circle represents a generator not state

Reactive Generators

• Work “just like DDS!”• Propagate values “downstream”

– Push-based instead of pull-based

• Subject-Observer (pub/sub) pattern– A single value is propagated to one or more dependent

attributes

• Dependent generators take the “right” action– Such as doubling the value, counting min, etc.

• map, flatmap, reduce etc. work exactly like pull generators

• Composable– Generator expressions can be mechanically translated to

reactive generator API

8/25/2015 © 2015 RTI 45

Basics of Reactive Generators

8/25/2015 © 2015 RTI 46

class Generator<T>

{

void listen(callback);

};

Where callback is a closure

callback = function (t) {

// provided by the user

}

You can register multiple callbacks

Create Listen Push Dispose

Lifecycle of a reactive (push) generator

n n

Create aPull

GeneratorSubject

transform, …

Optional

Composing Reactive Generators

• Producing more complex Generators from one or more simpler Generators

8/25/2015 © 2015 RTI 48

ReactGen

map

reduce

append

where

zipMany

concatMap

take

scan

Round-trip property-test for ShapeType

8/25/2015 © 2015 RTI 51

void test_roundtrip_shape(const ShapeType & in)

{

ShapeType out; // empty


reflex::write_dynamicdata(dd, in);


assert(in == out);

}

• The assertion must hold for all instances.

void test_many_shapes()

{

auto generator = gen::make_aggregate_gen<ShapeType>();

for (int i = 0;i < 100; ++i) {

test_roundtrip_shapetype(gen.generate());

}

}

A generic property-test in RefleX

8/25/2015 © 2015 RTI 52

template <class T>

void test_roundtrip_property(const T & in)

{

T out; // empty


reflex::update_dynamicdata(dd, in);


assert(in == out);

}

• What are the input types?

• Given a type, how to produce data?

Random Numbers at Compile-time

• Demo

– In C

– Then in C++

8/25/2015 © 2015 RTI 54

Live code

http://coliru.stacked-crooked.com/a/aaa81c1d489f9db5

Linear Feedback Shift Register (LFSR)

• Simple random number generator• 4 bit Fibonacci LFSR• Feedback polynomial = X4 + X3 + 1• Taps 4 and 3

8/25/2015 © 2015 RTI 55

Linear Feedback Shift Register (LFSR)

• Live code feedback polynomial (16 bit)– X16 + X14 + X13 + X11 + 1

• Taps: 16, 14, 13, and 11

8/25/2015 © 2015 RTI 56

Type Generators Demystified

• Random seed as command line #define

• Compile-time random numbers using LFSR

• C++ meta-programming for type synthesis

• std::string generator for type names and member names

• RefleX to map types to typecode and dynamic data

8/25/2015 © 2015 RTI 57

RefleX Property-test Type Generator

• “Good” Numbers– 31415 (3 nested structs)

– 32415 (51 nested structs)




– 7937 ( 5 nested structs)

– ...

• “Bad” Numbers– 2152 (test seg-faults) sizeof(tuple) > 2750 KB!

8/25/2015 © 2015 RTI 58

Github: https://github.com/rticommunity/rticonnextdds-reflex/blob/develop/test/generator

https://github.com/rticommunity/rticonnextdds-reflex/blob/develop/test/generator

property-based testing and generators (lua)

Software