QuickCheckClojureCheckScalaCheck

Dave Gurnell

Property-based testing with...

Testing(it’s good ...but it’s time consuming)

cost(maintenance) > cost(development)

test("reverse an empty list") { assert(List().reverse === List())}

test("reverse a list of length 2") { assert(List("a", "b").reverse === List("b", "a"))}

test("reverse a list with a null in it") { assert(List("a", null).reverse === List(null, "a"))}

test("reverse a palindrome (?!)") { assert("TACOCAT".toList.reverse === "TACOCAT".toList)}

Lots of tests = lots of maintenance...

How can we get similar coveragewith less code?

Properties(aka “Invariants”)

reverse(a ++ b) == reverse(b) ++ reverse(a)

reverse(reverse(a)) == a

endpoint(request).status != 500

i > 0 ==> i/2 < i

∀a,b reverse(a ++ b) == reverse(b) ++ reverse(a)

∀a reverse(reverse(a)) == a

∀request endpoint(request).status != 500

i > 0 ==> i/2 < i∀i

Problem - these are all universally quantified...

We can’t test all possible inputs to our properties.

Can we find a representative sample instead?

Generators(define inputs to properties)

import org.scalacheck.Gen

val ints = Gen.oneOf(1, 2, 3) // Gen[Int]

ints.sample // => Some(1)ints.sample // => Some(3)ints.sample // => Some(2)

Generators samples values from large datasets

import org.scalacheck.Gen

val lists = Gen.listOf(Gen.oneOf(1, 2, 3))

lists.sample // => Some(List(1, 3, 2, ... lists.sample // => Some(List())lists.sample // => Some(List(2, 3, 3, ...

Generators samples values from large datasets

import org.scalacheck.Genimport org.scalacheck.Prop.forAll

val lists = Gen.listOf(Gen.oneOf(1, 2, 3))

val prop = forAll(lists) { list => list.reverse.reverse == list}

Use generators to approximate universal quantification

import org.scalacheck.Genimport org.scalacheck.Prop.forAll

val lists = Gen.listOf(Gen.oneOf(1, 2, 3))

val prop = forAll(lists) { list => list.reverse.reverse == list}

prop.check // stdout: OK, passed 100 tests.

Use generators to approximate universal quantification

import org.scalacheck.Genimport org.scalacheck.Prop.forAll

val lists = Gen.listOf(Gen.oneOf(1, 2, 3))

val prop = forAll(lists, lists) { (a, b) => (a ++ b).reverse == b.reverse ++ a.reverse}

prop.check // stdout: OK, passed 100 tests.

Use generators to approximate universal quantification

import org.scalacheck.Genimport org.scalacheck.Prop.forAll

val lists = Gen.listOf(Gen.oneOf(1, 2, 3))

val prop = forAll(lists, lists) { (a, b) => (a ++ b).reverse == a.reverse ++ b.reverse}

prop.check // stdout: Falsified after 2 tests. // ARG_0: List(0) // ARG_1: List(1)

Use generators to approximate universal quantification

“Arbitraries”(materialize generators from types)

import org.scalacheck.Arbitrary.arbitrary

val ints = arbitrary[Int] // Gen[Int]

ints.sample // => Some(2147483647)ints.sample // => Some(-1)

val lists = arbitrary[List[Int]] // Gen[List[Int]]

lists.sample // => Some(List(2120334425, 0, 0, ...lists.sample // => Some(List(-1639755040, 1, ...

import org.scalacheck.Prop.forAll

forAll { (a: List[Int]) => a.reverse.reverse == a}

forAll { (a: List[Int], b: List[Int]) => (a ++ b).reverse == b.reverse ++ a.reverse}

forAll can grab Arbitraries implicitly

But wait, there’s more...

Automatic test case reductionDSL for testing side-effects

Generator and property combinators

See the ScalaCheck web site for info!

References

Haskell QuickCheckhttp://fsl.cs.illinois.edu/images/3/3d/QuickL.pdf

http://www.haskell.org/haskellwiki/Introduction_to_QuickCheck1

ScalaCheckhttps://github.com/rickynils/scalacheck

Java QuickCheckhttps://bitbucket.org/blob79/quickcheck

ClojureCheckhttps://github.com/jbondeson/clojurecheck