lessons learnt while building pex
DESCRIPTION
Peli de Halleux Senior Research Software Design Engineer Microsoft Research. Lessons Learnt While BUILDING Pex. What is Pex?. Pex is…. A Unit Test Generation Tool for .NET That Uses Dynamic Symbolic Exception Uses Constraint Solving. Pex is…. Developed at Microsoft Research - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Lessons Learnt While BUILDING Pex](https://reader035.vdocument.in/reader035/viewer/2022081422/56816156550346895dd0df9a/html5/thumbnails/1.jpg)
LESSONS LEARNT WHILE BUILDING PEX
Peli de HalleuxSenior Research Software Design EngineerMicrosoft Research
![Page 2: Lessons Learnt While BUILDING Pex](https://reader035.vdocument.in/reader035/viewer/2022081422/56816156550346895dd0df9a/html5/thumbnails/2.jpg)
WHAT IS PEX?
![Page 3: Lessons Learnt While BUILDING Pex](https://reader035.vdocument.in/reader035/viewer/2022081422/56816156550346895dd0df9a/html5/thumbnails/3.jpg)
Pex is…
A Unit Test Generation Tool for .NET
That Uses Dynamic Symbolic Exception
Uses Constraint Solving
![Page 4: Lessons Learnt While BUILDING Pex](https://reader035.vdocument.in/reader035/viewer/2022081422/56816156550346895dd0df9a/html5/thumbnails/4.jpg)
Pex is…
Developed at Microsoft Research 5 years x 2 developers
Nikolai TillmannPeli de Halleux
Z3 Constraint Solver - 5 years x 2 developersNikolaj BjornerLeonardo de Moura
40000 downloads last year, 3000 forum posts, …
![Page 5: Lessons Learnt While BUILDING Pex](https://reader035.vdocument.in/reader035/viewer/2022081422/56816156550346895dd0df9a/html5/thumbnails/5.jpg)
CHALLENGES FACED BY PEX(OR ANY OTHER TEST GENERATION TOOL)
![Page 6: Lessons Learnt While BUILDING Pex](https://reader035.vdocument.in/reader035/viewer/2022081422/56816156550346895dd0df9a/html5/thumbnails/6.jpg)
GUT FEELING AHEAD...DISCLAIMER
![Page 7: Lessons Learnt While BUILDING Pex](https://reader035.vdocument.in/reader035/viewer/2022081422/56816156550346895dd0df9a/html5/thumbnails/7.jpg)
Universal Low of Constraint Solving
“If you cannot collect constraints,you cannot solve them”
Famous test generation guru
![Page 8: Lessons Learnt While BUILDING Pex](https://reader035.vdocument.in/reader035/viewer/2022081422/56816156550346895dd0df9a/html5/thumbnails/8.jpg)
Closed with Friends
internal void Test(int i){ if (i == 123) throw new BugException();}
Private!!!Easy
challenge
![Page 9: Lessons Learnt While BUILDING Pex](https://reader035.vdocument.in/reader035/viewer/2022081422/56816156550346895dd0df9a/html5/thumbnails/9.jpg)
Assertions Deficit Syndrome
“Your tool only finds null references” “Do you have any assertion?” “Assertion???”
Conversation between a developer
and a test generation tool writer
![Page 10: Lessons Learnt While BUILDING Pex](https://reader035.vdocument.in/reader035/viewer/2022081422/56816156550346895dd0df9a/html5/thumbnails/10.jpg)
Hidden Complexity
“Why do something in a couple instructions, when you can do it in millions”
CPU Spoiled Developer
![Page 11: Lessons Learnt While BUILDING Pex](https://reader035.vdocument.in/reader035/viewer/2022081422/56816156550346895dd0df9a/html5/thumbnails/11.jpg)
String Arithmetics
void Sum(int[] numbers) { string sum = “0”; foreach(int number in numbers) { sum = (int.Parse(sum) + number) .ToString() } if (sum == “123”) throw new BugException();}
Hidden Complexit
yHidden
Complexity
![Page 12: Lessons Learnt While BUILDING Pex](https://reader035.vdocument.in/reader035/viewer/2022081422/56816156550346895dd0df9a/html5/thumbnails/12.jpg)
XML “Mon Amour”
void DoSomething(string xml) { var doc = new XmlDocument(xml); var node = doc.Select(“//foo[@bar = 5]”); if(node.Value == 123) throw new BugException();}
Hidden Complexit
yHidden
Complexity
![Page 13: Lessons Learnt While BUILDING Pex](https://reader035.vdocument.in/reader035/viewer/2022081422/56816156550346895dd0df9a/html5/thumbnails/13.jpg)
Technology Soup
“XML, XAML, Linq, Javascript, Aspx, Silverlight, .NET, COM, C/C++, Java, PHP, …….”
Pex
![Page 14: Lessons Learnt While BUILDING Pex](https://reader035.vdocument.in/reader035/viewer/2022081422/56816156550346895dd0df9a/html5/thumbnails/14.jpg)
The life of a generated test
“What do we do with the generated tests?” A developer in front of 100 generated tests
![Page 15: Lessons Learnt While BUILDING Pex](https://reader035.vdocument.in/reader035/viewer/2022081422/56816156550346895dd0df9a/html5/thumbnails/15.jpg)
Test Poetry
“Your tool generated Foo001. I don’t like it.”
“What did you expect?” “Foo_Should_Fail_When_The_Bar_Is_Negative.” Conversation between a
developer and a test generation tool writer
![Page 16: Lessons Learnt While BUILDING Pex](https://reader035.vdocument.in/reader035/viewer/2022081422/56816156550346895dd0df9a/html5/thumbnails/16.jpg)
Test Poetry II
“Your tool generated “\0”” “What did you expect?” “”Marc”.”
Conversation between a developer and a test generation tool writer
![Page 17: Lessons Learnt While BUILDING Pex](https://reader035.vdocument.in/reader035/viewer/2022081422/56816156550346895dd0df9a/html5/thumbnails/17.jpg)
Dangers of a Great Demo
“Your tool is Magic!”A developer wowed by a
cool demo
![Page 18: Lessons Learnt While BUILDING Pex](https://reader035.vdocument.in/reader035/viewer/2022081422/56816156550346895dd0df9a/html5/thumbnails/18.jpg)
Toxic Code Challenge
“New Testing Tools are always evaluated against toxic code”
![Page 19: Lessons Learnt While BUILDING Pex](https://reader035.vdocument.in/reader035/viewer/2022081422/56816156550346895dd0df9a/html5/thumbnails/19.jpg)
Last Week’s Bug Challenge
“New Testing Tools are evaluated against previously found bugs”
![Page 20: Lessons Learnt While BUILDING Pex](https://reader035.vdocument.in/reader035/viewer/2022081422/56816156550346895dd0df9a/html5/thumbnails/20.jpg)
Unit Testing Utopia
“I do not need test generation, I already practice Unit Testing (and/or TDD)”
A Unit Testing Enthusiast
![Page 21: Lessons Learnt While BUILDING Pex](https://reader035.vdocument.in/reader035/viewer/2022081422/56816156550346895dd0df9a/html5/thumbnails/21.jpg)
Test Driven Development Dogma
“Test Generation does not fit into the TDD process”
TDD Convert
![Page 22: Lessons Learnt While BUILDING Pex](https://reader035.vdocument.in/reader035/viewer/2022081422/56816156550346895dd0df9a/html5/thumbnails/22.jpg)
Dr Unit vs Mr Integration
“Most Unit Tests are Integration Tests in disguise”
Testing tool writer gut feeling
![Page 23: Lessons Learnt While BUILDING Pex](https://reader035.vdocument.in/reader035/viewer/2022081422/56816156550346895dd0df9a/html5/thumbnails/23.jpg)
Isolate First Development
“Test Generation cannot work without Isolation”
Peli’s gut feeling
![Page 24: Lessons Learnt While BUILDING Pex](https://reader035.vdocument.in/reader035/viewer/2022081422/56816156550346895dd0df9a/html5/thumbnails/24.jpg)
Welded Coding
void ReadFile(string fileName) { if(!File.Exists(fileName)) throw new FileNotFoundException(); ... ... ...}
Hard coded
dependency
![Page 25: Lessons Learnt While BUILDING Pex](https://reader035.vdocument.in/reader035/viewer/2022081422/56816156550346895dd0df9a/html5/thumbnails/25.jpg)
The Change Function
Change =
Pip Coburn
Find a project with lots of
pain
Make it as easy as possible to get
started
![Page 26: Lessons Learnt While BUILDING Pex](https://reader035.vdocument.in/reader035/viewer/2022081422/56816156550346895dd0df9a/html5/thumbnails/26.jpg)
The Chicken and The Egg
“Who is using your tool?” “Do you want to be the first?”
“I love your tool but no.”
![Page 27: Lessons Learnt While BUILDING Pex](https://reader035.vdocument.in/reader035/viewer/2022081422/56816156550346895dd0df9a/html5/thumbnails/27.jpg)
Conclusions
Test Generation one piece of the puzzle
Isolation is key Testing is an “integrated
experience”