automated software testing - software...

Automated Software TestingMaurício Aniche

m.f.aniche@tudelft.nl@mauricioaniche

Delft University of Technology, 2019

Agile is not only about processes…

• Technical practices are also very important:• Automated testing• Test-Driven Development• Pair programming• Refactoring• Continuous Integration

• Extreme Programming (XP) has a strong focus on the technical aspects of agility.

Agile is not only about processes…

• Technical practices are also very important:• Automated testing• Test-Driven Development• Pair programming• Refactoring• Continuous Integration

• Extreme Programming (XP) has a strong focus on the technical aspects of agility.

Yes, I can do it!!

Given a list of numbers, I want to know the smallest and the largest numbers there!

def find(nums):smallest = 9999largest = -9999for num in nums:

if(num < smallest):smallest = num

elif(num > largest):largest = num

print(smallest)print(largest)

find([29, 37, 4, 5])

Ø 4Ø 37

Yes, it works!

find([29, 28, 27, 26])

Ø 26Ø -9999

My bad…. :/If I pass {29, 28, 27, 26}, the program gets crazy!!

def find(nums):smallest = 9999largest = -9999for num in nums:

if(num < smallest):smallest = num

elif(num > largest):largest = num

print(smallest)print(largest)

find([29, 37, 4, 5])

def find(nums):smallest = 9999largest = -9999for num in nums:

if(num < smallest):smallest = num

if(num > largest):largest = num

return min, maxmin, max = find([29, 37, 4, 5])print(min)print(max)

Have you made such mistakes before?

Beller, Moritz, Andy Zaidman, and Andrey Karpov. "The last line effect." Proceedings of the 2015 IEEE 23rd International Conference on Program Comprehension. IEEE Press, 2015.Beller, M., Zaidman, A., Karpov, A., & Zwaan, R. A. (2017). The last line effect explained. Empirical Software Engineering, 22(3), 1508-1536.

A (bad) fairy tale

What’s the solution for that?

Test your software!!

What’s wrong with manual testing?

• Too slow• Too expensive• Not easy to reproduce• Susceptible to failures• … boring!

Let’s automate some tests!

• Unit testing framework• Pytest!

• Triple A (Arrange-Act-Assert)

•Let’s do it!

def test_decreasing_order():min, max = find([29, 28, 27, 26])

assert min == 26assert max == 29

def test_any_random_numbers():min, max = find([29, 37, 4, 5])

assert min == 4assert max == 37

https://gist.github.com/mauricioaniche/73366e8e00686e3d44cb48f971599716

Source code

• Implementation here:• https://gist.github.com/mauricioaniche/9735db28d7091dac0f701dac

a327313a

”But if you write 100 lines of production code,now you’ll write only 50, as the other 50 are testing. Therefore, you are less productive.”

– says a bad manager.

Not true.

• You spend a lot of time in manual testing.• Now, you will spend it only once: to write the test.

• Teams with automated test suites spend less time debugging.

George, B., Williams, L., An Initial Investigation of TDD in Industry. ACM Symposium on Applied Computing. Melbourne, Florida, USA, 2003.Janzen, D., Software Architecture Improvement through Test-Driven Development. Conference on Object Oriented Programming Systems Languages and Applications, ACM, 2005

Why is a failing test a good thing?

• If it fails here, it didn’t fail in production! J• You have the possibility to fix the bug before going to production.• ”If you have a test that never fails, you have a useless test.” –

pragmatic developer.• Always see your test failing at least once.

• How do I test the test?

The bug cycle

• You found a bug!• Write a test to make sure it exists• Fix the bug• The bug will never haunt you again.

FIRST!

• Fast• Isolated• Repeatable• Self-validating• Timely

Fast

• If they are fast, you run them all day long.• If they are not, you stop using them.• Make your tests fast!

• Unit tests tend to be fast• Mock dependencies (Curious? Google for it! Or ask Steve Freeman!)

Isolated

• A test should not depend on another test.• Test dependencies lead to flaky tests + hard to diagnose failures.

• Isolate your tests as much as you can.

Repeatable

• Tests that produce different results in different executions are hard to diagnose.

• If you run your test twice, the results should be the same.

• Write small, focused, isolated tests to avoid such issue.

Self-validating

• Your tests should be 100% automated.• If you need a ”small manual test” to arrange or to assert the test, it is not

automated anymore.

• We want our tests to be automatically executed by machines without any human intervention.

Timely

• You should write tests all day long.• Do not wait until the end to start writing tests.• Your team should understand the importance of such thing.

• ”peer pressure” helps.

A ATest

B BTest

C CTest

That’s what we call Unit Testing.

Some definitions

• ISQTB: “Searches for defects in, and verifies the functioning of software items (e.g., modules, programs, objects, classes, etc) that are separately testable”.

• Osherove: “A unit test is an automated piece of code that invokes a unit of work in the system and then checks a single assumption about the behavior of that unit of work. [...] A unit of work is a single logical functional use case in the system that can be invoked by some public interface (in most cases). A unit of work can span a single method, a whole class or multiple classes working together to achieve one single logical purpose that can be verified.”

Advantages Disadvantages

• Very fast• Easy to control• Easy to write

• Less real• Some bugs can’t be

reproduced at such level

When do we need more reality?What can we do to gain reality?

BigFeatureTest

We can do System Testing!

Advantages Disadvantages

• Very realistic• Captures the user

perspective

• Slow• Hard to write• Flaky

Testing pyramid

Unit tests

Integration tests

System tests

Manual

Mor

e re

ality

Mor

e co

mpl

exity

How I (Maurício) do the trade-off

All business rules should betested here.

Avoid at all cost. But do itwhen needed.

Complex integrations with external services.

Main/Risky flow of the apptested.

Unit tests

Integration tests

System tests

Manual

”Whenever you are tempted to type something into a print statement or a debugger expression, write it as a test instead”.

--- Martin Fowler

License

• You can use and share any of my material (lecture slides, website).• You always have to give credits to the original author.• You agree not to sell it or make profit in any way with this.

• Material that I refer has its own license. Please check it out.

Images in this presentation

• Female engineer by GDJ: https://openclipart.org/detail/230143/female-engineer-9

• Lady bug, by Scout: https://openclipart.org/detail/191074/lady-bug• Business man, by GDJ:

https://openclipart.org/detail/279728/business-presentation• Rocket explosion, by bf5man:

https://openclipart.org/detail/203747/rocket-explosion