software testing with python

7/27/2019 Software Testing With Python

http://slidepdf.com/reader/full/software-testing-with-python 1/35

Software Testing with Python

Magnus LyckåThinkware AB

www.thinkware.se

EuroPython Conference 2004Chalmers, Göteborg, Sweden

© 2004, Magnus Lyckå



In the next 30 minutes you should...

● Learn about different aspects of software testingsuch as unit tests and acceptance tests.

●

See how the standard modules for unit tests areused, learn about some alternatives and have agrasp of their respective pros and cons.

● Know about some options for acceptance tests.

● Know about some standard Python modules andthird party tools that are helpful in softwaretesting and related activities.



What is software testing?

● “Testing means verifying that your code isrunning correctly by exercising the code under known conditions and checking that the results

are as expected.” – Alex Martelli, Python in a Nutshell

● If we are serious with our requirements on asoftware system, we should make sure that we

verify them, and testing is the most common kindof verification for software.



Software testing in a context

Software Quality Assurance

Software Verification

Software Testing



Automating tests

● Writing automated tests is usually more work than testing things manually once, but they makeit much easier to...

– Work in a repeatable and predicable way

– Run tests more often

– Run tests unattended

– Find regression bugs

– Test more operating systems

– Manage the test load as the system grows



Example of requirements and

tests on different levels

System Specification Acceptance Test

Detailed Design

API Specification

Unit Test

Integration Test



Requirements and tests on different

levels in Extreme Programming (XP)

System specified through customer tests

Design by writing

API validated through

programmer tests

continuous integration



Consequences of the XP approach

● Requirements are written in a verifiable way.

● No discrepancies between requirements and tests.

●

We get unambiguous and repeatable ways of verifying requirements.

● Continuous picture of project progress

But...● Customers must be able to understand the tests.

● We need test automation frameworks.

●

Automated tests → more software to maintain.




Unit testing



Unit tests in Python

● Unit tests, or programmer tests, are tests the programmer writes to make sure that the codedoes what the programmer intended it to do.

● Unit tests can also help document how a piece of code (e.g. a class) is supposed to be used.

● Python has two standard modules for unit testing,

unittest and doctest.

● There are several third party modules that aim tocomplement or replace unittest.



The unittest module

● The Python unittest module is modeled after

the unit test modules developed within the XPcommunity by Kent Beck and Erich Gamma.

● It's intended for a test-first approach, where testsare written before the actual code that it tests.

● Always writing tests first is a bit like always

eating your meat before you eat your dessert...

● When your tests pass, you're done. (XPerscombine this with refactoring.)



Concepts in unittest

● A test fixture consists of the actions needed tosetup for a test, and clean up afterwards.

●

Atest case

is the smallest unit of testing. It mightfor instance call a function and check the results.

● A test suite is a collection of test cases and/or testsuites that should be executed together.

● A test runner runs test suites (or individual testcases), collects the results and presents it as textor graphically to the user.



The TestCase class

● Test cases sharing a common fixture will beimplemented as methods (whose names start withtest) in a sub class of unittest.TestCase.

● The test runner will create one instance object per test case, and run SetUp, followed by the test casemethod and finally tearDown.

● Checks are made with the methods fail, failIf,assert_/failUnless, assertEqual/failUnlessEqual,assertNotEqual/failIfEqual andassertRaises/failUnlessRaises.



The test runner

● In many cases, running tests is as simple asexecuting unitest.main() in the file

containing the test cases.

● Testing progress will be reported, and the testrunner makes a distinction between FAILUREand ERROR.

– FAILURE means that the check didn't produce theexpected result.

– ERROR means that something else in the test wentwrong. (I.e. You got an exception.)



Unittest template

import unittest import MyModule

class MyModuleTest(unittest.TestCase): def testOneCase(self): pass

if __name__ == '__main__':unittest.main()



Unittest examples

● divtest.py

– Trivial example to demonstrate features of unittest.

●

container_ut.py – Real world example from SystemSpecifyer (see

http://sourceforge.net/projects/systemspecifyer)



The doctest module

● The Python doctest module was written by

Tim Peters to check that coding examples in doc-strings were correct, by testing them.

● It has also found use as a more general unittesting module, and from Python 2.3, this has been made more convenient.

● Using doctest helps you combine tests anddocumentation in a more readable way thanunittest does.



Doctest example

def divide(a,b): """

Return a divided with b.

Divide will return the result of an integer division>>> divide(6,3)

2

Division by zero is handled as expected...>>> divide(1,0)Traceback (most recent call last):

...ZeroDivisionError: integer division or modulo by zero

"""

result = a / b return result

if __name__ == '__main__': import doctest, sys

doctest.testmod(sys.modules[__name__])



Unittest vs doctest

● Unittest mantra: Test a little, code a little

– The logical choice if you use extreme programmingor test-driven development of some other kind.

– Maybe better for complex tests?

● Doctest mantra: Code a little, test a little

– The natural companion if you prefer to experiment

interactively with your classes and functions in thePython interactive environment.

– Probably better for adding tests to already writtencode.



Unittest vs doctest on the web

● Charming Python: Testing frameworks in Python

– http://www-106.ibm.com/developerworks/linux/library/l-cptest.html

● Literate Testing: Automated Testing with doctest

– http://www.python.org/pycon/dc2004/papers/4/

● DocTest at WikiWiki

– http://c2.com/cgi/wiki?DocTest



Other contenders...

● Sancho – a unit testing framework from MEMSExchange. Adds coverage analysis etc.

– http://www.mems-exchange.org/software/sancho/

● Peckcheck by Darius Bacon.

– http://www.accesscom.com/~darius/software/clickcheck.html

● The “second standard library” std.

– Here, 11:00 today...



Unittest complements

● Pester – the Python version of Jester.

– It finds code that is not covered by tests, makes somechange to your code, runs your tests, and if the tests

pass it displays a message saying what it changed. – http://jester.sourceforge.net/

● Test coverage support

–

E.g. http://www.garethrees.org/2001/12/04/python-coverage/● Mock objects: Here, 9:30

● A better test runner: Here, 10:00




Integration testing



Python Integration Test

● No static linking. We must run code to testinterfaces! (On the other hand, compilers andlinkers fail to see lots of integration problems...)

● Since building/linking isn't an issue in a purePython project, we can use either unit test tools or acceptance test tools for integration tests.

● Python is good at gluing things together, and thushelpful in all sorts of integration work. See e.g.http://www.thinkware.se/cgi-bin/thinki.cgi/UsingPythonWithOtherLanguages




Acceptance testing



Test Frameworks using Python

● QMTest – Uses Python for test expressions.

– http://www.codesourcery.com/qmtest/

●

PyFIT – Python clone of Ward Cunningham'sFramework For Integrated Testing (FIT).

– http://www.xprogramming.com/software.htm

● Software Testing Automation Framework (STAF)Big framework from IBM with Python API.

– http://staf.sourceforge.net/

● TextTest – See more here, today 11:30!




Not quite software testing...



The usual suspects...

● Python is excellent for analysis and manipulationof data. A great tool for test related work.

● The re library is useful but use special tools

when available, such as for parsing XML files.

● For dealing with files you might use the modulesgzip, zipfile, codecs, filecmp,

struct, sgmllib, xml.* etc● Python interfaces well with internet services,

database, and all sorts of external programs.



difflib

● More or less like the unix diff utility, but as a

Python module. (New in version 2.1.)

● Useful when we want a more detailed responsethan FAILED from a test – particularly if we aretrying to spot small changes in big amounts of data.

● Great for spotting differences in configurations,for instance database schemas.



Difflib example

>>> import difflib; d=difflib.Differ()>>> diff =d.compare(['Hello World', "This is the same.",

"Time flies like an arrow. Isn't that great?"],['Hello World!', "This is the same.","Fruit flies like a banana. Isn't that great?"])

>>> print "\n".join(diff)- Hello World + Hello World!? +

This is the same.- Time flies like an arrow. Isn't that great?

? ^ ^^ - ^^^^

+ Fruit flies like a banana. Isn't that great?? ^^^ ^^ +++ ^^



AT&T Graphviz

● Tools to generate graphs from C-like text files.

– Dot – for directed graphs.

– Neato – for undirected graphs

– Fairly clever algorithms for decent looking layout

– Generates graphs in many file formats

– Suitable for automatic generation of graphs

– Some control over placement is possible

● Great for dependency analysis!

● Not Python, but good anyway... :)



Remember this?

System Specification Acceptance Test

Detailed Design

API Specification

Unit Test

Integration Test



AT&T Graphviz example 1

digraph G {label = "V-model";

sysspec[label="System Specification"];

systest [label="Acceptance Test"];apispec [label="API Specification"];apitest [label="Integration Test"];unitspec [label="Detailed Design"];unittest [label="Unit Test"];

sysspec -> apispec -> unitspec;unittest -> apitest -> systest;sysspec -> systest;apispec -> apitest;unitspec -> unittest;

}



AT&T Graphviz example 2

digraph G {label = "V-model";node [shape=box, style=filled,

color="#CCCCFF"];sysspec [label=

"System Specification"];apispec [label="API Specification"];unitspec [label="Detailed Design"];node [shape=box, style=filled,

color="#FFFF99"];systest [label="Acceptance Test"];

apitest [label="Integration Test"];unittest [label="Unit Test"];

{rank = same; sysspec; systest};{rank = same; apispec; apitest};{rank = same; unitspec; unittest};

...



Useful Books

● Python in a Nutshell, by Alex Martelli

● Text Processing in Python, by David Mertz

●

Software Test Automation, by Fewster & Graham● Just Enough Software Test Automation, by

Daniel J. Mosley & Bruce A. Posey

●

Testing Extreme Programming, by Lisa Crispinand Tip House

● Test-Driven Development By Example, by KentBeck

software testing with python

Documents