4th ucaat poster automation at system level case tetra pmr ... · automation at system level: case...

Automation at System Level: Case TETRA PMR Network

UCAAT 2016

• Achievement (at System Level)� Initial feasibility study showed that 50% of test suites to be automatable� Continuous integration (CI) using test automation implemented and used� Night/weekend executions, collection of statistics.� In average 47 % of tests are automated in several test phases:

SW build (”smoke”), regression, upgrade and some load tests

• Lessons Learned• Choose suitable tools: test execution, result analysis, result reporting, etc.• Build test automation portable• Minimize the effor required for re-configuration• Build test automation maintainable• Test automation challenging to apply if product is not designed to be ”test

automation friendly”• Transparency• Not directly comparable with manual testing

•Introduction• System level end-to-end verification• Continuous Integration & Automation from Sub-system to

System level• Test tools set for Agile, CI process combining manual and

automated test

•Target• Improve feedback cycle in all test levels (Unit, component,

sub-system, system level)• Reduce the time of non-functional tests• Increase requirement coverage• Improve quality of products & SW Release• Define test automation process & best practices • Select, Define and Optimize interface between tools (CI,

Automation, Test mngt tools)

2014 2015 2016

automated tests

Test Automation at System Level

SW Release

Product

Test Automation

UnitIntegration

Acceptance (functional)Capacity (and other non-functional)

Manual

Application p

Application 1

Dispatcher

UserTerminal

Network Elements

Gateways

Generation ofTest Cases

MBT

Generation ofTest CasesManually

Execute

Analysis

Report

Test AutomationFramework

Test AutomationFramework

SW DevelEnvironment

Continuous Integration

(@Sub-System)

Continuous Integration (@System)

Automated testsLibrariesTest data

Manual tests

Libraries ofSW Rel of

SUT

SW Repositoryfor test at System level

SW Repositoryfor test at Sub-system level