bob harnisch & tim koomen - mixing waterfall, agile & outsourcing at dutch rail - eurostar...
DESCRIPTION
EuroSTAR Software Testing Conference 2013 presentation on Mixing Waterfall, Agile & Outsourcing at Dutch Rail by Bob Harnisch. See more at: http://conference.eurostarsoftwaretesting.com/past-presentations/TRANSCRIPT
Bob Harnisch, ProRailTim Koomen, Consultant
Mixing
Waterfall, Agile and Outsourcing
at Dutch Rail
www.eurostarconferences.com
@esconfs#esconfs
Who am I
◦ Bob Harnisch
◦ Manager Testservices at ProRail
ProRail
◦ Railway company in The Netherlands
◦ Provides a secure, reliable, punctual and sustainable rail network
◦ Develops and maintains rail traffic control systems for rail traffic controllers
◦ Problem:
Control systems must Go-Live first-time-right!
Multiple vendors and suppliers, mixed project approaches
ICT Services
ICT Projects
TestservicesProject
managers
Planning
Control
Messaging
Security
Infra elements
• Real-time control systems• Safe and reliable• High available systems 24/7• Multiple system chains• Legacy and new• Mixed platform (VAX VMS, Linux)
Testservices◦ policy: formal testing department for IT projects
◦ involved at project start-up
◦ responsible for acceptance testing
◦ knowledgeable and experienced test experts
Structured test approach tailored to ProRail
Acceptance Test Centre Modelpost (close to 100% representative)
Technical, complex, real-time, high-availability system environments
Non-functional testing very important
But also office automation (SAP, Sharepoint, GIS)
Different project approaches◦ Policy: outsourcing of SW development
◦ Agile/scrum approach
◦ Waterfall approach
(Many) external SW suppliers◦ Little business knowledge
◦ No representative test environment
Changing architectures◦ From old legacy to SOA
Testing is vital for operation◦ Installation window 01:00 – 04:00 hrs weekend night
1st time right => next window in 6 months
◦ Any installation or operational failure:
We’re instantly on the news!
How to cope with issues …?
Anyone here that can give me advice……?
Tim Koomen, independent testconsultant
8
Baseline of testing is good
Last few years explosive increase of agile/scrum development at suppliers. Questions:◦ Is separate acceptance testing still necessary?
◦ If so, how to align planning of sprints with acceptance tests at test centre
◦ How to coordinate all testing activities?
◦ Controlling the quality of testing?
Shift to Service Oriented Architectures (ESB, Tibco)◦ Different testing?
◦ Other measures?
9
Subsysteem Ontwerp
SysteemSpecificatie
Systeem Ontwerp
Gebruikerswensen
SubsystemenIntegratie
Business Beleid Acceptatietest
(Pilot)
Busi-nessCase
Context
VL
(OCD
Post21)
Proces
boek
VL
Proces
boek
VL
- Proces-
sen (SSS
Post21)
-KPI’s
Proces
boek
VL
-Bedrijfs-
objecten
(SSDD
Post21)
-Externeeisen
- wetgeving- e.d.
OCD
BeheerProcedures
SSS
(Use
Cases)
IRS
(externe
systemen)
SSDD
IDD
-Organi-satie en
Geautoma-tiseerdeOnderst.
Project
Brief
SubsysteemSpecificatie
Systeem
Systeem =>Subsystemen
SRS
IRS
sub
systemen
Subsystemen
LocatieSpecifiekeTest
Subsystemen
SDD DBDD IDD
STD STR
STD STR
STD STR
STD STR
STD STR
STD STR
STD STR
STD STRPilotPlan
Vrijgave
BusinessRequirements
BusinessOntwerp
MTPsysteem
STPsysteem
UnitCoderen &
Test
MegaIntegratieTest
ValidatietestSysteem
STPSysteem/
Sub-systemen
Decharge leverancier
Systeem
Knip
HLOB
IRS
sub
systemen
SysteemKwalificatieTestIntegratie
TestSubsystemen
SubsysteemKwalificatieTest
Review
Review
Legenda:• ProRail architecten,
ontwerpers of andere ProRailonderdelen, in ieder geval buiten het werkgebied van TS
• project, ontwerpers • leverancier• integrator• Test Services
10
11
Business
policy
Business
requirements
Business
design
Architecture
design
Acceptance
test
Goinglive
Scrum
development
and system
test
YES, so far ProRail still needs it
Representative test environment Testing non-functionals Testing the chain of systems: at ProRail 1st time large scale system integration
Operational management tests Many suppliers, many scrum teams: quality not always predictable
Few demands on supplier testing: mostly deliverables like test plan, design and report
Suppliers have little business knowledge Configuration management of application/environment: not always as it should
12
Business
policy
Business
requirements
Business
design
Architecture
design
Scrum
development
and system
test
Acceptance
test
Going
livePoC
or
Pilot
Shadow run
13
Business
policy
Business
requirements
Business
design
Architecture
design
Scrum
development
and system
test
Acceptance
test
Going
livePoC
or
Pilot
Shadow run
14
Typical scrum tests:
Unit testCode reviewFunctional test (user story test)(Service) integration testNon-functional testsRegression test
Typical acceptance tests:
(Service) integration testValidation testChain testUATNon-functional testsLocation specific testOperational acc.test
15
Business
policy
Business
requirements
Business
design
Architecture
design
Going
livePoC
or
Pilot
Shadow run
System
requirements,
user stories
System-
design &
specification
Coding
Code reviews
Unit tests
Non-
functional
tests
(automated)
Regression
test
Functional
(qualification)
tests
(Service)
integration
tests
Validation
Test
Chain test
Location
Specific
Test
User
acceptance
test
Non-
functional
tests
Deliverables:
• Testplan
• Test scripts
• Test report
(Service)
Integration
Tests
Architect / designer
Business architect
Deliverables:
• Project Start Architecture
Product Owner / Supplier/ ProRail DevelopmentTeam
Definition of Done / deliverables:
• Test scripts
• Test Report
• User Manual
• Installation Guide
Testservices
Integrator
cafetaria
Architectural changes to SOA
Measure 1: test effort on services, non-functionals, platform changes◦ Experience:
non-functional requirements missing or not well described (architecture documentation )
(Lack of attention for) non-functionals is a major cause for project delay
Improve quality at beginning of project
Measure 2: improve review process◦ We have initiated an improved review process, together with architects
◦ Review the Product from 4 Directions:
Up: Norms, standards, reference architecture
Left: Consistency with other system documentation
Right: Usable (for users), testable (testers)
Down: Designable, Buildable, Implementable
◦ Use of roles versus directions
(Peer) architect reviews architecture document from Up and Left
Designer/ Tester/ Developer: reviews from Down, Left, and Right
Sr user reviews from Right
◦ Reviews form includes checklist per role
16
4DR
17
Business
policy
Business
requirements
Business
design
Architecture
design
Going
livePoC
or
Pilot
Shadow run
System
requirements,
user stories
System-
design &
specificatio
n
Coding
Code reviews
Unit tests
Non-
functional
tests
(automated)
Regression
test
Functional
(qualification)
tests
(Service)
integration
tests
Validation
Test
Chain test
Location
Specific
Test
User
acceptance
test
Non-
functional
tests
(Service)
Integration
Tests
Architect /
designer
Business
architect
Product Owner / Supplier/ ProRail DevelopmentTeam
Test services
Integrator
4DR4DR
4DRcafetaria
Deliverables:
• Project Start Architecture
Deliverables:
• Testplan
• Test scripts
• Test report
Definition of Done / deliverables:
• Test scripts
• Test Report
• User Manual
• Installation Guide
Business
policy
Business
requirements
Business
design
System
requirements,
user stories
System-
design &
specification
Coding
Code
reviews
Unit tests
Non-
functional
tests
(automated)
Regression
test
Functional
(qualification)
tests
4DR
(Service)
integration
tests
Validation
Test
Chain test
Location
Specific
Test
User
acceptance
test
Non-
functional
tests
Going
live
PoC
or
Pilot
Shadow running
Test services
Architect /
designer
Business
architect
Users
Deliverables:
Testplan,
testscripts,
Testreport
Definition of Done / deliverables:
Testscripts, testreport, user maual,
installation guide
Legenda:• ProRail architects, designers or other
ProRail roles, outside scope Test services
• project, developers• Supplier, ProRail product owner• Test Services
(Service)
Integration
TestsArchitecture
design
4DR
4DR
Deliverables:
Project Start
Architecture
Integrator
Product Owner / Supplier/ ProRail DevelopmentTeam
18
cafetaria
Business
policy
Business
requirements
Business
design
System
requirements,
user stories
System-
design &
specification
Coding
Code reviews
Unit tests
Non-
functional
tests
(automated)
Regression test
Functional
(qualification)
tests
4RR
(Service)
integration tests
Validation
Test
Chain test
Location
Specific
Test
User
acceptance
test
Non-
functional
tests
Going
live
PoC
or
Pilot
Shadow running
Test services
Architect /
designer
Business
architect
Users
Master Test PlanMTP
Legenda:• ProRail architects, designers or
other ProRail roles, outside scope Test services
• project, developers• Supplier, ProRail product owner• Test Services
(Service)
Integration
Tests
Architecture
design
4RR
4RR
Integrator
Product Owner / Supplier/ ProRail DevelopmentTeam
19
cafetaria
Deliverables:
Testplan,
testscripts,
Testreport
Definition of Done / deliverables:
Testscripts, testreport, user maual,
installation guide
Deliverables:
Project Start
Architecture
Implement it Train testmanagers, implement in projects
Build up experience 1st Project
2nd Project
……
Next Project
Some time later….
20
• Satisfied with– Roll-out and acceptance of model– Good fit to current practice, clarifies testing– Early identification & alignment of test activities at suppliers and ProRail
gives flexibility in finding test solutions– Early involvement– Coaching of test managers – Largest changes: choice and frequency of tests, tighter control of
supplier tests and working in or with scrum teams
• To be improved– 4 Direction Reviews, as Testservices is not the natural owner– Influencing and controlling the supplier tests
• Not a test issue– Agile / scrum in general: product owner is the (weakest!) link …– … and the hardest role!
21
Product owner
Stakeholders
Scrum team
Waterfall
Agile
Combined
United
Outsourcing
Testing
ProRail
…
22
WAUW
Business
policy
Business
requirements
Business
design
System
requirements,
user stories
System-
design &
specification
Coding
Code reviews
Unit tests
Non-
functional
tests
(automated)
Regression test
Functional
(qualification)
tests
4RR
(Service)
integration tests
Validation
Test
Chain test
Location
Specific
Test
User
acceptance
test
Non-
functional
tests
Going
live
PoC
or
Pilot
Shadow running
Test services
Architect /
designer
Business
architect
Users
Master test planMTP
Legenda:• ProRail architects, designers or
other ProRail roles, outside scope Test services
• project, developers• Supplier, ProRail product owner• Test Services
(Service)
Integration
Tests
Architecture
design
4RR
4RR
Integrator
Product Owner / Supplier/ ProRail DevelopmentTeam
23
cafetaria
Deliverables:
Testplan,
testscripts,
Testreport
Definition of Done / deliverables:
Testscripts, testreport, user maual,
installation guide
Deliverables:
Project Start
Architecture
ProRail’s test solution for hybrid situations: WAU! - testmodel
Acceptance testing is a part of this solution but only if it delivers value
Flexibility in choosing the required tests coordinated by a Master Test Plan
Early test involvement in reviews 4DR
Model for communication at start project and for awareness for testers and rest of project organisation
In future to 100% agile?
We’re ready!
24
25
Between 40-60% of IT-projects use scrum
Always a project manager
Product owners: usually 2, one big project 5
Product owner: mostly an information analist or architect
Number of suppliers: 1-2
Number of scrum teams: 1-4
Test role(s) in scrum team? Usually, but not always. If yes, then usually from supplier, sometimes from ProRail
Frequency of acceptance testing: from 1 x at the end to almost each sprint
27