agile systems engineering & agile at ska scale

Agile Software Engineering & System Engineering at SKA scale Juande Santander-Vela SKA SW Systems Engineer

SQUARE KILOMETRE ARRAYExploring the Universe with the worlds’ largest radio telescope

Astronomical Data AnalysisSoftware & Systems

(on behalf of the SE team)

SKA Key Science Drivers: The history of the Universe

Cosmic Dawn(First Stars and Galaxies)

Galaxy Evolution(Normal Galaxies z~2-3)

Cosmology(Dark Energy, Large Scale Structure)Cosmic Magnetism

(Origin, Evolution)

Cradle of Life(Planets, Molecules, SETI)

Testing General Relativity(Strong Regime, Gravitational Waves)

Exploration of the Unknown

Broadest range of science of any facility, worldwide

SKA Observatory Vision

Exploring the Universe with the worlds’ largest radio telescope

3 sites 2 telescopes 1 observatoryDesign Phase:

~ €200M; 600 scientists+engineers, 80% complete

SKA Phase 1 (SKA1) Construction: 2019 – 2025

Construction cost cap: €674.1M (2016 inflation-adjusted)

Operations cost: (estimate) €89M/yrMeerKAT integrated

Observatory Development Programme (€20M/year planned)

SKA Regional centres out of scope of centrally-funded SKAO.

SKA Phase 2: start mid-2020s ~2000 dishes across 3500km of

Southern AfricaMajor expansion of SKA1-Low

across Western Australia

>50 years lifetime! Drives need for reliability,

and adaptability

SKA Observatory Vision

3 sites 2 telescopes 1 observatoryDesign Phase:

~ €200M; 600 scientists+engineers, 80% complete

SKA Phase 1 (SKA1) Construction: 2019 – 2025

Construction cost cap: €674.1M (2016 inflation-adjusted)

Operations cost: (estimate) €89M/yrMeerKAT integrated

Observatory Development Programme (€20M/year planned)

SKA Regional centres out of scope of centrally-funded SKAO.

SKA Phase 2: start mid-2020s ~2000 dishes across 3500km of

Southern AfricaMajor expansion of SKA1-Low

across Western Australia

>50 years lifetime! Drives need for reliability,

and adaptability

SKA Organisation

! Australia (DoI&S)" Canada (NRC-HIA)# China (MOST)$ India (DAE)% Italy (INAF)& Netherlands (NWO)' New Zealand (MED)( South Africa (DST)) Sweden (Chalmers)* UK (BEIS/STFC)

In discussion with: + Germany , France - Portugal . Spain / Switzerland 0 Japan 1 South Korea

In the process of becoming an Inter-

Governmental Organisation

SKA1 Design Consortia

Software

Elements

~6-10 × FTE

wrt SKAO

Filter ConsortiaAssembly, Integration & Verification

Central Signal Processor

Infrastructure Australia

Infrastructure South Africa

Low-Frequency Aperture Array

Mid-Frequency Aperture Array

Science Data Processor

Signal and Data Transport

Telescope Manager

Wideband Single Pixel Feeds

Check/uncheck all

GMT+12

19 hour timezone span

SKA1-Lowlocations

1600km

➤➤

NDataSIO,NOAA,U.S.Navy,NGA,GEBCODataSIO,NOAA,U.S.Navy,NGA,GEBCO

ImageLandsat/CopernicusImageLandsat/Copernicus

SKA1-Midlocations

➤➤

NImageLandsat/CopernicusImageLandsat/Copernicus

DataSIO,NOAA,U.S.Navy,NGA,GEBCODataSIO,NOAA,U.S.Navy,NGA,GEBCO

SKA1 Sites

SKA1-LOW: 50 – 350 MHz Phase 1: ~130,000 antennas across 65km

SKA1-Mid: 350 MHz – 24 GHz Phase 1: 200 15-m dishesacross 150 km

SKA1 Telescopes

SKA1-LOW: 50 – 350 MHz Phase 1: ~130,000 antennas across 65km

SKA1-Mid: 350 MHz – 24 GHz Phase 1: 200 15-m dishes

across 150 km

SKA1 Notional Data Flow

2 Pb/s 7.2 Tb/s

7.2 Tb/s

8.8 Tb/s8.8 Tb/s

~50 PFLOPS

~5 Tb/s

~250 PFLOPS

Long Term Storage

300 PB/year

LFAA Data Network Digital Data Backhaul

Digital Data Backhaul

CSP-SDP Network

2 Pb/s 7.2 Tb/s

7.2 Tb/s

8.8 Tb/s8.8 Tb/s

~50 PFLOPS

~5 Tb/s

~250 PFLOPS

Long Term Storage

300 PB/year

CSP-SDP Network

2 Pb/s 7.2 Tb/s

7.2 Tb/s

8.8 Tb/s8.8 Tb/s

~50 PFLOPS

~5 Tb/s

~250 PFLOPS

Long Term Storage

300 PB/year

CSP-SDP Network

2 Pb/s 7.2 Tb/s

7.2 Tb/s

8.8 Tb/s8.8 Tb/s

~50 PFLOPS

~5 Tb/s

~250 PFLOPS

Long Term Storage

300 PB/year

CSP-SDP Network

Similar to LOFAR

in flexibility.

See O4.2 by

Adriaan Renting

How to develop so much software?

Agile, Scaled Agile

(Framework)

SAFe for Lean Enterprises Large Solution SAFe®

Agile Teams, 5-7 people (including Scrum Master), ±2 people

Agile Release Trains (ARTs) in Definition

https://scaledagileframework.com/

Agile Manifesto

• Individuals and interactions over processes and tools

• Working software over comprehensive documentation

• Customer collaboration over contract negotiation

• Responding to change over following a plan

http://agilemanifesto.org

The emphasis is on Quality

Nick Rees talk I10.1 (Thursday 26th, 9:00 AM)

What about Systems Engineering?

INCOSE, 2004

“Systems engineering [SE] is an interdisciplinary approach and means to enable the realisation of successful systems […] [SE] integrates all the disciplines and specialty groups into a team effort forming a structured development process […] from concept to production to operation. [SE] considers both the business and the technical needs of all customers with the goal of providing a quality product that meets the user needs.”

INCOSE, 2004

David Long, CEO Vitech Corp

“The role of Systems Engineers is to encode and translate engineering to all stakeholders.”

Nick Rees, SKAO Head of Computing

“Systems Engineers capture and remind us of all the bullshit stakeholders say about

the system.”

Juande Santander-Vela, SKAO Software SE

“Systems Engineers have to provide useful engineering artefacts to stakeholders, notice still missing definition, and supply it ASAP”

Freshly coined for

#ADASS2017CL!

Juande Santander-Vela, SKAO Software SE

“Systems Engineers have to provide useful engineering artefacts to stakeholders, notice still missing definition, and supply it ASAP”

Or assess and record the risk of

not doing it

Freshly coined for

#ADASS2017CL!

INCOSE, 2004

With an assessment of what are the risks against

quality and user needs

SE must be Agile, or it provides diminished value

Agile Manifesto

• Individuals and interactions over processes and tools

• Working software over comprehensive documentation

• Customer collaboration over contract negotiation

• Responding to change over following a plan

http://agilemanifesto.org

Agile SE

• Keep communication with all stakeholders involved in the project, with speedy and clear dissemination of agreements

• Continuous integration of the engineering artefacts… and code, preferably self-documented

• Get devs (and SEs) that are invested, not just contracted → Gain‑share frameworks

• Have systems & processes that accommodate system (and code) change and prove requirements are satisfied → don’t do document-centric requirements management

Agile SE

At all phases! Including design,

construction, commissioning, and

operations…

Don’t disband SE!

Agile SE

operations…

Don’t disband SE!

Agile SE

operations…

Don’t disband SE!

Agile SE

Partners, not

contractors/suppliers

mentioned by

JC Guzman on I2.1

operations…

Don’t disband SE!

Agile SE

Partners, not

contractors/suppliers

mentioned by

JC Guzman on I2.1

operations…

Don’t disband SE!

Agile-ish Practices at SKAO• SE planning• We have a full plan, from which the backlog is constructed• Progress reviewed every 2 weeks, with forward look to another 2 weeks

• Systems Modelling• Integrated System Model for SKA1_Common and SKA1_Low in Cameo SysML• Second model for SKA1_Mid in Vitech CORE• Used for ICDs, discussions

• SKA1_Mid and SKA1_Low Block Diagrams• Used for reference, updated in agile way, but low priority

Agile-ish Practices at SKA1

• Consortia work-planning• Agile for SDP, TM• LFAA uses agile for software development of AAVS1 prototype• Others want to embrace SAFe for construction, just not now

• Multimodal communication• Email, SMS, Confluence, JIRA, Jama comments, Slack, Vidyo, Skype…

Not so Agile-ish Practices at SKAO

• Engineering Change Process (ECPs) and Change Notes (CNs)• Bug, or Feature? Still TBD for ECPs → more time to reflect• Need for formality/signature/buy-in across consortia requires time; we need to

be able to move from as agreed to as designed before as signed… but how?

• Lack of JIRA/backlog culture → starting to improve

• Lack of self-awareness of performance metrics → starting to improve

Agile SE is not for free…

• Coordination/communication takes time!

• Self-organisation frees time from management, but needs assistance from monitoring → back to coordination!

• Automation of monitoring and communication can help… but still the effectiveness of the communication needs to be monitored.

a lot of

But it makes it easier to

focus on the right things

Conclusions

• Agile SE can be seen as a tailoring of SE; you still need to identify which are the processes that provide value to your system/organisation.

• It is a challenge to create teams that can work together, but making it on a cadence makes it easier to achieve it… through repetition.

• We have to improve a lot at SKAO… but we’re working on it.

• Agile for Agile’s sake is a fad; you become Agile it to build quality, and respond to mistakes, sooner.

Thanks!

SQUARE KILOMETRE ARRAY

Questions?

SQUARE KILOMETRE ARRAY

j.santander-vela@skatelescope.org

agile systems engineering & agile at ska scale

Engineering

large-scale adoption of agile development lessons...

scale enteprise agile and service mode

preparing agile for scale - van dusen

agile@scale, second chance

agile programmemanagement - agile business conference...

demystifying the role of (management) leadership in agile -...

creating agile tribes: the secret to succeeding with agile @...

agile governance at scale...2.3 agile governance and...

comparing ways to scale agile at agile product and project...

agile transformation at scale - itsm academy · agile...

flowcracker agile @ scale

how not to scale agile - tamara nation - agile israel 2014

large scale agile development practices

does agile scale to the portfolio?

agile at scale - agile orlando lightning talk

agile@scale: portfolio level

scale agile framework

introduction to agile teams at scale

challenges of implementing agile at scale

managing large scale agile transformation