The world leader in serving science

Andrei Rădulescu

4-Oct-2017

Renaissance: Tools for Large-Scale Code Refactoring

2 Public

Introduction

3 Public

• Multi-national company

• Precision laboratory equipment and services

• 65000 employees worldwide

• >$18 billion revenue

• Via the 2016 acquisition of FEI Company,

>65 years of experience in electron microscopy

• Main markets

• Science: Material science, Life science

• Industry: Electronics, Oil & Gas

• R&D activities

• Hillsboro, US

• Eindhoven, The Netherlands

• Brno, Czech Republic

• Bordeaux, France

Introduction to Thermo Fisher Scientific

Elemental map of a RAM cell

Phase image of graphene atomsDiffraction pattern from an icosahedral quasicrystal

3.8 Å Cryo-EM structure of Zika virus

4 Public

• High-end transmission electron microscopes (TEM)

• R&D, production, service, sales, marketing, etc.

• Multi disciplinary R&D

• Physics

• Electronics

• Mechatronics

• Software

• R&D Eindhoven

• ~250 engineers

• ~110 software engineers including outsourcing

Thermo Fischer Scientific – Eindhoven

5 Public

Introduction to TEM

Key figures• HT: 300 kV• Vacuum• Drift < 5 Å/min (3.8M years/km)

– Noise (Acoustic)– Vibrations / Pressure– Stray fields (EMI)– Temperature

• Objective Lens: 11 A ± 20 mA• Magnification: > 1 Mx

– “See a tennis ball on the moon”

– Atomic resolution: < 1 Å

• Detection– Imaging (TEM)– Scanned imaging (STEM)– X-ray (EDX)– Energy loss (EELS)

electron source

condenser lenssystem

specimen

projector lenssystem

fluorescent screen & cameras

objective lens

Vacuum

6 Public

TEM Software Landscape

PC

TEM Server

Emb

ed

de

d

pla

tfo

rms

SCU CAPPSCU CAPP

ApplicationsProvides general purpose UIor tailored for specific use-case

TEM serveroffers unified API to control all microscope configurations

Embedded platformsReal-time control of microscopeand processing of detector data

HardwareElectron gun, optics, vacuum, detectors, etc.

C# (WPF), C++ (Qt or MFC)x100k LOC / application

C++ & COMx1M LOC

C++, ThreadX, Linux, FPGAx100k LOC / platform

x100 hardware configurations,including specials

7 Public

Need for Large-Scale Refactoring

8 Public

• Complex software code base

• Reflects the complexity of the system

• Has evolved over tens of years, inevitably accumulating technical debt

• Increasing costs over time for both adding features and maintenance

• Addressing technical debt

• Manual refactoring or rewriting software hard to justify to business

• Tools can increase the speed and efficiency of large-scale refactoring of legacy code

Rationale

9 Public

• TEM Server has grown over years into a complex code base of ~6000 KLOC

• One huge component: 3300 kLOC large build / smoke time, multiple stream delivery / rebase overhead

• Large / complex dependency structure large integration overhead

• COM overused + issues large overhead to change / propagate interfaces; multiple interface registration

• End-user applications, service tools use internal interfaces lifecycle issues, system safety is bypassed

• etc

• We must transform the TEM Server code to enable a more efficient code development

• Decompose TEM into smaller components, simplify the dependency structure between and within components

• Each module should have a well-defined and tested interface

• Solve COM issues, reduce COM usage

• Port applications / tools to external interfaces

• etc

• Such large-scale transformations must be efficient, consistent and repeatable

• Use model-based refactoring tools Renaissance

Legacy Code

10 Public

Renaissance

11 Public

• Public / Private funded project lasting 2 years (September 2016 – August 2018)

• Partners

• Thermo Fisher Scientific: Lead industrial partner

• TNO-ESI: Applied research partner

• Cornerstone BV: Software quality measurement & analysis specialists

• Goal: Tools to analyze, visualize and refactor a large C++ / C# / COM code base

• Approach

• Semi-automated

• Most repetitive tasks can be easily automated; remaining one-offs are done manually

• Iterative

• Lower risks (no “big bang”)

• Visible project progress

• Early tool validation (“learn as we go” – e.g. task is repetitive enough to warrant automation)

About the Renaissance Project

12 Public

Analysis and Transformation of Code

Legacy code

(today)

Improved code

(tomorrow)

Extr

act

vis

ual

model

analyze

Restructure

Auto-rewrite

[Repeat]

(Valid

ate

)

chaos order

13 Public

• Unnecessary COM complexity requires cleanup

• Suboptimal design choices due to early adoption

• Complex interdependency structure = maintenance overhead

• Cleanup is a prerequisite for SW decomposition

• Business case

• 3x lower refactoring effort

• First-time right due to code generation

• Tools reusable in the next project phases

Renaissance #1: COM Cleanup

Refactoring Effort

0 50 100 150 200 250 300

Manual only

Renaissance

Effort extrapolation from manual interface cleanup

Spent

men/weeks

14 Public

Visual Analysis Using Neo4j Graph Database + Cypher Queries

Chaos (entangled code):

refactoring required

Well-structured projects Input: existing code base

16 Public

Renaissance Interface-Grouping Tool + yEd Visualization

• Interface grouping into

libraries

• Automatic procedure,

based on interface

implementation project

• Interface dependencies

are also identified

• Interface users

• Interface abusers(include idl i.o. import tlb)

• Subject to cleanup

17 Public

• Improve TEM-server architecture

• Make dependencies explicit through interfaces

• Reduce module coupling through interfaces (and interface versioning)

• Interface abstractions allow platform strategies around product families

• Reduce complexity by removing cross-module dependencies

• Smarter testing (e.g., component-level tests, integration tests based on use cases)

• Speedup TEM-server build time 25x (6 hr 15 min)

• Break large components smaller components

• Break long dependency chains interface components

• IncrediBuild, incremental builds

• Decomposition is a complex, significant-effort, tedious activity

Automate and guarantee quality with Renaissance tools

TEM: Decomposition

TAD

AutoLoader

Motion

TEM_TAD

TEM

Acquisition

TIA

TEM_HAL

VACUUM

HEP

TEM_ISO

IMAGING

AutoStar

LICENSING

Smoke tests

18 Public

Break long dependency chains

Break large components

• Renaissance (building on results of Phase 1)

• Identify and move module projects to separate components

• Refactor clients to use the new interface locations

Renaissance #2: Decomposition

TEM Test

TAD

AutoLoader

Motion

TEM_TAD

TEM

Acquisition

TIA

TEM_HAL

VACUUM

HEP

TEM_ISO

IMAGING

AutoStar

LICENSING

TEM_TAD ITEM_TAD

TEM ITEM

Acquisition IAcquisition

TIA ITIA

IMAGING IIMAGING

LICENSING ILICENSING

AutoStar IAutoStar

TEM_HAL ITEM_HAL

VACUUM IVACUUM

AutoLoader IAutoLoader

Motion IMotion

TAD ITAD

HEP IHEP

TEM_ISO

Acq Test

Lic Test

Imaging Test

TIA Test

TemHal Test

Vacuum Test

TAD Test

Motion Test

AL Test

HEP Test

TEM Test

AutoStar Test

TemTad Test

Va

cM

R

Op

tic

s

Va

cu

um

Sh

are

d

Va

cH

E

IVac

Op

t M

R

IOpt

Gu

nG

un

MR

IGun

Mo

tio

n

Sh

are

d

IMot

Mo

tio

n

TE

M

Au

to L

oa

de

rIAL

An

cil

lary

IAnc

TEM Control BHV

IOM

PeoUI, Service Tools, etc

TEM

19 Public

Renaissance yEd-based Exploration / Decomposition

• Specify new components

at project level

• The tool identifies

component dependencies

• Interface implementations

• Interface usages

• Circular dependencies are

marked red

TEM_COMMON TEM (legacy)

GUN (new) GUN_ITF (new)

20 Public

• Speed + efficiency

• Decouple teams interfaces + versioning

• Short build + test feedback cycles (wait time = waste)

• Early integration (see our last year presentation)

• Tools to automate repetitive work

• DSL + tools to increase abstraction

• Quality

• Extensive + smart testing

• Tools to (formally) verify SW correctness

• Allow (tool-based) refactoring to avoid building technical debt

• Tools to measure the code-base health

• Keep the measured scores up

Conclusions

SW Mission: Develop correct functionality fast. Be able to deliver high-quality SW at any time.

21 Public

Thank You