Software Development& Software Quality

Lessons Learned – Knowing The Limits

Herwig Habenbacherherwig dot habenbacher at gmail dot com

Background: About Me• Writing Software since over 30 years• Broad Spectrum: PC Software, Supercomputers, Embedded Systems,

Cloud. Mainly Technical and Scientific Software• Highly interested in Programming Languages• Leading Software Group for Measuring Devices over 11 years:

customers must trust the Software or you are out of business• No legal need (SIL Level) but still verified some parts:

Used PolySpace, played with SPIN, Maude, Frama-C and Coq.Being aware of the limits …

Musings About Software DevelopmentSoftware Development is not the same for all needs!

The best metaphor I found yet is to compare it to the Construction Industry – their needs, material, planning, processes and tools.

What do we expect from the Construction Industry to get reliable buildings? How do they adapt to the changing environment?

Built for Eternity: PyramidsAre we building very long standing Software which will be unchanged nearly forever? Who use them over time? They use a lot of cheap material, primitive tools, even with not so complicated architecture.It is pretty hard to adapt them to current needs. On the other side they are very robust and withstands nearly everything...But built for dead God’s?

Sophisticated Architecture: Cathedrals

Cathedrals are mainly known for their beautiful architecture. They were in detail planned, but nearly always unique pieces so that nothing was reused. Once build (often over centuries) they last for centuries - their use is not very adaptable, and changing more on the architecture may lead to a crash. Everything is more an art than technique - with a lot of handcrafted work and good tools.

Adapted To The Environment: Grown Cities

Most older cities are not planned - they are grown. We can see different styles of architectures and materials. They are full of life and adapt continuously on all changes as needed and possible. Planning and processes are applied mostly for subprojects and is only partially done for the city itself. Replacing parts is common and natural.

Driven By Economics: Planned CitiesMost planned cities are driven by economic needs: they should be cheap for the customers. This is only possible by reusing the design with only few adoptions or by using a modular approach.The material is rather cheap and standardized, the building follow a standardized process and tools. Some life happens outside the city, otherwise the city needs changes - and this means cost.

Special Needs: SkyscrapersIn some places space is a rare resource. So people started to build into height - Skyscrapers. Here we need a good planning and design, good materials and a well defined process. We need special tools to build them - without it would be impossible. They need maintenance during their full lifetime, otherwise they would become fragile and dangerous, even for the Neighbors!

Comparison To Software DevelopmentSo far we have seen that we need:

• A Plan - Architecture is dominating material but depends on it• Material - what the building is made of and stays there.

We cannot build a Skyscraper out of Sand only …• Tools - defined by used material, they gone after job is finished• Processes - defined rules and steps how to build it

We have to look at all of them if we want Quality …

Software Development: Actual StateSo what is the actual state for Software Development?

- Plan / Architecture: Requirements with Use-Cases, UML- Material: Mostly Imperative and Object-Oriented – Java, C++- Tools: IDE’s (Eclipse, Visual Studio), Unit Tests, Jenkins, …- Processes: Mostly own form of “Agile” – “Iterative V-Model”

Software is still developed by people, trend to Outsourcing still valid.

Development: V-Model vs. IterativeThe V-Model is mainly driven by the metaphor compared to Mechanical Engineering:

• Detailed planning at the beginning• Standardized components and processing methods• Purpose and mission clear and stable• Predictability: time, cost, performance• Build on each other phase-oriented implementation• Clear division of labor between specialists• Independence from specific performing persons• Communication over plans

Development: V-Model vs. IterativeWe know that in reality the V-Model has some flaws:Top-down approaches emphasize planning and a complete understanding of the system, and it means that the large-scale aspects of a program must be engineered first. It is inherent that no coding can begin until a sufficient level of detail has been reached in the design of at least some part of the system.

Iterative wants to shorten the feedback cycle.

How To Get Faster To The Product?Iterative Development is mainly to get faster feedback for doing the right thing.

“If everything seems under control, you’re not going fast enough.”

(Mario Andretti)

But how can we develop faster without loosing Quality?

Planning And SpecificationsNearly all Software people agree about one fact:

As longer a Software Bug is undetected in the Development cycle,as more expensive it becomes to correct the errorSo you have to shorten the feedback cycle between Development and Validation. This may be easy and cheap for a Web Application, but of very high cost for e.g. medical devices (you may loose a life) or space probes (not easy to correct the landing on a comet…)

So to get Software which behaved the way you want you first have to specify it!

"A program that has not been specified

cannot be incorrect, it can only be surprising."

How To Get Specifications: RequirementsHow do you know that you have the right Requirements?Tools like Doors help for tracing Requirements, but not to get them.Methodologies like such from the Sophist Group helps sometimes …

- Create a Vision- Concentrate on customer value- Gather Use-Cases- Build a Risk List- + Think how you want to test the Requirements

Test The Requirements: ValidationBut here we are not talking how to get the right Requirements.We want to talk how to validate them.

The most common way are Unit-Tests:

“For a well defined input we expect a well defined output.”

This may be a lot of work, because you have a lot of inputs to check.How to check if the tests are good enough? Code Coverage?

Problems With Unit-Tests: CoverageFor simple cases Unit-Tests work – but the can give a wrong sense of the correctness of the Software.

Assuming a simple function: f(x)=2/(3.4–x)

If you input vector for testing is [0,1,2,3,4,5,6] everything works fine – you even get the metrics for Code Coverage of 100%.

You experience an exception if you input one time becomes 3.4

Finding Remaining Bugs: Fuzzing TestsBut how can we find such remaining bugs?

One way is “Fuzzing” and “Horror Cabinet”:

You generate random input values or use corrupt data.We don’t look at the output values, just if the Software stays stable and maybe detect the corruption.In Embedded Systems this was a great way to handle wrong Sensor values caused by Electronic Failure or Environment Influences.

Test The Test: Code SabotageFor critical parts we wanted to know if the tests are “good enough”

One way to find out was that we were allowed to change the Code afterwards: changing values, operations (e.g. switching from “+” to “-”) and data flow – we made “Code Sabotage”.

If we run the tests they should detect the changes.If the don’t find them, they are not good enough.This works only on small critical parts, because it is very expensive.

Testing PropertiesInstead of testing single values you can also check for properties or a behavior you expect from a function:

- There is a trend: e.g. bigger inputs always give bigger outputs- The delta is always the same: e.g. |sin(x)-sin(x+d)| < r- There is a Symmetry: e.g. cos(x) = cos(-x)- There is always a neutral element: e.g. to 0: sin(-x)+sin(x)

If we have to work with external libraries this is one way for checking

Example: Check If The Function Is Symmetric

This was an example from Anchor Calculation:

The whole layout is symmetric!It doesn’t matter how big and in which direction the shear forces are: the results should be the same if we mirror the direction of the force.

We generate a shear force in direction and value randomly and check if the results are the same for mirrored directions. This can be automated.

Example: Check If The Function Is Continuous

This was an example from Anchor Calculation too:

We only change one value: the torque moment MFrom a big negative value over zero to a big positive value.We can have check points (e.g. Zero), but mainly we are not so much interested in the values but looking at the previous and next point if the results have only a small delta – they are continuous. This can be automated too.

Generate Tests: QuickCheckWriting tests is a lot of work – so it would be better to generate them.QuickCheck is coming from Haskell and is heavily used in Erlang.

“In QuickCheck the programmer writes assertions about logical properties that a function should fulfill. Then QuickCheck attempts to generate a test case that falsifies these assertions. Once such a test case is found, QuickCheck tries to reduce it to a minimal failing subset by removing or simplifying input data that are not needed to make the test fail.”(Wikipedia)

Limits Of Testing: Ashby’s LawAshby was a Cybernetican who defined the “The Law of Requisite Variety”It is about how you want to control complex systems.

“If a system is to be stable, the number of states of its control mechanism must be greater than or equal to the number of states in

the system being controlled.”

Can you manage the Effort to control your System?

Limits Of Testing: Bremermann’s LimitBremermann’s Limit is the maximum computational speed of a self-contained system in the material universe.Actually a lot of people look at it to judge cryptographic algorithms – but we can look for testing effort too.

At first look it is a high number – but if you look how many variables you have in you code it becomes clear that even for a few hundred bits of variables you cannot fully test them – even if the whole Earth would work as a Supercomputer!This makes it very clear that we cannot bet on full testing…

Proof Of Absence Of Bugs: VerificationSo if testing is not enough – are there other methods like verification?

- Can we use tools like PolySpace for Abstract Interpretation?- If we have verified Code, do we need a verified Compiler?- Do we need a verified Operating System?- What about 3rd party libraries?

Actually it helps a lot to verify our own Code, but for big Systems there is nothing yet available. Verification needs educated people!

How Can We Improve?If we compare to Construction Industry an look for improvement,do we need:

- Better Plans / Architecture?- Better Materials?- Better Tools?- Better Processes?

Probably something from everywhere…

Better Architecture: Adapt To Future NeedsTruly complex things are grown, not build.

Examples are a tree which grows to adapt to wind, rain, and the polethat was put too near to it. Or a city which grows as the focus of lifeor commerce changes from the seaport to the railroad to the airport.What does this mean for Software Architecture? Since we cannothandle the full Feature Set in a Monolithic System, we have to thinkabout Extensibility: only if we divide we can conquer! Withdecoupling we can bring the Complexity down to a manageable part.

Better Architecture: Divide And ConquerOne way is to decouple them – even the architecture.

- For different demands you need different architecture- Different architecture may need different materials- Different material needs different tools- Different architectures may need different processes

Different needs for different parts may need different solutions.How to work together? One Solution is Messaging!

Messaging Architectures: TrendsMessaging driven Architecture is not new (MOM)They are used in Embedded Systems, in SOA or Microservices …

The common thing is that they share nothing and communicate via Messaging – how close they are to Actor based Systems depends on the history of the implementation.The idea is coming from Biology where cells are some small independent structures which communicates via Messages – this influenced people like Alan Kay for their first Object-Oriented Systems.One promising way is Domain Driven Design.

Sidenote: Does The UML Help?Does Modeling help? Architecting with UML?

In my opinion very little – the UML is more a Notation than an Abstraction.The Syntax is defined, but not the Semantic.So it is very much bound to the underlying Programming Language.

For example in State Machines you can easily generate parallel regions.The UML says nothing about Semantic or Implementation: are OS Threads used, Green Threads or even no Threads? Ordering? You can only guess…

Sidenote: The UML Doesn’t Help!A lot of things are not specified in UML – therefore you have Profiles (like MARTE in Embedded Systems) and the OCL.The aspect of Inheritance, Interfaces or Threading depends a lot of the Language you want to use.

Notations can be useful – see arithmetic in Roman Notation versus the Arabic Notation and their influence on multiplication!

But the UML doesn’t really help you to get a better architecture.Even Patterns shows only the Defects of the Language…

Better Processes: Requirements & Specifications

What are the defined rules and steps how to build the Software?

It is not only about designing and building Software:

- What are the Risks and Opportunities?- What are the Steps for Risk Mitigation?- How do we manage Change?- How do we handle Technical Debts?

(Picture from «The Agile Samurai»)

Thinking Ahead: Anticipating The Future

Situation AppraisalPr

oblem Analy

sisDecision Analysis

Oppor

tunit

y Ana

lysis

Problem appeared

Decision taken

Problem determined

Cause determined

There are different Method-ologies available which support Decision making and Anticipating of Future Problems.

FMEA and Sensitivity Models may give you hints where Problems may occur.A systematic Approach is seldom used in Software Development.

Use Better Material: LanguagesProgramming Languages are the same than material is for construction:

- They allow you a special or the wished architecture- Architecture dominates material, but material defines what you can

build: there may also be some legal Requirements…- They define a lot of tools you need to build

A lot of Managers overlook the Importance of Programming Languages.They look more on processes and tools.

Languages: FeaturesFeatures of Languages can improve Quality and Productivity:

- If you don’t want to worry about Memory: use Memory Management- If you don’t want to worry about Null Pointers: use Options- If you don’t want to worry about Side Effects: use Pure Functions- If you don’t want to worry about Type mismatch: use Type Checking- If you don’t want to worry about Threads and Locks: use Actors

There is no Silver Bullet – but some Features may help a lot!

Languages: Java And Pointers

“Java has no pointers but a NullPointerException”

Are null references the billion-dollar mistake? (By Tony Hoare)

Languages: TrendsDue to trends like MultiCore and Distributed Systems we can see a trend to more “Functional” Languages:

Scala, Clojure, F#, Elixir, Elm, … and more advanced ones: Idris, Agda, …

They are going more to Strongly or Dependent Typed Languages.Constructs like Algebraic Types allows to prevent the representation of illegal states – a nice Example was given by Yaron Minsky “OCaml for the Masses” in ACM queue 9/9 2011.

Languages: Things To WatchAnother trend is to bring the Language closer to the Specification.Avoid verbose Constructs and work with the Human Short-Time Memory.

It is about composability of functions and how to handle both success and errors. The term “Monadic” may loose its frightening sound over time hopefully…

Scott Wlaschin created the term “Railway Oriented Programming”.Look at his wonderful Slides on SlideShare!

Languages: Better Replacements?What are better Language Alternatives Today?

- Use Rust instead of C++- Use F# instead of C#- Use Scala instead of Java- Use Elm instead of JavaScript

For Distributed Systems: Look at Erlang OTP and Elixir!

ToolsWhat are tools for?I found the best explanation on Bret Victor’s page:

A tool addresses human needs by amplifying human capabilities.That is, a tool converts what we can do into what we want to do. A great tool is designed to fit both sides.Which tools do we need to build our Software?

Redundancy: Need Of Distributed SystemsTo quote the father of Erlang, Joe Armstrong:

- The world is concurrent- Things in the world don’t share data- Things communicate with messages- Things fail

What happens if your perfect, error-free Software is hit by a lightning?(Happened to Google Datacenter in Belgium August 2015)

New Dragons Around The CornerTo minimize the Risk of Hardware Failure you need at least 2 Computers.

But now a full new can of worms are open:

- How to Synchronize the Systems?- How to find Consensus? (Hint: Paxos or Raft Protocol)- Maybe no common Clock, order of Messages…- Some other Byzantine Failures …- What happens to the CAP Theorem?

Erlang OTP and AkkaThere are some Attempts how we can manage Distributed Systems.A widely used one is based on the Actor Model.

Such Systems were pioneered by the Telecom Industry, namely Erlang.Other Systems took the ideas to other platforms like JVM or .NET.

For integrating them users prefer to stay in their ecosystem – that is the main reason why Akka get more focus than Erlang.

Maybe that changes with Elixir – the future will tell.

The Human FactorSince Software is mainly written and used by Humans we have to take the Human Nature into account.

We can train them, but better than that would be to know the Human limits and try to prevent errors based on his weakness.

This doesn’t only affect users with Human Interface Guidelines, but also Programmers and Designers who have to understand Mental Models, Programming Languages and Operating Systems.

The Human Factor: CultureAccording to Valentin Turchin Life had some Evolutionary Quanta’s:

1) Control of Position = Motion: the animal or agent develops the ability to control its position in space

2) Control of Motion = Irritability: the movement of the agent is no longer given, but a reaction to elementary sensations or stimuli

3) Control of Irritability = Reflex: different elementary sensations and their resulting actions are integrated into a coordinated, but still rigid, reflex-like behavior

4) Control of Reflex = Association: behavioral routines become flexible or adaptive, through the learning of new associations between experienced stimuli and actions

5) Control of Association = Thought: new routines no longer need to be learned through experience; they can be developed by abstract, symbolic reasoning

6) Control of Thought = Culture: symbols and concepts are no longer fixed entities; they adapt through a process of cultural evolution

The Human Factor: Culture Of DevelopmentOn the first moment it seems weird that culture is the control of thought. But on a closer look it makes sense:

- You may come from the culture of Object-Oriented Programming- You may come from the culture of Functional Programming- Your development culture was to deliver the minimal valuable product- Your development culture was from Aerospace or Medicine

Mixing cultural styles may create innovation – but can lead to conflicts

The Human Factor: Culture Of Developers“The Problem Is Not The Problem. The Problem Is Your Attitude About The Problem.

Do You Understand?”- Captain Jack Sparrow

Are the Developers aware of the Problem?Some times they often don’t know that their Code can cause a bug…

The Human Factor: EnvironmentsThe Bullshit Asymmetry Principle:

“The amount of energy needed to refute bullshit is an order of magnitude bigger than to produce it.”

- Alberto Brandolini

How to work with external libraries or legacy code?

The Human Factor: UsersBut even if our Soft- and Hardware is perfect: our users are Humans which can err and make mistakes:

- Habituation- Complicated Non-Monotone Interfaces- Missing or wrong Feedback- Engineering Constraints leading to wrong Focus- Not respecting the Human Nature- Cognitive Psychology for different cultures

(Picture from «Cognitive Psychology»)

ConclusionFor writing correct and error-free Code there is still no Silver-Bullet.Architecture, materials, tools and processes are important.We still have to follow our journey for better Software…