verify, validate and document models and code · • ‘going smart’, ‘internet of things ......
TRANSCRIPT
![Page 1: Verify, Validate and Document Models and Code · • ‘Going Smart’, ‘Internet of Things ... Finding Unintended Behavior Converting floating-point model to integer calibrations,](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4eba514ce2730d3a0d4bf4/html5/thumbnails/1.jpg)
1© 2015 The MathWorks, Inc.
Verify, Validate and Document Models and Code
Dr. Michael Kitz
Senior Applications Engineer, Physical Modelling and Control Design
Applications Engineering Group
MathWorks Switzerland
June 09, 2015
![Page 2: Verify, Validate and Document Models and Code · • ‘Going Smart’, ‘Internet of Things ... Finding Unintended Behavior Converting floating-point model to integer calibrations,](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4eba514ce2730d3a0d4bf4/html5/thumbnails/2.jpg)
2
The Challenge
• Product Innovation
• ‘Going Smart’, ‘Internet of
Things’
Software everywhere
How will you know your system
works?
![Page 3: Verify, Validate and Document Models and Code · • ‘Going Smart’, ‘Internet of Things ... Finding Unintended Behavior Converting floating-point model to integer calibrations,](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4eba514ce2730d3a0d4bf4/html5/thumbnails/3.jpg)
3
Three Key Takeaways
Find problems and bugs early in the design and code
Use mathematical analysis methods to prove software correctness
Reproduce field issues via property proving
![Page 4: Verify, Validate and Document Models and Code · • ‘Going Smart’, ‘Internet of Things ... Finding Unintended Behavior Converting floating-point model to integer calibrations,](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4eba514ce2730d3a0d4bf4/html5/thumbnails/4.jpg)
4
Issues during testing
1) Wasting a couple of days on the test
bench to find a code integration
issue without success
2) While going downhill, target speed
increase with “reduce speed” button
Application example: Cruise Control
![Page 5: Verify, Validate and Document Models and Code · • ‘Going Smart’, ‘Internet of Things ... Finding Unintended Behavior Converting floating-point model to integer calibrations,](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4eba514ce2730d3a0d4bf4/html5/thumbnails/5.jpg)
5
Verification and Validation in Context
Ad-hoc tests
Design error checks
Functional tests/ Model coverage
Equivalence tests
Effort / Time
Confidence
TE
ST
&
VE
RIF
ICA
TIO
N
INTEGRATION
IMPLEMENTATION
ANALYSIS – SPECIFICATION- DESIGNANALYSIS – SPECIFICATION- DESIGN
MODEL
RESEARCH REQUIREMENTS
MCU DSP FPGA ASIC
Structured
TextVHDL, VerilogC, C++
Architecture
Algorithms
Schematics
TEST
CASES
Environment
Constraints
Physical Domains
TEST
CASES
PLC PAC
![Page 6: Verify, Validate and Document Models and Code · • ‘Going Smart’, ‘Internet of Things ... Finding Unintended Behavior Converting floating-point model to integer calibrations,](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4eba514ce2730d3a0d4bf4/html5/thumbnails/6.jpg)
6
Ad-hoc testing
![Page 7: Verify, Validate and Document Models and Code · • ‘Going Smart’, ‘Internet of Things ... Finding Unintended Behavior Converting floating-point model to integer calibrations,](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4eba514ce2730d3a0d4bf4/html5/thumbnails/7.jpg)
7
Customizable Reports
![Page 8: Verify, Validate and Document Models and Code · • ‘Going Smart’, ‘Internet of Things ... Finding Unintended Behavior Converting floating-point model to integer calibrations,](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4eba514ce2730d3a0d4bf4/html5/thumbnails/8.jpg)
8
Verification and Validation in Context
Ad-hoc tests
Design error checks
Functional tests/ Model coverage
Equivalence tests
Effort / Time
Confidence
TE
ST
&
VE
RIF
ICA
TIO
N
INTEGRATION
IMPLEMENTATION
ANALYSIS – SPECIFICATION- DESIGNANALYSIS – SPECIFICATION- DESIGN
MODEL
RESEARCH REQUIREMENTS
MCU DSP FPGA ASIC
Structured
TextVHDL, VerilogC, C++
Architecture
Algorithms
Schematics
TEST
CASES
Environment
Constraints
Physical Domains
TEST
CASES
PLC PAC
![Page 9: Verify, Validate and Document Models and Code · • ‘Going Smart’, ‘Internet of Things ... Finding Unintended Behavior Converting floating-point model to integer calibrations,](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4eba514ce2730d3a0d4bf4/html5/thumbnails/9.jpg)
9
Finding Unintended Behavior
Converting floating-point model to integer calibrations, signals…
Dead logic due to “uint8” operation
![Page 10: Verify, Validate and Document Models and Code · • ‘Going Smart’, ‘Internet of Things ... Finding Unintended Behavior Converting floating-point model to integer calibrations,](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4eba514ce2730d3a0d4bf4/html5/thumbnails/10.jpg)
10
Finding Unintended Behavior
Dead logic due to “uint8”
operation on incdec/holdrate*10
Fix change the order of operation
10*incdec/holdrate
Condition can never be false
![Page 11: Verify, Validate and Document Models and Code · • ‘Going Smart’, ‘Internet of Things ... Finding Unintended Behavior Converting floating-point model to integer calibrations,](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4eba514ce2730d3a0d4bf4/html5/thumbnails/11.jpg)
11
Finding Unintended Behavior
Dead logic due to “uint8”
operation on incdec/holdrate*10
Fix change the order of operation
10*incdec/holdrate
Condition can never be false
![Page 12: Verify, Validate and Document Models and Code · • ‘Going Smart’, ‘Internet of Things ... Finding Unintended Behavior Converting floating-point model to integer calibrations,](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4eba514ce2730d3a0d4bf4/html5/thumbnails/12.jpg)
12
Verification and Validation in Context
Ad-hoc tests
Design error checks
Functional tests/ Model coverage
Equivalence tests
Effort / Time
Confidence
TE
ST
&
VE
RIF
ICA
TIO
N
INTEGRATION
IMPLEMENTATION
ANALYSIS – SPECIFICATION- DESIGN
MODEL
RESEARCH REQUIREMENTS
MCU DSP FPGA ASIC
Structured
TextVHDL, VerilogC, C++
Architecture
Algorithms
Schematics
TEST
CASES
Environment
Constraints
Physical Domains
PLC PAC
TEST
CASES
![Page 13: Verify, Validate and Document Models and Code · • ‘Going Smart’, ‘Internet of Things ... Finding Unintended Behavior Converting floating-point model to integer calibrations,](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4eba514ce2730d3a0d4bf4/html5/thumbnails/13.jpg)
13
Design
Simulation Testing Workflow
Structural coverage
report
Did we completely
test our model?
Did we meet
requirements?
Review functional
behavior
![Page 14: Verify, Validate and Document Models and Code · • ‘Going Smart’, ‘Internet of Things ... Finding Unintended Behavior Converting floating-point model to integer calibrations,](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4eba514ce2730d3a0d4bf4/html5/thumbnails/14.jpg)
14
Verification and Validation in Context
Ad-hoc tests
Design error checks
Functional tests/ Model coverage
Equivalence tests
Effort / Time
Confidence
TE
ST
&
VE
RIF
ICA
TIO
N
INTEGRATION
IMPLEMENTATION
ANALYSIS – SPECIFICATION- DESIGN
MODEL
RESEARCH REQUIREMENTS
MCU DSP FPGA ASIC
Structured
TextVHDL, VerilogC, C++
Architecture
Algorithms
Schematics
TEST
CASES
Environment
Constraints
Physical Domains
PLC PAC
TEST
CASES
![Page 15: Verify, Validate and Document Models and Code · • ‘Going Smart’, ‘Internet of Things ... Finding Unintended Behavior Converting floating-point model to integer calibrations,](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4eba514ce2730d3a0d4bf4/html5/thumbnails/15.jpg)
15
Equivalence Testing
Model
Generated Code
![Page 16: Verify, Validate and Document Models and Code · • ‘Going Smart’, ‘Internet of Things ... Finding Unintended Behavior Converting floating-point model to integer calibrations,](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4eba514ce2730d3a0d4bf4/html5/thumbnails/16.jpg)
16
Equivalence Testing
Model
Generated Code
![Page 17: Verify, Validate and Document Models and Code · • ‘Going Smart’, ‘Internet of Things ... Finding Unintended Behavior Converting floating-point model to integer calibrations,](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4eba514ce2730d3a0d4bf4/html5/thumbnails/17.jpg)
17
Verification and Validation in Context
Ad-hoc tests
Design error checks
Functional tests/ Model coverage
Equivalence tests
Effort / Time
Confidence
TE
ST
&
VE
RIF
ICA
TIO
N
INTEGRATION
IMPLEMENTATIONIMPLEMENTATION
ANALYSIS – SPECIFICATION- DESIGNANALYSIS – SPECIFICATION- DESIGN
MODEL
RESEARCH REQUIREMENTS
MCU DSP FPGA ASIC
Structured
TextVHDL, VerilogC, C++
Architecture
Algorithms
Schematics
Environment
Constraints
Physical Domains
PLC PAC
TEST
CASES
TEST
CASES
REQUIREMENTS
![Page 18: Verify, Validate and Document Models and Code · • ‘Going Smart’, ‘Internet of Things ... Finding Unintended Behavior Converting floating-point model to integer calibrations,](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4eba514ce2730d3a0d4bf4/html5/thumbnails/18.jpg)
18
Verification and Validation in Context
Ad-hoc tests
Design error checks
Functional tests/ Model coverage
Equivalence tests
Effort / Time
Confidence
TE
ST
&
VE
RIF
ICA
TIO
N
INTEGRATION
IMPLEMENTATION
ANALYSIS – SPECIFICATION- DESIGN
MODEL
RESEARCH REQUIREMENTS
MCU DSP FPGA ASIC
Structured
TextVHDL, VerilogC, C++
Architecture
Algorithms
Schematics
Environment
Constraints
Physical Domains
PLC PAC
TEST
CASES
TEST
CASES
![Page 19: Verify, Validate and Document Models and Code · • ‘Going Smart’, ‘Internet of Things ... Finding Unintended Behavior Converting floating-point model to integer calibrations,](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4eba514ce2730d3a0d4bf4/html5/thumbnails/19.jpg)
19
Cruise Control
Module (MBD)
System
InputsOutputsFuel Rate Control
Module
Shift Logic
Control Module
Engine
Control
Unit
Le
ga
cy c
od
e
Application Example: Cruise Control
![Page 20: Verify, Validate and Document Models and Code · • ‘Going Smart’, ‘Internet of Things ... Finding Unintended Behavior Converting floating-point model to integer calibrations,](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4eba514ce2730d3a0d4bf4/html5/thumbnails/20.jpg)
20
Checking Source Code
What Polyspace does
Checks coding rule conformance (MISRA-C/C++, JSF++, Custom)
Provides metrics (Cyclomatic complexity etc)
Quickly finds potential errors
Proves the existence of errors
Proves the absence of errors
Indicates when you’ve reached the desired quality level
No test cases
No compilation
![Page 21: Verify, Validate and Document Models and Code · • ‘Going Smart’, ‘Internet of Things ... Finding Unintended Behavior Converting floating-point model to integer calibrations,](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4eba514ce2730d3a0d4bf4/html5/thumbnails/21.jpg)
21
Identify Run-Time Error in Integrated Code
![Page 22: Verify, Validate and Document Models and Code · • ‘Going Smart’, ‘Internet of Things ... Finding Unintended Behavior Converting floating-point model to integer calibrations,](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4eba514ce2730d3a0d4bf4/html5/thumbnails/22.jpg)
22
Identify Run-Time Error in Integrated Code
![Page 23: Verify, Validate and Document Models and Code · • ‘Going Smart’, ‘Internet of Things ... Finding Unintended Behavior Converting floating-point model to integer calibrations,](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4eba514ce2730d3a0d4bf4/html5/thumbnails/23.jpg)
23
Identify Run-Time Error in Integrated Code
![Page 24: Verify, Validate and Document Models and Code · • ‘Going Smart’, ‘Internet of Things ... Finding Unintended Behavior Converting floating-point model to integer calibrations,](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4eba514ce2730d3a0d4bf4/html5/thumbnails/24.jpg)
25
Verification and Validation in Context
Ad-hoc tests
Design error checks
Functional tests/ Model coverage
Equivalence tests
Integration tests
Effort / Time
Confidence
Is there more?
TE
ST
&
VE
RIF
ICA
TIO
N
INTEGRATION
IMPLEMENTATION
ANALYSIS – SPECIFICATION- DESIGN
MODEL
RESEARCH REQUIREMENTS
MCU DSP FPGA ASIC
Structured
TextVHDL, VerilogC, C++
Architecture
Algorithms
Schematics
Environment
Constraints
Physical Domains
PLC PAC
TEST
CASES
TEST
CASES
![Page 25: Verify, Validate and Document Models and Code · • ‘Going Smart’, ‘Internet of Things ... Finding Unintended Behavior Converting floating-point model to integer calibrations,](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4eba514ce2730d3a0d4bf4/html5/thumbnails/25.jpg)
26
Using Model-Based Design to Reproduce Field Issue
Problem: While going downhill, target speed increases with “reduce speed”
button and assumes random values
– Functional tests pass for model
– No redundancies in model (100% coverage achieved)
– Nominal signal and parameter values worked in simulation
![Page 26: Verify, Validate and Document Models and Code · • ‘Going Smart’, ‘Internet of Things ... Finding Unintended Behavior Converting floating-point model to integer calibrations,](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4eba514ce2730d3a0d4bf4/html5/thumbnails/26.jpg)
27
Using Model-Based Design to Reproduce Field Issue
Construct a model of field issue:
Constrain inputs to represent field issue
Create model of field issue behavior
Ask tool to produce a test case
![Page 27: Verify, Validate and Document Models and Code · • ‘Going Smart’, ‘Internet of Things ... Finding Unintended Behavior Converting floating-point model to integer calibrations,](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4eba514ce2730d3a0d4bf4/html5/thumbnails/27.jpg)
28
Generated Test Case to Reproduce Field Issue
Automatic generation of harness model
![Page 28: Verify, Validate and Document Models and Code · • ‘Going Smart’, ‘Internet of Things ... Finding Unintended Behavior Converting floating-point model to integer calibrations,](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4eba514ce2730d3a0d4bf4/html5/thumbnails/28.jpg)
29
Debugging
Going downhill, actual
speed could increase (say to
25mph)
If coast button (reduce
speed) is set again, target
speed takes on actual speed
value of 25mph (jumps up
from 20mph)
Sneak Path
Intended Outcome
![Page 29: Verify, Validate and Document Models and Code · • ‘Going Smart’, ‘Internet of Things ... Finding Unintended Behavior Converting floating-point model to integer calibrations,](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4eba514ce2730d3a0d4bf4/html5/thumbnails/29.jpg)
30
Verification and Validation Best Practice
Ad-hoc tests
Design error checks
Functional tests/ Model coverage
Require-mentproofs
Integration tests
Effort / Time
Confidence
Equivalence tests
Field tests
![Page 30: Verify, Validate and Document Models and Code · • ‘Going Smart’, ‘Internet of Things ... Finding Unintended Behavior Converting floating-point model to integer calibrations,](https://reader033.vdocument.in/reader033/viewer/2022050118/5f4eba514ce2730d3a0d4bf4/html5/thumbnails/30.jpg)
31
Three Key Takeaways
Find problems and bugs early in the design and code
Use formal methods to prove software correctness
Reproduce field issues via property proving