© 2017 Nokia1
Petros Nanouris, Loukas Leontopoulos, Thalia Ikonomou, Marina Kesoglou, Giota Eftaxia, Panos Karantzias, Dimitris Fotiadis
30/3/2019
Augmented intelligence case studies for DevOps
Nokia Internal Use
© 2017 Nokia2
Presentation overview (Duration 20 min)
Nokia Internal Use
Short intro to DevOps
• Problem – symptoms• What is DevOps• The three pillars of DevOps
Conclusions - Summary
• Benefits• Next steps
Augmented intelligence solutions• Monitoring• Test strategy selection and• Intelligent Resource Manager (IRM)• Software Quality Evaluation Platform
Case study: Software Quality Evaluation Platform
• Solution design• Quality analytics lab• Use case
© 2017 Nokia3
• Defects into production.
• Late diagnosis.
• Problems appear in some environments only.
• Blame shifting/finger pointing.
• Long delays.
• “Manual errors”.
• Releases slip/fail.
• Delivering a service is too slow and error-prone.
• Internal friction points.
• Delay=loss
• IT is frequently the bottleneck.
Problem Symptoms
Nokia Internal Use
© 2017 Nokia4
What is DevOps?
Nokia Internal Use
Culture Automation Measurement Sharing
Product Dev
QA
Ops
Concept Cash
That is …
© 2017 Nokia5 Nokia Internal Use
Infrastructure automation Continuous delivery
DevOps three pillars
Reliability engineering
© 2017 Nokia6
DevOps augmented intelligence projects
Nokia Internal Use
Test log analysis to identify abnormal
behavior and produce metrics
Intelligentselection & weighting
of test cases
Nokia Internal Use
CI Orchestrator
StagingDevelopment Build
Software QualityEvaluation
Guidelines for quality improvement
SW Quality Evaluation platform
Test Strategy Analysis &
Optimization
ContinuousSystem Monitoring
Anomaly DetectionSelf Healing
GTC
IRM
Request Environment
provision
Intelligent Resource Manager
DPC
Quality Left shift
OptimizationTime reduction
Static Analysis
Testing dynamic analysis
Optimize HW utilization
Maximize availabilityGreen computing
© 2017 Nokia7
FEATURE EXTRACTION & ENGINEERING
Variety of quality metric data sources (e.g. static
analyzers, source control systems etc.)
Metrics per code fragment
Software quality evaluation platform case study
Nokia Internal Use
✓ Software Quality Evaluation Platform is a collection of prediction models.
✓ Models can be created on demand.
✓ Every instance of the code is analyzed.
✓ Feedback is in turn provided to the user.
✓ Prioritization of improvement actions.
RANDOM FOREST
TRAIN A REGRESSION MODEL DASHBOARD
SUPPORT VECTOR MACHINE
DEEP NEURAL NETWORKS
© 2017 Nokia8
Is static analysis adequate for continuous quality evaluation and improvement?
Nokia Internal Use
Static analysis
Dynamic Analysis
Test results & log analysis
Architecture -design analysis
Data persistence quality analysis
and the list continues …
Data report
Data report
Data report
Data report
Quality lab (platform) creating analytics
Data report
Data report
Data report
Data report
Prediction and quality evaluation models
Decision makingVisualization,
feedback, action
Data sources Reporting
Data report
Continuous quality evaluation and improvement loop
© 2017 Nokia9
Common quality metric extraction and collection (Feature Extraction)
Nokia Internal Use
C/C++
C#Java
Python
Technology agnostic
unused
bad-practice
code history
security
pitfall
code smell
convention
Testresults
suspicious
clumsy
Userexperience
design
bug
confusing
brain-overload
test coverage
unpredictable
Lock-in
Common quality metric extraction and collection
Various quality KPI areas
Code fragment
Metric 1 Metric 2 … Metric N
Data Data Data Data
Data Data Data Data
Transform extracted data into a suitable form for data analysis and machine
learning purposes
Data persistence layer
Map quality metrics to specific lines of code
© 2017 Nokia10
Graph and document query languagesQuality reporting and analytics
Nokia Internal Use
Abstract Syntax Trees
Code Dependency Graphs
• Is there a complete set of software quality KPIs out there?
✓ We need a platform (lab) used to create novel, custom and more efficient quality KPIs & analytics.
Basis for creating new quality metrics - features
Data
Analytics
Quality Lab
✓ Cypher.
✓ Mongo DB document querylanguage.
✓ Native Java API to Neo4j.
© 2017 Nokia11
Visualization of code quality metrics
Nokia Internal Use
Module
File
Class
Method
Nodes: code fragments (Classes. Methods, etc.)
Colour intensity and node size corresponds to depicted metric value (heat graph-map)
Code as a graph representation
Scale selection Cluster selection (neighborhoods or logical clusters that share common
characteristics)
Graph clusteringFocusing on data relationships
© 2017 Nokia12
Software EpidemiologyQuality reporting and analytics
Nokia Internal Use
How a deadly virus could spread around the world. The world is a metaphor for the product.
✓ “Source of infection” areas ✓ Risk assessment to add new code ✓ Prioritization and planning
© 2017 Nokia13
Case study: Software fault proneness analyticsQuality reporting and analytics
Nokia Internal Use
Fault proximity: defines how far in terms of change is a code fragment revision from an actual bug correction.
Fault dissemination: defines how much a change in a particular code fragment affects also its neighbouring codefragments.
Change proneness: defines how prone is a code fragment to change and estimates the amount of change.
Relative change proneness =1
NREVISIONS, Mean velocity of change =
ΔΧ
NREVISIONS
Exposure: defines how much a code fragment is affected by others in recent history, and estimates a future trend.
Exposure = σ𝟏
𝒅𝒔𝒑𝒂𝒄𝒆 𝒅𝒕𝒊𝒎𝒆
© 2017 Nokia14
Prediction & consulting (a use case)
Nokia Internal Use
Classification
Regression
Clustering
Machine Learningmodule
Predictionmodel
Training on Data
How long before our code gets a ticket correction?How prone is our code to change?How much is it going to change? How much a code fragment change affects the rest of the system?How much a code fragment is exposed to the neighbourhood’s bad quality code? … and so on …Do you have the analytic (target variable)? Create your custom model and use it.
The developer retrieves the current quality state of the code.
The developer Changes the code.
The model evaluates the new quality state of the code.
The model provides specific guidelines (at detail level) to avoid
quality compromisation.
The developer follows the guidelines and reassesses the code with the
support of the model.
Data
© 2017 Nokia15
Risks
• Data acquisition.
• Availability of test cases.
• Test suite running time
✓ Testing optimization module of DevOps DSS could mitigate these impediments.
Summary
Nokia Internal Use
Benefits
✓ User experience.
✓ Effectiveness.
✓ Continuous quality evaluation and improvement.
✓ State of the art data analytics and prediction models.
Next steps
➢ Performance evaluation.
➢ Dashboard.
➢ Feedback mechanism.
➢ Integrate additional quality data.
➢ Complete pilot and demo.
© 2017 Nokia16 Nokia Internal Use
Many thanks for your attention!Please feel free to ask questions