continuous integration testing for - tmcnet · continuous integration testing for iot endpoint...
TRANSCRIPT
Continuous Integration Testing for IoT Endpoint Integrity
Arthur “Code Curmudgeon” Hicken
Parasoft
Arthur Hicken is Chief Evangelist at Parasoft where he has been involved in automating various software development and testing practices for over 20 years. He has worked on projects including cybersecurity, database development, the software development lifecycle, web publishing and monitoring, and integration with legacy systems and maintains the IoT Hall-of-Shame http://bit.ly/iotshame
Follow him @codecurmudgeon
Blog: http://codecurmudgeon.com
Web: http://parasoft.com
Agenda
IoT specifics
Challenges in quality assurance
Effectiveness of end-to-end testing
Doing it better
Things are everywhere
Industrial Automation Smart Health
Smart Home Smart City
Things are vulnerable The IoT Hall-of-shame
http://codecurmudgeon.com http://bit.ly/iotshame
• Communication • Backbone • Identification • Hardware • Sensors • Protocols • Software • Security • Machine Learning • Data Brokers / Cloud Platforms
Aspects of IoT
IoT specifics • Multi-layered systems
• Disparate technologies
• Functionality
spanning across many
layers
Why it is difficult to test IoT • Disparate development technologies hamper
introducing consistent testing strategies
• “Not enough code to justify testing” syndrome
• Limited user interface make the solution to look deceptively simple
• Inverted testing pyramid problem – Strong temptation to rely entirely on system testing
Effectiveness of end-to-end testing
• Labor intensive
• Difficult to automate
• Difficult to start early
• Requires inter-team synchronization
• Difficult to test error handlers
• Failed tests do not provide precise information about failure points
Effectiveness of end-to-end testing
• End-to-End testing for diabetes health system example
• Difficult to apply stimulation
• Difficult to validate test result
• Difficult to test error handlers
Effective IoT Testing • Avoid manual end-to-end testing • Invest more into automated tests at component • Follow testing pyramid rules • Measure:
– test results – test effectiveness
• Determine: – Risk
How to do it better ?
• Give absolute priority to automated tests
• Invest time into designing interfaces (API)
• Use API (service) testing tools to cover interfaces
• Measure the quality of the test
How to do it better ?
• Use stubs and mocks to improve:
– Facilitate test automation
– Increase code penetration
How to do it better ?
• service virtualization to improve test automation
– Isolate at the message layer
– Simulate functional scenarios & performance
Test scenario example
Test Scenario: REQUEST_NEW_BOLUS_UPDATE_PUMP
Tested System
Tested System
Elements of Continuous Testing
Continuous Testing
Risk Assessment
Policy Analysis
Requirements Traceability
Test Environment
Access
Test Optimization
Advanced Analysis
Ensure access to complete test environments
Automate defect prevention and Policy measurement
Expand test coverage and measure test effectiveness
Define actionable practices
Connect functional with non-functional requirements
Process improvement opportunities
Feature Release
Business
Stakeholder
Customer Support
Beyond Automation
Software Feature
Continuous measurements mean continuous refinement of the process
Real-time feedback from
Objective assessment and go/no go Defects are eliminated at the
point that they are easiest to fix
Business
Stakeholder
Developer Customer Support
Quality gates: Organizations can automatically promote software through quality gates when business expectations have been met
Summary
IoT adds complexity - It’s not just a device
Stubs/mocks/virtualization enables testing
Continuous testing catches it before the user
Q&A
• Arthur Hicken: – Email: [email protected]
– Web: https://parasoft.com
– Twitter: @CodeCurmudgeon
– Facebook: https://facebook.com/CodeCurmudgeon
– Blog: http://codecurmudgeon.com
– IoT Hall-of-Shame: http://bit.ly/iotshame