standards-based iot testing with open-source test equipmenttest suites / test cases generators (ts)...
TRANSCRIPT
Standards-BasedIoT Testing withOpen-Source Test Equipment Alexander Kaiser, Sascha Hackel
Sofia, 24.07.2019
Speaker
Sascha Hackel
Fraunhofer FOKUS
Kaiserin-Augusta-Allee 31
10589 Berlin
Alexander Kaiser
Relayr GmbH
Bergmannstraße 102/103
10961 Berlin
IoT-TProject
Testlab and Testware for the Internet of Things
http://www.iot-t.de/en/
Motivation
IoT (protocols) quite young and implementations often immature
Complexity through interconnectivity and diversity
(Mostly) standardized protocols
Challenges in testing IoT
IoT Application Logic
IoT Services Layer
Cellular: 4G, NB-IoT, Cat-M1, EC-GSM, 5GNon-Cellular: Wifi, LoRa, Sigfox, Zigbee, BLE
IPv4 / IPv6
TCP UDP
CoAPMQTT-SN
DTLSTLS / SSL
MQTTHTTP…OPC*
C
Conformance
I
Interop.
T
Compliance-Tool
C
Certification
C I T C
C I T C
CITC
B
Battle Proofed
B B
B
B
B B
C T
I I
C C
Newcomer vs. Fieldbuses
Level 0
Field
Level 4
Business
Level 3
Operations Support
Level 2
Supervisory
Level 1
Control
› PLCs
› Edge Devices
› Sensors
› Machinery
› DCS / SCADA
› HMI
› MES
› Control DMZ
› Enterprise Cloud
› ERP
› Office
Protocol OpenSpecification
Stand.Testcases
Testwareavailable
🌍
🌍
🌍
🌍(🌍)(🌍)🌍
(🌍)
🌍
🌍
🌍🌍
🌍
🌍
🌍🌍
ConformanceCertification
🌍
🌍
🌍🌍
• IT ensures Quality with Ref. Implementations and Plugfests• OT is driven by standards and independent certifications
EclipseIoT-Testware
Open-Source Test Suites for standardized IoT protocols
https://projects.eclipse.org/projects/technology.iottestware
Approach
ISO/IEC 9646 – Conformance testing methodology and framework
SUT
Test executionCertification
TSS: Test Suite StructureTP: Test Purpose (Catalogue)ATS: Abstract Test SuiteETS: Executable Test SuiteSUT: System Under Test
Test Suite Structure (MQTT)
Client(TS)
Broker(SUT)
MQTT
Client(TS)
Broker(SUT)
Client(TS)
Broker(TS)
Client(SUT)
cfg_01
cfg_02
cfg_03
Broker as SUT
All mandatory message data fields
Regular and illegal data
(Fixed/variable header, payload)
Protocol features
Connect/disconnect (session)
Subscribe/unsubscribe
Immediate publish
Last will and Testament (LWT)
Heartbeats keepAlive values
Topic
Error handling
Client as SUT
…
Test Purposes
…
Test description
Verification &Validation
Requirements
Design
ImplementationTest
Evolution
ExecutableIoT-Testware
SUT
…
Input
„executablespecification“
MQTT Evaluation
Broker PASS FAIL INCONC
Name Version # % # % # %
Mosquitto 1.5.5 90 85,71% 11 10,48% 4 3,81%
HiveMQ CE 2019.1 86 81,90% 15 14,29% 4 3,81%
lannister v0.9.8 68 64,76% 33 31,43% 4 3,81%
Apache ActiveMQ 5.15.9 58 55,24% 43 40,95% 4 3,81%
Aedes 0.38.0 58 55,24% 43 40,95% 4 3,81%
RSMB 1.3.0.2 50 47,62% 51 48,57% 4 3,81%
Mosca 2.8.3 43 40,95% 58 55,24% 4 3,81%
Apache Apollo 1.7.1 34 32,38% 70 66,67% 1 0,95%
April 2019 – 105 Test Cases
Usability
stdout
IUT
Bro
wse
r
nmap
Fuzzer
Ch
ild P
roce
sses
OS
Do
cker
| U
bu
ntu
| D
ebia
n
tcpdump
spawn
IP [MQTT | CoAP | OPC-UA | *]
SU
T
…
tail -f
Usability
Server Client
Protocol Fuzzing
Test Suites /Test Cases Generators
(TS)
SUTFuzzing Proxy<pro�o><proto>
Fuzzing Engine
Codecs
Mutators
FuzzerConfiguration
Scapy: MQTT / CoAP / HTTP /OPC(-DA) / Modbus /
IEC 104 / CAN / OBD /….
Rules Engine
ProbabilityEngine
Logging
e.g. PRNG, Fuzzino, Radamsa, zzuf…
Fuzzing analysis and reproduction
Security Regression Testing
reports to
implements
reference to
Issue / CVE_[abc]*
TC: BrokerCVE_[xxx]
SUT
TP: Broker CVE_[xxx]
Vulnerable Broker
1..*
1
run against
Performance Testing
TS
…MTC /RIoT
PTC MITMF
SUT
PTC MITMF
MTC: Main Test ComponentPTC: Parallel Test ComponentMITMF: Man-in-the-Middle Fuzzer / Fuzzing ProxySUT: System Under Test
Takeaway
Open standards require open-source test equipment
Test equipment must be flexible though easy to use
Reuse of standards-based test equipment frees resources for additional testing activities
Cooperate on (Black-Box/conformance) testing, compete in implementation
Questions?
https://projects.eclipse.org/projects/technology.iottestware