qa / testing challenges the gods must be crazy 1980 sped ... · the gods must be crazy 1980 sped up...
TRANSCRIPT
![Page 3: QA / Testing challenges The Gods Must Be Crazy 1980 Sped ... · The Gods Must Be Crazy 1980 Sped Up Stuck In The Mud Scene. Driverless cars? it has no hand brake and if it stalls](https://reader035.vdocument.in/reader035/viewer/2022071219/6055fd1ca1baad6dc573a893/html5/thumbnails/3.jpg)
QA Challenges with IoT Systems
Bede Ngaruko
@bedengaruko
![Page 4: QA / Testing challenges The Gods Must Be Crazy 1980 Sped ... · The Gods Must Be Crazy 1980 Sped Up Stuck In The Mud Scene. Driverless cars? it has no hand brake and if it stalls](https://reader035.vdocument.in/reader035/viewer/2022071219/6055fd1ca1baad6dc573a893/html5/thumbnails/4.jpg)
What on earth is IoT?
![Page 5: QA / Testing challenges The Gods Must Be Crazy 1980 Sped ... · The Gods Must Be Crazy 1980 Sped Up Stuck In The Mud Scene. Driverless cars? it has no hand brake and if it stalls](https://reader035.vdocument.in/reader035/viewer/2022071219/6055fd1ca1baad6dc573a893/html5/thumbnails/5.jpg)
Not good at definitions, but …
here goes
▶ Interconnected physical devices, vehicles,
buildings, etc.
▶ Smart devices
▶ Embedded Electronics
▶ Software
▶ Sensors and Actuators
▶ Networks connectivity
▶ Collect data and exchange data
![Page 6: QA / Testing challenges The Gods Must Be Crazy 1980 Sped ... · The Gods Must Be Crazy 1980 Sped Up Stuck In The Mud Scene. Driverless cars? it has no hand brake and if it stalls](https://reader035.vdocument.in/reader035/viewer/2022071219/6055fd1ca1baad6dc573a893/html5/thumbnails/6.jpg)
The ‘things’: 20 billion devices
by 2020
▶ Wearables
▶ Smart phones
▶ Tablets
▶ Sensors
▶ Health devices
▶ Cars
▶ Planes
▶ Collecting data
▶ Real time
![Page 7: QA / Testing challenges The Gods Must Be Crazy 1980 Sped ... · The Gods Must Be Crazy 1980 Sped Up Stuck In The Mud Scene. Driverless cars? it has no hand brake and if it stalls](https://reader035.vdocument.in/reader035/viewer/2022071219/6055fd1ca1baad6dc573a893/html5/thumbnails/7.jpg)
Typical IoT architecture
![Page 8: QA / Testing challenges The Gods Must Be Crazy 1980 Sped ... · The Gods Must Be Crazy 1980 Sped Up Stuck In The Mud Scene. Driverless cars? it has no hand brake and if it stalls](https://reader035.vdocument.in/reader035/viewer/2022071219/6055fd1ca1baad6dc573a893/html5/thumbnails/8.jpg)
Typical Use Cases
![Page 9: QA / Testing challenges The Gods Must Be Crazy 1980 Sped ... · The Gods Must Be Crazy 1980 Sped Up Stuck In The Mud Scene. Driverless cars? it has no hand brake and if it stalls](https://reader035.vdocument.in/reader035/viewer/2022071219/6055fd1ca1baad6dc573a893/html5/thumbnails/9.jpg)
Typical IoT Applications
![Page 10: QA / Testing challenges The Gods Must Be Crazy 1980 Sped ... · The Gods Must Be Crazy 1980 Sped Up Stuck In The Mud Scene. Driverless cars? it has no hand brake and if it stalls](https://reader035.vdocument.in/reader035/viewer/2022071219/6055fd1ca1baad6dc573a893/html5/thumbnails/10.jpg)
![Page 11: QA / Testing challenges The Gods Must Be Crazy 1980 Sped ... · The Gods Must Be Crazy 1980 Sped Up Stuck In The Mud Scene. Driverless cars? it has no hand brake and if it stalls](https://reader035.vdocument.in/reader035/viewer/2022071219/6055fd1ca1baad6dc573a893/html5/thumbnails/11.jpg)
Zipline (SFO)
![Page 12: QA / Testing challenges The Gods Must Be Crazy 1980 Sped ... · The Gods Must Be Crazy 1980 Sped Up Stuck In The Mud Scene. Driverless cars? it has no hand brake and if it stalls](https://reader035.vdocument.in/reader035/viewer/2022071219/6055fd1ca1baad6dc573a893/html5/thumbnails/12.jpg)
Challenge: The things
▶ Hardware
▶ Car
▶ Air-conditioning unit
▶ Refrigerated truck
▶ Home security systems
▶ Etc.
![Page 13: QA / Testing challenges The Gods Must Be Crazy 1980 Sped ... · The Gods Must Be Crazy 1980 Sped Up Stuck In The Mud Scene. Driverless cars? it has no hand brake and if it stalls](https://reader035.vdocument.in/reader035/viewer/2022071219/6055fd1ca1baad6dc573a893/html5/thumbnails/13.jpg)
Sensors
▶ Temperature
▶ Humidity
▶ Gyro
▶ GPS
▶ Motion
▶ Etc.
![Page 14: QA / Testing challenges The Gods Must Be Crazy 1980 Sped ... · The Gods Must Be Crazy 1980 Sped Up Stuck In The Mud Scene. Driverless cars? it has no hand brake and if it stalls](https://reader035.vdocument.in/reader035/viewer/2022071219/6055fd1ca1baad6dc573a893/html5/thumbnails/14.jpg)
Actuators / Effectors
![Page 15: QA / Testing challenges The Gods Must Be Crazy 1980 Sped ... · The Gods Must Be Crazy 1980 Sped Up Stuck In The Mud Scene. Driverless cars? it has no hand brake and if it stalls](https://reader035.vdocument.in/reader035/viewer/2022071219/6055fd1ca1baad6dc573a893/html5/thumbnails/15.jpg)
Embedded Systems / Firmware
![Page 16: QA / Testing challenges The Gods Must Be Crazy 1980 Sped ... · The Gods Must Be Crazy 1980 Sped Up Stuck In The Mud Scene. Driverless cars? it has no hand brake and if it stalls](https://reader035.vdocument.in/reader035/viewer/2022071219/6055fd1ca1baad6dc573a893/html5/thumbnails/16.jpg)
Challenge: no UI (I freak out!)
![Page 17: QA / Testing challenges The Gods Must Be Crazy 1980 Sped ... · The Gods Must Be Crazy 1980 Sped Up Stuck In The Mud Scene. Driverless cars? it has no hand brake and if it stalls](https://reader035.vdocument.in/reader035/viewer/2022071219/6055fd1ca1baad6dc573a893/html5/thumbnails/17.jpg)
API Testing
▶ No UI?
▶ Testers or Devs
▶ Web servers,
▶ Http, SSL/TLS
▶ Web APIs, REST
▶ JSON, Xml
▶ Authentication
▶ Network protocols
![Page 18: QA / Testing challenges The Gods Must Be Crazy 1980 Sped ... · The Gods Must Be Crazy 1980 Sped Up Stuck In The Mud Scene. Driverless cars? it has no hand brake and if it stalls](https://reader035.vdocument.in/reader035/viewer/2022071219/6055fd1ca1baad6dc573a893/html5/thumbnails/18.jpg)
Mobile devices
▶ Things have short range comms (Bluetooth, Zigbee,
Intranet)
▶ Mobile as gateway
▶ Fragmented market
▶ Lots of configs
![Page 19: QA / Testing challenges The Gods Must Be Crazy 1980 Sped ... · The Gods Must Be Crazy 1980 Sped Up Stuck In The Mud Scene. Driverless cars? it has no hand brake and if it stalls](https://reader035.vdocument.in/reader035/viewer/2022071219/6055fd1ca1baad6dc573a893/html5/thumbnails/19.jpg)
The Cloud
▶ Added level of complexity
▶ Increased attack surface
?
![Page 20: QA / Testing challenges The Gods Must Be Crazy 1980 Sped ... · The Gods Must Be Crazy 1980 Sped Up Stuck In The Mud Scene. Driverless cars? it has no hand brake and if it stalls](https://reader035.vdocument.in/reader035/viewer/2022071219/6055fd1ca1baad6dc573a893/html5/thumbnails/20.jpg)
Third party libraries: the hack culture
![Page 21: QA / Testing challenges The Gods Must Be Crazy 1980 Sped ... · The Gods Must Be Crazy 1980 Sped Up Stuck In The Mud Scene. Driverless cars? it has no hand brake and if it stalls](https://reader035.vdocument.in/reader035/viewer/2022071219/6055fd1ca1baad6dc573a893/html5/thumbnails/21.jpg)
Added dimensions in IoT
▶ Security
▶ Availability
▶ Reliability
▶ Safety
▶ Resilience
▶ Compliance
![Page 22: QA / Testing challenges The Gods Must Be Crazy 1980 Sped ... · The Gods Must Be Crazy 1980 Sped Up Stuck In The Mud Scene. Driverless cars? it has no hand brake and if it stalls](https://reader035.vdocument.in/reader035/viewer/2022071219/6055fd1ca1baad6dc573a893/html5/thumbnails/22.jpg)
Security
![Page 23: QA / Testing challenges The Gods Must Be Crazy 1980 Sped ... · The Gods Must Be Crazy 1980 Sped Up Stuck In The Mud Scene. Driverless cars? it has no hand brake and if it stalls](https://reader035.vdocument.in/reader035/viewer/2022071219/6055fd1ca1baad6dc573a893/html5/thumbnails/23.jpg)
Availability and Reliability
▶ Remote
▶ Moving
▶ Low battery
▶ Rely on wireless
![Page 24: QA / Testing challenges The Gods Must Be Crazy 1980 Sped ... · The Gods Must Be Crazy 1980 Sped Up Stuck In The Mud Scene. Driverless cars? it has no hand brake and if it stalls](https://reader035.vdocument.in/reader035/viewer/2022071219/6055fd1ca1baad6dc573a893/html5/thumbnails/24.jpg)
Safety
▶ Irresponsible operators
▶ Decoupling the fate of the driver/pilot from that of the passengers
▶ Unattended devices and machinery
▶ Missing human judgement in critical circumstances
![Page 25: QA / Testing challenges The Gods Must Be Crazy 1980 Sped ... · The Gods Must Be Crazy 1980 Sped Up Stuck In The Mud Scene. Driverless cars? it has no hand brake and if it stalls](https://reader035.vdocument.in/reader035/viewer/2022071219/6055fd1ca1baad6dc573a893/html5/thumbnails/25.jpg)
Resilience
▶ Physical environments
▶ Noise & Interferences
▶ Frequently on and off
▶ Pinged frequently
![Page 26: QA / Testing challenges The Gods Must Be Crazy 1980 Sped ... · The Gods Must Be Crazy 1980 Sped Up Stuck In The Mud Scene. Driverless cars? it has no hand brake and if it stalls](https://reader035.vdocument.in/reader035/viewer/2022071219/6055fd1ca1baad6dc573a893/html5/thumbnails/26.jpg)
So what do we do?
![Page 27: QA / Testing challenges The Gods Must Be Crazy 1980 Sped ... · The Gods Must Be Crazy 1980 Sped Up Stuck In The Mud Scene. Driverless cars? it has no hand brake and if it stalls](https://reader035.vdocument.in/reader035/viewer/2022071219/6055fd1ca1baad6dc573a893/html5/thumbnails/27.jpg)
Traditional structured testing
▶ GUI
▶ Web
▶ Integration
![Page 28: QA / Testing challenges The Gods Must Be Crazy 1980 Sped ... · The Gods Must Be Crazy 1980 Sped Up Stuck In The Mud Scene. Driverless cars? it has no hand brake and if it stalls](https://reader035.vdocument.in/reader035/viewer/2022071219/6055fd1ca1baad6dc573a893/html5/thumbnails/28.jpg)
Testing ‘in the wild’
▶ Drive the truck
▶ Fly the drone
▶ Go to sea
▶ Physical ‘things’
![Page 29: QA / Testing challenges The Gods Must Be Crazy 1980 Sped ... · The Gods Must Be Crazy 1980 Sped Up Stuck In The Mud Scene. Driverless cars? it has no hand brake and if it stalls](https://reader035.vdocument.in/reader035/viewer/2022071219/6055fd1ca1baad6dc573a893/html5/thumbnails/29.jpg)
Service virtualisation
![Page 30: QA / Testing challenges The Gods Must Be Crazy 1980 Sped ... · The Gods Must Be Crazy 1980 Sped Up Stuck In The Mud Scene. Driverless cars? it has no hand brake and if it stalls](https://reader035.vdocument.in/reader035/viewer/2022071219/6055fd1ca1baad6dc573a893/html5/thumbnails/30.jpg)
Security / Penetration testing
▶ Start with the easy stuff
▶ Password strength
▶ Cookies expiry
▶ SQL injection
▶ Ultimately QA includes Security
![Page 31: QA / Testing challenges The Gods Must Be Crazy 1980 Sped ... · The Gods Must Be Crazy 1980 Sped Up Stuck In The Mud Scene. Driverless cars? it has no hand brake and if it stalls](https://reader035.vdocument.in/reader035/viewer/2022071219/6055fd1ca1baad6dc573a893/html5/thumbnails/31.jpg)
User Experience!
![Page 32: QA / Testing challenges The Gods Must Be Crazy 1980 Sped ... · The Gods Must Be Crazy 1980 Sped Up Stuck In The Mud Scene. Driverless cars? it has no hand brake and if it stalls](https://reader035.vdocument.in/reader035/viewer/2022071219/6055fd1ca1baad6dc573a893/html5/thumbnails/32.jpg)
Remember how Glade (a rival brand from SC Johnson) used to have
the jingle that went,“Plug it in, plug it in”? That is not the case with
connected devices. The new theme song should go something like,
“Plug it in, plug it in. And then go through a 50-step process that we
will now explain to you in painstaking detail, and you will still have to
call that 15-year-old kid who hooks up your TV and shows you how to
add people on Snapchat.”
David Berkowitz
![Page 33: QA / Testing challenges The Gods Must Be Crazy 1980 Sped ... · The Gods Must Be Crazy 1980 Sped Up Stuck In The Mud Scene. Driverless cars? it has no hand brake and if it stalls](https://reader035.vdocument.in/reader035/viewer/2022071219/6055fd1ca1baad6dc573a893/html5/thumbnails/33.jpg)
“The entire setup process felt absurd. First, I needed to find and
download the Febreze Connect app on my iPhone – for my air
freshener. Next, I had to connect to a custom WiFi network, requiring
me to look up my home network’s password – for my air freshener.
Then, I had to wait for a firmware update (really) – for my air freshener.
Finally, to differentiate multiple devices should I ever buy a case of
them, I had to come up with a name – for my air freshener (mine goes
by the name Feberky). The user experience will need to improve,
especially for lower-cost devices, so that they are truly plug-and-play.”
David Berkowitz
![Page 34: QA / Testing challenges The Gods Must Be Crazy 1980 Sped ... · The Gods Must Be Crazy 1980 Sped Up Stuck In The Mud Scene. Driverless cars? it has no hand brake and if it stalls](https://reader035.vdocument.in/reader035/viewer/2022071219/6055fd1ca1baad6dc573a893/html5/thumbnails/34.jpg)
So what now?
▶ Upskill
▶ Hardware/Firmware
▶ Basic Electronics
▶ Wireless technologies
▶ Networks
▶ Web technologies
▶ Work with devs
▶ Cloud platforms
▶ Security testing
![Page 35: QA / Testing challenges The Gods Must Be Crazy 1980 Sped ... · The Gods Must Be Crazy 1980 Sped Up Stuck In The Mud Scene. Driverless cars? it has no hand brake and if it stalls](https://reader035.vdocument.in/reader035/viewer/2022071219/6055fd1ca1baad6dc573a893/html5/thumbnails/35.jpg)
Back to the future...
▶ IoT collects Data
▶ Big data
▶ Analytics
▶ Machine Learning
▶ Artificial Intelligence
▶ Bots
▶ Oh my!
![Page 36: QA / Testing challenges The Gods Must Be Crazy 1980 Sped ... · The Gods Must Be Crazy 1980 Sped Up Stuck In The Mud Scene. Driverless cars? it has no hand brake and if it stalls](https://reader035.vdocument.in/reader035/viewer/2022071219/6055fd1ca1baad6dc573a893/html5/thumbnails/36.jpg)
A word on automation
▶ Hard to test end to end
▶ Combination of Desktop, Web, Mobile and Hardware
testing
▶ We are all familiar with Desktop, Web, Mobile
▶ Hardware integration test tools such as LabView
▶ Service Virtualisation
▶ Testing APIs: Vendor UI based such as SoapUI or http client
proxy such as Postman or better yet http client APIs and
libraries such as RestSharp (.Net) and Rest Assured (Java)
▶ In-house, custom-built test rigs or test jigs
![Page 37: QA / Testing challenges The Gods Must Be Crazy 1980 Sped ... · The Gods Must Be Crazy 1980 Sped Up Stuck In The Mud Scene. Driverless cars? it has no hand brake and if it stalls](https://reader035.vdocument.in/reader035/viewer/2022071219/6055fd1ca1baad6dc573a893/html5/thumbnails/37.jpg)
Conclusion
▶ IoT here to stay and expected to grow
▶ Blend existing strategies with innovative ones
▶ Testers to upskill
▶ Increase cooperation with devs/devOps/IT staff
▶ Include security/penetration testing in test plans
But…...
![Page 38: QA / Testing challenges The Gods Must Be Crazy 1980 Sped ... · The Gods Must Be Crazy 1980 Sped Up Stuck In The Mud Scene. Driverless cars? it has no hand brake and if it stalls](https://reader035.vdocument.in/reader035/viewer/2022071219/6055fd1ca1baad6dc573a893/html5/thumbnails/38.jpg)
![Page 39: QA / Testing challenges The Gods Must Be Crazy 1980 Sped ... · The Gods Must Be Crazy 1980 Sped Up Stuck In The Mud Scene. Driverless cars? it has no hand brake and if it stalls](https://reader035.vdocument.in/reader035/viewer/2022071219/6055fd1ca1baad6dc573a893/html5/thumbnails/39.jpg)
Questions, Queries &
Comments ….