automated test system - build blocks' - denx · introduction system description building...
TRANSCRIPT
. . . . . .
Introduction System description Building blocks Summary Discussion
”Automated Test System - build blocks”
Łukasz Majewski
Samsung R&D Institute Poland
Embedded Linux Conference EuropeDublin, 05-10-2015
Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 1/25
. . . . . .
Introduction System description Building blocks Summary Discussion
Outline
1 Introduction
2 System description
3 Building blocks
4 Summary
5 Discussion
Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 2/25
. . . . . .
Introduction System description Building blocks Summary Discussion
Section 1
Introduction
Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 3/25
. . . . . .
Introduction System description Building blocks Summary Discussion
A few words about me
Embedded systems programmerUsing u-boot since 2008”Blame me” for:
Device Firmware Upgrade (DFU)Trats/Trats2 devices
Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 4/25
. . . . . .
Introduction System description Building blocks Summary Discussion
Presentation goal
Presentation goalSharing knowledge and experience after building simple and costeffective setup for automated u-boot and kernel tests with usingopen source SW and off-the-shelf HW.
Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 5/25
. . . . . .
Introduction System description Building blocks Summary Discussion
Testing our u-boot
2011: Build test infrastructure (u-boot-CI)
2014: USB DFU/USB Mass Storage test scripts in mainline2014: Boot and flashing tests - HardWare Test (HWT) v12015: HWT v2
Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 6/25
. . . . . .
Introduction System description Building blocks Summary Discussion
Testing our u-boot
2011: Build test infrastructure (u-boot-CI)2014: USB DFU/USB Mass Storage test scripts in mainline
2014: Boot and flashing tests - HardWare Test (HWT) v12015: HWT v2
Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 6/25
. . . . . .
Introduction System description Building blocks Summary Discussion
Testing our u-boot
2011: Build test infrastructure (u-boot-CI)2014: USB DFU/USB Mass Storage test scripts in mainline2014: Boot and flashing tests - HardWare Test (HWT) v1
2015: HWT v2
Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 6/25
. . . . . .
Introduction System description Building blocks Summary Discussion
Testing our u-boot
2011: Build test infrastructure (u-boot-CI)2014: USB DFU/USB Mass Storage test scripts in mainline2014: Boot and flashing tests - HardWare Test (HWT) v12015: HWT v2
Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 6/25
. . . . . .
Introduction System description Building blocks Summary Discussion
Automated Test System
Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 7/25
. . . . . .
Introduction System description Building blocks Summary Discussion
Automated Test System - overview
Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 8/25
. . . . . .
Introduction System description Building blocks Summary Discussion
Automated Test System - block diagram
Board1
BoardN
Host PCWiki
USB HUB
JenkinsTests
USB HUB USB HUBPWR IO
RelaysUSB <->UART
USB <->UART
Power
Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 9/25
. . . . . .
Introduction System description Building blocks Summary Discussion
Section 2
System description
Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 10/25
. . . . . .
Introduction System description Building blocks Summary Discussion
Architecture
N testsSetup
Instance
TestInstance
ConfigPer board:- Toolchain- HW config
- Board quirks
Odroid U3/XU3
Trats Board N
Setupscripts
N testsTest
scripts
SetupInstance
TestInstance
SetupInstance
TestInstance
SetupInstance
TestInstance
SetupInstance
TestInstance
SetupInstance
TestInstance
ConfigPer board:- Toolchain- HW config
- Board quirks
ConfigPer board:- Toolchain- HW config
- Board quirks
Jenkins and WIKIresultsvisualization
Toolchains
GIT reposTest trigger:GIT push
Test trigger:Schedule
- Build bootloader and kernel- Download rootfs- Flash target board
- Execute script on target board- Use procfs, sysfs,debugfs, etc.- Test result:0 → OKOther → FAIL
Links to proper tests/setuprepositories.Test/setup instances representedas directories.
Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 11/25
. . . . . .
Introduction System description Building blocks Summary Discussion
Key design decisions
Why: New vs. Reuse
SW: Python vs. ShellConsole: Python (pexpect) vs. expect (CLI program)Flashing: Ethernet (TFTP) vs. USB (DFU)Scheduling: JenkinsPresentation: Wiki
Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 12/25
. . . . . .
Introduction System description Building blocks Summary Discussion
Key design decisions
Why: New vs. ReuseSW: Python vs. Shell
Console: Python (pexpect) vs. expect (CLI program)Flashing: Ethernet (TFTP) vs. USB (DFU)Scheduling: JenkinsPresentation: Wiki
Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 12/25
. . . . . .
Introduction System description Building blocks Summary Discussion
Key design decisions
Why: New vs. ReuseSW: Python vs. ShellConsole: Python (pexpect) vs. expect (CLI program)
Flashing: Ethernet (TFTP) vs. USB (DFU)Scheduling: JenkinsPresentation: Wiki
Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 12/25
. . . . . .
Introduction System description Building blocks Summary Discussion
Key design decisions
Why: New vs. ReuseSW: Python vs. ShellConsole: Python (pexpect) vs. expect (CLI program)Flashing: Ethernet (TFTP) vs. USB (DFU)
Scheduling: JenkinsPresentation: Wiki
Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 12/25
. . . . . .
Introduction System description Building blocks Summary Discussion
Key design decisions
Why: New vs. ReuseSW: Python vs. ShellConsole: Python (pexpect) vs. expect (CLI program)Flashing: Ethernet (TFTP) vs. USB (DFU)Scheduling: Jenkins
Presentation: Wiki
Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 12/25
. . . . . .
Introduction System description Building blocks Summary Discussion
Key design decisions
Why: New vs. ReuseSW: Python vs. ShellConsole: Python (pexpect) vs. expect (CLI program)Flashing: Ethernet (TFTP) vs. USB (DFU)Scheduling: JenkinsPresentation: Wiki
Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 12/25
. . . . . .
Introduction System description Building blocks Summary Discussion
Section 3
Building blocks
Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 13/25
. . . . . .
Introduction System description Building blocks Summary Discussion
Power control
FTDI FT232RLNote: https://github.com/lmajewski/HWT_io_ctl.git
RelaysNote: GPIO-controlled relays module for Arduino
USB HUBNote: External and stable power supply
Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 14/25
. . . . . .
Introduction System description Building blocks Summary Discussion
Host PC
Ubuntu Long Term Support (LTS)USB udev configurationRuns:
Dash testsJenkins clientRepository mirroringMediaWiki
Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 15/25
. . . . . .
Introduction System description Building blocks Summary Discussion
Host PC - Test U-boot booting
Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 16/25
. . . . . .
Introduction System description Building blocks Summary Discussion
Host PC - Test kernel
Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 17/25
. . . . . .
Introduction System description Building blocks Summary Discussion
Host PC - Jenkins [optional for simple setup]
Responsible for scheduling testsProvides more sophisticated functionality to the test setup:
Gerrit push triggerRepository/branch change trigger
For simple test system it can be omitted - cron managed testtriggering with results uploaded to wiki
Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 18/25
. . . . . .
Introduction System description Building blocks Summary Discussion
Host PC - Wiki
Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 19/25
. . . . . .
Introduction System description Building blocks Summary Discussion
Minimal HW costs
Device Price [EUR] RemarksFT232RL 10Relays 5USB HUB 40 External powerHost PC 0 Old reusedCables/connectors 10Power supply 0 Included with target boardTotal 65
Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 20/25
. . . . . .
Introduction System description Building blocks Summary Discussion
Section 4
Summary
Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 21/25
. . . . . .
Introduction System description Building blocks Summary Discussion
Solved problems
Problem: Breaking the default boot
Solution: Use C/C++ with nanosleep()Problem: Jenkins logs readability
Solution: Concatenate logs and link them to Wiki
Problem: Target power control
Solution: HW rework to also control VUSB
Problem: Lack of persistent USB configuration
Solution: Test system connectivity managed by udev rules,which use USB device serial numbers
Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 22/25
. . . . . .
Introduction System description Building blocks Summary Discussion
Solved problems
Problem: Breaking the default bootSolution: Use C/C++ with nanosleep()
Problem: Jenkins logs readability
Solution: Concatenate logs and link them to Wiki
Problem: Target power control
Solution: HW rework to also control VUSB
Problem: Lack of persistent USB configuration
Solution: Test system connectivity managed by udev rules,which use USB device serial numbers
Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 22/25
. . . . . .
Introduction System description Building blocks Summary Discussion
Solved problems
Problem: Breaking the default bootSolution: Use C/C++ with nanosleep()
Problem: Jenkins logs readability
Solution: Concatenate logs and link them to WikiProblem: Target power control
Solution: HW rework to also control VUSB
Problem: Lack of persistent USB configuration
Solution: Test system connectivity managed by udev rules,which use USB device serial numbers
Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 22/25
. . . . . .
Introduction System description Building blocks Summary Discussion
Solved problems
Problem: Breaking the default bootSolution: Use C/C++ with nanosleep()
Problem: Jenkins logs readabilitySolution: Concatenate logs and link them to Wiki
Problem: Target power control
Solution: HW rework to also control VUSB
Problem: Lack of persistent USB configuration
Solution: Test system connectivity managed by udev rules,which use USB device serial numbers
Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 22/25
. . . . . .
Introduction System description Building blocks Summary Discussion
Solved problems
Problem: Breaking the default bootSolution: Use C/C++ with nanosleep()
Problem: Jenkins logs readabilitySolution: Concatenate logs and link them to Wiki
Problem: Target power control
Solution: HW rework to also control VUSBProblem: Lack of persistent USB configuration
Solution: Test system connectivity managed by udev rules,which use USB device serial numbers
Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 22/25
. . . . . .
Introduction System description Building blocks Summary Discussion
Solved problems
Problem: Breaking the default bootSolution: Use C/C++ with nanosleep()
Problem: Jenkins logs readabilitySolution: Concatenate logs and link them to Wiki
Problem: Target power controlSolution: HW rework to also control VUSB
Problem: Lack of persistent USB configuration
Solution: Test system connectivity managed by udev rules,which use USB device serial numbers
Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 22/25
. . . . . .
Introduction System description Building blocks Summary Discussion
Solved problems
Problem: Breaking the default bootSolution: Use C/C++ with nanosleep()
Problem: Jenkins logs readabilitySolution: Concatenate logs and link them to Wiki
Problem: Target power controlSolution: HW rework to also control VUSB
Problem: Lack of persistent USB configuration
Solution: Test system connectivity managed by udev rules,which use USB device serial numbers
Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 22/25
. . . . . .
Introduction System description Building blocks Summary Discussion
Solved problems
Problem: Breaking the default bootSolution: Use C/C++ with nanosleep()
Problem: Jenkins logs readabilitySolution: Concatenate logs and link them to Wiki
Problem: Target power controlSolution: HW rework to also control VUSB
Problem: Lack of persistent USB configurationSolution: Test system connectivity managed by udev rules,which use USB device serial numbers
Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 22/25
. . . . . .
Introduction System description Building blocks Summary Discussion
Found bugs:
u-boot: SD card timeout for writing datau-boot: Wrong error status passing from FAT to upper layersLinux: ”rootwait” problem in Trats/Trats2
Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 23/25
. . . . . .
Introduction System description Building blocks Summary Discussion
Conclusion
YES, it was worth to build this setup.
Benefits:Tizen.org: build, boot and functionality testsu-boot-dfu: build, boot (u-boot and kernel), USB: MassStorage, DFU, THOR
Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 24/25
. . . . . .
Introduction System description Building blocks Summary Discussion
Conclusion
YES, it was worth to build this setup.Benefits:
Tizen.org: build, boot and functionality testsu-boot-dfu: build, boot (u-boot and kernel), USB: MassStorage, DFU, THOR
Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 24/25