lab 0 example from encm511 webfaq where more tests are ...smithmr/2014webs/encm... · m. smith...
Post on 27-Sep-2020
1 Views
Preview:
TRANSCRIPT
10/8/2014 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
1
Automated Testing Environment
Concepts required for testing embedded systems adopted in this course
(quizzes, assignments and laboratories)
Lab 0 Example from ENCM511 WebFAQ where more tests are needed
10/8/2014 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
2 / 28
Corrected test with proper system initialization - see FAQ sheet
10/8/2014 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
3 / 28 10/8/2014 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
4 / 28
To be tackled today
Why test, and what kinds of tests are there? Difference between
TLD – Test Last Development TDD – Test Driven Development
(Test First Development) How do you add tests? More details on the testing process E-TDD
and the testing toll E-UNIT Other kinds of available tests – time and
access Design of custom TESTs and ASSERTS
10/8/2014 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
5 / 28
Why test? Unit test
I AM CURRENTLY BUILDING A NEW FUNCTION –Does it work the way I expect?
Testing my and my partner’s understanding of the Design WE ARE DESIGNING A NEW PRODUCT
-- If we can’t explain (to ourselves via a test) the product ideas – how do we know when product is working?
Regression testing MY PARTNER ADDED A NEW FEATURE to a product –
how do I know it did not “break something” I wrote? System testing
How do I PROVE TO THE CUSTOMER that the product (Lab. or assignment) works the way that was requested?
10/8/2014 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
6 / 28
What are the problems with this sort of “Human intervention” test?
This TEST codewould
(confusingly) be placed in
Assignments1directory with the rest of the customer code
10/8/2014 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
7 / 28
CCES development environment(IDDE) – Communication overhead
DebuggingInterface
“JTAG” boundary scan operation
Processor
Peripherals
BF533 Evaluation Board
CCES IDE
(VisualDSP++IDE,
MIPS IAR IDE)
Debugging Interface
HPUSB-ICE
Lab. station (ENA301 / 305)
Each time we want to send a message (via printf( ) or cout) we must “pause” the processor from doing what we want it to do, and allow sending of messages over the interface – Slow, plus the testing can impact on peripheral performance.
E.g. no sound while doing (debug) printing in Assignment 2 and Lab. 1
THIS PROBLEM IS COMMON TO ALL EMBEDDED SYSTEM DEVELOPMENT; NOT JUST THE BLACKFIN USED IN THE LAB.
10/8/2014 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
8 / 28
More issues with this sort of test!
Tests are mixed up “with production code”, in many places Must remove tests before release to customer
Unreliable when done manually – may result in test messages at wrong time in released code (confusing to user)
Difficult to “add” and “remove tests” from production code without introducing new errors
Results must be checked manually – therefore unlikely to have tests run often during development – leads to unreliable product
10/8/2014 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
9 / 28
E-TDD / E-UNIT Tool E-TDD – embedded test driven development
Build-the-tests-first (during design) ideas for product development has been made popular by the “Agile” approach
Requires an automated testing approach (test framework) In this class, we are going to use the E-Unit tool for
its automated behaviour Will use in both “test last” development (What most people
currently do) and in “test first” development (What I am trying to do , and manage 60% of the time, and encourage you to do, expectation that you will succees at least 20% of the time).
I (the customer) will provide tests so that you can check that you have developed the “right” stuff at the “high” level
You will build additional tests to check your own code at the “low level”
10/8/2014 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
10 / 28
Test Environment LabX directory is where you keep
your “customer-specific” code (.dxe)
LabXLibrary is where you keep your“utility’s code”. Archive or .dlb for re-use in other labs
LabXTests is where you keep the testsfor the customer specific code and for the Library code you will reuse
Refactoring – You may have to addadditional tests, and move thingsbetween Library and Customer
“2013 example” Assign1 directory – Customer code
The 2013 Assign1 code uses many library calls ReadSwitches( ) andWriteLED( ) Use Open-CCES-ame plug-in
learned in Lab0 Use Open-CCES-ame plug-in to add
in paths to Assign1Library.h and Assign1Library.dlb
2013 Assign1 code may just be a file with a small main( )
I developed code in Assign1 directory and then refactored into Assign1Library
10/8/2014 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
11 / 28 10/8/2014 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
12 / 28
Build a new CCES project “Assign1Tests” and place in “Assign1Tests” directory (2013 example)
I developed TESTs as I added new code to Assign1 and Assign1Library
Certain standard files in a TEST project Same set of files for all TESTs
in assignments and laboratory Use the automated E-UNIT
HELPER features of the OPEN-CCES-ame plugin to “convert” project to TEST project
10/8/2014 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
13 / 28
Testing your (re-usable) code stored in the Assign1Library directory with E-UNIT HELPER
Don’t move or copy the files into the test directory – You will end up with two “different” versions – BAD PRACTICE
Add Path to Assign1Library.h header file by clicking on Assign1Library directory (NOT src” or the specific .h file)
Add Path to Assign1Library.dlb library (or archive) file in Assign1Library directory “debug” (Click on desired .dlb)
10/8/2014 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
14 / 28
Open-CESS-ame plug-in has automatically added the “Tests_main.cpp”(2013 E-Unit GUI was refactored across the summer with new features)
Open-CCES-ame has automatically added a “TestGroup” file for you
10/8/2014 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
15 / 28 10/8/2014 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
16 / 28
Modify “Assign1Tests.cpp” to perform the needed testing
10/8/2014 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
17 / 28
Tests and Test Groups have a standard format. NOTE – Write tests but no code -- common quiz question
TESTGROUP NAME
TESTGROUP NAME,
TEST NAME
Add further tests at this location .
E-UNIT can do this automatically but CUT-AND-PASTE
seems faster after a while
10/8/2014 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
18 / 28
Tests then need to be customizedCan be automated by Open-CCESS-ame
2013 example
10/8/2014 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
19 / 28
Test ComponentsRemember format for “exams” and “quizzes”
The ACTUAL Test CHECK(result1 == result2) If an error occurred what was ‘result1’ being used for? Error result1 != result2 We would get better test error messages if we
“self-documented the code”int resultCPPCode = ; int resultASMCode = CHECK(resultCPPCode == resultASMCode);
CHECK_EQUAL(resultCPPCode, resultASMCode);
Error resultCPPCode != resultASMCode is a UBH Test CONTROL statements
TEST_CONTROL( testgroup_name) TEST_FILE_RUN_NOTIFICATION( testgroup_name)
Add stub versions of equivalent CPP and ASM functions into Assign1Library and Build library (2013 e.g.)
10/8/2014 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
20 / 28
Note “code DEFECT” (logical error NOT discovered as Assembler syntax error)
There is NO return value placed in R0
Assign1CPP.cpp
Assign1ASM.asm
Now build Assign1Tests source files to make executable
10/8/2014 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
21 / 28
Note the format of the error messages with both C / C++ and name-mangled ASM name shown
Fix and rebuild Library code. Review the code for syntax errors and “logical” errors.
10/8/2014 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
22 / 28
DON’T IGNORE WARNINGSFIX THE PROBLEM, e.g. add .extern or fix type
10/8/2014 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
23 / 28
Now “build and run” the testsAll the tests MUST fail (no ASM code written) otherwise using bad tests
Detailed test failure information
Something wrong – how did the ASM test pass with incorrect code (missing R0 return value)?
The test that “passed” has “failed” as a test
10/8/2014 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
24 / 28
Clicking on the “detailed test info”automatically takes you to the “test” that failed
What happens if reverse call order
Assign1_ASM( ) thenAssign1_CPP( )
Better tests? Which one “will fail” correctly – and why (quiz question)?
10/8/2014 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
25 / 28
Actually – I don’t know Depends on how compiler decides to implement the code and (bad) luck
(100 – 2 * 0.42)% chance that version 2 will fail
(2.25 * 42)% that version 1 will fail
10/8/2014 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
26 / 28
Test information modes arecontrolled in “Test_main” Show Failures and Expected Failures only
Show Failures and Successes
10/8/2014 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
27 / 28
Use helper to add a separate test to show that result2 (from the ASM code) is equal to 42
Add further tests at this location
10/8/2014 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
28 / 28
Write the tests for “self-documenting” messages(What does “result1 == result2” mean if 200 tests present?)
HAVE – What is a ‘result1’ and a ‘result 2’
WANT
10/8/2014
Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
29 / 28
Write the “CPP” code to get to workbut “Write the ASM for speed (ISR)”
No need to measure times in Assignment1 or Labs unless doing Interrupt Service Routine
Very key concepts on real-time DSP (515)
FAIL
SUCCESS
SUCCESS
USELESS TOHAVE FAST CODE
THAT FAILSCUSTOMER
REQUIREMENTS
10/8/2014 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
30 / 28
Available commonly used ASSERTSMore can be “customized”
CHECK(expected == actual) – Success if passesXF_CHECK(expected == actual) – Success if “fails”CHECK_EQUAL(expected, actual)XF_CHECK_EQUAL(expected, actual) – Success if failsCHECK_CLOSE(expected, actual, tolerance)CHECK_ARRAY_EQUAL(expected, actual, length)CHECK_ARRAY_CLOSE(expected, actual, length, tolerance)
REPORT(“character message”);
Changes to editor properties can make these ASSERTS into highlighted key-words
10/8/2014 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
31 / 28
Handled today – some easy “non-programming” quiz and midterm questions
Why test, and what kinds of tests are there? Difference between
TLD – Test Last Development TDD – Test Driven Development
(Test First Development) How do you add tests for Laboratory 1? More details on the testing process E-TDD
and the testing tool E-UNIT Other kinds of available tests – time and
access Design of custom TESTs and ASSERTS
top related