california institute of technology requirements decomposition analysis, model based testing of...
TRANSCRIPT
![Page 1: California Institute of Technology Requirements Decomposition Analysis, Model Based Testing of Sequential Code Properties Allen P. Nikora, John D. Powell](https://reader036.vdocument.in/reader036/viewer/2022072016/56649ef05503460f94c00789/html5/thumbnails/1.jpg)
California Institute of Technology
Requirements Decomposition Analysis,Model Based Testing of Sequential Code
PropertiesAllen P. Nikora, John D. Powell
Jet Propulsion Laboratory,California Institute of Technology
Pasadena, [email protected] D. [email protected]
The work described in this paper was carried out at the Jet Propulsion Laboratory, California Institute of Technology. This work is sponsored by the National Aeronautics and Space Administration’s Office of Safety and Mission Assurance under the NASA Software Program led by the NASA Software IV&V Facility. This activity is managed locally at JPL through the Assurance Technology Program Office (ATPO).
![Page 2: California Institute of Technology Requirements Decomposition Analysis, Model Based Testing of Sequential Code Properties Allen P. Nikora, John D. Powell](https://reader036.vdocument.in/reader036/viewer/2022072016/56649ef05503460f94c00789/html5/thumbnails/2.jpg)
20 July, 2004 OSMA Software Assurance Symposium 2
California Institute of Technology
Requirements Decomposition AnalysisTask Description
Requirements Decomposition Analysis Problem Statement: Requirements play a pivotal role in
planning, selection, development, testing and operation of NASA's missions. Starting from mission objectives, requirements are successively decomposed. The correctness of this decomposition is critical, yet V&V of this crucial step is limited to manual inspection and pointwise testing, which are cumbersome and fallible (e.g., Mars Polar Lander).
Task: Rigorous lightweight analysis methods for requirements decomposition have been developed by the software engineering research community, and have shown promise in successful application to critical systems (e.g., rail transportation). We study their application to the V&V of spacecraft software requirements, to ascertain if, when and how they are suitable for use by NASA.
![Page 3: California Institute of Technology Requirements Decomposition Analysis, Model Based Testing of Sequential Code Properties Allen P. Nikora, John D. Powell](https://reader036.vdocument.in/reader036/viewer/2022072016/56649ef05503460f94c00789/html5/thumbnails/3.jpg)
20 July, 2004 OSMA Software Assurance Symposium 3
California Institute of Technology
Requirements Decomposition AnalysisTask Description (cont’d)
Model Based Testing of Sequential Code Properties Problem Statement: A common issue with model checking is
that the state space to be explored can become too large to search in a reasonable time. This may prevent the application of model checking techniques in situations where it may be desirable to do so. Reducing the number of states visited would make it possible to apply model checking to larger systems.
Task: Recent results indicate that checking a small number of states in a system model is as effective as checking a large number of states. We explore this conjecture by comparing the results of applying traditional model checkers, such as SPIN, to the results of applying the LURCH tool, which does not visit all of the system`s states.
![Page 4: California Institute of Technology Requirements Decomposition Analysis, Model Based Testing of Sequential Code Properties Allen P. Nikora, John D. Powell](https://reader036.vdocument.in/reader036/viewer/2022072016/56649ef05503460f94c00789/html5/thumbnails/4.jpg)
20 July, 2004 OSMA Software Assurance Symposium 4
California Institute of Technology
Requirements Decomposition AnalysisGoals and Objectives
Requirements Decomposition Analysis Goal: study the applicability to NASA spacecraft requirements of
rigorous analysis methods for requirements decomposition that have been developed by the software engineering research community.
Objectives:1. Manually apply decomposition analysis methods applied to
spacecraft requirements.2. Based on the results of of these application studies, emerge with
recommendations for the application of these methods, identify needed extensions to those methods, and indicate the opportunities for their support (e.g., via checklists, procedures and/or tool support).
3. Develop the most promising support approaches identified by the first phase to make them suitable for application to NASA's spacecraft requirements.
![Page 5: California Institute of Technology Requirements Decomposition Analysis, Model Based Testing of Sequential Code Properties Allen P. Nikora, John D. Powell](https://reader036.vdocument.in/reader036/viewer/2022072016/56649ef05503460f94c00789/html5/thumbnails/5.jpg)
20 July, 2004 OSMA Software Assurance Symposium 5
California Institute of Technology
Requirements Decomposition AnalysisGoals and Objectives (cont’d)
Model Based Testing of Sequential Code Properties Goal: Determine whether reduced-state model
checking can be as effective as traditional model checking
Objectives:1. Manually translate an existing formal model of a
JPL system into LURCH MER Arbiter
2. Apply LURCH and traditional model checking techniques (i.e., SPIN) to the models
3. Compare the effectiveness of LURCH and SPIN in identifying errors in the models.
![Page 6: California Institute of Technology Requirements Decomposition Analysis, Model Based Testing of Sequential Code Properties Allen P. Nikora, John D. Powell](https://reader036.vdocument.in/reader036/viewer/2022072016/56649ef05503460f94c00789/html5/thumbnails/6.jpg)
20 July, 2004 OSMA Software Assurance Symposium 6
California Institute of Technology
Requirements Decomposition AnalysisImportance and Benefits
As the complexity of spacecraft systems increases, and as increasing reliance is placed on software as an enhancing or enabling technology, the need for analysis of requirements decomposition is expected to grow further.
Improved methods for assuring correctness of requirements decomposition advance the state of the practice in software V&V.
Early detection of flaws in requirements decomposition permits early-lifecycle repair, thus avoiding costlier downstream repairs.
Reducing the state space required for model checking will enable analytical verification of larger systems with less effort.
![Page 7: California Institute of Technology Requirements Decomposition Analysis, Model Based Testing of Sequential Code Properties Allen P. Nikora, John D. Powell](https://reader036.vdocument.in/reader036/viewer/2022072016/56649ef05503460f94c00789/html5/thumbnails/7.jpg)
20 July, 2004 OSMA Software Assurance Symposium 7
California Institute of Technology
Requirements Decomposition AnalysisRelevance to NASA
The study uses requirements of actual NASA spacecraft The New Millennium Program ST6 Autonomous
Rendezvous eXperiment (ARX) Purpose: demonstrate autonomous rendezvous between
the spacecraft and a passive in-orbit payload Key technology demonstration in preparation for the
Mars Sample Return mission Mission requirements decompose into software
requirements which must control the laser rangefinder, the on-board calculation of trajectory maneuvers, the commanding of the propulsion system, and the orchestration of data downlinks to report the mission's results to Earth.
![Page 8: California Institute of Technology Requirements Decomposition Analysis, Model Based Testing of Sequential Code Properties Allen P. Nikora, John D. Powell](https://reader036.vdocument.in/reader036/viewer/2022072016/56649ef05503460f94c00789/html5/thumbnails/8.jpg)
20 July, 2004 OSMA Software Assurance Symposium 8
California Institute of Technology
Requirements Decomposition AnalysisRelevance to NASA (cont’d)
The study uses requirements of actual NASA spacecraft (cont’d) Mars Reconnaissance Orbiter
Goals:• Study the history of water on Mars• Become first link in “interplanetary network”
Mission requirements decompose into software requirements for typical on-board functions
• Commanding• Data handling• Navigation• Attitude Control
Mars Exploration Rover arbiter (for model checking investigation)
![Page 9: California Institute of Technology Requirements Decomposition Analysis, Model Based Testing of Sequential Code Properties Allen P. Nikora, John D. Powell](https://reader036.vdocument.in/reader036/viewer/2022072016/56649ef05503460f94c00789/html5/thumbnails/9.jpg)
20 July, 2004 OSMA Software Assurance Symposium 9
California Institute of Technology
Requirements Decomposition AnalysisHighlights
Examined ST-6 Autonomous Sciencecraft Experiment requirements (approx. 9 pages)
Got a feel for the potential complexity of analyzing the decomposition of resource requirements, while working with a relatively small set of requirements (approximately 9 pages of technical detail)
Focused on the Mars Reconnaissance Orbiter requirements (approx. 1,370 in all) Developed a means to use the project-provided traceability information to
extract all the requirements that are related to a requirement of interest WHY: Convenience & comprehension – extracts just those requirements
connected, directly or indirectly, assembling the results into a (web-browser-viewable) table. The result is easier to study than following individual links within the large set of requirements, and is more focused than the “graphic mode” that the requirements tool DOORS provides. In the event of the need to make a change to a requirement, this capability has potential utility, by finding and reporting all the requirements related (directly or indirectly) to that requirement.
HOW: This is in the form of an automatic script, which takes as input the user’s identification of the requirement of interest, and returns the requirements linked to that (both “parents” of that requirement, and “children” of that requirement), the requirements linked to those requirements, etc.
![Page 10: California Institute of Technology Requirements Decomposition Analysis, Model Based Testing of Sequential Code Properties Allen P. Nikora, John D. Powell](https://reader036.vdocument.in/reader036/viewer/2022072016/56649ef05503460f94c00789/html5/thumbnails/10.jpg)
20 July, 2004 OSMA Software Assurance Symposium 10
California Institute of Technology
Requirements Decomposition AnalysisHighlights (cont’d)
Focused on the Mars Reconnaissance Orbiter requirements (cont’d) Developed a means independent of the project traceability
information to extract relevant requirements. WHY: avoids reliance on the potentially incomplete or
incorrect the traceability information within the existing documentation, thus giving a means in independently assure the correctness of requirements decomposition.
HOW: text-based search for keywords (e.g., search for the word “mass” and the string “kg” – for kilograms), and regular-expression textual searches for more refined patterns (e.g., a digit, then a space, then the letters “kg”)
![Page 11: California Institute of Technology Requirements Decomposition Analysis, Model Based Testing of Sequential Code Properties Allen P. Nikora, John D. Powell](https://reader036.vdocument.in/reader036/viewer/2022072016/56649ef05503460f94c00789/html5/thumbnails/11.jpg)
20 July, 2004 OSMA Software Assurance Symposium 11
California Institute of Technology
Query 1: yields this set of requirements
Query 2: yields this set of requirements
As before, for easy of viewing the results are presented in HTML tables that provide hyperlinks to the requirements themselves.
Simplest example, of two result sets
Example: requirements that trace to a mass-related requirement
Example: requirements that involve the string Kg
Result: mass-related requirements that do NOT involve the string Kg
Result: requirements that do involve the string Kg but are not related to mass requirements
Result: requirements that do involve the string Kg AND are related to mass requirements
Requirements Decomposition AnalysisHighlights (cont’d)
Trace- and string- based means to query a set of requirements have been developed. The result of such a query is a set of requirements. We have also developed capabilities to compare such result sets. Given two or more queries each of which yields a set of requirements, the capabilities developed allow the calculation of the intersection, difference and union among the several returned sets.
The simplest example, of two results sets, is shown below. The results are placed into one of three categories: occurs in only the results returned by the first query; occurs in only the results returned by the second query; occurs in both sets of results (see diagram below).More generally, for N queries, results are distributed among 2N – 1 categories.
![Page 12: California Institute of Technology Requirements Decomposition Analysis, Model Based Testing of Sequential Code Properties Allen P. Nikora, John D. Powell](https://reader036.vdocument.in/reader036/viewer/2022072016/56649ef05503460f94c00789/html5/thumbnails/12.jpg)
20 July, 2004 OSMA Software Assurance Symposium 12
California Institute of Technology
Requirements Decomposition AnalysisHighlights (cont’d)
Tools Developed TraceRequirements.pl
Traces one or more requirements through a set of documents
For each specified requirement, parents and children are found, as well as siblings and all other possible relatives
FindPatterns.pl A specified set of documents is searched for one or more
patterns Patterns can be specified as regular expressions
CompareResults.pl Two or more trace and/or search results are compared to
identify requirements common to the results
![Page 13: California Institute of Technology Requirements Decomposition Analysis, Model Based Testing of Sequential Code Properties Allen P. Nikora, John D. Powell](https://reader036.vdocument.in/reader036/viewer/2022072016/56649ef05503460f94c00789/html5/thumbnails/13.jpg)
20 July, 2004 OSMA Software Assurance Symposium 13
California Institute of Technology
Model Based Testing ofSequential Code Properties
Highlights
Simplicity of Modeling Systems with Embedded C code Lurch Modeling Language annotates C code to randomly exercise
the code User Modeled a system with only 15 hours training Model allows the set of legal calling sequence Model prohibits the set of legal calling sequence
Promela (SPIN Modeling Language) exhaustively search the model’s state space Steep Learning Curve Semester long College course required C code faults reported as having occurred at the model lever
where the C call is made Make debugging embedded C code hard Confusion over whether errors are
Errors Embedding C in Promela Errors in the imbedded C code itself
![Page 14: California Institute of Technology Requirements Decomposition Analysis, Model Based Testing of Sequential Code Properties Allen P. Nikora, John D. Powell](https://reader036.vdocument.in/reader036/viewer/2022072016/56649ef05503460f94c00789/html5/thumbnails/14.jpg)
20 July, 2004 OSMA Software Assurance Symposium 14
California Institute of TechnologyModel Based Testing of
Sequential Code PropertiesHighlights (cont’d)
Accuracy Lurch confirmed deadlock violations found earlier with SPIN Lurch provided easier access (than SPIN) to information
about the location of C code faults After faults repaired
The new C code is used in conjunction with the old SPIN model
The number of SPIN reported verification problems was reduced by half
SPIN subsequently ran out of memory before quitting• RA-RRE Model still too big for LURCH
![Page 15: California Institute of Technology Requirements Decomposition Analysis, Model Based Testing of Sequential Code Properties Allen P. Nikora, John D. Powell](https://reader036.vdocument.in/reader036/viewer/2022072016/56649ef05503460f94c00789/html5/thumbnails/15.jpg)
20 July, 2004 OSMA Software Assurance Symposium 15
California Institute of Technology
Model Based Testing of Sequential Code PropertiesHighlights (cont’d)
Scalability of LURCH LURCH scales better than SPIN over larger systems X-axis is # of entities / size Y-axis is Time / Memory Used
![Page 16: California Institute of Technology Requirements Decomposition Analysis, Model Based Testing of Sequential Code Properties Allen P. Nikora, John D. Powell](https://reader036.vdocument.in/reader036/viewer/2022072016/56649ef05503460f94c00789/html5/thumbnails/16.jpg)
20 July, 2004 OSMA Software Assurance Symposium 16
California Institute of Technology
Requirements Decomposition AnalysisFuture Work
Reduce false positives for traces, searches Develop guidelines for consistently expressing
Static values Temporal properties
Natural language processing? Integrate with measurement tools
In theory, measurable changes in requirements could be related to number of faults inserted/number of failures observed
Requires traceability to implementation
![Page 17: California Institute of Technology Requirements Decomposition Analysis, Model Based Testing of Sequential Code Properties Allen P. Nikora, John D. Powell](https://reader036.vdocument.in/reader036/viewer/2022072016/56649ef05503460f94c00789/html5/thumbnails/17.jpg)
20 July, 2004 OSMA Software Assurance Symposium 17
California Institute of TechnologyModel Based Testing of
Sequential Code PropertiesFuture Work
Automatic Generation of LURCH models from State Charts Source Code
Extension of LURCH for Test Case Generation Application of LURCH as a C source code debugging agent Application of LURCH to ongoing software projects Future work contingent on securing additional funding
Current work performed with minimal funds (12K) and achieved
Solid value added with respect to cost (ROI) Valuable Lessons learned LURCH established as viable and practical testing tool
Clarified and Next Steps and for Maximizing future ROI