a work-centered perspective on research needs for systems engineering with models

Procedia Computer Science 8 (2012) 315 – 320

1877-0509 © 2012 Published by Elsevier B.V.doi:10.1016/j.procs.2012.01.066

Available online at www.sciencedirect.com

Procedia Computer Science Procedia Computer Science 00 (2012) 000–000

www.elsevier.com/locate/procedia

New Challenges in Systems Engineering and Architecting Conference on Systems Engineering Research (CSER)

2012 – St. Louis, MO Cihan H. Dagli, Editor in Chief

Organized by Missouri University of Science and Technology

A work-centered perspective on research needs for systems engineering with models

Lisa Murphy, Ph.D.a*, Paul Collopy, Ph.D.b a Atura Integration, LLC, Huntsville, AL 35811

b Center for System Studies, University of Alabama in Huntsville, Huntsville, AL 35899

Abstract

Compared to document-based systems engineering (SE) models, IT-enabled ones can introduce radical changes in how SE is practiced. Such models are increasingly important today, but how well do we understand their use, impact, and issues? Real-world cases are helpful but tell us little about how and why models work (or don't), their suitability for a particular context, or how to guide selection and use. Lacking sound knowledge and theory, we cannot assess the value of Model-Based Systems Engineering (MBSE) [1] or predict its outcomes [2]. Instead, we rely on heuristics and judgment, e.g., to make process tailoring decisions [3]. Our motivations include identifying core research needs, provoking discussion on key issues and questions [4] and generating interest by applied and theoretical researchers, including from outside engineering. Other authors have noted that SE would benefit from broader research methodologies and methods (e.g., [5]; [6]) and greater participation by non-engineering disciplines (e.g., [2]; [7]). This paper identifies MBSE research topics culled from literature and briefly discusses some key ones based on a work-centered organizational approach with the goal of sparking interest in foundational research. © 2012 Published by Elsevier Ltd. Selection Keywords: model-based; model-driven; practice; research; theory; application; MBSE; agenda; architecting; methodology

1. Systems Engineering Practice with Technology-enabled Models

We have always used models in Systems Engineering (SE); what is different today is the power available through their representation as complex digital entities – what Haskins calls the “continuation of trend toward increasing levels of abstraction and automation” [8]. When the content of SE models are not

* Corresponding author. Tel.: +1-256-503-5014. E-mail address: [email protected].

316 Lisa Murphy and Paul Collopy / Procedia Computer Science 8 (2012) 315 – 320

Murphy & Collopy/ ProcediaComputer Science 00 (2012) 000–000

just entries in a document or passive objects on an electronic diagram but instead are addressable, logical, and relatable elements maintained in IT-empowered data structures, a new range of interactions and outcomes are possible. INCOSE MBSE Vision 2020 [9] and the System 2020 from DoD [3] identify basic improvements in model interaction such as direct support for multiple views, linking of models to other structured and ill-structured work products, and easier and faster changing, inspecting, and sharing models. More advanced capabilities include comparison and metrics for model instances, improved discipline applied to systems definition via constraints and parameterization, new options for automating model quality checking, use of formal transformations, and creating executable models. These changes have an impact on the daily work of planning, doing, and managing systems engineering. Our focus here is on the need for research about Model-Based Systems Engineering (MBSE) – or as we choose to call it, the practice of conducting systems engineering with technologically-enabled models.

2. Need for Research in MBSE

MBSE research needs are a subset of those for systems engineering and architecting (e.g., Kalawsky, 2009 [4]; Valerdi et al., 2008 [10]) focused on the particulars of technology-enabled modeling for systems work. Work with IT-enabled models has been a focus of research in software engineering for decades (e.g., Wand & Weber, 2002 [11]).

Guidance for MBSE and supporting technologies and methods are still evolving. Case reports such as Wheeler and Brooks [12] and Karban et al. [13] provide exposure to MBSE in practice; however, unless cases are addressed in a rigorous and systematic way [6], they can show us what happened, but not why. Without insights into why, it is difficult to answer what makes a particular practice suitable for a particular context or how models and associated practices should be selected and used. It is not sufficient to invoke general principles of systems engineering (which themselves are often not theoretically grounded [2]); we need to address the specifics of working with advanced capability models. The claims made for MBSE are non-trivial [9]; it will take discipline and a diverse set of research approaches [5; 2] to separate the hype from reality.

3. Research Topics for Systems Engineering Practice with Models

It is our hope to begin a dialog around the application and theory of SE models and working with them in context – that is, the activities, decisions, cognitions, and other intra- and inter-personal engagements that go on with and around models and their use in engineering systems. SE is a collaborative effort for which we need to both address the mathematical basis for model use and transforms (e.g., Paredis, 2011 [14]) and recognize that models, practices, and techniques are selected and used in organizational and social contexts where humans act in concert with implicit, opaque, and taken-for-granted theories [15].

We regret that space limitations keep us from listing all the inspiring and relevant sources discovered while preparing this paper. Instead, we have settled for providing a table of key topics (Table 1) from which we have selected a few for additional attention.

Before we address the agenda contents, a word about how we have organized it: while compiling this agenda, we considered schemes that classified the topics by systems types, phases in the lifecycle, classes of models, and research methods before settling on one associated with how the work of people intersects with models. This is represented by our choice of terminology as “systems engineering practice with models” rather than “model based systems engineering.” From a socio-technical perspective [16], we want to capture the full breadth of situations in which people are creating, making decisions, extracting information from, collaborating around, judging the quality of, and dozens of other engagements with IT-dependent SE models. While our primary focus is on a conventional SE product development context, we recognize that IT-enabled SE models proliferate across the product life cycle, technical disciplines, and organizational boundaries. A framework that addressed the nature of the work seemed the most inclusive,

317Lisa Murphy and Paul Collopy / Procedia Computer Science 8 (2012) 315 – 320

and so our work-centered perspective follows the activity categories of planning, creating, managing, using, and assessing. Issues of the adequacy and understanding of models as representations are foundational to all work with SE models, and we have chosen to group issues such as theory, education, and value outcomes into a single category we call “cross-cutting.” Systems engineering is holistic and recursive at its core [1;3;9], meaning that some topics will have implications in many activity areas; we offer these as useful (if provisional) categories and to generate interest.

Table 1. Work-centered research topic structure for systems engineering practice with technology-enabled models

Category Research Area Representative Research Topics

PLANNINGPlanning & Estimating WBS & Deliverables; Cost/Resource Planning; Progress Reporting; Risk/Recovery;

Facilitating Change/Adaptability

Selection & Application Goals & Objectives; Product Life Stage; System Type(s); Project Type(s); Tailoring; Acquisition Strategy

CREATINGMethods/Tools Methodologies; Process; Notations/Languages; Tools; Standards/Guidance

Combining/Comparing/Integrating

Comparing Models/Equivalence; Patterns; Assumptions/ Scale/Constraints; Frameworks (DODAF, FEA); Parameters; Search & Retrieval; Model Re-use

MANAGINGArtifacts Selection & Format; Identification/Nomenclature; Configuration Management;

Documents to Data; Access Control; ITAR; Provenance; Data Supply Chain/Exchange Relationship to Other Eng.

WorkTraceability; Tech Baseline; Trades, Analyses; Product Definition; Simulations, Prototyping, Testing; Downstream Uses/Users

USING

Role in Life Cycle SE Life Cycle; Product Life Cycle; Decision Making & Reviews; Supply Chain/Sustaining Logistics; Interfaces, Integrations; End-of-life

Organiz. & Individual Effects Level: Individual, Organizational, Teams; Identity; Culture; Policy; Forming; Roles

Engineering Effectiveness Result/Process Improvement; Quality of Requirements/Constraints; Closed Loop Modelling; Consistency across specialties; Impact on final product

ASSESSINGModel Quality Auditing/Ground Truth; Metrics; Revisions/Errors/Gaps; Sufficiency; Heterogeneity;

Scale; Transforms Merging & Transitivity; Integrating Software & Hardware

System-Level Perspectives Systems of Systems, Complex Systems, Families of Systems; -ilities/Safety; Technology Maturity /Integration/Change; Risk/Cost/Timing

REPRESEN-TATION

Representational Sufficiency Uncertainty/Ambiguity; Nominal vs. Off-Nominal; Time & Change; Aspects/ -ilities;Views/Viewpoints; System Boundaries & Interfaces; Descriptive Logic

Cognition/Communication Detecting Changes/Errors; Convergence of Understanding; Views vs. Aspects; Relations Between Models; Usability of Models and Notations

CROSS-CUTTING

Outcomes Value-Driven Design; Elegant, Secure, Adaptable Solutions; Contributions to SEBoKMBSE BoK Theories; Good Practice; Patterns; Metrics; Education/Development

4. A Few Areas of Research Focus

Due to space limitations, we focus our discussion on a few topic areas that we consider to be particularly significant: Selection & Application (Planning), Organization and Individual Effects (Using), Model Quality and Sufficiency (Assessing), Cognition and Communication (Representation), and Outcomes (Cross-Cutting). For each, we identify one or more highly pertinent research questions and discuss their significance for advancing the work of SE with technology-enabled models.

4.1. Selection & Application (PLANNING)

At a minimum, using models affects the work to be done, how tasks are related, skills of team members, how you will work with partners, and the cost of tools and the timing of their provision [12;13]. A SE project which begins with insufficient attention to the choice of modeling approaches will struggle. Because MBSE is an evolving territory in practice [3], methods [17] and tools [12;13], each project has limited ability to draw on the successes and failures of prior projects or on practice guidance, standards, or patterns [18]. Tailoring a standard framework to a particular system project takes on new meaning when in addition to addressing the extend and coverage of work products, core content is no longer in familiar




documents but in data records and executable models requiring definition and management of elements, decision rules, relationships, parameters, and constraints.

Some research questions to address include: What resources are needed; what new tasks or assets support their deployment and use? What about time and effort: how long does developing models take? How will we know if we are on track? What belongs on the critical path? What characteristics of my system or project should factor into selecting and applying MBSE? What does tailoring look like (i.e., if the model is the master rather than documents)? Do we understand how to identify, measure, and mitigate risks for doing SE with models?

If we take one question: “What does tailoring look like?” and look for direction from traditional systems engineering theory and practice, we discover that there is little or no scientific basis offered for tailoring decisions. Boehm and colleagues [3] recognize that guidance for tailoring is a gap but stop short of proposing that we need a “theory of tailoring.” Developing a scientific foundation for tailoring will go a long way toward answering the challenges of building highly complex hybrid hardware and software systems, of dynamic systems of systems, and of the need for repurposing and extending the life of existing systems [2;4;7]. Suggesting why some parts of SE practice should receive more or less attention implies that you can characterize the project, the system, the context, the roles of models, which aspects are critical, and how practice domains relate. This is a tall challenge – perhaps even a grand challenge [4].

4.2. Organizational & Individual Effects (USING)

Systems engineering is done with and by humans; Collopy [2] proposes that an important contribution of SE has been to organize disparate efforts of hundreds or thousands of people to achieve complex outcomes. What happens to people’s jobs, their self-identity, and their team relationships when the system of interest is a pure abstraction for which there may be no single universal representation? Familiar SE topics must be addressed (i.e., concepts developed, requirements engineered, alternatives analyzed, baselines prepared) while recognizing that these tasks are related and relatable in new ways. Beyond differences in day-to-day work, this change is shifting boundaries between roles, altering the dynamics of teams, changing how we organize, and modifying how we think of SE work and work products [12]. Research on other model-centric technologies such as 3D CAD, analysis models, simulations, ontologies, knowledge representation, and virtual reality may highlight fruitful paths [15].

Technology has a long history of interacting with social contexts to produce emergent forms and properties not anticipated by their developers [16]. Effects on roles, expectations, and the flow of information and control are specific topics of interest which will benefit from – or even require – participation by investigators from outside engineering [2;5;6;15]. As a hybrid technology itself – a combination of computing, methods, techniques, representations, policy, and expectations – MBSE is a good candidate to surprise us [5;6].

4.3. Model Quality & Sufficiency (ASSESSING)

The quality of SE models has always been important and has always been difficult to measure [18]. By moving to models composed of abstract digital objects and their relationships, we can exploit IT to help assess the adequacy of our models [3;11;14]. Frankly, this would be easier if the available model set was restricted with well-defined intents and uses, but we aren’t holding our breath since SE needs models for systems of different scales, scope, and configurations (e.g., sub-systems, hybrid hardware/software, systems-of-systems; autonomous system aggregations) [2;4;17;19]. It is not enough to ask if a model is sufficient for a need; we must ask what makes it so, or would an alternative be as good or better [11]. From such premises we can derive metrics of model quality addressing relationships between content, coverage, representational sufficiency and thereby to make assertions about what is a mistake and why.

319Lisa Murphy and Paul Collopy / Procedia Computer Science 8 (2012) 315 – 320

Important research questions include: When is a model done? How important are completeness and consistency? What constitutes an error? Are some omissions or errors more serious than others? What makes a change significant? Does that differ by level of abstraction or use/intent of model? How can we combine models with confidence? What does validity mean for different types of

models? Theoretically and practically, all models are incomplete [3;7]; we must decide when they are sufficient for a specific use or need. And, SE requires that we confine our attention to a specific domain and choose to how to spend scarce resources considering uncertainty. Even if we can converge on a handful of model metrics, they will likely need to be applied differently to different uses. We must answer how to select and apply such metrics, e.g., by variations in system type, level of abstraction, role in the life cycle, or by other characteristics to be established [e.g. 18].

4.4. Cognition & Communication (REPRESENTATION)

The core of SE work with models requires comprehending how people understand SE models. The significance of how people think, collaborate, communicate using models cannot be understated, especially as SE must capture and represent multiple views of the system (e.g., functional, structural, behavioral, informational) which are incompletely transitive [7;14;20]. If we succeed at getting systems engineers to understand the models but not the users, stakeholders, developers, and other domain specialists, we cannot say we are successful. We can and must take advantage of existing research done in the area of conceptual and process modeling, visualization, languages, model grammars, and cognition with models arising out of the information systems, software development, ontology, and simulation modeling communities, e.g., Wand & Weber [11]; Davis, Shrobe, & Szolovits [15]; Herdlick, Mazzuchi,& Sarkani [19]; Moody [21]; Renger, Kolfschoten, & De Vreede [22].

4.5. Outcomes (CROSS-CUTTING)

The ultimate question as we see it: does systems engineering with modern IT-enabled models produce better outcomes? Corollary questions include: Are equivalent products achieved faster or at less cost? Are some types of risks reduced or errors avoided? Do systems have a longer life expectancy or lower life cycle costs, take less effort to adopt or use,

demonstrate more security or reliability, or are easier to modify or re-purpose? Whether and how SE with IT-enabled models contributes to success in producing systems that satisfy objectives is in need of systematic and theoretically-grounded research attention [2;3;4;5;7;9]. This requires that we also define and measure systems success whether framed as mission success, value-added, or elegance [3;23;24].

5. Conclusion We wrap up our abbreviated discussion by reiterating the impact of modeling technology on the daily

practice of systems engineering needs our systematic attention. We cannot be satisfied with best practices that lack context, unvalidated choices of representations, or assertions of traditional process hegemony while technologies change roles, procedures, interactions, decision-making, and information flow. MBSE has the potential to bring new discipline and reach for systems engineering; we need to build a foundation based on methodologically sound investigations that address the full breadth of relevant topics and generate a theoretical basis for this important domain. We offer this paper as a start in that direction.




References [1] Honour EC. Understanding the value of systems engineering. Proceedings of the INCOSE International Symposium,

Toulouse, France, 2004. [2] Collopy PD. A research agenda for the coming renaissance in systems engineering. Accepted at 2012 Aerospace Sciences

Meeting; Center for System Studies, University of Alabama in Huntsville, 2011. [3] Boehm B, ed.. Systems 2020 Strategic Initiative. DoD Systems Engineering Research Center Final Technical Report SERC-

2010-TR-009, August 29, 2010. [4] Kalawsky R. Grand challenges for systems engineering research. Seventh Annual Conference on Systems Engineering

Research (CSER 2009), April, 2009. [5] Valerdi R, Davidz H. Empirical research in systems engineering: challenges and opportunities of a new frontier. Fifth

Annual Conference on Systems Engineering Research (CSER 2007), March, 2007. [6] Ferris TLJ. On the methods of research for systems engineering. Seventh Annual Conference on Systems Engineering

Research (CSER 2009), April, 2009. [7] Hybertson DW. Model-oriented Systems Engineering Science: A Unifying Framework for Traditional and Complex Systems.

Auerbach Publications, 2009. [8] Haskins C. MBSE – A Historical Perspective Promises and Pitfalls. Keynote presentation to 3rd International Conference on

Model-Based Systems Engineering, George Mason University, September, 2010. [9] International Council on Systems Engineering. INCOSE MBSE Vision 2020, INCOSE-TP-2004-004-02: 2.03 September,

2007. [10] Valerdi R, Axelband E, Baehren T, Boehm B, Dorenbos D, Jackson S et al. A research agenda for systems of systems

architecting. International J. of System of Systems Engineering, Vol1 (1/2), pp. 171-188, 2008. [11] Wand Y, Weber R. Research commentary: Information systems and conceptual modeling--a research agenda. Information

Systems Research, 13 (4), pp. 363 -376, Dec 2002. [12] Wheeler TM, Brooks MD. Experiences in applying architecture-centric model based systems engineering to large-scale,

distributed, real-time systems., The Mitre Corporation Technical Papers Case 07-0838, July, 2007. [13] Karban R, Zamparelli M, Bauvir B, Koehler B, Noethe L, Balestra A. Exploring Model Based Engineering for Large

Telescopes - Getting started with descriptive models, SPIE Proceedings, Vol. 7017, July, 2008. [14] Paredis C. Model-Based Systems Engineering: A roadmap for academic research. Frontiers in Model-Based Systems

Engineering, Atlanta, GA. April, 2011. [15] Davis R, Shrobe H, Szolovits P. What is a Knowledge Representation? AI Magazine, 14(1):17-33, 1993. [16] Bijker WE. Of Bicycles, Bakelites, and Bulbs: Toward a Theory of Sociotechnical Change. MIT Press, 1997. [17] Estefan J. Survey of Model-Based Systems Engineering (MBSE) methodologies. International Council on Systems

Engineering, INCOSE-TD-2007-003-01, June, 2008. [18] Morse KL, Coolahan JE, Lutz B, Horner N, Vick S, Syring R. Best Practices for the Development of Models and

Simuations Final Report NSAD-R-2010-037 prepared by Johns Hopkins University Applied Physics Lab for Office of the Under Secretary of Defense for Acquisition, Technology and Logistics, US Department of Defense, June, 2010.

[19] Herdlick B, Mazzuchi T, Sarkani S. Establishing confidence in Federations-of-Models. Presentation to System of Systems Engineering Collaborators Information Exchange (SoSECIE), Office of Deputy Assistant Secretary of Defense, June, 2011.

[20] Wymore AW. Model-Based Systems Engineering: An Introduction to the Mathematical Theory of Discrete Systems and to the Tricotyledon Theory of System Design, CRC Press, Boca Raton FL, 1993.

[21] Moody DL. Theoretical and practical issues in evaluating the quality of conceptual models: current state and future directions. Data & Knowledge Engineering 55, pp. 243–276, 2005.

[22] Renger M, Kolfschoten GL, De Vreede, G-J. Challenges in collaborative modeling: a literature review and research agenda. Int. J. Simulation and Process Modeling, 4(3-4), pp. 248-263, 2008.

[23] Griffin MD. How do we fix systems engineering? (IAC-10.D1.5.4) Int. Astronautical Congress, Prague, Czech Republic, November, 2010.

[24] Collopy PD, Hollingsworth PM. Value-driven design. J. of Aircraft, 48(3); pp. 749-759, 2011.