Phu H. NguyenPostdoc, Software Engineering department

Section on Computing and Software Monthly Lunch Seminar (March 16th, 2017)

SMS & SLR

Agenda

U-Test MBT4CPS

Wrap-up

SMS & SLR

Agenda

U-Test MBT4CPS

U-Test = Testing Cyber-Physical Systems under Uncertainty: Systematic, Extensible, and

Configurable Model-based and Search-based Testing

Methodologies

MBT4CPS = Model-Based Testing for Cyber-Physical Systems

The consortium

Overall U-Test objective: Improving CPS

dependability via systematic and automated testing of

Uncertainty in CPS

SUnMBT4CPS: Security-related Uncertainty Model-

Based Testing for CPS

A taxonomy of security-related uncertainties

Model-Based Security-Related Uncertainty Testing

A security-related uncertainty test model of Advanced Metering Infrastructure

SUnMBT4CPS: Security-related Uncertainty Model-

Based Testing for CPS

NIST: IR 7628-Guidelines for Smart Grid Cyber Security The ANSI C12 protocol suite

AvailabilityIntegrity Accountability

Advanced Metering Infrastructure in Smart GridConfidentiality

Smart Meters

Collector Firewall

Internet/Private Network/CellularHeadend System

Smart Grid: an important CPS

Agenda

U-Test MBT4CPS

SMS & SLR

http://matt.might.net/articles/phd-school-in-pictures/

How to systematically check the boundary of knowledge for your research work with SMS and SLR?

Boundary of Knowledge

Systematic Mapping Study (SMS) vs. Systematic Literature Review (SLR) vs. Tertiary Review (TR)

Secondary study: “a study that reviews all the primary studies relating to a specific research question with the aim of integrating/synthesizing evidence related to a specific research question.”

SLR: “A form of secondary study that uses a well-defined methodology to identify, analyze and interpret all available evidence related to a specific research question in a way that is unbiased and (to a degree) repeatable.”

SMS: “A broad review of primary studies in a specific topic area that aims to identify what evidence is available on the topic.”

TR: “A review of secondary studies related to the same research question.”

B.A. Kitchenham , D. Budgen , O.P. Brereton, Using mapping studies as the basis for further research–a participant-observer case study, Inf. Softw. Technol. 53 (6) (2011) 638–651.

http://epub.wu.ac.at/4431/

P.H. Nguyen et al., Model-based security engineering for cyber-physical systems: A systematic mapping study, Information and Software Technology (2016), http://dx.doi.org/10.1016/j.infsof.2016.11.004

Three Research Questions (13 Sub-Questions)

MBSE4CPS = Model-Based Security Engineering for Cyber-Physical Systems

Where and how to find your Primary Studies?

Database Search

Manual Search

Snowballing

Search String (PICO) = Population AND Intervention AND Comparison AND Outcome

Population: (“cyber-physical system” OR CPS OR “smart grid” OR “power grid” OR “smart car” OR “automotive cyber-physical system” OR “pervasive healthcare system” OR “unmanned air- craft system”)

Intervention: (model OR modelling OR model-based OR model- driven)

Comparison: (security OR confidentiality OR integrity OR avail- ability OR accountability OR authentication OR authorisation OR “access control” OR attack OR threat OR vulnerability OR uncertainty)

Outcome: (architecture OR design OR verification OR validation OR test OR analysis)

Inclusion criteria (IC)

Exclusion criteria (EC)

Database Search and Selection Process

* Inclusion and Exclusion Criteria were carefully used to find primary studies.

Snowballing: backwardand forward

Hybrid Search: Database Search + Snowballing

SLR

RQ1: What are the publication statistics of the existing primary MBSE4CPS studies in the literature?

RQ2: What are the existing primary MBSE4CPS studies & their characteristics?

RQ3: What are the open issues of MBSE4CPS research?

SLR

More in-depth analysis is needed!

Agenda

U-Test MBT4CPS

Wrap-up

SMS & SLR

•  Cyber-Physical Systems Engineering is an emerging but important research area. U-Test and MBT4CPS focus on addressing the reliability and security of CPS.

•  SMS: discover research trends with general research questions, at a high level of granularity.

•  SLR: detailed aggregated evidence in terms of the research outcomes, very specific research questions.

•  Snowballing can complement well for database search and manual search to achieve primary studies.

•  Snowballing can be employed in a follow-up SMS or SLR of an existing one.

•  Important to explicitly discuss the threats to validity of any SMS and SLR

Main points

•  SMS and SLR are worth to do because we can: •  Systematically review and condense the evidences for the

boundary of knowledge in a research area,•  Evaluate the current research landscape of a research area, •  Point out the current research challenges, •  Explore new directions for research,•  Have good citations J

•  Cons:•  Time consuming•  Good background knowledge in the research area•  Not well-defined review protocol and search pilot could lead to unconvincing review results.

Main points

•  For PhD students: •  Would you like to do a SMS or SLR? •  What would be the most difficult steps for you?

•  For supervisors: •  If a PhD student wants to conduct a SMS or SLR, would you

provide your support?•  For all: can we create a tool for automate most of the

steps in conducting SMS and SLR?•  Data Science approach

Open questions

Guidelines for conducting SMS and SLR

K. Petersen , S. Vakkalanka , L. Kuzniarz , Guidelines for conducting systematic mapping studies in software engineering: an update, Inf. Softw. Technol. 64 (2015) 1–18.

B. Kitchenham , Guidelines for performing systematic literature reviews in software engineering, Technical Report, EBSE, 2007.

C. Wohlin, Guidelines for snowballing in systematic literature studies and a replication in software engineering, in: Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering, ACM.