some insights from a systematic mapping study and a systematic review study: from snowballing to...
Post on 21-Mar-2017
96 Views
Preview:
TRANSCRIPT
Phu H. NguyenPostdoc, Software Engineering department
Section on Computing and Software Monthly Lunch Seminar (March 16th, 2017)
SMS & SLR
Agenda
U-Test MBT4CPS
Wrap-up
SMS & SLR
Agenda
U-Test MBT4CPS
U-Test = Testing Cyber-Physical Systems under Uncertainty: Systematic, Extensible, and
Configurable Model-based and Search-based Testing
Methodologies
MBT4CPS = Model-Based Testing for Cyber-Physical Systems
The consortium
Overall U-Test objective: Improving CPS
dependability via systematic and automated testing of
Uncertainty in CPS
SUnMBT4CPS: Security-related Uncertainty Model-
Based Testing for CPS
A taxonomy of security-related uncertainties
Model-Based Security-Related Uncertainty Testing
A security-related uncertainty test model of Advanced Metering Infrastructure
SUnMBT4CPS: Security-related Uncertainty Model-
Based Testing for CPS
NIST: IR 7628-Guidelines for Smart Grid Cyber Security The ANSI C12 protocol suite
AvailabilityIntegrity Accountability
Advanced Metering Infrastructure in Smart GridConfidentiality
Smart Meters
Collector Firewall
Internet/Private Network/CellularHeadend System
Smart Grid: an important CPS
Agenda
U-Test MBT4CPS
SMS & SLR
http://matt.might.net/articles/phd-school-in-pictures/
How to systematically check the boundary of knowledge for your research work with SMS and SLR?
Boundary of Knowledge
Systematic Mapping Study (SMS) vs. Systematic Literature Review (SLR) vs. Tertiary Review (TR)
Secondary study: “a study that reviews all the primary studies relating to a specific research question with the aim of integrating/synthesizing evidence related to a specific research question.”
SLR: “A form of secondary study that uses a well-defined methodology to identify, analyze and interpret all available evidence related to a specific research question in a way that is unbiased and (to a degree) repeatable.”
SMS: “A broad review of primary studies in a specific topic area that aims to identify what evidence is available on the topic.”
TR: “A review of secondary studies related to the same research question.”
B.A. Kitchenham , D. Budgen , O.P. Brereton, Using mapping studies as the basis for further research–a participant-observer case study, Inf. Softw. Technol. 53 (6) (2011) 638–651.
http://epub.wu.ac.at/4431/
P.H. Nguyen et al., Model-based security engineering for cyber-physical systems: A systematic mapping study, Information and Software Technology (2016), http://dx.doi.org/10.1016/j.infsof.2016.11.004
Three Research Questions (13 Sub-Questions)
MBSE4CPS = Model-Based Security Engineering for Cyber-Physical Systems
Where and how to find your Primary Studies?
Database Search
Manual Search
Snowballing
Search String (PICO) = Population AND Intervention AND Comparison AND Outcome
Population: (“cyber-physical system” OR CPS OR “smart grid” OR “power grid” OR “smart car” OR “automotive cyber-physical system” OR “pervasive healthcare system” OR “unmanned air- craft system”)
Intervention: (model OR modelling OR model-based OR model- driven)
Comparison: (security OR confidentiality OR integrity OR avail- ability OR accountability OR authentication OR authorisation OR “access control” OR attack OR threat OR vulnerability OR uncertainty)
Outcome: (architecture OR design OR verification OR validation OR test OR analysis)
Inclusion criteria (IC)
Exclusion criteria (EC)
Database Search and Selection Process
* Inclusion and Exclusion Criteria were carefully used to find primary studies.
Snowballing: backwardand forward
Hybrid Search: Database Search + Snowballing
SLR
RQ1: What are the publication statistics of the existing primary MBSE4CPS studies in the literature?
RQ2: What are the existing primary MBSE4CPS studies & their characteristics?
RQ3: What are the open issues of MBSE4CPS research?
SLR
More in-depth analysis is needed!
Agenda
U-Test MBT4CPS
Wrap-up
SMS & SLR
• Cyber-Physical Systems Engineering is an emerging but important research area. U-Test and MBT4CPS focus on addressing the reliability and security of CPS.
• SMS: discover research trends with general research questions, at a high level of granularity.
• SLR: detailed aggregated evidence in terms of the research outcomes, very specific research questions.
• Snowballing can complement well for database search and manual search to achieve primary studies.
• Snowballing can be employed in a follow-up SMS or SLR of an existing one.
• Important to explicitly discuss the threats to validity of any SMS and SLR
Main points
• SMS and SLR are worth to do because we can: • Systematically review and condense the evidences for the
boundary of knowledge in a research area,• Evaluate the current research landscape of a research area, • Point out the current research challenges, • Explore new directions for research,• Have good citations J
• Cons:• Time consuming• Good background knowledge in the research area• Not well-defined review protocol and search pilot could lead to unconvincing review results.
Main points
• For PhD students: • Would you like to do a SMS or SLR? • What would be the most difficult steps for you?
• For supervisors: • If a PhD student wants to conduct a SMS or SLR, would you
provide your support?• For all: can we create a tool for automate most of the
steps in conducting SMS and SLR?• Data Science approach
Open questions
Guidelines for conducting SMS and SLR
K. Petersen , S. Vakkalanka , L. Kuzniarz , Guidelines for conducting systematic mapping studies in software engineering: an update, Inf. Softw. Technol. 64 (2015) 1–18.
B. Kitchenham , Guidelines for performing systematic literature reviews in software engineering, Technical Report, EBSE, 2007.
C. Wohlin, Guidelines for snowballing in systematic literature studies and a replication in software engineering, in: Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering, ACM.
top related