design, implementation and evaluation of a collaborative ...€¦ · software engineering for...
TRANSCRIPT
Software Engineering for Business Information Systems (sebis) Department of InformaticsTechnische Universität München, Germany
wwwmatthes.in.tum.de
Design, Implementation and Evaluation of a Collaborative Synthesis Stage in the Context of Computer-Supported Morphological AnalysisBastian Rodehüser | Final Presentation Master Thesis | 05.12.2016
1. Introduction § What is Morphological Analysis?§ What Support Systems for Morphological Analysis do exist?
2. Collaborative Morphological Analysis§ Group Decisions§ Real-Time-Delphi Technique§ Computer-Supported Collaborative Cross-Consistency Assessment§ Computer-Supported Collaborative Constraint Management
3. Evaluation§ Hypotheses§ Solution Space and Actions§ Satisfaction with process, decision and application
4. Conclusion
Overview
© sebis 2Bastian Rodehüser | Final Presentation Master Thesis | 05.12.2016
What is Morphological Analysis?
§ Creativity technique for groups§ Iterative process
© sebis 3
The general iterative process model cf. Zec et al. 2015
Synthesis
à Cross-Consistency Assessmentà Constraint Management
Analysis
Define parameters &
values
Exploration
Examine consistent
configurations
Bastian Rodehüser | Final Presentation Master Thesis | 05.12.2016
What is Morphological Analysis?
© sebis 4
Decomposition of system• Identification of parameters• Definition of alternatives/ values for each
parameter
Morphological Box/Field
Motorization Range Operation area Number of Passengers
Control
CombustionEngine
< 100 km Private City Traffic
1 Driver
Electric Motor 100 - 299 km Travelling 2 Driver / Autopilot
Fuel Cell 300 - 599 km Delivery Vehicle 4-5 Only AutopilotHybrid 600 - 999 km > 5
> 1000 km
K is the set of simple configurations.à Cardinality: |K| = 720
Bastian Rodehüser | Final Presentation Master Thesis | 05.12.2016
What is Morphological Analysis?
© sebis 5
LK
In synthesis phase, all configurations in K are examined if they are consistent.
All consistent configurations of K are elementof the solution space L.à 𝐿 ⊆ K
Elements of L can be ordered.à Rank order
Bastian Rodehüser | Final Presentation Master Thesis | 05.12.2016
What is Morphological Analysis?
© sebis 6Bastian Rodehüser | Final Presentation Master Thesis | 05.12.2016
Cross-Consistency Assessment• Configurations with incompatible parameter values
can be excluded from further consideration• Rank order by consistency scores
Cross-Consistency Matrix
What is Morphological Analysis?
© sebis 7
Constraint Management• Positive Constraints:
• All configurations that fit to a positive constraint receive a recommendation point for the rank order
• Negative Constraints: • All configurations that fit to a negative
constraint are excluded from the solution space
Motorization: Electric Motor OR Fuel CellOperation Area: Delivery DeviceNumber of Passengers: 1 OR 2
NOT Range: <100 kmNOT Range: 100-299 km
Example constraint;can be both, positive and negative
Bastian Rodehüser | Final Presentation Master Thesis | 05.12.2016
Captures 32 of 720 configurations!
What is Morphological Analysis?
© sebis 8
Exploration of the remaining set (L) of consistent configurations
Bastian Rodehüser | Final Presentation Master Thesis | 05.12.2016
Existing Support Systems
§ Existing Support Systems (e.g. MA/Carma, Parmenides Eidos):
à [Analysis] Help to collect parameters and their corresponding valuesà [Synthesis] Offer tools for Cross-Consistency Assessment
• Calculate the solution space
à [Exploration] Help to explore the solution space of consistent configurations
§ BUT: existing Support Systems are not collaborative!àGroup members have to use these applications at
• the same place and • the same time
àThey have to discuss and to agree on inputs to the applicationàSoftware does not help to enhance social-psychological process gains und
to avoid process losses
© sebis 9Bastian Rodehüser | Final Presentation Master Thesis | 05.12.2016
Computer-Supported Synthesis in Practice
© sebis 10
• Cross-Consistency Assessment (three approaches)1. For each cell: Discussion in group and building consensus; input by
moderator2. Individual work by participants; discussion of contradictions and building
consensus; input by moderator3. Individual work by participants; use of all created individual solution spaces
in exploration phase
• Weaknesses• complex (example: 143 cells)• time-consuming• labor-intensive
Not suitable for many participants respectively high complex projects
Bastian Rodehüser | Final Presentation Master Thesis | 05.12.2016
cf. interviews by Rodehüser (2016, appendix master thesis)
1. Introduction § What is Morphological Analysis?§ What Support Systems for Morphological Analysis do exist?
2. Collaborative Morphological Analysis§ Group Decisions§ Real-Time-Delphi Technique§ Computer-Supported Collaborative Cross-Consistency Assessment§ Computer-Supported Collaborative Constraint Management
3. Evaluation§ Hypotheses§ Solution Space and Actions§ Satisfaction with process, decision and application
4. Conclusion
Overview
© sebis 11Bastian Rodehüser | Final Presentation Master Thesis | 05.12.2016
§ Advantages1. Combination and integration of:
a) Different knowledgeb) Different ideasc) Different perspectives
2. Identification and integration of individual points of view Ø higher acceptance of decision
§ Quality• Comprehensive exchange of ideas and information is important• Set union of all individual idea / information sets
§ Preferences• Case: similar information set but different individual preferences
Ø Cause: different objectives / different risk attitudesØ Aggregation of individual preferences to group preferenceØ Discussions and use of voting procedures
Group Decisions
© sebis 12Bastian Rodehüser | Final Presentation Master Thesis | 05.12.2016
Real-time Delphi technique
© sebis 13Bastian Rodehüser | Final Presentation Master Thesis | 05.12.2016
Goal: Avoid process losses | support process gains | achieve comprehensive information exchange
cf. Gordon and Pease 2006cf. Gordon 2009cf. Gnatzy et al. 2011
Implementation
© sebis 14Bastian Rodehüser | Final Presentation Master Thesis | 05.12.2016
LIVE DEMO
1. Introduction § What is Morphological Analysis?§ What Support Systems for Morphological Analysis do exist?
2. Collaborative Morphological Analysis§ Group Decisions§ Real-Time-Delphi Technique§ Computer-Supported Collaborative Cross-Consistency Assessment§ Computer-Supported Collaborative Constraint Management
3. Evaluation§ Hypotheses§ Solution Space and Actions§ Satisfaction with process, decision and application
4. Conclusion
Overview
© sebis 15Bastian Rodehüser | Final Presentation Master Thesis | 05.12.2016
Hypotheses
The process of Cross-Consistency Assessment is easier to learn than the process of Constraint Management.
Both implemented methods are suitable for solving hidden profiles.
Both implemented methods are suitable for creating a consistent solution space.
Constraint Management can be carried out in a shorter time than Cross-Consistency Assessment.
© sebis 16
1
2
3
Bastian Rodehüser | Final Presentation Master Thesis | 05.12.2016
4
Test Procedure
§ Morphological Box is given
§ Every participants got individual information• unshared• sharedà Hidden Profiles in distribution of information
§ “Correct” solution space exists
§ 4 group with 4 participants• 2 groups conduct Cross-Consistency Assessment• 2 groups conduct Constraint Management
© sebis 17Bastian Rodehüser | Final Presentation Master Thesis | 05.12.2016
Solution Space over time
© sebis 18Bastian Rodehüser | Final Presentation Master Thesis | 05.12.2016
Number of Actions
© sebis 19Bastian Rodehüser | Final Presentation Master Thesis | 05.12.2016
Solution Space: Time
© sebis 20Bastian Rodehüser | Final Presentation Master Thesis | 05.12.2016
Left: Last change of solution spaceRight: Last action
→ Does not support hypothesis 4. (Constraint Management can be carried out in a shorter time than Cross-Consistency Assessment.)
→ Future research: time advantage with more complex Morphological Boxes or more experienced teams?
Solution Space: Quality
© sebis 21Bastian Rodehüser | Final Presentation Master Thesis | 05.12.2016
→ Created solution spaces approximate to the sample solution.
→ Supports hypothesis 3. (Both methods are suitable for creating a consistent solution space.)
Resolution of Hidden Profiles
© sebis 22Bastian Rodehüser | Final Presentation Master Thesis | 05.12.2016
Information3 Information4 Information5 Information6 Information9
Group Correct SI CAC SI CAC SI CAC SI CAC SI CAC
CCA1 3/5
CCA2 4/5
CM1 4/5
CM1 5/5
SI = Shared Information (transformation of unshared information into shared information)
CAC = Correct Assessment/Constraint(creation of correct assessment/constraint related to information)
→ Supports hypothesis 2. (Both methods are suitable for solving hidden profiles.)→ All unshared information could be transformed into shared information.→ This has led to 16 of 20 correct assessments/constraints.
Satisfaction with process
© sebis 23Bastian Rodehüser | Final Presentation Master Thesis | 05.12.2016
1,0
1,5
2,0
2,5
3,0
3,5
4,0
4,5
5,0
1efficient5inefficient
1coordinated5uncoordinated
1fair5unfair
1understandable5confusing
1satisfying5dissatisfying
Howwouldyoudescribeyourgroup'sproblemsolvingprocess?
CCA CM CCA1 CCA2 CM1 CM2
→ Can indicate the correctness of hypothesis 4 (Constraint Management can be carried out in a shorter time than Cross-Consistency Assessment), if the problem complex is large.
Questions by Green und Taber (1980, p. 102) and Reinig (2003, p. 82)
Satisfaction with application (1/2)
© sebis 24Bastian Rodehüser | Final Presentation Master Thesis | 05.12.2016
1,01,52,02,53,03,54,04,55,0
1StronglyDisagree5StronglyAgree
1StronglyDisagree5StronglyAgree
1StronglyDisagree5StronglyAgree
1StronglyDisagree5StronglyAgree
1StronglyDisagree5StronglyAgree
IthinkthatIwouldliketousethis
productfrequently.
Ithoughttheproductwaseasytouse.
Ifoundthevariousfunctionsintheproductwerewell
integrated.
Iimaginethatmostpeoplewouldlearntousethisproductvery
quickly.
Ifeltveryconfidentusingtheproduct.
CCA CM CCA1 CCA2 CM1 CM2
→ Supports hypothesis 1. (The process of Cross-Consistency Assessment is easier to learn than the process of Constraint Management.)
Questions by Brooke (1996), Bangor et al. (2008), Bangor et al. (2009)
Satisfaction with application (2/2)
© sebis 25Bastian Rodehüser | Final Presentation Master Thesis | 05.12.2016
1,01,52,02,53,03,54,04,55,0
1StronglyDisagree5StronglyAgree
1StronglyDisagree5StronglyAgree
1StronglyDisagree5StronglyAgree
1StronglyDisagree5StronglyAgree
1StronglyDisagree5StronglyAgree
Ifoundtheproductunnecessarilycomplex.
IthinkthatIwouldneedthesupportofatechnicalpersontobeabletousethis
product.
Ithoughttherewastoomuch
inconsistencyinthisproduct.
Ifoundtheproductveryawkwardtouse.
IneededtolearnalotofthingsbeforeI
couldgetgoingwiththisproduct.
CCA CM CCA1 CCA2 CM1 CM2
→ Previous knowledge is important.
Questions by Brooke (1996), Bangor et al. (2008), Bangor et al. (2009)
System Usability Scale (SUS)
© sebis 26Bastian Rodehüser | Final Presentation Master Thesis | 05.12.2016
83,171,6
88,877,5
66,376,9
0,0
10,0
20,0
30,0
40,0
50,0
60,0
70,0
80,0
90,0
100,0
CCA CM CCA1 CCA2 CM1 CM2
Notacceptable
Marginal
Acceptable
Calculation according to Brooke (1996, p. 194)
1. Introduction § What is Morphological Analysis?§ What Support Systems for Morphological Analysis do exist?§ Social Psychology Considerations
2. Collaborative Morphological Analysis§ Quality of Group Decisions§ Real-Time-Delphi Technique§ Computer-Supported Collaborative Cross-Consistency Assessment§ Computer-Supported Collaborative Constraint Management
3. Evaluation§ Hypotheses§ Solution Space and Actions§ Satisfaction with process, decision and application
4. Conclusion
Overview
© sebis 27Bastian Rodehüser | Final Presentation Master Thesis | 05.12.2016
Summary & Conclusion (1/2)
© sebis 28Bastian Rodehüser | Final Presentation Master Thesis | 05.12.2016
§ Methods• Cross-Consistency Assessment
- More structured- Higher certainty that no inconsistent configurations are element of the solution space- Encourages participants to consider all parameters and values
• Constraint Management- More powerful- More effective under certain conditions
§ Both methods can be combined (cf. master thesis)
§ Existing support systems• No support of decision-making, consensus-building and collaborative work• No enhancement of process gains and no avoidance of process losses• Only support of Cross-Consistency Assessment (time-consuming / labor-intensive
à suitable for few participants)
§ Good group decisions require a comprehensive information exchange
Summary & Conclusion (2/2)
© sebis 29Bastian Rodehüser | Final Presentation Master Thesis | 05.12.2016
§ Web Application• Frameworks: Meteor & React• Implements both methods• Based on Real-Time-Delphi approach à collaborative work is possible
- Enhancement of process gains and avoidance of process losses- Encourages participants to a comprehensive information exchange
§ The results suggest that hypothesis 1 is correct. (The implemented Cross-Consistency Assessment seems to be easier to learn)
§ Both implemented methods are suitable to create consistent solution space (Hypothesis 2)
§ Both implemented methods are suitable to solve hidden profiles (Hypothesis 3)
§ Hypothesis 4 could not be verified. (Constraint Management can be carried out in a shorter time than Cross-Consistency Assessment)
§ Future Research• Varied group size• Varied group experience• Varied group knowledge• Varied problem complexities
Literature§ Gnatzy, Tobias; Warth, Johannes; Gracht, Heiko von der; Darkow, Inga-Lena (2011): Validating an
innovative real-time Delphi approach - A methodological comparison between real-time and conventional Delphi studies. In: Technological Forecasting and Social Change 78 (9), S. 1681–1694.
§ Gordon, Theodore J.; Pease, Adam (2006): RT Delphi. An efficient, “round-less” almost real time Delphi method. In Technological Forecasting and Social Change 73 (4), pp. 321–333.
§ Gordon, Theodore J. (2009): The Real-Time Delphi Method. Available online at http://www.millennium-project.org/millennium/RTD-method.pdf.
§ Green, Stephen G.; Taber, Thomas D. (1980): The effects of three social decision schemes on decision group process. In: Organizational Behavior and Human Performance 25 (1), S. 97–106.
§ Reinig, Bruce A. (2003): Toward an Understanding of Satisfaction with the Process and Outcomes of Teamwork. In: Journal of Management Information Systems 19 (4), S. 65–83.
§ Ritchey, Tom (2006): Problem structuring using computer-aided morphological analysis. Available online at http://www.swemorph.com/pdf/psm-gma.pdf.
§ Ritchey, Tom (2015): Principles of Cross-Consistency Assessment in General Morphological Modelling. In : Acta morphologica generalis. AMG ; on-line journal of the Swedish Morphological Society. Vällingby: Swedish Morphological Society (Vol. 4 No. 2).
§ Stasser, Garold; Stewart, Dennis (1992): Discovery of hidden profiles by decision-making groups. Solving a problem versus making a judgment. In: Journal of Personality and Social Psychology 63 (3), S. 426–434.
§ Zec, Marin; Schneider, Alexander W.; Matthes, Florian (2015): Towards a Process Model for Computer-Supported Collaborative Morphological Analysis.
© sebis 30Bastian Rodehüser | Final Presentation Master Thesis | 05.12.2016
Technische Universität MünchenDepartment of InformaticsChair of Software Engineering for Business Information Systems
Boltzmannstraße 385748 Garching bei München
wwwmatthes.in.tum.de
Bastian RodehüserB. Sc.
Thank you for your attention! Any questions?
Technische Universität MünchenDepartment of InformaticsChair of Software Engineering for Business Information Systems
Boltzmannstraße 385748 Garching bei München
wwwmatthes.in.tum.de
Bastian RodehüserB. Sc.
BACKUP
• How to avoid process losses with an application?• Coordination Losses• Production Blocking• Social Loafing• Free Riding• Dispensability Effect• Sucker Effect• Cognitive Restriction• Hidden Profiles• Common-Knowledge-Effect• Discussion Distortion• Assessment Distortion• Conformity• Group Polarization
• How to support process gains with an application?• Social Competition• Social Compensation• Köhler Effect• Cognitive Stimulation
Social Psychology Considerations
© sebis 33Bastian Rodehüser | Final Presentation Master Thesis | 05.12.2016
Computer-Supported Collaborative Morphological Analysis
© sebis 34
Synthesis
- Cross-Consistency Assessment- Constraint Management
Analysis
Define parameters & values
Exploration
Examine consistent
configurations
Process model for CMA derived from Delphi technique cf. Zec et al. 2015
General Delphi Techniqueà Round based
1. Individual judgements and justifications2. Anonymous summary by a facilitator
Bastian Rodehüser | Final Presentation Master Thesis | 05.12.2016
Real-time Delphi technique§ Technique to conduct group assessments
§ Round-less
§ For each element that shall be assessed there are two phases for each participant:
1. Each participant makes an assessment without the knowledge of other assessments in the group
2. After the first own assessment participants get further information:
§ The average of all of the responses of the group so far§ The number of responses made so far§ Reasons that others have given for their responses
Responses that stay in conflict to the group answer should be marked.
Participants should observe marked responses§ Reassess own response§ Provide a comprehensible justification for the own response
§ Process can be synchronous or asynchronous
§ Participants remain anonymous
© sebis 35
cf. Gordon and Pease 2006cf. Gordon 2009
Bastian Rodehüser | Final Presentation Master Thesis | 05.12.2016
Constraint Management Process
© sebis 36Bastian Rodehüser | Final Presentation Master Thesis | 05.12.2016
Hidden Profile
“In a hidden profile, a superior decision alternative exists but its superiority is hidden from individual group members because they each have only a portion of information that supports this superior alternative.”
(Stasser and Steward 1992, p. 426)
© sebis 37Bastian Rodehüser | Final Presentation Master Thesis | 05.12.2016
Type of Actions
© sebis 38Bastian Rodehüser | Final Presentation Master Thesis | 05.12.2016
CCA
CM
Satisfaction with decision
© sebis 39Bastian Rodehüser | Final Presentation Master Thesis | 05.12.2016
1,0
1,5
2,0
2,5
3,0
3,5
4,0
4,5
5,0
1verydissatisfied5verysatisfied
1notatall5toaverygreat
extend
1notatall5toaverygreat
extend
1notatall5toaverygreat
extend
1notatall5toaverygreat
extend
Howsatisfiedordissatisfiedareyouwiththequalityof
yourgroup’ssolution?
Towhatextentdoesthefinalsolution
reflectyourinputs?
Towhatextentdoyoufeelcommittedtothegroupsolution?
Towhatextentareyouconfidentthatthegroupsolutionis
correct?
Towhatextentdoyoufeelpersonallyresponsibleforthecorrectnessofthegroupsolution?
CCA CM CCA1 CCA2 CM1 CM2
Questions by Green und Taber (1980, p. 102) and Reinig (2003, p. 82)