can you study right brain processes with left brain tools? · 2015-12-15 · design automation lab...

DESIGN AUTOMATION LAB

ARIZONA STATE UNIVERSITY


Can you study Right Brain Processes with Left Can you study Right Brain Processes with Left Brain Tools? Brain Tools?

Jami J. ShahMechanical and Aerospace Engineering

Arizona State University, Tempe, AZ

NSF Innovation & Discovery WorkshopMay, 2006

Financial support provided by NSF grant DMI-9812646 NSF grant DMI-0115447 Ford Research Labs

In-kind support provided byHewlett-Packard, San Diego

2

DESIGN AUTOMATION LABORATORY

©J. Shah: NSF Innovation & Discovery Workshop


Our 15 year journey

Multi-level aligned experiments; Outcomes assessment

Can design studies be related to microscopic cognitive experiments?

2002-05

Review past studies: Cognitive Sci & Design Theory

Identification of ideation componentsDevelopment of ideation metricsStatistical DOE procedure

2000-02

Statistical DOEanalysis of sketches/ text

Hypothesis Testing: Expressiveness of graphical vs. textual representations

1998-2000

Iteration based snapshots; analysis of sketches & text

Evaluate effectiveness of collaborative idea generation methods (6-3-5, C-Sketch, Gallery)

1996-2000

Protocol studiesMatch ideation cognitive processes to GENEPLORE model

1995

Protocol studiesIdentify variables, cognitive processe; Hypothesis finding?

1991-93Experimental methodObjectiveYear

3




The Dark Age: Process identification; Hypothesis finding

Perform protocol study first, then see what you can “discover” in the transcriptExamples: processes, types, sequence, patterns, ….

Clamped Coulomb Friction Device on

Leg R t i t

Sliding Di ti End

Constraint

Maximum Tension to Secure, Then Relax Slightly

Shah J, Nico J, Kraver T: "Non-intrusive protocol study of idea generation in mechanical design", ASME Design Theory & Meth. conference, Albuquerque, pp 213-222, Sep. 1993.

4




What can we learn from Cognitive Psychologists?

Match cognitive processes to known model (Geneplore)Protocol study

16 subjects; 30 mins/each

Key ReferenceShah, J., 1998, “Experimental Investigation of Progressive Idea Generation Techniques,”

Proceedings, ASME Design Theory and Methodology Conference, Atlanta, GA.

5




Getting more "scientific": Hypothesis Testing

Pictures aid ideation89 students; statistical DOEOutcome measures: Novelty, Quality

McKoy F., Hernandez N., Summers J., Shah J., “Influence of design representation on effectiveness of idea generation”, ASME Design Theory & Methodology conf., September 10-13, 2001, Pittsburgh, Paper#DTM-21685.

6




Focus back on Engineering: Effectiveness of ideation methods

% Retention = ((XY∩X)/X) x 100 = 69%% Modification = (∆Y/X) x 100 = 31%% Fixation = (∆Y+/Y) x 100 = 3.2%

Comparison of C-Sketch, Method 6-3-5 and Gallery200 hrs of data, 3 years, 44 designers: undergraduates, graduates, and experienced engineers from industry

7




Apply left brain tools:Step 1: Identify variables

Human Factors• Designer’s background, education, experience• Designer’s personal characteristics, MBTI profile, creativity, intelligence,

group interaction skills• Nuisance variables that need to be controlled by “equivalent” sets of

designers or random selection of designers to “average out” bias Environment Variables

• Time constraint (deadlines), incentives, working conditions • Nuisance variables that need to be controlled by maintaining an identical

environmentDesign Method Variables

• Specific to idea generation method; identified from its procedure • Examples: group size, cycle time, number of iterations.

Design Problem Variables• Characterize nature and difficulty of design problem• Complexity, Degree of innovation needed, Decomposability.

8




DESIGN IDEATION METHODS:Classification

INVERSION

FORWARD STEPS

MORPHOLOGICAL CHARTSBRAINSTORMINGK-J METHOD

CHECKLISTSRANDOM STIMULI

METHOD 6-3-5C-SKETCH (Collaborative Sketching)GALLERY METHOD

AFFINITY METHODSTORYBOARDINGFISHBONE

SYNECTICS

GERMINAL

TRANSFORMATIONAL

PROGRESSIVE

ORGANIZATIONAL

INTUITIVE

EXPERENTIAL

FORMAL IDEATION METHODS

INVENTIVE PRINCIPLES

HYBRID

DESIGN CATALOGS

PHYSICAL EFFECTS

OF SOLUTIONS

TRIZ

SIT/USIT

WORKING PRINCIPLES

Many ideation methods have similar components: e.g., suspended judgment, provocative stimuli, etc.

9




Step 2: What outcomes to measure?

Bottom line for an engineer:How well does a design meet the specified requirements? (Design quality)

Is that enough? The Kano model says “NO!”

Fulfillment of basic requirements is necessary but not sufficient.

The discovery of new attributes that will “surprise and delight” the customer, is what results in a competitive advantage.

Thus, quality and novelty must be pursued together.

10




Measures of design ideation effectiveness

Engineer's "End Goal" measuresNovelty: how unusual or unexpected an idea is as compared to other ideasQuality: feasibility and conformance to design specifications

Refinements to ideation measuresBest Quality and Best NoveltyPlans to include “Efficiency” in the future (above metrics divided by time)

"Process" measuresHow effective is the ideation method in expanding design space?How effective is the ideation method in exploring design space?Variety: how different concepts are from each other Quantity: total number of ideas generated

KEY REFERENCEShah, J. J., Smith, S. M., Vargas-Hernandez, N., 2003, “Metrics for Measuring Ideation Effectiveness”, Design Studies, V24 (2), pp. 111-134.

11




Step 3: Statistical DOE

evaluate the outcome not the processevaluate the ideation method and neutralize human and environment variablesUse appropriate outcome metricstest ideation components rather than methods in their entiretyuse fractional factorial DOE & ANOVA

Key ReferenceShah, J. J., Kulkarni, S. V., Vargas-Hernandez, N., 2000, “Guidelines for Experimental Evaluation of Idea Generation Methods in Conceptual Design”, ASME Transactions, Journal of Mechanical Design, vol. 122, no. 4, pp. 377-384.

12




Simulating Real World Design

Predict result

REAL WORLD DESIGN Corporate experience Designer expertise Technical complexity Environment variables Hard constraints Fixed roles/job functions Multiple interacting processes; no control Serious consequences for failure

DESIGN EXPERIMENT Limited designer expertise & incentive Fictitious problem “Play” environment Maximum freedom Synthetic group Involves group dynamics Multiple interacting processes; limited control No penalty for failure

LAB EXPERIMENT Highly controlled environment Simple tasks Study single cognitive process or structure Tests individuals No direct relation to engineering design

Extract key components

Combine models + interactions

Simulate by

DESIGN EXPERIMENTLimited designer expertise & incentiveFictitious problem“Play” environmentMaximum freedomSynthetic groupInvolves group dynamicsMultiple interacting processes; limited controlNo penalty for failure

LAB EXPERIMENTHighly controlled environmentSimple tasksStudy single cognitive process or structureTest individualsNo direct relation to engineering design

COMPONENT EXPERIMENT

Use single or multiple components

•Fictitious methods Interactions between components Reduced experiments•Some relation to design

ALIGNMENT

low ecological validityhigh intrinsic validity

moderate ecological validitymoderate intrinsic validity

high ecological validitylow intrinsic validity.

Single task; Well known variables

simple tasks, few variables

complex tasks, many variables; poorly knownValidity of

Experiments at Multiple

Levels

13




Aligning Experiments at Multiple Levels

Combine the strengths of lab experiments and ideation component experiments:• Break idea generation methods into key ideation components• Run experiments for identified key components, derive models for component

interactions• Keys to alignment: equivalent components, same fractional factorial

experiments, same levels, same outcome metrics, same ANOVA for significant effects & interactions

Check Correlation

Alignment: Same Ideation Components Same Effectiveness Metrics

Same DOE

Design Experiments Done by Engineers

Lab Experiments

Done by Psychologists

Main effects of components &

interactions

Construct Cognitive models of components

Main effects of components &

interactions

Make corrections & repeat

Key ReferenceShah J, Smith S, Vargas N, Gerkens D, Muqi W, “Empirical Studies Of Design Ideation: Alignment

of Design Experiments with Lab Experiments” ASME Design Theory & Meth. conference, 2003.

14




Multi-level alignment Test

Reference for Results:Vargas-Hernandez, Shah, Smith, “Multi-level aligned empirical studies of ideation: Final Results”,

ASME Design Theory Conference, 2006 (accepted)

15




Alignment results

The results are mixed; alignment failed in some waysthe most clearly aligned component is Judgment. Incubation shows 75% alignment, despite differences in the length of the interruption. FORS is also well aligned despite different implementations. Alignment failed completely for example exposure. This may be due to the different timing of example exposure in the different levels. In the presence of Incubation, FORS had consistently a detrimental effect on all measures in design experiments, but a positive effect on all measures in the lab experiments! No explanation is obvious at this time.the process measures are much better aligned (variety and quantity). The outcome measures are poorly aligned. Novelty was poorly aligned because the lab problems were less constrained and required little technical expertise. Quality poorly aligned maybe because the measurement of quality at lab level is somewhat “loose”, while engineering evaluation follows some well-defined and formal procedures.

16




Summary

Our thinking has evolved gradually, from hypothesis finding in protocol studies to statistical DOE based methodology based on outcomesWe have developed and used ideation effectiveness measure and distinguished between outcomes (end goals) and process (means)Further, we have demonstrated a methodology for conducting aligned experiments at multiple levels Although perfectly alignment was not achieved, experience has been gained on how to do that in the future; Tighter controls are needed, particularly future lab studies need to use methods closer to engineering evaluation and better problems

Some surprises: Our effectiveness metrics are now being used at several schools, not only for design theory research, but for curriculum design and student grading. Also, the C-Sketch method has found its way into design textbooks and Corporate training organizations

Shah J, "Identification, Measurement & Development of Design Skills for Engineering Education", Intl. Conference on Engineering Design, Melbourne, Australia, Aug 2005.

Shah J., Vargas-Hernandez, Summers, Kulkarni, “Collaborative Sketching (C-Sketch) - an Idea Generation Technique for Engineering Design”, J. Creative Behavior, V35(3), 168-198, 2001

17




Where do we go from here?

If we were to repeat multi-level experiments, we expect to do better under tighter controls and better coordination, there are differences that we know are not reconcilable.Should we continue this line of enquiry? Can you really study “right brain” processes with left brain tools (scientific method)Can we find better, faster methods, that will have both ecological validity and intrinsic validity? That will lead to the development of better methods for “innovation on demand”What happens when other factors, such as human variables and problem complexity are added to the mix?

ACKNOWLEDGEMENTWe are grateful to National Science Foundation for financial support (grant DMI-9812646 & DMI-0115447) and Ford Research LabsViews expressed in this presentation are those of the authors, not NSF

18




NSF DTM/ED Funding History

11.6%11.9%17.6%16.5%Others

0.0%1.5%2.7%3.5%DfM/IPPD

2.3%4.5%5.4%4.7%Mfg process design

2.3%4.5%2.7%3.5%Function, Behavior modeling

7.0%1.5%1.4%2.4%GD&T

2.3%4.5%1.4%0.0%VR/visualization

11.6%7.5%5.4%0.0%DfE/ EbDM

2.3%1.5%1.4%5.9%Ideation, innovation

7.0%10.4%5.4%3.5%Kinematics/Robotics

11.6%6.0%16.2%5.9%CAD/CAGD

32.6%23.9%5.4%1.2%Decision/Utility, DBD

0.0%6.0%8.1%22.4%AI/KBS

9.3%13.4%25.7%9.4%Optimization

0.0%3.0%1.4%21.2%Design process/cognition

2001-041996-001990-951985-90Research area

can you study right brain processes with left brain tools? · 2015-12-15 · design automation lab...

Documents