how cife does translational research for industrial … cife does translational research for...
TRANSCRIPT
CIFE Research Questions and Methods
How CIFE Does Translational Research for Industrial Sponsors
10/1/2009 Research Methods 1
John Kunz, CIFE
Construction Management Today
• Large unique projectsprojects
• Lots of– stakeholders – different types
of infoNeed to relate– Need to relate data
– Paper– Time for
coordination• Time and cost
Needs: implemented, tested theory w/• Power
10/1/2009 Research Methods 2
• Time and cost pressures • Generality
Research question at CIFE
• What is a theory, based on fundamental y,principles, that can be used to describe and explain or predict elements of (sustainable)
?design, construction and operations?• What is a way to operationalize a new theory
i t th d ?using computer methods?• How well does the theory describe and
predict reality; how general is it?predict reality; how general is it?
10/1/2009 Research Methods 3
Research methods
• Careful observation of practice– Embedded participation
Ethnographic interviews– Ethnographic interviews• Formal models of project elements, i.e., products,
organizations and processes ( ) ( G– Usually symbolic (non-numeric) + graphical (4D, Gantt, tables,
many specialized views)– Often computational
C f ll d i d lid ti t di t h id• Carefully designed validation studies to show evidence of power and generality– Yes: Study real projects (n); compare performance of innovative
d b li tiand baseline practices– No:
• Solve hard problems: every good engineer does it; not a “mouse”D th t d t b k i t
10/1/2009 Research Methods 4
• Demo that does not break is not a mouse• “One mouse is no mice”
Objective for Ph.D. research
• Make a believable claim that an innovative theorycontributes to knowledge in a particular subarea of AEC research
• Theory (scientific sense): analytic structure to that• Theory (scientific sense): analytic structure to that can be used to describe and explain or predict empirical observations– Universal requirement of the Ph.D. in all fields, all countries
10/1/2009 Research Methods 5
Translational Medicine
Phases:• Observe and prototype: explore needs, develops
potential treatments in basic laboratory research, and tests safety and efficacy, principally in randomized
CIF y y p p y
clinical trials. • Field intervention: examine how findings from clinical
science shown to be efficacious and safe treatments
FE Focu science, shown to be efficacious and safe treatments
established in phase 1 translational research, function when are applied in routine practice.
• Implement broadly: add necessary information to
us
• Implement broadly: add necessary information to convert treatments and prevention strategies, shown to be effective and cost-effective in Phase 2 Translational Research into sustainable solutions
IndustrFocus
10/1/2009 Research Methods 6
Translational Research, into sustainable solutions.– Based on Wikipedia
ry s
Translational research at CIFE
• Integrate: g– Baseline observations of practice– Theory that integrates fundamentals of
engineering, basic sciences, social and political sciences
– Results of field intervention studiesResults of field intervention studies • Goal: improve project (virtual) design and
construction in ways that measurably improveconstruction in ways that measurably improve project success and life cycle quality.
10/1/2009 Research Methods 7
CIFE “Horseshoe” Research Method
Metrics (performance),Doable?
Research M th d
Observed Problem
Intuition
scope (domain)
Generality
Power Testable?
MethodsTheoretical POD
Research Ei t i Questions
R Tasks: • Theory
Einstein: The only real
valuable thing is intuition.
ValidationClaimed
Evidence?• Model• Test
Legend:
ResultsClaimed
ContributionsPredicted Impact
10/1/2009 Research Methods 8
Leads toCompare
IntellectualMeritImpact
Example of Horseshoe in UseRyan Orr
Observed Problem
Theoretical POD
Research Questions
ValidationResultsClaimed
ContributionsPredicted
Impact
Intuition
R Tasks: •Theory•Model• Test
Research Methods
y
Observed Problem Theoretical POD/ Gap RQs Data Source
Lack an integrated theoretical
It takes firms many years to evolve a good strategy. What
are they learning?
- Lack an integrated theoretical framework for why firms adopt
different entry & staffing strategies.
- Literature tells us strategy selection is discretionary.
(1) What strategies do firms use?
(2) How do they select these t t i ?
(1) Case studies w. 3 firms:Kone, Fluor, Hines
(2) Asia data(Melin quote about deepare they learning? selection is discretionary.
- Classic mgmt theory has been overlooked.
strategies? (Melin quote about deep understanding)
R d t OD'<- Identifies framework to <- Actors, scripts, set, props>
(1) GTRecommends a concrete OD strategy & show that not all strategies are appropriate for
all firms.
describe environment>-(1) Recognize set of strategies
that firms adopt- (2) Implies a KBV contingency
approach to strategy
, p , , p p- (1) Strategies: incr. KS, decr.
KD, incr. except handling, knowl. integration
- (2) Strategy = F(EMB, KD)
(1) GT(2) Inference
<copy the paper b/w US & Canada>
Predicted Impact Contribution Findings Data Analysis/Presentation
10/1/2009 Research Methods 9
Example of Horseshoe in UseClaudio Mourgues
Observed Problem
Theoretical POD
Research Questions
ValidationResultsClaimed
ContributionsPredicted
Impact
Intuition
R Tasks: •Theory•Model• Test
Research Methods
g
10/1/2009 Research Methods 10
AgendaObserved Problem
Theoretical POD
Research Questions
ValidationResultsClaimed
ContributionsPredicted
Impact
Intuition
R Tasks: •Theory•Model• Test
Research Methods
• Motivating Engineering • ExamplesMotivating Engineering Problem
• Intuitions
p
• Properties• What: Research
Questions
Properties
• How: – Research Methods
Conclusions• Conclusions
10/1/2009 Research Methods 11
Steps in the Horseshoe Research MethodObserved Problem
Theoretical POD
Research Questions
ValidationResultsClaimed
ContributionsPredicted
Impact
Intuition
R Tasks: •Theory•Model• Test
Research Methods
Observed Problem
Theoretical POD
Research Questions
ValidationResultsClaimed
ContributionsPredicted
Impact
Intuition
R Tasks: •Theory•Model• Test
Research Methods
• Observed problem: – Some problem faced in practice– Motivates your research, not just another problem
t lto solve– May relate to one or more of product (technical),
organization and process limitso ga a o a d p ocess s
10/1/2009 Research Methods 12
Observed Motivating ProblemObserved Problem
Theoretical POD
Research Questions
ValidationResultsClaimed
ContributionsPredicted
Impact
Intuition
R Tasks: •Theory•Model• Test
Research Methods
Embedded Measure of successparticipation method
• Spend time with practitioners (days to
• You find some theory to address fundamental moti ating iss es (laterpractitioners (days to
many months)• Identify a recurring
problem
motivating issues (later in intuition)
problem• Identify (complex)
problem context: staffing, no time, bad site conditions, …
• Identify fundamental
10/1/2009 Research Methods 13
ymotivating issue(s)
Example-1: ObservationSheryl Staub-French
Observed Problem
Theoretical POD
Research Questions
ValidationResultsClaimed
ContributionsPredicted
Impact
Intuition
R Tasks: •Theory•Model• Test
Research Methods
Observed Problem
Theoretical POD
Research Questions
ValidationResultsClaimed
ContributionsPredicted
Impact
Intuition
R Tasks: •Theory•Model• Test
Research Methods
y
Hard to (re)estimate costsW ll H i h 8’ 10 5’Wall Height: 8’ → 10.5’ Wall Rating: Unrated → Fire-Rated
D i i QTY Res. TotalProd.
Description QTY uom Res. Costs
Total Cost
Metal Stud Wall, Type A (using Crew C-1)Install Metal Stud 1550 LF 6.9 lf/hr $3,125 $3,513
Prod. Rate
Description QTY uom Res. Costs
Total Cost
Metal Stud Wall (Crew C-1 & Rolling Scaffolding)Install Metal Stud 1895 LF 4.3 lf/hr $6,000 $6,474Hang Drywall 5040 SF 65 sf/hr $6,978 $8,037Apply Tape 5040 SF 45 sf/hr $8,960 $9,464
Prod. Rate
10/1/2009 Research Methods 14
Hang Drywall 3840 SF 65 sf/hr $5,317 $6,123Apply Tape 3840 SF 45 sf/hr $6,827 $7,211Install Insulation 1920 SF 100 sf/hr $1,440 $2,304Framing for Openings 6 EA 75 sf/hr $450 $476
Install Insulation 2520 SF 100 sf/hr $1,890 $3,024Framing for Openings 6 EA 75 sf/hr $450 $476Frame Wall-Beam Intersec 12 EA 1 hr/ea $540 $552Apply Fire Caulking 48 LF 40 lf/hr $54 $222Cut Drywall 240 LF 60 lf/hr $180 $180
Steps in the Horseshoe Research MethodObserved Problem
Theoretical POD
Research Questions
ValidationResultsClaimed
ContributionsPredicted
Impact
Intuition
R Tasks: •Theory•Model• Test
Research Methods
Intuition
• Intuition:– How you want to approach the problem … your
focus on one or more of product (technical), organization and process limitsorganization and process limits
– Motivates and guides your library and internet search for enabling theoretical methods
– No need to justify it -- if it helps you make progress
10/1/2009 Research Methods 15
Intuition: Features drive costsObserved Problem
Theoretical POD
Research Questions
ValidationResultsClaimed
ContributionsPredicted
Impact
Intuition
R Tasks: •Theory•Model• Test
Research Methods
Resources>Features <Objects Actions Cost
C t I t ti M
-- Property Set
Component Features
Intersection Features
Macro Features
• Fire-rating• Fire-rated• Height
Length
• Area• Bending Radius• Bottom Elevation
Curved
• Thickness• Top Elevation• Width
-- Feature Set
• Penetrations• Wall-Beam Intersection
• Dimension Change • Direction Change
• Length• Curved
10/1/2009 Research Methods 16
Wall Beam Intersection• Wall-Countertop Connections
Direction Change• Grouping(Similarity)• Openings
Type OfInstance OfImpacts
Legend
Steps in the Horseshoe Research MethodObserved Problem
Theoretical POD
Research Questions
ValidationResultsClaimed
ContributionsPredicted
Impact
Intuition
R Tasks: •Theory•Model• Test
Research Methods
• Theoretical Point of Departure (POD): – Literature that relates to the motivating problem in
the focus of interest (based on intuition)C id ll th l t CIFE lit t– Consider all the relevant CIFE literature
– Be scholarly: consider all the relevant literature from journals, potential reviewers of your worko jou a s, po e a e e e s o you o
– Distinguish • Fundamental: a few theoretical constructs on which you
b ildbuild• Courtesy: many theoretical and practical references of
broad interest to you or potential readers
10/1/2009 Research Methods 17
Example POD: Work InstructionsObserved Problem
Theoretical POD
Research Questions
ValidationResultsClaimed
ContributionsPredicted
Impact
Intuition
R Tasks: •Theory•Model• Test
Research Methods
Job Assignment Sheets Design principles for assembly instructions
Project’s 3D Model Company’s Process ModelsKey plan
• Step-by-step, one for each major step
(Oglesby et al., 1989)assembly instructions (Agrawala et al., 2003)
j Company s Process Models
Work area
Model view
view
Color coding
• Clear and explicit order
• Parts added in each step should be visible
• Mode of attachment
Activity Work InstructionKey planMethod to Generate Key plan
Details view
Color coding view
Mode of attachment should be visible
• Action diagrams rather than structural
A d id liWork area
Model view
Details view
Key plan view
Color coding view
Method to Generate Work Instructions from Product and
Work area
Model view
Details view
Key plan view
Color coding view
• Arrows and guidelines to indicate attachment
Work Packaging
18
Process Models(missing reference)
Steps in the Horseshoe Research MethodObserved Problem
Theoretical POD
Research Questions
ValidationResultsClaimed
ContributionsPredicted
Impact
Intuition
R Tasks: •Theory•Model• Test
Research Methods
Research Questions
• Research questions: specific questions that the• Research questions: specific questions that the research intends to (or did) address:– Answer gives you a contribution no matter what it
b h t h t b tmay be, e.g., how, not what or best– Relate to motivating problem and theoretical POD– Typically involveTypically involve
• A model, or “ontology,” which is a vocabulary in the computerReasoning method to anal e the inp t data• Reasoning method to analyze the input data given the model
• Framework that links the model and reasoning
10/1/2009 Research Methods 19
method
Example: Research questions, theoryObserved Problem
Theoretical POD
Research Questions
ValidationResultsClaimed
ContributionsPredicted
Impact
Intuition
R Tasks: •Theory•Model• Test
Research Methods
1. How do product features in a building product model1. How do product features in a building product model affect construction costs?
2 How can the relationship between product features and2. How can the relationship between product features and construction cost be formalized in computer models?
3 How can reasoning mechanisms leverage the3. How can reasoning mechanisms leverage the representation in (2) to automatically generate cost-loaded and resource-loaded activities?
10/1/2009 Research Methods 20Sheryl Staub-French, 19 December 2001
Steps in the Horseshoe Research MethodObserved Problem
Theoretical POD
Research Questions
ValidationResultsClaimed
ContributionsPredicted
Impact
Intuition
R Tasks: •Theory•Model• Test
Research MethodsResearch Methods
• Research methods: Broad approaches to research t ktasks, e.g., – Hypothesis testing using statistical comparison of
a performance of baseline and innovative methodsp– Grounded theory to develop a conceptual
framework when none existsComputer demonstration of an existence proof– Computer demonstration of an existence proof that a conceptual model and analysis of it can be successfully implemented I f l th i t l– Informal, e.g., synthesize an ontology
– Computational modeling: develop a method
10/1/2009 Research Methods 21
Research Methods, TasksObserved Problem
Theoretical POD
Research Questions
ValidationResultsClaimed
ContributionsPredicted
Impact
Intuition
R Tasks: •Theory•Model• Test
Research Methods
How Research Methods:How – Research Methods:– Observe: e.g., Hathaway project – Computational model and statistical analysis of results p y
of use – Validate by analysis of charrette, retrospective or
prospective applicationprospective application– n(cases) ~100 Ideally; at least >= ~10
• What: Staub-French created– Feature and cost ontology (i.e., vocabulary in
computer)Model of effects of feature changes on costs
10/1/2009 Research Methods 22
– Model of effects of feature changes on costs
Steps in the Horseshoe Research MethodObserved Problem
Theoretical POD
Research Questions
ValidationResultsClaimed
ContributionsPredicted
Impact
Intuition
R Tasks: •Theory•Model• Test
Research Methods
R Tasks: •Theory•Model• Test
• Research tasks: specific segments of the work to address the research questionsaddress the research questions
• Operationalize research methods, e.g.,– Collect survey data test a hypothesisy yp– Observe how practitioners use a traditional and
innovative method test the hypothesis that the two are the sameare the same
– Do retrospective or (ideally) prospective testing statistical evidence for power and generality of the underlying model reasoning and frameworkunderlying model, reasoning and framework
– Modeling or theory formation– Implement reasoning method and Framework
10/1/2009 Research Methods 23
g– Initial testing to validate the concept
Research tasks: Example-1
Observed Problem
Theoretical POD
Research Questions
ValidationResultsClaimed
ContributionsPredicted
Impact
Intuition
R Tasks: •Theory•Model• Test
Research Methods
• Build
Research tasks: Example 1
• Build– Graphical product
model– Symbolic product
model– Symbolic process
ComponentFeatures
<F> and/or <O>
IntersectionFeatures
<F> and/or <O>Actions
<A>Symbolic process model (Cost analysis procedure)
P f t ti
Subcomp-onents<O>
Resource Activity• Perform retrospective & charrette studies
Activities
Resources Costs
ResourceSpecification
ActivitySpecification
10/1/2009 Research Methods 24
Resources<R>
Costs<C>
Steps in the Horseshoe Research MethodObserved Problem
Theoretical POD
Research Questions
ValidationResultsClaimed
ContributionsPredicted
Impact
Intuition
R Tasks: •Theory•Model• Test
Research Methods
ValidationResults
• Validation results: quantitative data that you interpret as• Validation results: quantitative data that you interpret as evidence for the power or generality of the innovative method in comparison with traditional methods
10/1/2009 Research Methods 25
Validation methodsObserved Problem
Theoretical POD
Research Questions
ValidationResultsClaimed
ContributionsPredicted
Impact
Intuition
R Tasks: •Theory•Model• Test
Research Methods
ValidationResults
E id f d lit f• Evidence for power and generality of theory: “milk-stool” consistency
10/1/2009 Research Methods 26
TheoryObservation
Model Analysis (Simulation)
Validation methodsObserved Problem
Theoretical POD
Research Questions
ValidationResultsClaimed
ContributionsPredicted
Impact
Intuition
R Tasks: •Theory•Model• Test
Research Methods
ValidationResults
E id f d lit f• Evidence for power and generality of theory: “milk-stool” consistency
Swiss milk stool
10/1/2009 Research Methods 27
TheoryObservation
Model Analysis (Simulation)
Swiss milk stool
Validation study results: Example-1Observed Problem
Theoretical POD
Research Questions
ValidationResultsClaimed
ContributionsPredicted
Impact
Intuition
R Tasks: •Theory•Model• Test
Research Methods
10/1/2009 Research Methods 28Δ Theoretical Ideal – ACE Process Δ ACE Process – Traditional Process
Validation Method: Charrette studyObserved Problem
Theoretical POD
Research Questions
ValidationResultsClaimed
ContributionsPredicted
Impact
Intuition
R Tasks: •Theory•Model• Test
Research Methods
ValidationResults
• Realistic engineering problem: 1-2 hours for titi t “ l ”practitioner to “solve”
• Two studies, same testers– Conventional methodConventional method– Experimental method
• Measured performance metrics: time, cost, accuracy, l t i tcompleteness, consistency, …
• Null hypothesis: Performance same for two methods• Test results provide evidence of power and generalityTest results provide evidence of power and generality
of the feature and relationship representation and associated reasoning method claim that theory of each is a contribution to knowledge
10/1/2009 Research Methods 29
each is a contribution to knowledge
Form of a Claim to make a contribution to k l d
Observed Problem
Theoretical POD
Research Questions
ValidationResultsClaimed
ContributionsPredicted
Impact
Intuition
R Tasks: •Theory•Model• Test
Research Methods
knowledge
• Based on theseBased on these• empirical data (test results) that• provide (verb)• evidence (information that an independent reviewer can
believe that helps confirm or deny a hypothesis or elucidate mechanisms) of)
• power (ability to solve a particular problem) and• generality (ability to solve a range of problems) of the
th ti l f k ( t l d l) d• theoretical framework (ontology or model) and • method (reasoning or analysis process), • I claim that <my> theory is a contribution to the knowledge of
10/1/2009 Research Methods 30
I claim that my theory is a contribution to the knowledge of <some subfield of AEC>
Example claimObserved Problem
Theoretical POD
Research Questions
ValidationResultsClaimed
ContributionsPredicted
Impact
Intuition
R Tasks: •Theory•Model• Test
Research Methods
Claimed Contributions
• Claimed contribution: Theory that adds to the POD with evidence– Theory that adds to the POD with evidence
– Not data (normally)– Universal requirement for the Ph.D.
• Form of a “claim” for a contribution:– Data: validation data using a (Prospective) method show
improved performance of the innovative method in p pcomparison with traditional methods.
– Interpretation: I interpret these data as evidence of the power and generality of the innovative method. p g y
– Claim: Based on the evidence, I claim the underlying theory (innovative ontology, reasoning method and framework) as contributions to knowledge in the field of
10/1/2009 Research Methods 31
) gVirtual Design and Construction.
Steps in the Horseshoe Research MethodObserved Problem
Theoretical POD
Research Questions
ValidationResultsClaimed
ContributionsPredicted
Impact
Intuition
R Tasks: •Theory•Model• Test
Research Methods
• Practical contribution:• Practical contribution: – Your speculation about impact on practice of broad
application of the innovative methodpp– Normal and appropriate for engineering– Good for your career and job prospects – Not required for the Ph.D.
10/1/2009 Research Methods 32
Example-2: ObservationChuck Han
Observed Problem
Theoretical POD
Research Questions
ValidationResultsClaimed
ContributionsPredicted
Impact
Intuition
R Tasks: •Theory•Model• Test
Research Methods
It is hard to ask:D thi d i• Does this design comply with building code?code?
• How do you know?
10/1/2009 Research Methods 33
IntuitionObserved Problem
Theoretical POD
Research Questions
ValidationResultsClaimed
ContributionsPredicted
Impact
Intuition
R Tasks: •Theory•Model• Test
Research Methods
• Intent is a crucial issue for compliance– Designer: rationale
f ffor forms– Code: requirements
of formsof forms
10/1/2009 Research Methods 34
Research questions, theory
Observed Problem
Theoretical POD
Research Questions
ValidationResultsClaimed
ContributionsPredicted
Impact
Intuition
R Tasks: •Theory•Model• Test
Research Methods
Research questions, theory
1. Can a generalized analysis framework be built to support automated architectural building code checking?
based on the design intent of building code– based on the design-intent of building code 2. What are the product model issues to support (1)
– Considering “IFC” standardCo s de g C s a da d– Reformulate IFCs to represent intent
10/1/2009 Research Methods 35Chuck Han, 5 October 1998
Research methods: Example-2
Observed Problem
Theoretical POD
Research Questions
ValidationResultsClaimed
ContributionsPredicted
Impact
Intuition
R Tasks: •Theory•Model• Test
Research Methods
Research methods: Example 2
• Build and testBuild and test– Graphical product
modelS b li d t– Symbolic product model
– Symbolic process model (Analysis procedure)
– GUIGUI• Retrospective,
Intervention studies
10/1/2009 Research Methods 36
Research Tasks
Observed Problem
Theoretical POD
Research Questions
ValidationResultsClaimed
ContributionsPredicted
Impact
Intuition
R Tasks: •Theory•Model• Test
Research Methods
Research Tasks
Research tasks – specifically how:Research tasks specifically, how:– Observe: Sunnyvale building permit group – Model and analyze (simulation) results y ( )– Validate
• Many what-if animations• Stanford career center (intervention study)
Research Tasks: Han created computer models ofP f k– Process – framework
– Product – building; wheelchair
10/1/2009 Research Methods 37
Intervention Study Method
Observed Problem
Theoretical POD
Research Questions
ValidationResultsClaimed
ContributionsPredicted
Impact
Intuition
R Tasks: •Theory•Model• Test
Research Methods
Intervention Study Method
• Tasks– Observe engineering case– Predict project performance: time, cost, quality, …
• With current plan• Following possible intervention (s)
– Intervene - at management discretion• Intervention ⇒ Evidence for “believability”• Small sample statistics w/o control• Small-sample statistics w/o control
10/1/2009 Research Methods 38
Research methods and tasks: Example-3John Haymaker
Observed Problem
Theoretical POD
Research Questions
ValidationResultsClaimed
ContributionsPredicted
Impact
Intuition
R Tasks: •Theory•Model• Test
Research Methods
• Build
John Haymaker
• Build– Graphical product
model (CAD)– Formal Symbolic
Project Model as a Directed Acyclic Graph of dependencies between geometric views
• Perform retrospective validation studies (of WDCH)
10/1/2009 Research Methods 39
WDCH)
Example-3: WDCH Concert Hall Cantilever Test CaseObserved Problem
Theoretical POD
Research Questions
ValidationResultsClaimed
ContributionsPredicted
Impact
Intuition
R Tasks: •Theory•Model• Test
Research Methods
10/1/2009 Research Methods 40
Cantilevered Ceiling Panels Perspector GraphObserved Problem
Theoretical POD
Research Questions
ValidationResultsClaimed
ContributionsPredicted
Impact
Intuition
R Tasks: •Theory•Model• Test
Research Methods
10/1/2009 Research Methods 41
Task: Develop and test RepresentationObserved Problem
Theoretical POD
Research Questions
ValidationResultsClaimed
ContributionsPredicted
Impact
Intuition
R Tasks: •Theory•Model• Test
Research Methods
Perspector
Perspective
Perspective
10/1/2009 Research Methods 42
Task: ReasoningObserved Problem
Theoretical POD
Research Questions
ValidationResultsClaimed
ContributionsPredicted
Impact
Intuition
R Tasks: •Theory•Model• Test
Research Methods
A Generic Perspector:- Analyzes the source Features in the source Perspective(s)
10/1/2009 Research Methods 43
y p ( )- Constructs dependent Feature(s) in the dependent Perspective- Relates the dependent Feature(s) and source Features(s)
Evidence For PowerObserved Problem
Theoretical POD
Research Questions
ValidationResultsClaimed
ContributionsPredicted
Impact
Intuition
R Tasks: •Theory•Model• Test
Research Methods
Deck attachments
Interpretation• Interpretation by staff
Perspector• Perspector interpretation
10/1/2009 Research Methods 44
Evidence for Power of the Perspective Approach: Deck Attachment Retrospective Test Case
Observed Problem
Theoretical POD
Research Questions
ValidationResultsClaimed
ContributionsPredicted
Impact
Intuition
R Tasks: •Theory•Model• Test
Research Methods
field welds 5 Perspective
Deck Attachment Retrospective Test Case
~ total attachments 100 WDCH Approach CommentsQuality Accuracy
- Found in test case 0 (shop weld)* 114Significant improvement over current practice is possible.
___________ ___
Found in test case 0 (shop weld) 114 - Missed in test case 86 (field weld)* 2 Completeness - Amount of detail
p pFurther improvement possible.
Automation could make creating more detail cost-effective.
Time to Specify - To conceptualize 0 sec. 10 hrs
To code Perspectors 0 sec 200 hrs
Re-use of Perspectors reduces programming time. Practice can
- To code Perspectors 0 sec. 200 hrs - To assemble graph 0 sec. 120 sec to Construct and Integrate - For project 140 hours no data
F t t d t 56
Once graph is specificed, representations can be constructed quickly
reduce conceptualization time.
10/1/2009 Research Methods 45
- For test case no data 56 secs constructed quickly.
Evidence for generalityObserved Problem
Theoretical POD
Research Questions
ValidationResultsClaimed
ContributionsPredicted
Impact
Intuition
R Tasks: •Theory•Model• Test
Research Methods
PerspectorPerspector method for
• Deck• Deck Attachments
• Cantilever Deck Attachment Perspector
conditions
Cantilevered Ceiling Panel Perspector
10/1/2009 Research Methods 46
Cantilevered Ceiling Panel Perspector
Types of Validation studiesObserved Problem
Theoretical POD
Research Questions
ValidationResultsClaimed
ContributionsPredicted
Impact
Intuition
R Tasks: •Theory•Model• Test
Research Methods
• Existence demonstration; stress test: not evidence!• Retrospective: Look at past cases; adjust theory to p p ; j y
get good results; analyze to identify– Quality of results: power– Context in which quality is good: generality
Mor Context in which quality is good: generality
– N >= 10 (hopefully)• Charrette: N >= 10 (hopefully)
Prospective:
re
• Prospective: – Look at current cases; do not adjust theory; analyze power
and generalityN 100 (hopefully)
Pow – N ~ 100 (hopefully)
• Intervention: – Look at a case; do not adjust theory; sponsor intervenes
b d l i
er
10/1/2009 Research Methods 47
based on analysis– N ~ 1 (typically)
(Empirical) Validation methods
Observed Problem
Theoretical POD
Research Questions
ValidationResultsClaimed
ContributionsPredicted
Impact
Intuition
R Tasks: •Theory•Model• Test
Research Methods
Observed Problem
Theoretical POD
Research Questions
ValidationResultsClaimed
ContributionsPredicted
Impact
Intuition
R Tasks: •Theory•Model• Test
Research Methods
(Empirical) Validation methods
Validation study methods: 1 2 3
• Retrospective test case(s)Retrospective test case(s)
• “Stress” test
• Engineering test cases
• Prospective (engineering) test case(s)Prospective (engineering) test case(s)
• Laboratory study - Charrette method
10/1/2009 Research Methods 48
• Field intervention study
Conclusions about Validation
Observed Problem
Theoretical POD
Research Questions
ValidationResultsClaimed
ContributionsPredicted
Impact
Intuition
R Tasks: •Theory•Model• Test
Research Methods
Observed Problem
Theoretical POD
Research Questions
ValidationResultsClaimed
ContributionsPredicted
Impact
Intuition
R Tasks: •Theory•Model• Test
Research Methods
Conclusions about Validation
• Serious validation studies are a realistic objective for university research
• Validation motivates studentsS• Skills– Define measurable performance objectives
Observe engineering practice– Observe engineering practice– Collect data: Formal instrument– Analyze data– Interpret results
10/1/2009 Research Methods 49
Contributions:Observed Problem
Theoretical POD
Research Questions
ValidationResultsClaimed
ContributionsPredicted
Impact
Intuition
R Tasks: •Theory•Model• Test
Research Methods
Based on evidence of generality and power, Haymaker claimsthat the underlying mechanismsthat the underlying mechanisms of his method are a contribution to knowledge, specifically
1. Project Model Ontology: dependencies, views, Features, relationshipsrelationships
2. Perspector: reasoning to formally construct a newformally construct a new Perspective from other Perspectives
10/1/2009 Research Methods 50
3. Graph Manager:
SummaryObserved Problem
Theoretical POD
Research Questions
ValidationResultsClaimed
ContributionsPredicted
Impact
Intuition
R Tasks: •Theory•Model• Test
Research Methods
• Motivating Observations: – Engineering practice still has many aspects of a craftg g p y p– Careful observation of practice can inform and test research
• Intuitions:Engineering Principles can guide practice– Engineering Principles can guide practice
– Implementing principles in computer can lead to testable theoretical models
• Research Methods: Horseshoe method• Research Methods: Horseshoe method• Research questions:
– What fundamental principles characterize civil engineering design d t?and management?
– How can we operationalize new ones in the computer– How how good are they?
10/1/2009 Research Methods 51
SummaryObserved Problem
Theoretical POD
Research Questions
ValidationResultsClaimed
ContributionsPredicted
Impact
Intuition
R Tasks: •Theory•Model• Test
Research Methods
• Build Theory: – Processes (e.g., estimate costs, check ADA, plan, create ( g , , , p ,
reasoning, …)– Products (Buildings, components, systems),
• Model: Object-oriented symbolic and graphical models ofModel: Object oriented symbolic and graphical models of products and processes
• Validation to show evidence of power and generality: Many CIFE Ph D projects– Many CIFE Ph.D. projects
– Integration into CEE curriculum: DCI, CEE 100, 111, 215, 222, 241, 242, VDC internshipsOther Stanford departments: CS Medicine Symbolic Systems– Other Stanford departments: CS, Medicine, Symbolic Systems
10/1/2009 Research Methods 52
Skills for success
• Good engineer: design analyze manage• Good engineer: design, analyze, manage• Research methods• Communication
– Written/Oral– Programming
Colleagues sponsors stakeholders– Colleagues, sponsors, stakeholders• Ability to identify, find examples in the chaos of
practice, propose “gold standard” methods• Integrated use of quick-response, careful analysis,
reflection
10/1/2009 Research Methods 53
Research: Benefits for Students
• Real engineering problems• Synthesize academic, work experience• Methods apply to big questions:
– What fundamental principles characterize civil engineeringWhat fundamental principles characterize civil engineering design and management?
– How can we operationalize new ones in the computer– How how good are they?How how good are they?
• Research methods necessary for careers in practice & researchInteresting• Interesting
• Support
10/1/2009 Research Methods 54
Research: Benefits to Industry
• Industry suggests problems– Helps identify big, pre-competitive problems
• University perspective adds value – Broad (“integrated”)– Theoretical: consistent, repeatable, exportable processes– Modern methods
• Unique opportunity to address hard problems– Focused effort of gifted graduate students working on industrial-class
problemsproblems• Results have been of interest
– Emergence of VDC and BIM in practice now big!– Applications (e g org analysis 4D)Applications (e.g., org analysis, 4D)– Methods (e.g., VDC, OOP, 4D, testing, POP, ICE)– Successful evangelists within companies– (Informal) benchmark evaluation of company processes wrt global best
10/1/2009 Research Methods 55
( ) p y p gpractices
Research Program Status
• Research questionsWhat fundamental principles characterize civil engineering design and management?How can we operationalize new ones in theHow can we operationalize new ones in the computerHow how good are they?
• Research methodsCareful observation of practiceSymbolic (non numeric) computer models ofSymbolic (non-numeric) computer models of products and processesEvidence of power and generality
10/1/2009 Research Methods 56
Research Program Status
Progress P ti l ( VDC lt )• Practice: real (see VDC survey results)
• Academic: – Questions and methods now well-validated by CIFE
research since 1988, especially last 10 years– CIFE now 20! New competition emergingCIFE now 20! New competition emerging– CIFE methods now part of Stanford CEE culture:
Integration requirement; DCI, CEE 100, 111, 115, 222, 241, 242, 243, …
– External recognition: “CIFE is the premier academic center for VDC research”
10/1/2009 Research Methods 57
center for VDC research– External influence: VDC, validation now real issues