Science &Scientific Method

DAIMI Henrik Bærbak Christensen 2

Literature

  [Wikipedia, 2005]– Scientific Method.

  [Carter, 1996]– The Scientific Method.

  [Zobel, 1997]– Writing for Computer Science, Chap 8. Springer-

Verlag 1997

DAIMI Henrik Bærbak Christensen 3

Definition

  Science is both a process of gaining knowledge, and the organized body of knowledge gained by this process.

  The scientific process is the systematic acquisition of new knowledge about a system. This systematic acquisition is generally the scientific method, and the system is generally nature.

  Science is also the scientific knowledge that has been systematically acquired by this scientific process.

DAIMI Henrik Bærbak Christensen 4

Example

  Science is the “opposite” of faith or belief. Often observations of the world can lead to false conclusions, or people generalize too quickly. People mistake truth for belief.

  Classical trick:  Which of the two horizontal lines is the longest?

DAIMI Henrik Bærbak Christensen 5

Example

  For centuries people believed in spontaneous generation. This was the idea that non-living objects can give rise to living organisms. It was common “knowledge” that simple organisms like worms, beetles, frogs, and salamanders could come from dust, mud, etc., and food left out, quickly “swarmed” with life.

  For example:    Observation: Every year in the spring, the Nile River flooded areas

of Egypt along the river, leaving behind nutrient-rich mud that enabled the people to grow that year’s crop of food. However, along with the muddy soil, large numbers of frogs appeared that weren’t around in drier times.

  “Conclusion”: It was perfectly obvious to people back then that muddy soil gave rise to the frogs.

DAIMI Henrik Bærbak Christensen 6

New ‘evidence’

  Microscope in the 1600s: observation of bacteria growth in soup seemed to support spontaneous generation.

  But – what exactly goes on?

  “Life force” present in the molecules of all inorganic matter, including air and the oxygen in it, that could cause spontaneous generation to occur?

DAIMI Henrik Bærbak Christensen 7

Scientific question

  Question: – The scientist then raises a question about what (s)he

sees going on. The question raised must have a “simple,” concrete answer that can be obtained by performing an experiment.

– For example, “How many students came to school today?” could be answered by counting the students present on campus, but “Why did you come to school today?” couldn’t really be answered by doing an experiment.

DAIMI Henrik Bærbak Christensen 8

Example question

  Question: – Is there indeed a “life force” present in air (or oxygen)

that can cause bacteria to develop by spontaneous generation? Is there a means of allowing air to enter a container, thus any life force, if such does exist, but not the bacteria that are present in that air?

DAIMI Henrik Bærbak Christensen 9

Hypothesis

  Hypothesis

  This is a tentative answer to the question: a testable explanation for what was observed. The scientist tries to explain what caused what was observed.

DAIMI Henrik Bærbak Christensen 10

Example

Hypothesis: • There is no such life force in air, and a container of

sterilized broth will remain sterile, even if exposed to the air, as long as bacteria cannot enter the flask.

DAIMI Henrik Bærbak Christensen 11

Aspects of Hypotheses

  In a cause and effect relationship, what you observe is the effect, and hypotheses are possible causes. A generalization based on inductive reasoning is not a hypothesis. An hypothesis is not an observation, rather, a tentative explanation for the observation. – For example, I might observe the effect that my throat

is sore. Then I might form hypotheses as to the cause of that sore throat, including a bacterial infection, a viral infection, or screaming too much at a ball game.

DAIMI Henrik Bærbak Christensen 12

Aspects of Hypotheses

  Hypotheses reflect past experience with similar questions (“educated propositions” about cause) and the work of others. Hypotheses are based on previous knowledge, facts, and general principles. Your answer to the question of what caused the observed effect will be based on your previous knowledge of what causes similar effects in similar situations. – For example, I know that colds are contagious, I don’t

know anyone with a cold, I was at the ball game yesterday, and I was doing a lot of yelling while I was there, so I think that caused my sore throat.

DAIMI Henrik Bærbak Christensen 13

Aspects of Hypotheses

  Multiple hypotheses should be proposed whenever possible. One should think of alternative causes that could explain the observation (the correct one may not even be one that was thought of!) – For example, maybe somebody sitting near me at the

ball game had a sore throat and passed it on to me.

DAIMI Henrik Bærbak Christensen 14

Aspects of Hypotheses

  Hypotheses should be testable by experimentation and deductive reasoning. – For example, throat culture and other tests yielded no

signs of a bacterial or viral infection, I have no fever or other signs/symptoms, and the doctor says my vocal cords are “swollen” in a way that would indicate overuse.

DAIMI Henrik Bærbak Christensen 15

Aspects of Hypotheses

  Hypotheses can be proven wrong/incorrect, but can never be proven or confirmed with absolute certainty. It is impossible to test all given conditions, and someone with more knowledge, sometime in the future, may find a condition under which the hypothesis does not hold true.

– Example: Newtonian physics does not hold at relativist speeds…

DAIMI Henrik Bærbak Christensen 16

Aspects of hypothesis

  That a hypothesis cannot be proven to be correct only incorrect has some consequences.

  First, you should actually try to falsify the hypothesis and if this fails repeatedly only then should your confidence in it grow.

  It is generally difficult for people to be brutally critical about their own work!

DAIMI Henrik Bærbak Christensen 17

Types of reasoning

– Inductive reasoning goes from a set of specific observations to general conclusions: I observed cells in x, y, and z organisms, therefore all animals have cells.

– Deductive reasoning flows from general to specific. From general premises, a scientist would extrapolate to specific results: if all organisms have cells and humans are organisms, then humans should have cells. This is a prediction about a specific case based on the general premises.

DAIMI Henrik Bærbak Christensen 18

Prediction

  Prediction:– Next, the experimenter uses deductive reasoning to test the

hypothesis.

  Example:  Prediction:

– If there is no life force, broth in swan-neck flasks should remain sterile, even if exposed to air, because any bacteria in the air will settle on the walls of the initial portion of the neck. Broth in flasks plugged with cotton should remain sterile because the cotton is able to filter bacteria out of the air.

DAIMI Henrik Bærbak Christensen 19

Experiment/testing

  Testing: – Then, the scientist performs the experiment to see if

the predicted results are obtained. If the expected results are obtained, they support (but does not prove) the hypothesis.

– In science when testing, when doing the experiment, it must be a controlled experiment. The scientist must contrast an “experimental group” with a “control group”. The two groups are treated EXACTLY alike except for the ONE variable being tested.

DAIMI Henrik Bærbak Christensen 20

Example

  Testing: – Wide-mouth jars each containing a piece of meat

were subjected to several variations of “openness” while all other variables were kept the same.control group — These jars of meat were set out without lids so the meat would be exposed to whatever it might be in the butcher shop.experimental group(s) — One group of jars were sealed with lids, and another group of jars had gauze placed over them.

DAIMI Henrik Bærbak Christensen 21

Theory

  A theory is a generalization based on many observations and experiments; a well-tested, verified hypothesis that fits existing data and explains how processes or events are thought to occur. It is a basis for predicting future events or discoveries. Theories may be modified as new information is gained. – This definition of a theory is in sharp contrast to

colloquial usage, where people say something is “just a theory,” thereby intending to imply a great deal of uncertainty.

Summary

from Wikipedia

DAIMI Henrik Bærbak Christensen 23

Process

  The essential elements of the scientific method are iterations and recursions of the following four steps:– Characterization – Hypothesis (a theoretical, hypothetical explanation) – Prediction (logical deduction from the hypothesis) – Experiment (test of all of the above)

DAIMI Henrik Bærbak Christensen 24

Characterization

  The scientific method depends upon the careful characterization of the subject of the investigation.

  Observation demands careful measurement and the use of operational definitions of relevant concepts.

– When the terms used are formally defining, they acquire exact meanings which do not necessarily correspond with their use in natural language:

– for example, mass and weight are quite distinct concepts. On the surface of the Earth, the distinction may not be obvious, but in orbit around Earth, we are weightless and not massless.

– Isaac Newton: "I do not define time, space, place and motion, as being well known to all"

DAIMI Henrik Bærbak Christensen 25

Hypothesis development

  A hypothesis includes a suggested explanation of the subject. It will generally provide a causal explanation or propose some correlation.

  Observations have the general form of existential statements, stating that some particular instance of the phenomena has some characteristic. Causal explanations have the general form of universal statements, stating that every instance of the phenomena has a particular characteristic. It is not deductively valid to infer a universal statement from any series of particular observations.

DAIMI Henrik Bærbak Christensen 26

Prediction from the hypothesis

  A useful hypothesis will enable predictions to be made, by deductive reasoning, that can be experimentally assessed. If results contradictory to the predictions are found, the hypothesis under test is incorrect or incomplete, requiring either revision or abandonment. If results confirming the hypothesis are found, the hypothesis might be correct, but is always subject to further test.

DAIMI Henrik Bærbak Christensen 27

Experiment

  Once a prediction is made, an experiment is designed to test it. The experiment may seek either confirmation or falsification of the hypothesis. Yet an experiment is not an absolute requirement. In observation-based fields of science, like astronomy, actual experiments are never performed, and control experiments are impossible.

  Scientists assume an attitude of openness and accountability on the part of those conducting an experiment. Detailed recordkeeping is essential, to aid in recording and reporting on the experimental results, and providing evidence of the effectiveness and integrity of the procedure. They will also assist in reproducing the experimental results.

DAIMI Henrik Bærbak Christensen 28

Evaluation and Iteration

  The scientific process is iterative.– Verification

• Science is a social enterprise, and scientific work will become accepted by the community only if they can be verified.

• Scientific journals use a process of peer review, in which scientists' papers describing experimental results and their conclusions are submitted to a panel of fellow scientists for evaluation

– Reproducibility • Your experiments should be reproducible so other scientists

can validate your hypothesis and conclusions.

– Reevaluation• All scientific knowledge is in a state of flux, for at any time

new evidence could be presented that contradicts a long-held hypothesis.

DAIMI Henrik Bærbak Christensen 29

Evidence and assumptions

  Evidence comes in different forms and quality, mostly due to underlying assumptions.

  An underlying assumption that 'objects heavier than air fall to the ground when dropped' is not likely to incite much disagreement. An underlying assumption like 'aliens abduct humans' however is an extraordinary claim which requires solid proof. Most extraordinary claims also do not survive Occam's razor.

Scientifc Process case

Debugging as science

DAIMI Henrik Bærbak Christensen 31

Debugging…

  The scientific method may sound heavy, but often you can use the

  observe hypothesis prediction experiment– and detailed record keeping

  ...in fast iteration to analyze precisely

  One good example is debugging...

DAIMI Henrik Bærbak Christensen 32

Case: Haken-Blostein algoritm

  Haken-Blostein is a complex algoritm for spacing sheet music.

  Basically it treats the graphical objects like a system of connected springs with rods.

DAIMI Henrik Bærbak Christensen 33

Observation

  Observation: Each invokation of the algoritm spaced the sheet music differently. Something gets accumulated or changed between invokations?

  Hypothesis: The observed behaviour is due to some kind of accumulating defect in the code. Some variable(s) change value during invocation

  But - Too vague to be useful...

DAIMI Henrik Bærbak Christensen 34

Iteration

  Parameters:– external algorithm parameters: a

– spring constants ci, rod sizes Rj

  Hypothesis 1:– The algorithm parameter, a, changes value

  Experiment:– Inspect the value of a before and after invokation– Easy: there is a dialogue box where it can be

inspected.

  Result: Hypothesis false.

DAIMI Henrik Bærbak Christensen 35

Iteration

  Hypothesis 2:– Spring constants changes

  Experiment:– Inspect the values of a before and after invocation– Difficult:

• There are a lot of springs and no user interface. • Inserting debug output into code.• Manually comparing long lists of double values.

  Result: Hypothesis false.

DAIMI Henrik Bærbak Christensen 36

Iteration

  Hypothesis 3:– Rod sizes changes

  Experiment:– Inspect the values of a before and after invocation– Even more Difficult:

• There are a hundreds of rods and no user interface. • Inserting debug output into code.• Using diff-tool to compare very long lists of double values.

  Result: Hypothesis accepted.– a few rod sizes somehow accumulate width…

… and computer science ?Software Engineering ?

DAIMI Henrik Bærbak Christensen 38

Some observations…

  Computer science / software engineering has some twists…

  Computer science deals more with artifacts than nature itself.

  Thus the material we make hypotheses about and experiment with are constructed objects.– i.e. there exists a relation between hypothesis and the

material in the experiment that we often do not see in physics or biology…

– gives problems with reproducibility• who else builds systems like ours?

– scope of hypothesis (our system versus all systems?)

DAIMI Henrik Bærbak Christensen 39

Some observations…

  Computer science/software engineering is a creative discipline just as much as a descriptive discipline.– You do not invent nature in physics or biology– but software engineers do!

  Part of computer science is inventing stuff– meaning there is room for presenting hypotheses

without experimentation, only convincing argumentation.

  Example: Inheritance in Simula– started object-orientation

DAIMI Henrik Bærbak Christensen 40

Some observation...

  Another research tradition: Mathematics– No experiments but proofs.– [but proofs are subject to errors as well ]

Diploma Thesis

Science in the small…

DAIMI Henrik Bærbak Christensen 42

Science in the small

  From the studieordning:

– Ved udarbejdelsen af hovedopgaven skal den studerende demonstrere fortrolighed med almindelige principper for videnskabelig metode og færdighed i at anvende metoder og teorier til selvstændigt at afgrænse og behandle problemstillinger inden for området softwarekonstruktion.

DAIMI Henrik Bærbak Christensen 43

… Thus

  This is the yardstick that we will use to evaluate your thesis!– softwarekonstruktion: the focus is on constructed

software objects (not necessarily constructing them!)

– selvstændigt at afgrænse og behandle problemstillinger: you must define your focus yourself.

– videnskabelig metode, færdighed i at anvende metoder og teorier : adherence to the process is an important aspect.

DAIMI Henrik Bærbak Christensen 44

Corollary

  Note an important implication:– building application X or Y is not interesting in

itself!

  The focus is on – A) the scientific process– B) the product is knowledge

  This is usually in contrast to engineering degrees where the product is the focus…

DAIMI Henrik Bærbak Christensen 45

Diploma considerations…

  Now – you are not supposed to become world leading scientists in software engineering :

– we do no expect ground-breaking new hypothesis

– comprehensive, large sample, double-blind experiments

– large software artifacts being build

DAIMI Henrik Bærbak Christensen 46

Diploma considerations…

  But – we do expect– Characterization

• that you master the terminology and precision and clarity in statements and writing

• that you provide clear definitions in new concepts or ideas you present or introduce

– Assumptions and scope• that you state assumptions• that you know what “variables” you control and what remain

fixed• that you clearly scope your work (what do you not consider

part of the work, and argue how this influence your conclusions).

DAIMI Henrik Bærbak Christensen 47

Diploma considerations…

  But – we do expect– Scientific method

• that you either– try hard to formulate your problem as a hypothesis to be

evaluated (prediction and experiment)

– perform an experiment to falsify/confirm an existing hypothesis/claim in a ‘novel’ environment

• that software construction techniques are used as part of the process

DAIMI Henrik Bærbak Christensen 48

Diploma considerations…

  But – we do expect– Scientific progress

• that you not replicate well known facts in well known situations

• that you consult the (scientific) literature for inspiration and relations to your work (including web, products, etc).

– Scientific experiment• “Detailed recordkeeping is essential, to aid in recording

and reporting on the experimental results”• that you document both experiments and process in a clear

and unambiguous way so your observations are easy to understand and reproduce

• record ‘open issues’ as you go along!

DAIMI Henrik Bærbak Christensen 49

Diploma considerations…

  But – we do expect

– Scientific evaluation and iteration• that you present and discuss your work with your fellow

‘researchers’• Short papers and presentations at seminars

– include as part of your final report

• Peer reviewing groups• Consulting senior researchers (course teachers )

DAIMI Henrik Bærbak Christensen 50

Record keeping

  If you do an interview– you analyse and conclude on the interview– ... but the interview raw data must be present also

  Not only the result but also the steps in achieving the result is important (you may make an error that a fellow scientist will correct!)

DAIMI Henrik Bærbak Christensen 51

’we build a system type’

  hypothesis: ”using our tool is better and easier”– better: ”will achieve result in half the used time”

  how to handle this:– describe the current situation using a couple of

concrete scenarios• inductive reasoning

– describe same output using the tool • deductive reasoning confirms hypothesis

DAIMI Henrik Bærbak Christensen 52

Rationale

  Every time you make a decision you potentially invalidate your analysis (because you may make the wrong choice!)

  Thus every decision you make must be supplemented with a rationale. Why did you choose as you did?

Designing a project

… or finding a problem…

DAIMI Henrik Bærbak Christensen 54

At least two ways to go…

  Question– observer some interesting aspect

• from the work place; open source system; old project…

– turn the scientific machine• formulate hypothesis; make predictions; experiment

  Evaluation– that is reproduce someone else’s results:– take existing hypothesis/technique– make predictions and experiment

DAIMI Henrik Bærbak Christensen 55

Process of making hypothesis

  Hypothesis making is difficult !  Example:

– General idea:• “I would like to write something about vegetation in Africa”

– Specific topic• “Trees in Africa”

– Problem • “Imbalance between felling and re-growth”

– Hypothesis• “Urban wood consumption is more harmful to tree cover than

rural consumption”.

DAIMI Henrik Bærbak Christensen 56

Software example

– General idea:• “I would like to work with frameworks”

– Specific topic• “Frameworks for classic board games”

– Problem • “Hot spots required to cover large set of board games”

– Hypothesis• “It is possible to construct a framework for classic board

games with 90% code reuse that covers backgammon, chess, and checkers”.

  It is only the latter that we can objectively measure…

DAIMI Henrik Bærbak Christensen 57

Software example

– General idea:• “We want to build software to plot weather data from the

school’s weather station”

– Specific topic• “Plotting software for weather data (time series)”

– Problem • “Adding hot spots to existing plot framework for weather

data”

– Hypothesis• “It is possible to extend framework X with relevant hotspots

using patterns A, B and C to provide hot spots for weather data manipulation”.

  Less “objectively measurable” but adds a lot of precision.

DAIMI Henrik Bærbak Christensen 58

Advice

  Even small questions take a lot of time! – Better to answer small, specific, questions and add a

few more questions than– … drown in work on a very broad hypothesis.

  Problem X turns out to be– problem A– problem B– problem C– problem D– ...

DAIMI Henrik Bærbak Christensen 59

Advice

  Important to be explicit about assumptions and limitations:

  Problem X consists of subproblems A..D. We will here delimit us to subproblem C.

DAIMI Henrik Bærbak Christensen 60

Example

  Health care scenario– [Nurse showing graph on TV using PDA]

  Subproblems:– locating TV to use– combining ctrl and viewer for PDA and TV– steady-state interaction between PDA and TV– MFC combines ctrl and viewer in concrete code

  ... but most of these are independent problems and can be abstracted away...

DAIMI Henrik Bærbak Christensen 61

Open issues

  ’Open issues’ is an important product!!!

  Scientific process leads to– results (typically only on a single subproblem)– open issues

• that is: all the unsolved subproblems that we encountered during the process of trying to produce the result!

  Scientists do not solve all the problems!  The usually identify more problems than they

solve! (We don’t want to become unemployed)

DAIMI Henrik Bærbak Christensen 62

Example: Open Issues

  Health care scenario– [Nurse showing graph on TV using PDA]

  Experiment– .NET on Laptop and PDA

• Laptop keeps model and viewer; PDA keeps controller

  Open issue– MFC’s architecture merge ctrl and view making

experiment really troublesome– analyze and propose alternative implementation.

DAIMI Henrik Bærbak Christensen 63

Evaluation

  Another path is to evaluate existing hypotheses/techniques and report results– Using Quality Attribute workshops to assess qualities

in an architecture– Report experience on architectural documentation

using multiple view – Use architectural prototyping to assess buildability of

new project

DAIMI Henrik Bærbak Christensen 64

Acknowledgements

  Parts of the presentation is work by J. Stein Carter.

  Parts are taken from Wiki pedia