Critical EvaluationCritical ReadingCritical Thinking

James Bisset (james.bisset@uwe.ac.uk ) Academic Liaison Librarian (Research Support)

Session outline- What is Critical Reading / Critical

Thinking- Ecology of Resources

- Cognitive bias and you, the researcher

- Evaluating Research Information

Part 1What is...

Critical Reading

Facts vs

Interpretation

The non-critical reader

- Reads a text as a source for...- memorising facts & statements - repeating facts & statements- building a narrative around facts & statements without analysing validity, reliability or applicability

The critical reader- Reads a text as a...

- One interpretation of facts

- Recognises the importance of...- what a text says- how the text evidences and portrays the subject matter

Critical Reading“ Critical Reading involves understanding the content of a text as well as how the subject matter is developed. Critical reading takes in the facts, but goes further. “

http://www.rimt.edu.au/studyandlearningcentre/

What is...

Critical Thinking

Critical Thinking“Critical thinking involves reflecting on the validity of what you have read in light of our prior knowledge and understanding of the world.“

http://www.criticalreading.com/critical_reading_thinking.htm

• Much training is about directing you to the right information = searching and retrieval

• As postgraduate researchers you have to be critical and reflect on what you find.

• Be aware of your impact on your own research, and the research of others.

• What defines your evaluative criteria?

The need to evaluate information

• Resources are interconnected and they evolve

• Information resources are transformed into knowledge

• Knowledge becomes a resource

• Therefore prior knowledge shapes what we go on to create

Ecology of Resources

• Resources are interconnected and they evolve

• Information resources are transformed into knowledge

• Knowledge becomes a resource

• Therefore prior knowledge shapes what we go on to create

Ecology of Resources

• In theory we can select almost any information to complete a task

• In practice we filter it by selecting resources we think most appropriate

• Motivation - affected by the learning we have already done

Role of the researcher

Role of the researcher

• In theory we can select almost any information to complete a task

• In practice we filter it by selecting resources we think most appropriate

• Motivation - affected by the learning we have already done

Other factors

• But, filtering is done for us BEFORE we get the chance to make a judgement

• People• Technologies• Cost• Skills• Copyright, IP

Other factors

• But, filtering is done for us BEFORE we get the chance to make a judgement

• People• Technologies• Cost• Skills• Copyright, IP

Other factors

• But, filtering is done for us BEFORE we get the chance to make a judgement

• People• Technologies• Cost• Skills• Copyright, IP

Other factors

• But, filtering is done for us BEFORE we get the chance to make a judgement

• People• Technologies• Cost• Skills• Copyright, IP

Other factors

• But, filtering is done for us BEFORE we get the chance to make a judgement

• People• Technologies• Cost• Skills• Copyright, IP

• You need to be critical and reflect on all of the sources you find and use.

• You have a professional responsibility, as your research will impact on others.

• You are creating knowledge…… which evolves, and will shape what others create…… similarly, the information you discover will shape the knowledge you create.

• You need to be aware of the filters already impacting upon the information you use.

Part 1 Summary

Part 2Cognitive bias

Cognitive biases• Subjectivity is vulnerable to bias &

hunches• Concept of cognitive bias was

developed in 1970s by Tversky and Kahneman

• Four main groups- Social - Probability/belief - Memory - Decision making

Social biases• Ascribe positive or negative traits to

self, individuals or groups• Loading values or anticipating action

based on prior experience or a bias against self, individuals or groups

• Academic impact: need to verify information and not rely on own views; important to remember when analysing human subjects

Memory biases• How you perceive past events• False memory, positive memory,

imbalanced memory• E.g. A Photo, a Suggestion, a False

Memory • Academic impact: importance of

accurate record keeping and note taking

Memory biases

Memory biases

Probability and belief

• To disregard or to pay too much attention to probability

• Academic impact: need to treat each research finding as distinct and to judge it in its own right

Decision-making biases

• Influences on your decisions by own biases or those of a group

• Academic impact: need to be objective and consider all possible routes of enquiry and treat all research findings as valid until proved otherwise e.g. Semmelweis reflex

Cognitive biases• On your table, group the

forty cards into four piles of ten

• Social • Memory • Probability• Decision

• Explored some of the key bias which may subconsciously be impacting upon how your (subjective) measure of ‘value’ might effect how you filter information and your evaluative criteria.#

Part 2 Summary

Part 3 Evaluation of Research Information

Evaluating information In a literature review you need to

evaluate:• Relevance to the topic• Authority of the author, publisher etc• Objectivity• Presentation• Methodology• Currency

Evaluating information In a literature review you need to

evaluate:• Relevance to the topic• Authority of the author, publisher etc• Objectivity• Presentation• Methodology• Currency

• Before reading the text…• Read the abstract, introduction or

summary.• Scan the bibliographic information

which may highlight key subject areas not specifically alluded to.

• Emphasis may not be clear until you read in full.

Relevance to the topic

• Upon reading the text…• What level is the information at?• Does it contain, and discuss in enough

detail the information you are seeking?• Is the research relevant to the subject

domain / geographical area / demographic / time period you are interested in?

• Three ways of reading: Comprehension, Analysis, Interpretation

Relevance to the topic

“In the course of a series of investigations into various aspects of mental inheritance an intensive study has been made of so-called ‘identical’ twins. The cases examined fall into two main groups: first, those reared together in their parents' homes; secondly, those separated in early infancy, and brought up apart. With the latter, despite wide differences in environmental conditions, the correlations for intelligence, unlike those for school attainments, prove to be surprisingly high. It is argued that this implies that ‘intelligence’, when adequately assessed, is largely dependent on genetic constitution.” http://10.1111/j.2044-8295.1966.tb01014.x“ Burt’s study of monozygotic twins reared apart … involved the largest number of separated twin pairs at the time and produced the highest estimate of heritability for IQ”

Relevance to the topic Exampl

e

Comprehension: Burke identified a link between IQ and inherited genes.

Analysis: This article compares the evidence for IQ being determined by inherited genes as opposed to IQ being affected by external environmental factors.

Interpretation: Evidence for IQ being an inherited trait rather than affected by external factors has potential implications for the development of social and education policy.

Relevance to the topic Exampl

e

• Key topics and ideas• Level of information presented• Relevance in terms of location/subject/scope.• Three ways of reading: Comprehension,

Analysis, Interpretation

Relevance to the topic

Be aware of what is filtering your choices… - Vocabulary and broadness of interpretation. Are you under-estimating the value of a source because it doesn’t match your choice of keywords precisely?

“The trouble with quotes from the internet is that

you never know if they are genuine.”

Abraham Lincoln(1809-1865, President of the United States of

America).

Authority

• Are the authors acknowledged experts in the field? - frequently cited? do they have an h-index? - have you or colleagues heard of them? - can you find any profile information where they work? - how well respected is the author, and their work, in their related field of research?

• Where is it published? - impact factors for a journal (not always an accurate measure of quality, but potentially one of prestige) - is it peer reviewed?

Authority

• Sir Cyril Burt• Fellow of British Academy• Author of over 350 articles and a number of books.

• “pioneer research on the inheritance of mental ability”

AuthorityExampl

e

Authority – Citations Exampl

e

””the most satisfactory attempt” to estimate hereditability of IQ” and

“”the most valuable” of all the separated twin research.”

”the largest of its kind and the only one where “the distribution of children into

foster homes was random ” “

Richard J Herrnstein

Arthur R Jenson Exampl

e

””the most satisfactory attempt” to estimate hereditability of IQ” and

“”the most valuable” of all the separated twin research.”

• Professor at the University of California, Berkeley. • Author of 400+ peer reviewed papers. • In 2002, was listed in the Review of General Psychology’s top 50 “most eminent psychologists of the 20th century”

”the largest of its kind and the only one where “the distribution of children into

foster homes was random ” “

• Edge Pierce Professor of Psychology at Harvard University.• Author of several peer reviewed papers and books. • Credited with discovering and developing several models and theories as one of the founding researchers in the field of quantitative analysis of behaviour.

Richard J Herrnstein

Arthur R Jenson

”the only one of its kind in which the calculation of heritability had any

meaning.”

””the best data“ on separated twins.”

William B Shockley

Hans Eysenck Exampl

e

”the only one of its kind in which the calculation of heritability had any

meaning.”

• Professor of Psychology at the Institute of Psychiatry, King’s College, London. • In 2002, was listed in the Review of General Psychology’s top 100 “most eminent psychologists of the 20th century” as the most cited living psychologist at the time of his death.

””the best data“ on separated twins.”

• Alexander M. Poniatoff Professor of Engineering and Applied Science at Stanford University.• Joint awarded Nobel Prize in Physics in 1956.• 1960’s and 1970’s moved also into area of hereditary behaviour.

William B Shockley

Hans Eysenck

- British Journal of Mathematical and Statistical Psychology - Impact Factor 1.258 - 5th of 13 journals in category “Psychology, Mathematical”

- British Journal of Educational Psychology - Impact Factor 2.093 - 11th of 50 journals in category “Psychology, Educational”

- British Journal of Psychology - Impact Factor 2.103 - 26th of 126 journals in category “Psychology, Multi-disciplinary”

Authority – Impact Factors

Exampl

e

• Are the authors acknowledged experts in the field? - frequently cited? do they have an h-index? - have you or colleagues heard of them? - can you find any profile information where they work? - how well respected is the author, and their work, in their related field of research?

• Where is it published? - Impact factors for a journal (not always an accurate measure of quality, but potentially one of prestige) and is it peer reviewed?

Authority

Be aware of what is filtering your choices y - Is the prestige of the author or the publication impacting on how you evaluate the content?

• Is the subject controversial?

• What is the authors purpose in writing the paper?

• If there are differing views on the subject area, does the author consistently fall into one ‘camp’?

Objectivity

** This is an over-simplification…. **

Objectivity

IQ is inherited IQ is affected by external

factorsBurt, C (1943) “Ability and Income” British Journal of Educational Psychology

Burt, C.L. (1957) “Heredity and Intelligence; A reply to criticisms” British Journal of Statistical Psychology

Burt, C.L. (1958). "The inheritance of mental ability", American Psychologist,

Burt, C.L. (1972). "Inheritance of general intelligence", American Psychologist,

Burt C (1966) “The Genetic Determination of Differences in Intelligence: A Study of Monozygotic Twins Reared Apart and Together.” British Journal of Psychology

Exampl

e

• Is the subject controversial?

• What is the authors purpose in writing the paper?

• If there are differing views on the subject area, does the author consistently fall into one ‘camp’?

Objectivity

Be aware of what is filtering your choices - Does the author demonstrate any hidden bias on the topic? - Evaluate yourself? Are you subconsciously over-valuing the resource because it confirms your own prejudices? Are you being objective?

• are the methodology and sources of data used clearly identified or explained?

• what evidence is presented to support the ideas and conclusions expressed?

• are the conclusions and assumptions made by the author consistent, logical and justified?

• does the research raise any unanswered questions?

Methodology

“ tests of the usual type… as a means of estimating genotypic differences, even the most carefully constructed tests are highly

fallible instruments, and … their verdicts are far less trustworthy than the judgments of the

pupil’s own teachers.”(on measuring IQ of subject children]

Methodology Exampl

e

“ tests of the usual type… as a means of estimating genotypic differences, even the most carefully

constructed tests are highly fallible instruments, and … their verdicts are far less trustworthy than the

judgments of the pupil’s own teachers.”Burt, C. (1957)British Journal of Statistical Psychology

“the unaided judgments even of the most experienced teachers … are nevertheless far less

trustworthy in the long run that the results obtained by properly applied intelligence tests.”

Burt, C. (1943)British Journal of Educational Psychology

Methodology Exampl

e

“ tests of the usual type… as a means of estimating genotypic differences, even the most carefully

constructed tests are highly fallible instruments, and … their verdicts are far less trustworthy than the

judgments of the pupil’s own teachers.”Burt, C. (1957)British Journal of Statistical Psychology

“the unaided judgments even of the most experienced teachers … are nevertheless far less

trustworthy in the long run that the results obtained by properly applied intelligence tests.”

Burt, C. (1943)British Journal of Educational Psychology

Methodology Exampl

e

2.23.39 – 2.25.56http://

news.bbc.co.uk/1/hi/help/3681938.stm

• Various criteria you can assess a resource by.

- a lot more ‘citation’ tools available for journal literature.

• How much time do you realistically have?

Part 3 Summary

• All quotes and opinions were taken from one article:

Tucker, W.H.(1994) “Fact and fiction in the discoevry of Sir Cyril Burt’s Flaws” Journal of the History of the Behavioral Sciences (30).

• Does that change your opinion on some of the previous slides assertions and emphasis?

Part 3 Summary

Bibliography• Kahneman, D. and Amos, A. (1972) ‘Subjective probability: a judgment of

representativeness’, Cognitive Psychology. 3(3): 430-454.

• Luckin, R. (2010) Redesigning learning contexts: technology-rich, learner-centred ecologies. Abingdon: Routledge.

• Strange, D., Hayne, H. and Garry, M. (2008) ’A photo, a suggestion, a false memory’, Applied Cognitive Psychology. 22: 587–603.

• Whitworth, D (2010) “The three domains of value: Why IL practitioners must take a holistic approach” Available at: http://prezi.com/rxqnzpoooolb/the-three-domains-of-value-why-il-practitioners-must-take-a-holistic-approach/

• Tucker, W.H.(1994) “Fact and fiction in the discoevry of Sir Cyril Burt’s Flaws” Journal of the History of the Behavioral Sciences (30).

• http://www.informationliteracy.ie/

Image Credits[Slide 11, 12] Via Flickr Creative Commons, by Martin LaBar. Original available at http://www.flickr.com/photos/32454422@N00/163107859/

[Slide 3, 8, 21, 36] Via Flickr Creative Commons, by Kevin Dooley. Original available at http://www.flickr.com/photos/12836528@N00/2577006675

[Slide 14] Via Flickr Creative Commons, by shellorz. Original available at http://www.flickr.com/photos/59198719@N00/2192821345

[Slide 17] Via Flickr Creative Commons, by Richard Cocks. Original available at http://www.flickr.com/photos/richardland/3999234316/

[Slide 16] Via Flickr Creative Commons, by Photo Extremist. Original available at http://www.flickr.com/photos/thevlue/4839060646/

[Slide 15] Via Flickr Creative Commons, by What Dave Sees. Original available at http://www.flickr.com/photos/whatdavesees/2487875504/

Image CreditsPicture of Professor Arthur Jensen at the 2002 ISIR meeting. Author: Tim Bates. Available at http://en.wikipedia.org/wiki/File:Arthur_Jensen_Vanderbilt_2002.jpg

Picture of Hans Eysenck. Author: Sirswindon. Available at https://en.wikipedia.org/wiki/File:Hans.Eysenck.jpg

Picture of William B Shockley. Author Chuck Painter / Stanford News Service. Available at http://en.wikipedia.org/wiki/File:William_Shockley,_Stanford_University.jpg

[Slide 18] Via Flickr Creative Commons, by vl8189. Original available at http://www.flickr.com/photos/27630470@N03/

[Slide 19] Via Flickr Creative Commons, by opensourceway. Original available at http://www.flickr.com/photos/47691521@N07/4371001458/

[Slide 22] From tdifh.blogspot.com

[Slide 28-29] Provided by colleague

[Slide 68] ‘Vitae®, © 2010 Careers Research and Advisory Centre (CRAC) Limited‘ Available at www.vitae.ac.uk/rdf

Measuring Researche

r Developm

ent

Vitae Researcher Development Framework [see image credits]