Effect of Experience in On-line Searching on Thinking about

Reliability of Information

By Sharon Kerr

Ideas: Issues Motivating Me

• The world is full of information nowadays. Not all of it is trustworthy. How do we make decisions using this information?

• Helping students doubt or be skeptical about what they hear, read, think and know is critical for their success.

• It will help them begin to know how to sort through the plethora of information and make well reasoned decisions.

Ideas: Added by Team

• Jerrod brought up the issue of the small number of students, five, I would use in my inquiry

• This echoed my own concern, but I hope the depth of the interviews, observations and reflections will make the inquiry meaningful.

Prediction

• If students have more experience searching on the internet, then they will not think more critically about the answers they find than students who have little experience searching.

• In other words, students will not think critically about the answers they find on the internet, regardless of their amount of experience with internet searches.

Explanation

• I think students tend to think they’ve found “the answer” too easily on the internet.

• I think that direct instruction about the range of answers on the internet and about the importance of healthy skepticism can help—

• But that the drive to finish tasks may over-ride this instruction, unless students are specifically reminded of it.

Significance to me

• My natural tendency is to focus on activity and accomplishments rather than the thinking behind them

• Lately I’ve been trying to uncover student thinking through writing and conversation, and have been surprised at how visible the learning process becomes

• Two families among my relatives home-school, so I am curious about the effectiveness of this alternative means of education

Inquiry Plan, Part 1

• I chose five home-schoolers from four different families to work with during this inquiry project

• Three were inexperienced in searching for answers on the internet, having never searched independently for an answer previously, and two were experienced, searching for answers once a day.

Inquiry Plan, Part 3

• The observation was followed by a reflective discussion, that attempted to uncover how each student was thinking during the search, with a particular focus on how they were thinking about reliability of web sites and the answers they contained

Interesting Patterns in Data

• Every student came up with a different answer to the question about lion population in the wild

• Their answers are shown on a chart on the following slide

How Many Lions Are in the Wild?

Initials Answer

Inexperienced

SS 20,000

JP 10,260

RH 100,300

ExperiencedAH 10000-50000

SH 23,000-39,000

Interesting Patterns in Data

• Both experienced internet users gave a range of values rather than a single value, which may indicate a degree of skepticism

• AH rounded the range he found on the web site a little, which could indicate skepticism

• But SH reported that range exactly as she found it on the web site

Interesting Patterns in Data

• JP, who is inexperienced, looked at a web site that reported a range of values, but chose only the top end of that value for his answer

• This seems to me to indicate lack of understanding of the meaning of the numbers, and to affirm that he was just looking for an answer rather than thinking about its meaning

Search Time vs. Experience (SS, JP, RH are inexperienced; AH and SH are experienced)

0

2

4

6

8

10

12

14

16

SS JP RH AH SH

Students' Initials

Tim

e o

f se

arch

Interesting Patterns in Data

• Students all seemed to have a limited attention time, with most students spending around five minutes looking for information about the number of lions

• SS, who looked longer, was the oldest student, but he also took a detour during that time to show me his favorite airplane site

• His internet connection was also the slowest

Number of Word Changes vs. Experience (SS, JP and RH inexperienced; AH and SH experienced)

0

1

2

3

4

5

6

SS JP RH AH SH

Searching Students' Initials

Nu

mb

er o

f W

ord

Ch

ang

es

Interesting Patterns in Data

• The experienced students and the oldest inexperienced student used more word combinations to search than the students who were younger and inexperienced

• Two inexperienced students used only one word to search, continuing to look deeper in the Google index, rather than re-directing the search

Interesting Patterns in Data

• Students only searched words that were in the question asked, “How many lions are living in the wild?”

• Words searched: lions in wild, number of lions in wild (2x), number of lions in wild around world, number of wild lions around world, number of lions around world, how many lions live in the wild? wild lions, lion, how many lions live

Number of Sites Searched vs. Experience (SS, JP and RH inexperienced; AH and SH experienced)

0

1

2

3

4

5

6

SS JP RH AH SH

Searching Students' Initials

Nu

mb

er o

f S

ites

Sea

rch

ed

Interesting Patterns in Data

• Number of sites searched varies across inexperienced and experienced users

• The experienced users stayed with favorite sites, exclusively, while the inexperienced users searched more widely

• SH explained that she had been taught to search only familiar sites

Interesting Patterns in Data

• Four out of five students chose the first answer they found that matched the question as the answer to report

• One student looked for a second answer, since his first answer only referred to “Lions in Africa”

• In the pre-search interview most said they decided they had the right answer when it “matched the question”

Interesting Patterns in Data

• Students first responded to questions about reliability by relying on personal intuition:– “The answer sounds very likely.” SS– “I knew it would be less than 100,000.” AH– “The site looked like a good place.” RH– “When you’re looking at Google, you have to

trust yourself so if you’re not at the right site, just go back and find a different site.” JP

Interesting Patterns in Data

• Three out of five students learned something else along the way to finding the answer to the question, using their “peripheral vision in the search” as recommended by the UC Berkeley Tutorial– “Google isn’t very good at getting exactly what you want. I got

sea lions, monkeys called lions, all kinds of things, lions clubs…” SS

– “A lot of lions live in Africa, but not many live in Asia. You could find out what breed a male lion is by how thick its mane is. Males have thicker manes than females.” JP

– “There are such things as ligers and they’re cuter than they sound. They eat a lot.” SH

– The other two students said they were only focusing on finding the answer to the question

Interesting Patterns in Data

• Beginning with the second student, I asked a follow-up question where I asked them to compare the previous answers to their own and think about which was more reliable– JP discovered a difference in the currency of the data, , “When you’re a

teacher, you should tell people to look at the bottom to check for the date.”

– AH was motivated by this to look at many more sites and to compare values with a reference book

– “I don’t think there’s even an answer. We need to send a real scientist to count! I am confused. There really doesn’t seem to be an answer. Could it be they reproduce and the number goes up and then people hunt them and the number goes down? That makes me think. I’m confused.” SH

– “The first site is more reliable than mine because it has more information. Maybe the sites are older and newer.” RH

Interesting Patterns in Data

• After showing the discrepancy in answers, I asked students how people might count lions anyway and they responded with genuine curiosity and a few guesses:– “They might get different answers because

they go to different places. There are different amounts of lions in different places so that could give them different estimates. They estimate based on what they see in one place.” RH

Interesting Patterns in Data

• More theories of how lions are counted– “Probably they count them by putting the lion to sleep,

and taking its blood so they know the breed. They’d have to do this one by one, so they estimate. It said 6,000-10,000 so it’s an estimate.” JP

– “Send some one out and they say 1,2, 3…Line up, I’ve got some turkey here. (laughs) Maybe they calculate it based on how they reproduce.” SH

– “They might have used books, pictures, or had scientists watch lions. Every year I think they count them up, like a lion census. You know how censuses work for people? They visit every house and ask how many people live there. They do that for lions and then add them up.” DH

Emergent Ideas

• Experienced web searchers were simultaneously more skeptical and less skeptical than inexperienced searchers

• They were less skeptical, because they had certain sites they trusted completely

• Their parents had recommended these sites previously. It seems the parents were “agnostic readers” of the web “acknowledg[ing] that there are both good and bad resources, and so develop[ing] schemes for finding good sites and separating one from the other. However the students read the sites that they were told were good “exegetically,” “treating the text [as] definitive.”

• They were more skeptical, because they answered with a range of numbers rather than a single one

Emergent Ideas

• Students tend to think of questions and answers having a one to one relationship. They look for “a match.” Fill in the blank worksheets most likely reinforce this perception

• Showing them that others found a different answer to the same question took their thinking to a much deeper level

• Searching the same question with others stretches students’ thinking. Bertram Bruce predicts, “Literacy practices in the future may become highly collaborative enterprises, corresponding to an intensification of emphasis on coordination and communication.”

Emergent Ideas

• Only after comparing their own answers to others’ answers did students begin thinking seriously about reliability and validity

• Only then did they begin to wonder where the published numbers came from and how scientists count animals in the wild

Emergent Ideas

• Bertram Bruce writes, “the view of Web searching as simply finding information limits the key to its importance for education or other life activities. The joy and true value of the Web lie in the way it can open up our questions. We ask one thing, but the Web leads us to ask more questions and to become aware of how much we do not know.”

Emergent Ideas

• Three out of five students naturally learned something wider than the question while searching

• The two others searched with “blinders” on looking only for the answer

• To broaden students’ perspectives as they search, Bertram Bruce suggests, “We might say: ‘Use the web to find the answer to such-and-such question. Now, report on three things you learned that you had never imagined before you did that search.’”

Emergent Ideas

• Two students reasoned independently to the idea that currency of the information was related to its validity

• Bertram Bruce writes, “Some sites, but not all, indicate when they were last updated, but it is usually difficult to determine.”

Emergent Ideas

• Some students implied that they considered authorship and publishing body as they chose web sites

• However at least two of them considered Wikipedia to be among the most reliable

• Encouraging students to think about the original source of information seems to be highly important

Questions

• How would students’ in traditional schools perceptions of reliability of internet answers compare to the homeschoolers I studied?

• What additional methods would be effective in encouraging students to be skeptical as they search for answers on the internet?

• How do traditional teaching methods, like fill-in-the-blank worksheets, encourage idea that questions only have one short answer?

Questions

• What strategies are best to help students assess whether or not an author or a publishing body is reliable, as recommended by Johns’ Hopkins?

• How can students learn to spot propaganda, misinformation and disinformation, as recommended by Johns Hopkins?

• How many lions ARE living in the wild?• How DO scientists count them?

Example of Change• Initially I thought of this inquiry as a series

of one-on-one encounters

• After noticing that the second student’s answer was 50% of the first student’s answer, I realized sharing and comparing answers had potential to produce more learning

• Even when using the internet, working together increases our effectiveness and uncovers the issue of reliability for students

Example of Change

• I used to think of the internet primarily as a place for people to find information, answers

• Five students finding five different answers to the same question helped me realize that questioning the answers and their origin more important than finding an answer

• Now I realize internet is an excellent resource for generating questions, particularly about reliability of information

Sources• Austhink, (10 September 2007). Critical Thinking on the Web: A Directory of Quality On-line

Resources. Retrieved September 21, 2007 from http://www.austhink.org/critical/pages/teaching.html

• Bruce, B. Credibility of the Web:Why We Need Dialectical Reading. Journal of Philosophy of Education (special issue), 34(1), 97-109.

• Bruce, B. C. (1999, April). Digital content: The babel of cyberspace. Journal of Adolescent and Adult Literacy, 42(7), 558-563.

• Bruce, B. C. (1999-2000, December/January). Searching the web: New domains for inquiry. Journal of Adolescent and Adult Literacy, 43(4), 348-354.

• Bruce, B. Twenty-First Century Literacy. A version of this paper appears (1998) under the title, "Current Issues and Future Directions" in J. Flood, S. B. Heath, & D. Lapp (Eds.), A handbook for literacy educators: Research on teaching the communicative and visual arts (pp. 675-684). New York: Macmillan

• Middletown Thrall Library (2202). Critical Thinking: Get Enlightened. Retrieved September 21, 2007 from http://www.thrall.org/criticalthinking/

• Regents of the University of California. (May 23, 2007). Finding Information on the Internet Tutorial. Retrieved September 21, 2007 from http://www.lib.berkeley.edu/TeachingLib/Guides/Internet/FindInfo.html

• Sheridan Libraries (2004). Johns Hopkins University Libraries: Evaluating Information Found on the Internet. Retrieved September 21, 2007 from http://www.library.jhu.edu/researchhelp/general/evaluating/