building brick by brick: a pragmatic approach to measuring impact
DESCRIPTION
Illustrates pragmatic approaches to building meaningful information literacy impact measures in an imperfect environment.TRANSCRIPT
![Page 1: Building brick by brick: A pragmatic approach to measuring impact](https://reader035.vdocument.in/reader035/viewer/2022062510/54b5337b4a7959f8228b4592/html5/thumbnails/1.jpg)
BUILDING BRICK BY BRICK: A PRAGMATIC APPROACH TO MEASURING IMPACT
Sarah Faye CohenInformation Literacy Librarian
Janet R. CottrellDirector of Academic Resources & the Library
Michelle G. MillerSenior Associate Provost
Champlain College, Burlington, Vermont, USA
![Page 2: Building brick by brick: A pragmatic approach to measuring impact](https://reader035.vdocument.in/reader035/viewer/2022062510/54b5337b4a7959f8228b4592/html5/thumbnails/2.jpg)
CHAMPLAIN COLLEGE, BURLINGTON VERMONT
2000 on-campus students
Professional FocusMultimedia
GraphicsE-gamingDigital ForensicsBusiness
New Interdisciplinary Core CurriculumRoll-out began
Fall 2007
http://www.flickr.com/photos/glemak/270321175/
![Page 3: Building brick by brick: A pragmatic approach to measuring impact](https://reader035.vdocument.in/reader035/viewer/2022062510/54b5337b4a7959f8228b4592/html5/thumbnails/3.jpg)
CHAMPLAIN’S UNIQUE “CORE” CURRICULUM
Sequential, linked, theme-based courses
Interdisciplinary Inquiry-based Common, master
syllabi Common texts Common, project-
based, research assignments
Apple Core (1992) by Claes Oldenberg (1929- ), Billy Rose Sculpture Garden, Israel Museum in Jerusalem.
http://www.flickr.com/photos/imagemd/827470717/
![Page 4: Building brick by brick: A pragmatic approach to measuring impact](https://reader035.vdocument.in/reader035/viewer/2022062510/54b5337b4a7959f8228b4592/html5/thumbnails/4.jpg)
CHAMPLAIN’S INFORMATION LITERACY PROGRAM
“All students, all four years”
Incremental Embedded Inquiry Based Rubric-based
assessments Common
curriculumhttp://www.flickr.com/photos/malloy/195204215/
![Page 5: Building brick by brick: A pragmatic approach to measuring impact](https://reader035.vdocument.in/reader035/viewer/2022062510/54b5337b4a7959f8228b4592/html5/thumbnails/5.jpg)
EPORT
Evidence that students are learning the skills and the concepts we are teaching.
“Map” IL outcomes taught and assessed in Core classes.
Track instruction and assessment of outcomes in a matrix.
Use the data to inform our teaching.
http://www.flickr.com/photos/prawnpie/72710206/
![Page 6: Building brick by brick: A pragmatic approach to measuring impact](https://reader035.vdocument.in/reader035/viewer/2022062510/54b5337b4a7959f8228b4592/html5/thumbnails/6.jpg)
DISCOVER DATA
![Page 7: Building brick by brick: A pragmatic approach to measuring impact](https://reader035.vdocument.in/reader035/viewer/2022062510/54b5337b4a7959f8228b4592/html5/thumbnails/7.jpg)
SOME DATA CHARTS ARE:EASY TO READ, HAPPY, USEFUL
Number of people entering the library
![Page 8: Building brick by brick: A pragmatic approach to measuring impact](https://reader035.vdocument.in/reader035/viewer/2022062510/54b5337b4a7959f8228b4592/html5/thumbnails/8.jpg)
SOME DATA CHARTS ARE:EASY TO READ, HAPPY, USEFUL
Number of people entering the library
Number of books checked out
![Page 9: Building brick by brick: A pragmatic approach to measuring impact](https://reader035.vdocument.in/reader035/viewer/2022062510/54b5337b4a7959f8228b4592/html5/thumbnails/9.jpg)
SOME DATA CHARTS ARE:EASY TO READ, HAPPY, USEFUL
Number of people entering the library
Number of books checked out
Number of reference questions answered
![Page 10: Building brick by brick: A pragmatic approach to measuring impact](https://reader035.vdocument.in/reader035/viewer/2022062510/54b5337b4a7959f8228b4592/html5/thumbnails/10.jpg)
HOW DATA IS PRESENTED ELICITS SOME REACTIONS.
![Page 11: Building brick by brick: A pragmatic approach to measuring impact](https://reader035.vdocument.in/reader035/viewer/2022062510/54b5337b4a7959f8228b4592/html5/thumbnails/11.jpg)
SOME DATA CHARTS ARE NOT.
![Page 12: Building brick by brick: A pragmatic approach to measuring impact](https://reader035.vdocument.in/reader035/viewer/2022062510/54b5337b4a7959f8228b4592/html5/thumbnails/12.jpg)
HOW DATA IS PRESENTED ELICITS SOME REACTIONS.
![Page 13: Building brick by brick: A pragmatic approach to measuring impact](https://reader035.vdocument.in/reader035/viewer/2022062510/54b5337b4a7959f8228b4592/html5/thumbnails/13.jpg)
COLLABORATE
![Page 14: Building brick by brick: A pragmatic approach to measuring impact](https://reader035.vdocument.in/reader035/viewer/2022062510/54b5337b4a7959f8228b4592/html5/thumbnails/14.jpg)
NO ONE IS AN ISLAND
Learn from other assessment efforts
Explore unexpected avenues Center for
Instructional Practice
Informational interviewing
Bring a beginner’s mind
http://www.flickr.com/photos/msojka/4227411105/
![Page 15: Building brick by brick: A pragmatic approach to measuring impact](https://reader035.vdocument.in/reader035/viewer/2022062510/54b5337b4a7959f8228b4592/html5/thumbnails/15.jpg)
ENCOUNTERING OBSTACLES
Expect some roads to be closed
Look for other partners, other opportunities
Understand and learn about the block and what it can mean for your assessment
http://www.flickr.com/photos/whartz/1209257477/
![Page 16: Building brick by brick: A pragmatic approach to measuring impact](https://reader035.vdocument.in/reader035/viewer/2022062510/54b5337b4a7959f8228b4592/html5/thumbnails/16.jpg)
FIND ALLIES AND LET THE WORD SPREAD
Cultivate individual connections Annotated
Bibliographies as an example
Ask for help Be patient Celebrate small
victories Express gratitude
http://www.flickr.com/photos/marcoveringa/3203134555/in/photostream/
![Page 17: Building brick by brick: A pragmatic approach to measuring impact](https://reader035.vdocument.in/reader035/viewer/2022062510/54b5337b4a7959f8228b4592/html5/thumbnails/17.jpg)
MAKE DECISIONS WHEN NECESSARY
![Page 18: Building brick by brick: A pragmatic approach to measuring impact](https://reader035.vdocument.in/reader035/viewer/2022062510/54b5337b4a7959f8228b4592/html5/thumbnails/18.jpg)
MAKE DECISIONS WHEN NECESSARY
Sometimes you can wait for a culture to evolve organically…
Sometimes you just have to help it along.
http://www.flickr.com/photos/24761036@N00/2904177517/
![Page 19: Building brick by brick: A pragmatic approach to measuring impact](https://reader035.vdocument.in/reader035/viewer/2022062510/54b5337b4a7959f8228b4592/html5/thumbnails/19.jpg)
BUILDING ON WHAT YOU HAVE AND BUILDING WHAT YOU NEED
We have a culture that is nimble and innovative.
http://www.flickr.com/photos/jeffica/3973440757/
![Page 20: Building brick by brick: A pragmatic approach to measuring impact](https://reader035.vdocument.in/reader035/viewer/2022062510/54b5337b4a7959f8228b4592/html5/thumbnails/20.jpg)
BUILDING ON WHAT YOU HAVE AND BUILDING WHAT YOU NEED
We have a culture that is nimble and innovative.
We did not have a foundation in information literacy or assessment.
http://www.flickr.com/photos/jeffica/3973440757/
![Page 21: Building brick by brick: A pragmatic approach to measuring impact](https://reader035.vdocument.in/reader035/viewer/2022062510/54b5337b4a7959f8228b4592/html5/thumbnails/21.jpg)
BUILDING ON WHAT YOU HAVE AND BUILDING WHAT YOU NEED
We have a culture that is nimble and innovative.
We did not have a foundation in information literacy or assessment.
We needed to create a plan quickly.
http://www.flickr.com/photos/jeffica/3973440757/
![Page 22: Building brick by brick: A pragmatic approach to measuring impact](https://reader035.vdocument.in/reader035/viewer/2022062510/54b5337b4a7959f8228b4592/html5/thumbnails/22.jpg)
MANDATE 1
http://www.old-picture.com/crimean-war/000/pictures/General-Bosquet-giving-orders-001.jpg
We WILL embed
Information Literacy in
the new Core Curriculum...
![Page 23: Building brick by brick: A pragmatic approach to measuring impact](https://reader035.vdocument.in/reader035/viewer/2022062510/54b5337b4a7959f8228b4592/html5/thumbnails/23.jpg)
MANDATE 2
http://www.old-picture.com/crimean-war/000/pictures/General-Bosquet-giving-orders-001.jpg
We WILL spread our instruction throughout
our students’ college
careers !
![Page 24: Building brick by brick: A pragmatic approach to measuring impact](https://reader035.vdocument.in/reader035/viewer/2022062510/54b5337b4a7959f8228b4592/html5/thumbnails/24.jpg)
MANDATE 3
http://www.old-picture.com/crimean-war/000/pictures/General-Bosquet-giving-orders-001.jpg
We WILL devise
meaningful impact
measures !
![Page 25: Building brick by brick: A pragmatic approach to measuring impact](https://reader035.vdocument.in/reader035/viewer/2022062510/54b5337b4a7959f8228b4592/html5/thumbnails/25.jpg)
EMPHASIZE MEANING AND USEFULNESS
![Page 26: Building brick by brick: A pragmatic approach to measuring impact](https://reader035.vdocument.in/reader035/viewer/2022062510/54b5337b4a7959f8228b4592/html5/thumbnails/26.jpg)
MAKING MEANING
Address the needs of various audiences
Revise Questions DataRepeat
Adjust data views until they can be seen http://www.flickr.com/photos/photobunny_earl/993311118/
![Page 27: Building brick by brick: A pragmatic approach to measuring impact](https://reader035.vdocument.in/reader035/viewer/2022062510/54b5337b4a7959f8228b4592/html5/thumbnails/27.jpg)
MEANINGFUL TO STUDENTS
The big question from students: What’s in it for me?
Tie into existing motivation
Be persistent
http://www.flickr.com/photos/seandreilinger/2326448445/
![Page 28: Building brick by brick: A pragmatic approach to measuring impact](https://reader035.vdocument.in/reader035/viewer/2022062510/54b5337b4a7959f8228b4592/html5/thumbnails/28.jpg)
MEANINGFUL TO FACULTY
Connect to what faculty care about Students Their courses
Speak their language
Model assessment Context
http://www.flickr.com/photos/drurydrama/2198820696/
![Page 29: Building brick by brick: A pragmatic approach to measuring impact](https://reader035.vdocument.in/reader035/viewer/2022062510/54b5337b4a7959f8228b4592/html5/thumbnails/29.jpg)
MEANINGFUL TO ADMINISTRATION
Capitalize on existing initiatives Quality Return on
investment
Capitalize on external pressures Accreditation Accountability
“Close the loop”http://www.flickr.com/photos/mutantmandias/3061053256/
![Page 30: Building brick by brick: A pragmatic approach to measuring impact](https://reader035.vdocument.in/reader035/viewer/2022062510/54b5337b4a7959f8228b4592/html5/thumbnails/30.jpg)
QUESTION: ARE STUDENTS LEARNING IL?DATA: MEAN SCORES
Fall 2007
![Page 31: Building brick by brick: A pragmatic approach to measuring impact](https://reader035.vdocument.in/reader035/viewer/2022062510/54b5337b4a7959f8228b4592/html5/thumbnails/31.jpg)
QUESTION: BETTER AT CITATION?DATA: PERCENT STUDENTS MEETING EXPECTATION
Fall 2007 Fall 2008
Note: First year students in 2007 vs different group of first year students in 2008
![Page 32: Building brick by brick: A pragmatic approach to measuring impact](https://reader035.vdocument.in/reader035/viewer/2022062510/54b5337b4a7959f8228b4592/html5/thumbnails/32.jpg)
QUESTION: BETTER AT CITATION?DATA: TRACKING 2007 COHORT
0102030405060708090
100
Met First YearMet Second Year
![Page 33: Building brick by brick: A pragmatic approach to measuring impact](https://reader035.vdocument.in/reader035/viewer/2022062510/54b5337b4a7959f8228b4592/html5/thumbnails/33.jpg)
QUESTION: ARE THEY THE SAME STUDENTS?DATA: INDIVIDUAL STUDENT SCORE CHANGES
5.6 Citation0
5
10
15
20
25
30
35
40
45
Both YrsImprovedNeither YrRe-gressed
![Page 34: Building brick by brick: A pragmatic approach to measuring impact](https://reader035.vdocument.in/reader035/viewer/2022062510/54b5337b4a7959f8228b4592/html5/thumbnails/34.jpg)
QUESTION: ARE THEY THE SAME STUDENTS?DATA: INDIVIDUAL STUDENT SCORE CHANGES
5.3 Use In-formation
0
5
10
15
20
25
30
35
40
45
Both YrsImprovedNeither YrRe-gressed
5.6 Citation0
5
10
15
20
25
30
35
40
45
Both YrsImprovedNeither YrRe-gressed
![Page 35: Building brick by brick: A pragmatic approach to measuring impact](https://reader035.vdocument.in/reader035/viewer/2022062510/54b5337b4a7959f8228b4592/html5/thumbnails/35.jpg)
NEXT STEPS
Our next question: Which students are we reaching?
New data needs Identify sub-
populations (gender, area of study)
Demographic information
Play with data display
http://www.flickr.com/photos/8829172@N02/3993827024/
![Page 36: Building brick by brick: A pragmatic approach to measuring impact](https://reader035.vdocument.in/reader035/viewer/2022062510/54b5337b4a7959f8228b4592/html5/thumbnails/36.jpg)
TOLERATE UNCERTAINTY
![Page 37: Building brick by brick: A pragmatic approach to measuring impact](https://reader035.vdocument.in/reader035/viewer/2022062510/54b5337b4a7959f8228b4592/html5/thumbnails/37.jpg)
HOW TO CULTIVATE TOLERATING UNCERTAINTY?
Create a culture in which it is OK for things to not work. Trial and error Pilot projects Model the behaviour
http://www.flickr.com/photos/slaff/1523973250/
![Page 38: Building brick by brick: A pragmatic approach to measuring impact](https://reader035.vdocument.in/reader035/viewer/2022062510/54b5337b4a7959f8228b4592/html5/thumbnails/38.jpg)
HOW TO CULTIVATE TOLERATING UNCERTAINTY?
Create a culture in which it is OK for things to not work. Trial and error Pilot projects Model the behaviour
Remember: “Fail often to succeed sooner”
http://www.flickr.com/photos/slaff/1523973250/
![Page 39: Building brick by brick: A pragmatic approach to measuring impact](https://reader035.vdocument.in/reader035/viewer/2022062510/54b5337b4a7959f8228b4592/html5/thumbnails/39.jpg)
KEEP YOUR EYE ON THE PRIZE
http://www.flickr.com/photos/91021420@N00/68366862/
![Page 40: Building brick by brick: A pragmatic approach to measuring impact](https://reader035.vdocument.in/reader035/viewer/2022062510/54b5337b4a7959f8228b4592/html5/thumbnails/40.jpg)
EMPHASIZE THE FORMATIVE NATURE OF PROJECT— “FORMATIVE ASSESSMENT”
http://www.flickr.com/photos/mike-bensalem/3438900700/
![Page 41: Building brick by brick: A pragmatic approach to measuring impact](https://reader035.vdocument.in/reader035/viewer/2022062510/54b5337b4a7959f8228b4592/html5/thumbnails/41.jpg)
SCALLOPING: UPS & DOWNS
http://www.flickr.com/photos/slimjim/2675461650/
![Page 42: Building brick by brick: A pragmatic approach to measuring impact](https://reader035.vdocument.in/reader035/viewer/2022062510/54b5337b4a7959f8228b4592/html5/thumbnails/42.jpg)
TOLERATING UNCERTAINTY
Uncertainty is key
Inquiry-based learning
Information Literacy (i.e. Kuhlthau)
Assessment
http://www.flickr.com/photos/sara/277695784/
![Page 43: Building brick by brick: A pragmatic approach to measuring impact](https://reader035.vdocument.in/reader035/viewer/2022062510/54b5337b4a7959f8228b4592/html5/thumbnails/43.jpg)
TOLERATING UNCERTAINTY
Uncertainty is key
Inquiry-based learning
Information Literacy (i.e. Kuhlthau)
Assessment
Our process mirrors our methodology http://www.flickr.com/photos/sara/277695784/
![Page 44: Building brick by brick: A pragmatic approach to measuring impact](https://reader035.vdocument.in/reader035/viewer/2022062510/54b5337b4a7959f8228b4592/html5/thumbnails/44.jpg)
QUESTIONS?
Feel free to contact us:
Sarah Cohen: [email protected]
Janet Cottrell: [email protected]
Michelle Miller: [email protected]