assessment of knowledge and skills in nursing and allied ... · (n = 604 ) full-text articles...

Assessment of knowledge and skills in nursing and allied health student information literacy instruction: results from a scoping review

Jill Boruff, MLIS McGill University Library Pamela Harrison, MLIS Alberta Health Services

MLA/CHLA/ICLC, May 2016

•  No conflicts of interest

Disclosure Statement

Boruff and Harrison, 2016

•  To determine how knowledge and skills are being assessed in nursing and allied health student information literacy instruction.

•  To determine whether these assessment methods have been tested for reliability and validity.

•  To provide librarians with guidance on assessment methods that could be used in their own instruction.

Study objectives


•  Teaching course-integrated information literacy in School of Physical and Occupational Therapy since 2008.

•  Wanted to improve assessment of learning, but did not know the best methods for finding whether instruction was effective.

•  Desire to learn about assessment in an allied health context.

Background: Our context


•  Shaneyfelt (2006)1 conducted a systematic review on validated instruments for evaluating evidence based practice education.

•  Brettle (2007)2 conducted a systematic review on measures used to evaluate information skills training in health libraries.

•  Schilling and Applegate (2012)3 reviewed the biomedical, arts, humanities and social sciences literature to compile an overview of the assessment methods being used by librarians.

•  Just (2012)4 updates the review done by Brettle (2007)2 and takes it a step further by conducting a literature review on the methods of instruction and assessment of that instruction with medical students and residents.

•  Maggio & Kung (2014)5 did a systematic review of information retrieval instruction in undergraduate medical education in 2007-2012, but do not collect data on assessment methods.

•  None of the above results included physical and occupational therapy or communication sciences and disorders.

Background: Previous reviews


Measures of attitude: “based on learners’ perceptions of their skills and learning, including what people think, feel, or believe about the training experience or about their skills.”3

Background: Attitude versus performance

I am glad we learned about Medical Subject Headings.

I feel confident in using Medical Subject Headings


Measures of performance: “based on actual student work”; direct measurement of their knowledge and/or skills.3

Background: Attitude versus performance

I know how to define Medical Subject Headings

I can demonstrate the use of Medical Subject Headings in PubMed.

Knowledge Skills


•  A scoping review of the literature was conducted, following the methods outlined by Arksey and O’Malley.6

•  A systematic search strategy was constructed by one author and reviewed by the second author.

•  This strategy was then run in Ovid Medline, and adapted for CINAHL, EMBASE, ERIC, LISA, LISTA, and ProQuest Theses and Dissertations from 1990 to January 16, 2015.

•  Articles included if information literacy skills were being taught and assessed with OR without a librarian involved.

Study Methods


Results: PRISMA Diagram

Records identified through database searching

(n = 4366)

Records after duplicates removed (n = 2965)

Records screened (title/abstract)

(n = 2747)

Records excluded (n = 2143)

Records after pre-1990 removed (n = 2747)


Results: PRISMA Diagram (cont’d)

Full-text articles assessed for eligibility (n = 604 )

Full-text articles excluded (n = 468): •  Librarian but no assessment described (n = 67) •  No assessment described (n = 58) •  No instruction program described (n = 83) •  No librarian and no IL skills (n = 166) •  Reasons from abstract screening (n = 40) •  Review articles (n = 4) •  Article not available (n = 11) •  Language (n = 33) •  Duplicates (n = 6)

Studies included in qualitative synthesis

(n = 136)

Nursing and midwifery (n = 112)

Other allied health

(n = 4)

Physical and Occupational Therapy;

Communication Sciences and Disorders

(n = 20)


Results: Country of study

Canada N = 3

United States N = 12

Australia N = 1

Republic of Ireland N = 1

Norway N = 1

Italy N = 1

The Netherlands N = 1

N = 20


Results: Subject area and level of learner

2 2 1 1

3

6

1 1

1

2

Physical Therapy Occupational Therapy

Physical & Occupational

Therapy combined

Communication Sciences and

Disorders

Graduate

Undergraduate

Mixed

N = 20 studies


Results: Measures of attitude

11

2 2

Self-report surveys

Focus groups Informal feedback

Published self-report surveys used: •  Knowledge, Attitude, and

Behavior Questionnaire (Johnston et al.7) N=1

•  EBP self-efficacy and task value (Spek et al., 20138) N=1

•  Research Self-Efficacy Scale (Bieschke et al.9) N=1

•  Evidence-Based Competency Scale (Sabus10) N=1

N = 14 studies


Results: Measures of performance

14

4

1 1

Course products Non-Info Lit standardized test

Info Lit standardized test

Locally developed test

N= 18 studies


Results: Measures of performance Study included in scoping review

Name of measure used by research study (Authors of study that created the measure)

Tested for Validity Tested for Reliability

Justiss and Swinehart11

Adapted Fresno Test of Competence in EBP (McCluskey and Lovarini12)

Original test developed by Ramos et al.13, yes Adaptation, no

Yes

Simoni et al.14

Italian Adapted Fresno Test (Translated into Italian measures tested by McCluskey and Bishop15; Tilson16)

Originals, yes Translation, no

Originals, yes Translation, no

Spek et al., 201317

Dutch Modified Fresno (Spek, et al, 201218)

Yes Yes

Ratcliff et al.19

Research Readiness Self-Assessment (Ivanitskaya, et al20)

Original, yes Adaptation, no

Yes

Thomas et al.21

Marking Rubric for Test Vignette21 No Yes


Theme 1: Role of assignments

•  14 studies had assignments that could be used for assessment of information literacy knowledge and skills.

•  3 of these 14 studies had assignments that were not considered assessment measures.

Critically appraised topics

Searching assignments

Research projects


Theme 2: Librarian involvement

Many opportunities for librarians to get involved in course integrated instruction.

6 of the 20 studies identified had librarians involved

2 involved in instruction and assessment

4 involved in instruction only

3 had librarians as authors


Theme 2: Librarian involvement

Examples of instruction of information literacy skills happening without the (explicit) involvement of a librarian:

The major objective of this class is to provide students opportunities to research current interventions and to develop research proposals based upon their findings.18

Students learn how to frame appropriate clinical questions…then to search for and appraise descriptive and assessment studies to obtain the best scientific evidence to inform their clinical decisions.19


Conclusions

•  Few validated tools are being used in PT, OT, CSD information literacy instruction.

•  The variety of assignments being used as assessments provide a rich resource for ideas.

•  Librarians have many opportunities to contribute to assessment of learning.


Future directions

•  Investigate the development of validated rubrics for marking assignments.

•  Investigate the use of the Adapted Fresno for information literacy assessment in EBP contexts.


Included studies: 1-10

Study Location of study

Degree level of learners

Health profession of learners

Librarian involved Measures of attitude

Assessment validated Measures of performance

Assessment validated

Assessment tested for reliablity

Boruff and Thomas, 201124 Canada Undergrad PT & OT

Yes, as author, instructor, assessor None N/A

Course products: Searching assignment No No

Cahill, 201225 Rep. of Ireland

Undergrad & Grad OT No

Self-reported survey: KAB questionnaire Yes None N/A Yes

Cohn et al., 201423 US Grad OT No

Self-reported survey: Locally developed (2 different measures) No

Course products: searching assignment No No

Crist, 201026 US Grad OT No

Self-reported survey: Research Self-efficacy scale; and locally developed measure Unclear Course products No Unclear

Griffin and Schumm, 199227 US Grad OT

Yes, as author, instructor

Self-reported survey: Locally developed No

Locally developed tests; Course products: searching strategy No No

Justiss and Swinehart, 201211 US Grad OT No None N/A

Non-IL questionnaire or test: Adapted Fresno Test Yes Yes

Lederer, 200422 US Grad OT No Informal feedback No Course products: searching assignment No No

Maritz et a.l, 201128 US Grad PT No


Course products: capstone research project No No

Olsen et al., 201429 Europe Undergrad PT No

Self-reported survey: Locally developed Face validity None No No

Portney, 200430 US Grad PT No None N/A

Course products: development of Critically Appraised Topic No No


Included studies: 11-20

Study Location of study

Degree level of learners

Health profession of learners

Librarian involved

Measures of attitude (indirect)


Measures of performance (direct)


Assessment tested for reliablity

Raghavendra, 200931 Australia Grad CSD

Yes, as instructor Informal feedback No

Course products: development of Critically Appraised Topic No No

Ratcliff et al., 201319 US

Undergrad & Grad CSD

No; one author specialist in IL None N/A

Standardized IL test: Research Readiness Self-Assessment Yes Yes

Sabus, 200810 US Grad PT No Self-reported survey: EBP Competency Scale Face validity

Course products: in-service presentation (few details) No Yes

Scott et al., 201132 US Grad PT & OT No

Self-reported survey: Course evaluation No

Course products: Poster presentation (not used as assessment) No No

Simoni et al., 201414 Europe Undergrad PT No Focus groups No

Non-IL questionnaire or test: Italian A-Fresno; Course products: Audit form (behaviours) Fresno, yes

Fresno: Yes; Audit: not tested, but noted

Spek, et al., 201317 Europe Undergrad CSD No

Self-reported survey: EBP self-efficacy and task value Yes

Non-IL questionnaire or test: Dutch Modified Fresno Yes Yes

Thomas et al., 201221 Canada Undergrad OT

Yes, as instructor None N/A

Non-IL questionnaire or test: Vignette written for study with grading rubric No Yes

Villeneuve and Maranda, 200533 Canada Undergrad OT

Yes, as author, instructor Focus group No

Course products: case based inquiry No No

Vogel, 201234 US Grad OT

Yes, as author, instructor, assessor


Course products: searching assignment No No

Wolter et al., 201135 US

Undergrad & Grad CSD Unclear None N/A

Course products: Research article critique No No


References


1. Shaneyfelt T, Baum KD, Bell D, Feldstein D, Houston TK, Kaatz S, et al. Instruments for evaluating education in evidence-based practice - A systematic review. JAMA. 2006;296(9):1116-27. 2. Brettle A. Evaluating information skills training in health libraries: a systematic review. Health Info Libr J. 2007;24(1):18-37. 3. Schilling K, Applegate R. Best methods for evaluating educational impact: a comparison of the efficacy of commonly used measures of library instruction. J Med Libr Assoc. 2012;100(4):258-69. 4. Just ML. Is literature search training for medical students and residents effective? a literature review. J Med Libr Assoc. 2012;100(4):270-6. 5. Maggio LA, Kung JY. How are medical students trained to locate biomedical information to practice evidence-based medicine? A review of the 2007-2012 literature. J Med Libr Assoc. 2014;102(3):184-91. 6. Arksey H, O'Malley L. Scoping Studies: Towards a Methodological Framework. Int J Soc Res Methodol. 2005;8(1):19-32. 7. Johnston JM, Leung GM, Fielding R, Tin KYK, Ho L-M. The development and validation of a knowledge, attitude and behaviour questionnaire to assess undergraduate evidence-based practice teaching and learning. Med Educ. 2003;37(11):992-1000. 8. Spek B, Wieringa-de Waard M, Lucas C, Dijk N. Competent in evidence-based practice (EBP): validation of a measurement tool that measures EBP self-efficacy and task value in speech-language therapy students. Int J Lang Commun Disord. 2013;48(4):453-7. 9. Bieschke KJ, Bishop RM, Garcia VL. The Utility of the Research Self-Efficacy Scale. J Career Assess. 1996;4(1):59-75.

References


10. Sabus C. The effects of modeling evidence-based practice during the clinical internship. J Phys Ther Educ. 2008;22(3):74-84. 11. Justiss M, Swinehart S. Occupational therapy master-level students' evidence-based practice knowledge and skills before and after fieldwork. Occup Ther Health Care. 2012;26(2-3):138-49. 12. McCluskey A, Lovarini M. Providing education on evidence-based practice improved knowledge but did not change behaviour: a before and after study. BMC Med Educ. 2005;5:40-. 13. Ramos KD, Schafer S, Tracz SM. Validation of the Fresno test of competence in evidence based medicine. BMJ. 2003;326(7384):319-21. 14. Simoni G, Balboni M, Fiorini F, Bombardi S, Bertin N, Da Roit M. Undergraduate physiotherapy students' competencies, attitudes and perceptions after integrated educational pathways in evidence-based practice: a mixed methods study. Physiother. 2014;30(8):557-71. 15. McCluskey A, Bishop B. The Adapted Fresno Test of competence in evidence-based practice. J Contin Educ Health Prof. 2009;29(2):119-26. 16. Tilson JK. Validation of the modified Fresno Test: assessing physical therapists' evidence based practice knowledge and skills. BMC Med Educ. 2010;10:38-. 17. Spek B, Wieringa-de Waard M, Lucas C, Dijk N. Teaching evidence-based practice (EBP) to speech-language therapy students: are students competent and confident EBP users? Int J Lang Commun Disord. 2013;48(4):444-52. 18. Spek B, de Wolf GS, van Dijk N, Lucas C. Development and validation of an assessment instrument for teaching evidence-based practice to students in allied health care: the Dutch Modified Fresno. J Allied Health. 2012;41(2):77-82. 19. Ratcliff A, Swartz B, Ivanitskaya L. Information Literacy Skills in Speech- Language Pathology Students: Skill Differences Across Academic Levels. Contemp Issues Commun Sci Disord. 2013;40:31-9.

References


20. Ivanitskaya L, O’Boyle I, Casey AM. Health Information Literacy and Competencies of Information Age Students: Results From the Interactive Online Research Readiness Self-Assessment (RRSA). J Med Internet Res. 2006;8(2):e6. 21. Thomas A, Saroyan A, Snider LM. Evidence-based practice behaviours: a comparison amongst occupational therapy students and clinicians. Can J Occup Ther. 2012;79(2):96-107. 22. Lederer JM. Evidence for occupational therapy interventions: a student educational assignment. Occup Ther Health Care. 2004;18(4):29-40. 23. Cohn E, Coster W, Kramer J. Facilitated learning model to teach habits of evidence-based reasoning across an integrated master of science in occupational therapy curriculum. Am J Occup Ther. 2014;68 Suppl 2:S73-82. 24. Boruff JT, Thomas A. Integrating evidence-based practice and information literacy skills in teaching physical and occupational therapy students. Health Info Libr J. 2011;28(4):264-72. 25. Cahill M. Self-reported knowledge, attitudes and behaviour towards evidence-based practice of occupational therapy students in Ireland. Occup Ther Int. 2012;19(1):7-16. 26. Crist P. Adapting research instruction to support the scholarship of practice: practice-scholar partnerships. Occup Ther Health Care. 2010;24(1):39-55. 27. Griffin N, Schumm RW. Instructing occupational therapy students in information retrieval. Am J Occup Ther. 1992;46(2):158-61. 28. Maritz CA, Thielman G, Campolo M. Using a Capstone Project to Prepare Students to Become Evidence-Based Practitioners. J Fac Dev. 2011;25(2):12-7. 29. Olsen N, Lygren H, Espehaug B, Nortvedt MW, Bradley P, Bjordal JM. Evidence-based practice exposure and physiotherapy students' behaviour during clinical placements: a survey. Physiother Res Int. 2014;19(4):238-47.

References


30. Portney LG. Evidence-based practice and clinical decision making: it's not just the research course anymore. J Phys Ther Educ. 2004;18(3):46-51. 31. Raghavendra P. Teaching evidence-based practice in a problem-based learning course in speech-language pathology. Evid Based Commun Assess Interv. 2009;3(4):232-7. 32. Scott P, Altenburger PA, Kean J. A collaborative teaching strategy for enhancing learning of evidence-based clinical decision-making. J Allied Health. 2011;40(3):120-7. 33. Villeneuve M, Maranda S. Preparing entry-level practitioners for evidence-based practice. J Can Health Libr Assoc. 2005;26(1):13-21. 34. Vogel K. Librarians and occupational therapy faculty: a collaboration for teaching evidence-based practice. J Allied Health. 2012;41(1):e15-20. 35. Wolter JA, Corbin-Lewis K, Self T, Elsweiler A. An Evidence-Based Practice Model Across the Academic and Clinical Settings. Commun Disord Q. 2011;32(2):118-32.

Images from the Noun Project

Slide 11: America by Bohdan Burmich Canada by Ted Grajeda Australia by Ted Grajeda Norway by Ted Grajeda Italy by Ted Grajeda Netherlands by Ted Grajeda

Slide 16: Writing by Magicon

Slide 17 Two people by Alfredo Hernandez Three people by Alfredo Hernandez Four people by Alfredo Hernandez Six people by Alfredo Hernandez

Slide 18 Quotes by Consumer Financial Protection Bureau

Slides 6 and 7: Speech-bubble by Blake Thompson


Slides 3 and 19: Target by Creative Stall

Slide 20: Arrow by Austin Condiff

Acknowledgments

•  McGill University for the sabbatic leave •  Skye Barbic, Robin Featherstone, and Lorie Kloda for

their guidance in the design and methods of this study Contact information

Slides on website: https://www.mcgill.ca/library/librarians/jill-boruff Email: [email protected]


assessment of knowledge and skills in nursing and allied ... · (n = 604 ) full-text articles...

Documents