Collaborative Research Scoring Rubric for Projects with Indigenous Peoples

Research and Presentation By:Dr. Ronald L. Trosper withKatherine BrooksElizabeth EklundMatthew SchwoebelKestrel SmithJuanita Francis Lisa PalaciosPeter Dubois

American Indigenous Research Association Meeting

Pablo, Montana

October 11, 2014

Introduction: Why a rubric is useful

• A rubric is useful for assessing the amount of collaboration and joint benefit that occurs in research projects with Indigenous peoples or communities.

• The rubric has four general areas of criteria, and under each of the criterion, it has levels of achievement described in observable terms with assigned numerical values for achievement. The four areas are;

1) Community Centered Control-Joint Planning, 2) Joint Methodology Implementation, 3) Community Benefit-Outcomes, and 4) Researcher Benefit-Outcomes

• The contribution of the rubric is to provide a helpful worksheet for the design of research projects, and to provide a potential set of standards for ranking different projects that are seeking funding. The rubric initially developed as a class project for a course on collaborative research and learning with Indigenous communities.

• What has come to be known as collaborative research developed recently, between 1950 and 1970, after western scholars had been studying and working with Indigenous communities for more than a hundred years.

• Collaborative research is defined as a process in which researchers, program developers, and community members are networked to do research for joint problem solving and positive social change (Van Willigen 2002: 101).

• The goal of collaborative research is to blend theory into practice, heading away from conceptual frameworks only applicable within a university setting, moving toward the application of action within communities.

Background on Collaborative Research

Literature ReviewTo frame the rubric and its ideas, an analysis of theoretical publications and examples of other attempts to create models or guidelines for effective collaborative research is necessary. Key pieces include:

• Wulfhorst et al. (2008) “Core Criteria and Assessment of Participatory Research”

• Fortmann (2008) “Participatory Research in Conservation and Rural Livelihoods: Doing Science Together”

• Lambert (2013) “Spider Web Conceptual Model”

• Arnstein (1969) “A Ladder Of Citizen Participation”

Lambert’s Spider Web Conceptual Model (2013)

• The majority of past research regarding Indigenous peoples has been conducted on them and not for them.

• Lambert’s Spider Web Conceptual Model (2013) accounts for elements that are important to Indigenous peoples not only when conducting research, but central to their daily lives.

• Despite the early date of Arnstein’s 1969 article the concepts presented prove relevant and directly transferable to the discussion of research collaboration with Indigenous peoples.

• Arnstein’s Ladder was deemed essential for Community-Centered Control of research, as well as projects involving Joint Planning, Joint Methodology Implementation, and Joint Benefits-Outcomes.

• Arnstein’s concepts of the identification and organization of vital aspects of community participation provided key elements in the development of the collaborative research assessment rubric that the current study has created.

Image from Arnstein 1969

Arnstein’s Ladder

Description of the RubricThe four categories of the rubric include: 1) Community Centered Control-Joint Planning, 2) Joint Methodology Implementation, 3) Community Benefit-Outcomes, and 4) Researcher Benefit-Outcomes

• Presented under these four larger categories are a series of more specific criteria, with each item assisting in the assessment of the design, implementation, final product, and dissemination of collaborative research projects.

• Each set of criterion is assigned a point value, on a scale of zero to five, with five representing the highest score based on the fulfillment of the criteria at an above average level, and zero serving as the lowest possible score, indicative of the absence of collaboration within a given criterion.

• The rubric provides a five to zero point scale for each set of criteria within each category. Twenty-one total criteria are present within the rubric, allowing for a maximum score of 105 or lower.

Category Criteria Scoring: Evidence that the criteria have been met

 

1) Community-Centered control –Joint Plannin

g

1.A) Community working with or conducting its own research. 1.B) Community plays prominent role in determining outcomes and goals of research. 1.C) Community has provided free, prior and informed consent for the project

__ (5) Community plays an active role in the study by working with their own community and conducting research.__ (3) Community plays a role in the study beyond research participant (serving as a researcher; interviewing, consulting, etc.)__ (0-1) Community plays no role beyond research/ study participant. __ (5) Community members are able to decide or change the study outcome, goals, and priorities before the researcher visits.__ (3) Community members know about and agree with researchers’ pre-determined study goals, outcomes, and priorities.__ (0-1) Community members find researchers’ study goals, outcomes, and priorities to be of minimal interest/ not important/ or not necessary. __ (5) A formal agreement with informed consent exists between the community and researcher__ (3) Some type of agreement exists between the community and researcher (IRB; Tribal Permission; Interview Consent, etc.)__ (0-1) No agreement exists between the community and researcher

Category Criteria Scoring: Evidence that the criteria have been met 1) Community-Centered control –Joint Planning

1.D) Political and Legal context understood.

1.E) Breadth of Community Participation (Participation of marginalized people i.e., gender, age, income, class, etc.)

__ (5) Outside researchers are aware of and discuss the political and legal context with the community’s representatives, and changes plans as required.__ (3) Outside researchers demonstrate awareness of the political and legal context. __ (0-1) The political and legal context is not discussed.

__ (5) Community participation is broad and inclusive of all members. Dialogue with all parties continues throughout the research project__ (3) Community participation is diverse in some aspects (gender, age range, income level, etc.)__ (0-1) Community participation is limited to one category (gender only; age only; etc.) Total Points: _________

Category Criteria Scoring: Evidence that the criteria have been met 2) Joint methodology implementation

2.A) Shared Control of Information or data gathered

2.B) Community members as researchers: Data gathering

___(5) Control over the data/information is balanced. The researcher and community equally manage data and develop protocol for use, storage, processing, disposal, etc.___(3) Community has some control over study related data/ information. Researcher follows community’s standard research protocol, but community input is not unique to the study.___(0-1) No shared control over data/ information or protocol for management.

___(5) Community members help design data gathering templates, questionnaires, and other tools. Community members also participate in data acquisition.___(3)Community members gather data following procedures that are determined by the external research team.___(0-1) No community members are hired to gather data.

Category Criteria Scoring: Evidence that the criteria have been met

2) Joint methodology implementation

2.C) Community members as researchers: analysis 2.D) Communication

2.E) Mutual trust between researcher and community

___(5) Community members participate in discovering the significant results of the research, including any findings that are surprises.___(3) Community members help interpret results. Researcher may compute statistical tests, but the results of the tests are discussed with community members in the process of determining what the data show.___(0-1) Researchers do all of the analysis. ___(5) Communication between researcher and community continues regularly throughout duration of project. (Quarterly reports, meetings, town hall, etc).___(3) Communication occurs only at major project milestones,: start, end, ___(0-1) ___(5) Researcher is confident in the abilities and knowledge of the community. Community is comfortable with the researcher/or institution, who respects the needs, views, and morals of the community. ___(3) Community and researcher/institution appear to have some degree of mutual trust (measurable through previous collaboration; community has extensive relationship w/ institution, i.e. University students work on projects every summer, etc.)___(0-1) Researcher is legally permitted to work with community, extent of mutual trust is unclear. Total _____

Category Criteria Scoring: Evidence that the criteria have been met Community Benefit

3) Outcomes-Community Benefit

3.A) Material Benefit

__(5) Food security enhanced; land tenure increased; new economic opportunities developed__(3)Positive material benefit, but small scale (Grants, Return of cultural materials, etc.) __(0-1) No material benefit was attained by the community

3.B) Control over publication

__(5) Full veto power; Co-authorship__(3) Complete review of drafts__(0-1) Informed of the publication; Received a copy after published

3.C) Empowerment

__(5) Membership on examination committee; changed procedures by government agencies which move the community to higher levels of Arnstein’s ladder__(3) Community experienced financial, educational, etc. empowerment__(0-1) Study did not empower community

3.D) Capacity Building

__(5) The community creates a research office or museum which stores all data collected and is capable of negotiating new projects with new researchers.__(3) Capacity for community members to work/hold authoritative positions __(0-1) Community capacity building was not encouraged/ No capacity building took place

Category Criteria Scoring: Evidence that the criteria have been met Community Benefit

3) Outcomes-Community Benefit

3.E) Research questions answered

__(5) Were the questions that the community developed answered; and there is one or more surprising finding from the community’s viewpoint.__(3) Findings were something previously unknown or surprising.__(0-1) Findings were not new or surprising; Community questions were answered.

3.F) Sustainability of outcomes

__(5) Outcomes of the study demonstrate long term effectiveness through implementation of policies, programs, etc. __(3) Outcome offers assistance in attaining sustainability, but does not implement a solution__(0-1) Outcomes of the study are not sustainable

3.G) Control of Information and gathered data

__(5) Community houses recordings, images, records in their own facility; Full control in physicality and legal rights__(3) Legal rights to information shared during a project/ ability to manage access__(0-1) No control beyond signing a consent form Total: ______

Category Criteria Scoring:Evidence that the criteria have been met

Researcher Benefit

4)Outcomes- Researcher benefits

4.A) Publications __(5) Journal accessible to non-scholarly readers; is it widely distributed; or does the publication fall into gray literature such as technical reports, etc?__(3) Publication is scholarly and has academic audience primarily __(0-1) Non-publishable

4.B) Degree Attainment

__(5) MA, MSc, or Ph.D. awarded to the researcher if a student. __(3) BA awarded__(0-1) Degree sought but not awarded based on shortcomings of study.

4)Outcomes- Researcher benefits

Category Criteria Scoring:Evidence that the criteria have been met

Researcher Benefit

4)Outcomes- Researcher benefits

4.C) Long term relationships

__(5) Community and researcher successfully collaborate on study, move forward in working together again and again over an extended period of time. Relationship does not end upon finalization of current project.__(3) Community and researcher maintain contact after study, possibility of future work but no solidified project known.__(0-1)Community and researcher end relationship upon finishing current project.

4.D) Number of loops in the learning

__(5) Triple loop: fundamental, unrecognized assumptions are recognized, causing a revision of worldview.__(3) Double loop The structure of an existing model is changed; new equations used with the same set of variables.__(0-1) Single loop: parameters in an existing model are changed (improved) based on more data.

Total Score:

The Rubric Applied

The Collaborative Research Scoring Rubric was applied to three articles. We have scored these pieces as both text based articles, and then a score for the project as a whole was assessed.

• Lloyd KWSS-PSBLCB. 2012. Reframing development through collaboration: towards a relational ontology of connection in Bawaka, North East Arnhem Land. Third World Quarterly 33(6).

• Sherwood J, and Kendall S. 2013. Reframing spaces by building relationships: Community collaborative participatory action research with Aboriginal mothers in prison. Contemporary Nurse 46(1):83-94.

• Stoffle, Richard and Richard Arnold. 2003. Confronting the Angry Rock: American Indians’ Situated Risks from Radioactivity. Ethnos 68.2: 230-248.

Scoring Comparison

Community Centered

Control-Joint Planning (Max=25)

Joint Methodology Implementa-

tion (Max=30)

Community Benefit

Outcomes (Max=35)

Outcome Re-searcher Benefits

(Max=20)

0

5

10

15

20

25

30

Lloyd et al. 2012Sherwood & Kendall 2013Stoffle & Arnold (2003)

2220

23 24

17

25

11

14

20

16

21

16

Scoring Comparison – Articles vs. Projects

Lloy

d et

al.

Proj

ect

Lloy

d et

al.

Artic

le

Sher

woo

d &

Ken

dall

Proj

ect

Sher

woo

d& K

enda

ll A

rtic

le

Stoffl

e &

Arn

old

Proj

ect

Stoffl

e& A

rnol

d Art

icle

0

10

20

30

40

50

60

70

80

90

100 89

71

93

72

97

86

The Rubric Applied

ReferencesArnstein SR. 1969. A Ladder Of Citizen Participation. Journal of the American Institute of Planners 35(4):216-224.Barsh RL, and Trosper RL. 1975. Title I of the Indian Self-Determination and Education Assistance Act of 1975. American Indian Law Review 3(2):361-395.Canby WC. 2009. American Indian law in a nutshell, fifth edition. St. Paul, MN: Thomson/West.Fortmann L. 2008. Participatory research in conservation and rural livelihoods : Doing science together. Hoboken, NJ: Wiley-Blackwell.Kishchuk N, Torrie J, Horn-Miller K, Légaré-Dionne C, and Gagnon C. 2008. Stepping Into A Flowing River: Evaluation of the Social Sciences and Humanities Research Council's Aboriginal Research Pilot Program. Final Evaluation Report. Ottawa, Ontario: Social Sciences and Humanities Research Council of Canada.Lambert L. 2013. Spider Conceptual Framework.Lloyd KWSS-PSBLCB. 2012. Reframing development through collaboration: towards a relational ontology of connection in Bawaka, North East Arnhem Land. Third World Quarterly 33(6).Meeds L. 1976. The Indian Policy Review Commission. lawcontprob Law and Contemporary Problems 40(1):9-11.National Congress of American Indians. 2014. American Indian and Alaska Native Genetics Resource Center. Havasupai Tribe and the Lawsuit Settlement Aftermath.Schensul and LeCompte. 1999. The Ethnographer’s Toolkit. 7 volumes. Book (190 - 250 pages each): Introduction to Ethnographic Research Methods (1); Essential Ethnographic Methods (2); Enhanced Ethnographic Methods (3); Researching Social Networks, Spatial Data and Hidden Populations (4) ; Analysis and Interpretation of Ethnographic Data (Book 5); Researcher’s Role and Research Partnerships (Book 6); Disseminating Ethnographic Data (7). Seven Oaks Innovation. CA: Altamira Press,Schensul, J. 1982. Issues in Applied Anthropology Training Programs. Practicing Anthropology, 4.Sherwood J, and Kendall S. 2013. Reframing spaces by building relationships: Community collaborative participatory action research with Aboriginal mothers in prison. Contemporary Nurse 46(1):83-94.Shirk JL, Ballard HL, Wilderman CC, Phillips T, Wiggins A, Jordan R, McCallie E, Minarchek M, Lewenstein BV, Krasny ME et al. . 2012. Public Participation in Scientific Research: A Framework for Deliberate Design. Ecology and Society 17(2):207-227.Sobeck J, Elizabeth C, and Charles F. 2003. Conducting research with American Indians: A case study of motives, methods, and results. Journal of Ethnic and Cultural Diversity in Social Work 12(1):69-84.Stoffle, Richard and Richard Arnold. 2003. Confronting the Angry Rock: AmericanIndians’ Situated Risks from Radioactivity. Ethnos 68.2: 230-248.Van Willigen, John. 2002. Applied Anthropology: An Introduction, 3rd Edition. Westport, CT.: Bergin and Garvey.Wulfhorst J.D, with Eisenhauer, B.W, Gripne, S.L, and Ward, J.M. 2008. Core Criteria and Assessment of Participatory Research. In Wilmsen C, Elmendorf W, Fisher L, Ross J, Sarathy B, and Wells G, editors. Partnerships for empowerment : Participatory research for community-based natural resource management. London: Earthscan.