aia web conference on culturally competent practices in program evaluation considering culture in...
TRANSCRIPT
AIA Web Conference on Culturally Competent Practices in Program
Evaluation
Considering Culture in Evaluation and Applied Research
Rodney K. HopsonDuquesne University
2
Acknowledgements Collaborative and funded work from lectures, co-
presentations, research grants, published papers, commissioned evaluation work:
National Science Foundation California Endowment American Evaluation Association/Center for Disease
Control Relevance of Culture in Evaluation Institute Heinz Endowments Lincoln U, Manchester Metropolitan U, Claremont Graduate
U Development of pipeline programs to encourage
additional practice AEA Graduate Diversity Internship:
http://www.eval.org/gedip.htm RWJF Evaluation Fellowship: http://www.rwjf-
evaluationfellows.org/
Learning objectives: Recognize relevance and value-addedness
of culture in evaluation theory and practice Identify strategies for assisting evaluators
and agencies in becoming culturally responsive Encourage the development of tips and tools
for working with agencies who work with traditionally underserved and/or minoritized groups
4
Warm up/Introductions Who are you? What do you do? What disciplines, connections,
experiences do you bring into evaluation? How are your disciplines,…important in the
way you think about evaluation? How do you currently think about the role
of culture in evaluation and applied research? How does your background help you think
about the role of culture in evaluation and applied research?
5
Reflections of an ethnoevaluator/culturally responsive evaluator/sociolinguistevaluator
Curry School bridge at UVa between education & arts/sciences
Linking (trans)disciplines: (educational) evaluation, sociolinguistics, anthropology (Hopson, 2000)
Exploring connections between ethnography and evaluation (Hopson, 2005)
Beyond “technocist imperatives of most program evaluations”
Extending Fetterman’s pioneering work and Stakes’ seminal notions on countenance of educational evaluation
Challenging and resisting mainstream thinking on ways of thinking about cultural diversity and difference (Hopson, 2009)
Legitimizing knowledge of communities of color or traditionally underserved communities
Building spaces of hope, praxis and social action for same communities
New answers to increasing complex questions @ culture
7
Warm up/Introductions/Review
Who are you? What do you do? What disciplines, connections, experiences
do you bring into evaluation? How do they help you think about evaluation? What previous backgrounds or experiences do
you bring that might assist you in thinking culture in evaluation?
How do you currently think about the role of culture in your practice? How do you intend to use this discussion of
culture in evaluation to influence your practice?
8
Culturethe way of life of a group of people, the complex of shared concepts and patterns of learned behavior that are handed down from one generation to the next through the means of language and imitation.
(Barnouw, 1985)
the ever-changing values, traditions, social and political relationships, and worldview created, shared and transformed by a group of people bound together by a combination of factors that include a common history, geographic location, language, social class, and religion…
(Nieto, 1999)
9
Cultural Contexts, Characteristics, Locations, and Perspectives Demographic, sociopolitical, contextual
dimensions Characteristics as dynamic, multifaceted,
learned, created, influenced From personal to global position Majority vs. minority More than about race but inclusive of
other identifications(SenGupta, Hopson & Thompson-Robinson, 2004;
Nieto, 1999)
Emerging approach/model used to guide evaluation
System and culmination of evaluation strategies
Theoretical and political positioned (as are all evaluations) Demographic, sociopolitical, and contextual
dimensions, locations, perspectives, and characteristics of culture matter
Privileging lived experiences, esp. communities and populations of color
Avoiding the phenomenon of “evaluating down”
Culturally Responsive Evaluation
(Hopson, 2009)
Theoretical and Practical Intersection of CRE: Advocacy, Race, Power
Decolonizing/ indigenous positions, epistemologies, and
frameworks
Critical theories and epistemologies of race
Social agenda and advocacy theories,
models and approaches in evaluation
12
Cultural CompetenceA set of academic and interpersonal skills that allow individuals to increase their understanding and appreciation of cultural differences and similarities within, among, and between groups. This requires a willingness and ability to draw on community-based values, traditions, and customs, and to work with knowledgeable persons of and from the community in developing focused interventions, communications and other supports.
(Orlandi, 1992)
Talking as culturally responsive
14
Scenario: Dialogue for Diversity and Social Change (DDSC)first impressions
What elements of culture, at what levels, seem salient to this scenario at first glance? What elements of culture are you assuming will not be as salient, based upon your initial impressions?
What is your own cultural position in relation to these cultural elements?How do we think about the relevance of culture in all stages of evaluation for those in public health, education, and other helping professions?
15
Culturally Responsive Evaluation Framework
Step 1: Prepare for the evaluation. Step 2: Engage stakeholders. Step 3: Identify the purpose of the evaluation. Step 4: Frame the right questions. Step 5: Design the evaluation. Step 6: Select and adapt instrumentation. Step 7: Collect the data. Step 8: Analyze the data. Step 9: Disseminate and use the results.
(Frierson, Hood & Hughes, 2002)
Cultural ResponsiveEvaluation
7Collect the data
8 Analyze the data
3 Identify purpose of the evaluation
4 Frame the
right questions
1 Prepare for
the evaluation
5 Design the evaluation
6 Select and adapt instrumentation
2 Engage
stakeholders
9 Disseminate and
use the results
16
17
1 Prepare for the Evaluation Examine the sociocultural context of
the evaluand, including History Community Intersecting cultural identifications
Assemble an evaluation team whose collective lived experience is appropriate to the context of the evaluand.
18
Engage Stakeholders
Develop a stakeholder group representative of the population served by program.
Seek to include direct and indirect consumers.
Pay attention to distributions of power.
Include multiple voices.
19
Identify Purpose of Evaluation
Alignment of purpose with its intended consumers
Program operates in ways that are respectful of cultural context
Program resources are equitably distributed
Benefits of program evaluation are equitable
Applying Stages 1-3 (LaFrance, 2004)
Reflections on embedding culturally competent evaluation in Indian Country Build understanding of values that underlie
programs and projects and create value-added evaluative contribution
Engage stakeholders in participatory manner Build ethic of participation and capacity building that
values community, relationships, respect Frame purpose by building conceptual
picture/model Careful of “too sequential and narrative driven”
logic model 20
21
Scenario: Stages 1-3 What elements of background and
context are important here? What more would you want to know?
Who was included on the evaluation team and what presumed skills, traits do they bring to the evaluation process?
Based on the stated purpose of this evaluation, who do you understand to be the major stakeholders?
22
Frame the Right Questions Include questions of relevance to
significant stakeholders. Determine what will be accepted as
evidence in seeking answers to the questions.
Examine whose voices are heard in the choice of questions and evidence.
Is the lived experience of stakeholders reflected in these choices?
23
Design the Evaluation Build design appropriate to both evaluation
questions and cultural context. Seek culturally appropriate methods that
combine qualitative and quantitative approaches.
Try to collect data at multiple points in time, extending the time frame of the evaluation as needed.
Construct control or comparison groups in ways that respect cultural context and values.
24
Select & Adapt Instrumentation
Establish reliability and validity of instruments for the local population.
Norms must be appropriate to the group(s) involved in the program.
Language and content of instruments should be culturally sensitive.
Adapt instruments as needed and conduct additional validation studies.
Applying Stages 4-6 (Jay, Eatmon, & Frierson, 2005)
Evaluation of undergraduate STEM research program designed for students of color Deliberate design of evaluation team intimately
connected with program of study and background of program, including similar lived experiences of participants
Questions were sensitive to lived experiences of participants and focused on substance of participant experiences
Beyond attention to traditional issues of success but exploring issues of persistence as students of color
Attempt to address nuances and subtleties relative to experiences and impact of program
25
26
Scenario: Stages 4-6 What/whose perspectives are
represented in the evaluation questions, and what other questions might have been posed?
Whose perspectives are accepted as credible evidence? Credible to whom?
How well does the time frame in this study match the needs and rhythms of this context?
27
Collect the Data Procedures used to collect both
qualitative and quantitative data must be responsive to cultural context.
Nonverbal as well as verbal communications provide qualitative data.
Careful training of data collectors in both technical procedures and culture is key.
Shared lived experience provides optimal grounding for culturally-responsive data collection.
28
Analyze the Data Cultural context is a necessary
component of accurate interpretation. Disaggregate data to examine diversity
within groups. Examine outliers, especially successful
ones. A cultural interpreter may be needed to
capture nuances of meaning. Stakeholder review panels can assist in
accurate interpretation.
29
Disseminate & Use the Results
Cultural responsiveness increases both the truthfulness and utility of the results.
Communication mechanisms must be culturally responsive.
Inform a wide range of stakeholders. Make use consistent with the purpose of
the evaluation. Consider community benefit
Applying Stages 7-9 (Manswell Butty, Reid, & LaPoint, 2004)
Discussion of urban school-to-career intervention program using culturally responsive evaluation approach Input derived from school stakeholders on how
best to analyze and interpret data in ways that provided meaning in particular contexts
Findings disaggregated by gender and age to get breakdown of career attitudes and beliefs for participants
Findings provided to numerous stakeholders in particular ways (e.g. student findings presented in student-friendly manner)
30
31
Scenario: Stages 7-9 What additional data collection
procedures might have been useful to consider in designing a culturally responsive evaluation?
Given the findings briefly summarized, what aspects of cultural context might add meaning to guide recommendations?
Were results shared in a culturally congruent way?
32
Take Home/Next Steps/Questions
Culture is relevant, if not central, to all aspects of the evaluation process. Thinking through project plans Incorporating CRE approaches and thinking
in project planning and evaluation steps What key ideas do you take away from
CRE? How can you inform your own evaluation project with cultural lenses?
What questions do you continue to have about CRE?
33
References Barnouw, V. (1985). Culture and personality (4th
ed.). Homewood, IL: The Dorsey Press.
Frierson, H. T., Hood, S., & Hughes, G. B. (2002). Strategies that address culturally-responsive evaluation. In Frechtling, J., The 2002 user-friendly handbook for project evaluation. National Science Foundation.
Hopson, R. (2009, forthcoming). Reclaiming knowledge at the margins: Culturally responsive evaluation in the current evaluation moment. In Ryan, K. & Cousins, B. (Eds.). International Handbook on Evaluation. Thousand Oaks, CA: Sage.
Hopson, R.K. (2005). Commentary: Reinventing Evaluation. Anthropology and Education Quarterly, 36(3).
References Hopson, R. K. (2003). Overview of multicultural and
culturally competent program evaluation: Issues, challenges and opportunities. Woodland Hills, CA: The California Endowment.Hopson, R.K., Ed. (2000). How and Why Language Matters in Evaluation. New Directions for Evaluation, No. 86. San Francisco: Jossey-Bass.
Jay, M., Eatmon, D., & Frierson, H. (2005). Cultural reflections stemming from the evaluation of an undergraduate research program. In S. Hood, R. K. Hopson, & H. T. Frierson (Eds.) The role of culture and cultural context: A mandate for inclusion, the discovery of truth, and understanding in evaluative theory and practice (pp. 201-216). Greenwich, CT: Information Age Publishing, Inc.
34
References King, J. A., Nielsen, J. E., & Colby, J. (2004). Lessons
for culturally competent evaluation from the study of a multicultural initiative. In M. Thompson-Robinson, R. Hopson & S. SenGupta (Eds.), In search of cultural competence in evaluation: Toward Principles and Practices, New Directions for Evaluation, Number 102 (pp.5-19). San Francisco: Jossey-Bass.
La France, J. (2004). Culturally competent evaluation in Indian Country. In M. Thompson-Robinson, R. Hopson, & S. SenGupta (Eds.), (pp. 39-50). San Francisco: Jossey-Bass.
35
36
References Nieto, S. (1999). Affirming diversity: The sociopolitical
context of multicultural education (3rd ed.). Boston: Allyn & Bacon.
Orlandi, M. A. (Ed.) (1992). Cultural competence for evaluators: A guide for alcohol and other drug abuse prevention practitioners working with ethnic/racial communities. U.S. Department of Health and Human Services, Office for Substance Abuse Prevention. DHHS Publication No. (ADM)92-1884.
SenGupta, S., Hopson, R., & Thompson-Robinson, M. (2004). Cultural competence in evaluation: An overview. In M. Thompson-Robinson, R. Hopson & S. SenGupta (Eds.), (pp.5-19). San Francisco: Jossey-Bass.