““Stuck Somewhere in the Stuck Somewhere in the Middle”Middle”

Anthropologists as Evaluators:Anthropologists as Evaluators:Mediating Between Community-Based Mediating Between Community-Based

Health Projects and the StateHealth Projects and the StatePanel: “Studying Up: Exploring the Exotic World of Panel: “Studying Up: Exploring the Exotic World of

Elites”Elites”2006 SfAA Annual Meeting2006 SfAA Annual Meeting

Beverly Hill & Sharon WatsonBeverly Hill & Sharon WatsonThe University of South FloridaThe University of South Florida

IntroductionIntroduction► In this presentation, we address Laura In this presentation, we address Laura

Nader’s concepts of “controlling processes” Nader’s concepts of “controlling processes” and “studying up” in order to explore:and “studying up” in order to explore:

Evaluation as a tool for empowermentEvaluation as a tool for empowerment Negotiating the expert-client relationshipNegotiating the expert-client relationship Adapting expertise to accommodate Adapting expertise to accommodate

democratic organizational practicesdemocratic organizational practices

Goal: To better inform our practice as applied Goal: To better inform our practice as applied anthropologists serving as evaluators.anthropologists serving as evaluators.

Description of ProjectDescription of Project► Two-year project, collaborative between a Two-year project, collaborative between a

private communications firm and a private communications firm and a hospital-based research centerhospital-based research center

► Funded by the state, our clientFunded by the state, our client► Evaluation teams represented in three Evaluation teams represented in three

regions, mostly university-basedregions, mostly university-based► Community based organizations with Community based organizations with

health promotion projects that served health promotion projects that served populations where ethnic and racial populations where ethnic and racial disparities existdisparities exist

The Big PictureThe Big Picture

General Population including minorityGroups suffering from health disparity

Legislation passed to reduce health Disparities in six target areas

Appropriate $5 million Department of HealthRecipient of $5 million

Office of Minority Health“Client”

AdvisoryCommittee

Evaluation Team“Experts”Directors

Field Visit Teams

“Grantees” OrganizationsImplementing Programs to address racial

And ethnic health disparities“the other client”

CBOs DH NGO Community Centers

Evaluation RoleEvaluation Role “ “Our roles are defined by our contract with Our roles are defined by our contract with

the DOH of the State of Florida to achieve the DOH of the State of Florida to achieve two overall lines of action: two overall lines of action: formative and formative and summative evaluationsummative evaluation. Briefly stated, the . Briefly stated, the formative evaluation is designed to help you formative evaluation is designed to help you (the projects) have success in your future (the projects) have success in your future efforts to improve your community’s health. efforts to improve your community’s health. Within the formative evaluation, we will Within the formative evaluation, we will assess your program’s strengths and assess your program’s strengths and challenges and offer suggestions for challenges and offer suggestions for improvement. improvement. Our efforts will focus on Our efforts will focus on helping you improve how you measure and helping you improve how you measure and demonstrate your performancedemonstrate your performance.”.”

Evaluation Role (cont’d)Evaluation Role (cont’d)“ “ On the other hand, the summative evaluation is what you On the other hand, the summative evaluation is what you

might consider a more traditional form of evaluation. Within might consider a more traditional form of evaluation. Within the summative evaluation, we will examine whether or not the summative evaluation, we will examine whether or not to to what extent your program is meeting it’s goals and what extent your program is meeting it’s goals and how much value it is providing to your communityhow much value it is providing to your community.”.”

“ “ In both of these lines of action, we will be fair and equitable. In both of these lines of action, we will be fair and equitable. Even though our assistance will be based on your challenges Even though our assistance will be based on your challenges and requests, most communications will be standardized and requests, most communications will be standardized through the use of protocol developed by our entire team. through the use of protocol developed by our entire team. Moreover, we will establish and use consistent standards of Moreover, we will establish and use consistent standards of evaluation across sites and we will report our findings to the evaluation across sites and we will report our findings to the State using the most objective and scientifically rigorous State using the most objective and scientifically rigorous methods that can be accomplished within the constraints we methods that can be accomplished within the constraints we all face. all face. However, we will also attempt to understand However, we will also attempt to understand your specific challenges well enough to permit your specific challenges well enough to permit sensitive and respectful communications to the Statesensitive and respectful communications to the State.”.”

Role of evaluatorsRole of evaluatorsYear oneYear one

► BoundariesBoundaries► AuthorityAuthority► RoleRole

Participatory, Participatory, objective researcher, objective researcher, advocacy, coachadvocacy, coach

► Task: to measure Task: to measure and evaluate and evaluate performanceperformance

Year twoYear two►On-line data system On-line data system

was emphasized as was emphasized as the primary tool for the primary tool for measurement and measurement and evaluation.evaluation.

► Technicians/Technicians/teachers of new teachers of new system, in addition system, in addition to the first year’s to the first year’s tasks.tasks.

Adapting expertise to Adapting expertise to accommodate democratic accommodate democratic organizational practicesorganizational practices

► Flow of power: Flow of power: It is acquired, used, maintained, and lostIt is acquired, used, maintained, and lost It is redistributed by professionsIt is redistributed by professions It involves embedded microprocesses, which It involves embedded microprocesses, which

challenge our ability to study upchallenge our ability to study up It can be analyzed through models of harmony via It can be analyzed through models of harmony via

mediation vs. the quest for justice (Nader 1997)mediation vs. the quest for justice (Nader 1997)

Q: Are we, as evaluators, complicit in the system’s Q: Are we, as evaluators, complicit in the system’s perpetuation of hegemony, rather than acting as perpetuation of hegemony, rather than acting as agents of change in it’s reversal? agents of change in it’s reversal?

Negotiating the expert-client Negotiating the expert-client relationshiprelationship

►Fundamental question: Is it possible to Fundamental question: Is it possible to restructure the largely undemocratic restructure the largely undemocratic expert-client relationship?expert-client relationship?

►Does autonomy have a place in the Does autonomy have a place in the evaluation process? evaluation process? Are the decisions made by those of the CBOs Are the decisions made by those of the CBOs

voluntary, or are they disguised as free will?voluntary, or are they disguised as free will? Example: MCHD project vs. LHANC projectExample: MCHD project vs. LHANC project

Evaluation as empowerment?Evaluation as empowerment?►Empowerment evaluation is Empowerment evaluation is

conceptualized by David Fetterman conceptualized by David Fetterman (1997) and described as, “the use of (1997) and described as, “the use of evaluation concepts, techniques, and evaluation concepts, techniques, and findings to foster improvement and self-findings to foster improvement and self-determination.”determination.”

►Harmony law model (“ethic of treatment” Harmony law model (“ethic of treatment” approach) vs. the explicit quest for approach) vs. the explicit quest for justice (“ethic of right and wrong”)justice (“ethic of right and wrong”)

Ethical ConsiderationsEthical Considerations► American Evaluation American Evaluation

Association guiding Association guiding principles:principles: Systematic inquirySystematic inquiry CompetenceCompetence Integrity/honestyIntegrity/honesty Respect for peopleRespect for people Responsibilities for Responsibilities for

general & public general & public welfarewelfare

► Constraints within Constraints within the boundaries of the boundaries of our authorization our authorization arose, limiting our arose, limiting our ability to serve as ability to serve as effective mediators.effective mediators.

Ethical Considerations Ethical Considerations (cont’d)(cont’d)

► Human agency (autonomy vs. conformity)Human agency (autonomy vs. conformity) Example: BP project, accepting funding but not Example: BP project, accepting funding but not

reporting.reporting.

► Lack of mechanism to ensure Lack of mechanism to ensure truthfulness/accuracy of data reportedtruthfulness/accuracy of data reported Example: SWP reporting numbers vs. health Example: SWP reporting numbers vs. health

outcomes.outcomes.

► Relationships between CBOs and contract Relationships between CBOs and contract managers—not equal across all projects.managers—not equal across all projects.

Revisit The Big PictureRevisit The Big Picture

General Population including minorityGroups suffering from health disparity

Legislation passed to reduce health Disparities in six target areas

Appropriate $5 million Department of HealthRecipient of $5 million

Office of Minority Health“Client”

AdvisoryCommittee

Evaluation Team“Experts”Directors

Field Visit Teams

“Grantees” OrganizationsImplementing Programs to address racial

And ethnic health disparities“the other client”

CBOs DH NGO Community Centers

Contract Managers

SummarySummary

►Why should we employ studying up to Why should we employ studying up to determine how individuals/groups are determine how individuals/groups are influenced and persuaded to influenced and persuaded to participate in their own domination or, participate in their own domination or, alternatively, to resist it?alternatively, to resist it?

Summary (cont’d)Summary (cont’d)► Mechanisms for ensuring that “studying up” will Mechanisms for ensuring that “studying up” will

result in more equitable outcomes:result in more equitable outcomes:

Create space for the reversal of top-down power Create space for the reversal of top-down power flows in order to maintain accountability by those flows in order to maintain accountability by those in positions of power, and to engage social actors in positions of power, and to engage social actors at every level of the evaluation process.at every level of the evaluation process.

Utilize our participation in evaluation roles to Utilize our participation in evaluation roles to address decision makers in their forum.address decision makers in their forum.

Move away from the “harmony law model,” which Move away from the “harmony law model,” which encapsulates coercive compromise and consensus encapsulates coercive compromise and consensus as a form of behavior modification, toward a as a form of behavior modification, toward a model of justice that is characterized by an ethic model of justice that is characterized by an ethic of right and wrong vs. an ethic of treatment. of right and wrong vs. an ethic of treatment.