The Art (and Science) of Defining a

Construct

Tamara ShoreDr. Robin Anderson

Center for Assessment and Research Studies

James Madison University

What is a Construct? An Attribute of people

Motivation Depression Sociocultural Competency

Qualitative Attributes or Structures People have it or they don’t (Amnesia)

Quantitative Attribute People have it to some degree (Anxiety)

Inferences are made about the attribute

(Cronbach & Meehl, 1955)

Validity

“The assessment instrument has content validity to the degree that it taps the targeted construct and facilitates valid...judgments.”

(Haynes, Richard, & Kubany, 1995)

Construct

Content

VALIDITY

Criterion

Validation of the Construct

Review existing research in literature Formulate a theory of the construct Position the construct with related

constructs (Nomological network) Generate hypotheses about construct

relationships Test, interpret and refine

(Cronbach & Meehl, 1955)

Three Aspects of Construct Validation

Substantive Stage: Definition of the Theoretical and Empirical Domains of the Construct

Structural Stage: Internal Relations among Observed Variables

External Stage: Relations Among Constructs (convergent & divergent)

(Benson, 1998)

Begin to Define the Construct (Substantive Stage)

TheoreticalCultural Intelligence- the ability to construct innovative

ways of conceptualizing, data gathering, and operating, in a new culture

General cultural competency - the meta-cultural perspective

Styles of Communication (verbal and nonverbal) Conflict styles Prejudice and power

Intercultural Communication Intercultural Competence (INCA Motivation,

Skills/Knowledge, & Behavior)

Empirical How the construct is operationalized

Challenges to the Process of Defining a Construct

Do you have the same definition? Are enough related constructs included to

adequately represent the full domain? Do you operationalize the construct in the

same way as others do? Have you teased apart related constructs

that should be included from those that should NOT be included?

Model of Construct

Specify content knowledge How detailed?

Specify procedural knowledge Cognitive processes, communication skills

Specify pathways of development from novice to expert on all facets of the construct.

Our current working definition

Socio-cultural Competence: An individual's ability to discern equitable and inequitable circumstances for persons involved in any situation, correctly identify environmentally and culturally influenced behavior and responses by all, and communicate effectively using all available data.

Construct Explication

Compile a list of specific behaviors, beliefs, and attitudes that demonstrate the presence of the construct

Also compile a list of behaviors, beliefs, and attitudes that do NOT indicate the presence of the construct

Use these lists to determine the nature and content of the test items.

Our List of Characteristics

I. Cognitive Skills    Critical Thinking       Clarification and analysis of information supporting own position     Identification and exploration of information supporting multiple

viewpoints.     Reflective Thinking Involves the assumption that knowledge is uncertain and can be created in the internal process of constructing meaning

      Dialectical Thinking

      Systems thinking     Comparative Thinking      Self – awareness                i.      Knowledge of own cultural programming               ii.      Knowledge of beliefs and values     Awareness of self in relation to others

Our List of Characteristics

II. Intercultural Competency Attitudes

    Valuing of all cultures

    Curiosity and discovery

    Interaction posture (non-judgmental)

    Role behavior

    Ethnocentrism – ethnorelativism

    Willingness to self-correct

Our List of Characteristics III.Procedural Skills  Communication

Verbalization   Mindful Listening       Face management     Non-verbalized communication – (un)conscious

  Identification & General Knowledge of:

     Conflict Styles     Value Patterns Value judgments result from ignorance of how our culture shapes our values - errors result from not using a systematic and integrated method of coming to value conclusions or unexamined value principles

   Cultural Adaptation    Prejudice and Power (Mindsets and Stereotypes)

 

Our List of Characteristics

IV. Motivation      Mindfulness (as a bridging concept between cognition and affect - a self-conscious concern for and evaluation of one’s thinking process)

     Open - mindedness

     Tolerance of Ambiguity

    Suspension of judgment

     Respectfulness

     Empathy

 

Wolcott’s Critical Thinking Rubric (Novice to Expert)

Clarifies and analyzes the information/ evidence supporting OWN position/thesis

Provides very limited support, primarily unexamined prior beliefs, clichés, expert opinions, or quotes

Provides support, but uses superficially understood evidence

Explores support for multiple viewpoints, but fails to identify adequate support for own position

Uses carefully evaluated evidence to reason logically in support of position

Articulates how a systematic process of critical inquiry was used to build position/thesis

Identifies and explores information/ evidence supporting multiple viewpoints or approaches

Portrays viewpoints dichotomously, e.g., right/wrong, good/bad, smart/stupid

Acknowledges more than one potential viewpoint, but provides superficial analyses

Explores arguments and evidence supporting multiple viewpoints

Evaluates information using general principles that allow comparisons across viewpoints

In addition to previous level, articulates a systematic process for evaluating viewpoints over time

Our Conceptual Map of the Domain Comparative Thinking

Self-awareness

Sees self as beliefs Realizes beliefs are formed from history

Suspends beliefs in new situations while processing new data

Awareness of self in

relation to others

See differences as problematic

Responding neutrally to difference

Politely states positions/respects

differences

Ethno-relation

Ethnocentrism Ethnorelativism

Social Justice

Assumes equality Notices inequity Understands causes

Intercultural

competency

Openness

Curiosity & discovery

Responding to situations

Have a plan for possible situations

Have large repertoire of strategies

Tolerating Ambiguity

Find ambiguous situations

unpleasant, threatening,

attempt to leave

Find ambiguous situations

uncomfortable, butis able to remain in

them

Find ambiguous situations involving different opinions to be interesting and

challenging

Where to go from here…

Allow a lot more time to define your construct than you were planning

Examine existing instruments to determine their ability to fully assess the domain

If no instrument fully assesses the domain and you must develop one, plan enough time for defining the construct.

Review the Standards for Educational and Psychological Testing

Have fun with the process!

ReferencesAPA Board of Educational Affairs Task Force on Psychology Major Competencies, APA Guidelines for the Undergraduate Psychology Major.

www.apa.org/ed/resources.html, retrieved 4/20/2008.Bennett, J. & Salonen, R. (2007). Intercultural communication and the new american

campus. Change March/April, 46-50.Benson, J. (1998). Developing a strong program of construct validation: A test anxiety

example. Educational Measurement: Issues and Practice, 17, 10-17.Earley, P.C, & Ang, S. (2003). Cultural intelligence: Individual interactions across cultures.

Educational Measurement: Issues and Practice, 17, 10-17.Ferrara, S., & DeMauro, G.E., (2006). Standardized assessment of individual achievement

in K-12. In R.L. Brennan (Ed.), Educational Measurement (4th ed., pp. 579-621). Westport, CT: American Council on Education/Praeger.

Ferrara, S. (2006). Toward a psychology of large-scale educational achievement testing: Some features and capabilities. Educational Measurement: Issues and Practice, 4, 2- 5.

Gorin, J. (2006). Test design with cognition in mind. Educational Measurement: Issues and Practice, 4, 21-35.

Haynes, S.N., Richard, D.C.S., & Kubany, E.S. 1995. Content validity in psychological assessment: A functional approach to concepts and methods. Psychological Assessment, 7(3), 238-247.

Yershova, Y., Dejaeghere, J., & Mestenhauser, J. (2000). Thinking not as usual: Adding the intercultural perspective. Journal of Studies in International Education, 4(1), 39-78.