running head: creative engagement€¦ · treat ideas as a product of creative thinking and...

Running head: CREATIVE ENGAGEMENT

Creative Engagement in Learning:

A New Way to Conceptualize and Measure the Middle School Experience

Ross Anderson1

Christine Pitts, 1

Keith Smolkowski2

Author Note

1Ross Anderson is Senior Lead Researcher and Christine Pitts is Doctoral Policy Fellow

at the Educational Policy Improvement Center. 2Keith Smolkowski, PhD is a Research Scientist

at the Oregon Research Institute.

This research was supported by a grant from the U.S. Department of Education

(PR/Award No. U351D140063).

Correspondence concerning this article should be addressed to Ross Anderson,

Educational Policy Improvement Center, 1700 Millrace, Eugene, OR 97405. E-mail:

[email protected].

CREATIVE ENGAGEMENT 2

Abstract

This current study examines measurement of creative engagement as part of a larger

program of inquiry designed to examine factors—cognitive, metacognitive, and social-

emotional—that relate to student success in middle school. This program focuses on the

multifaceted dimensions of student creativity and engagement during a critical period of identity

formation and in the context of a new approach to teaching and learning through the arts across

content areas. Situated within a model of creative engagement in learning, this study includes

confirmatory factor analyses, cross-validation, and invariance testing of two extant measures,

using responses from samples of 6th grade students. Our key findings show that a reduced

version of the Runco Ideational Behavior Scale for Students (RIBS-C) and a reduced version of

the Student Engagement Instrument (SEI) demonstrated reliability and validity. Results provide

greater precision for future measurement and highlight the complexity of measuring contextually

specific constructs with diverse students.

Keyword: measurement, factor analysis, creativity, engagement, middle school


Creative Engagement in Learning:

A New Way to Conceptualize and Measure the Middle School Experience This present study begins to conceptualize the factors that govern creative engagement in

middle school through a process of measurement refinement and validation. In order to

understand how components of creativity affect and are affected by components of cognitive,

behavioral, and affective engagement in school over the middle school period, this study applies

a multiphase process to refine and validate two distinct measures. As Beghetto (2016) suggests, a

paucity of research studying the interdependence of intrapersonal and interpersonal dimensions

of creative learning in school has left the field with few measurement models to capture and

study these phenomena. The models investigated in this study begin to fill this gap.

Program of Inquiry

This current study is part of a larger program of inquiry designed to examine factors of

student success and changes in creativity and engagement over three years of middle school—

from grades six to eight. In this program, we focus on the complex processes of both creativity in

learning and learning in creativity (Beghetto, 2016; Truman, 2011) alongside the multifaceted

dimensions of affective, behavioral, and cognitive engagement in learning and school (Fredricks,

Blumenfeld, & Paris, 2004). Our program of inquiry responds to the call from researchers

(Beghetto, 2016; O’Neal, Paek, & Runco, 2015; Peppler, 2013) to employ new approaches to

understand test theories about the role of creativity in learning, the impact of the arts as a

pedagogical tool for transformation, and the effect on outcomes of success in middle school.

Research demonstrates that the middle years entail a critical period of identity formation (Meeus,

van de Schoot, Keijsers, Schwartz, & Branje, 2010) and that learning in and through the arts (i.e.,

arts integration) provides expressive experiences that can be transformative for the learner


(Peppler, 2013) and disrupt “fossilized curricula” for teachers (Beghetto, in press, p. 5) . Further,

the construction of the middle school experience to serve as a transition between elementary

school and high school has been a contested dimension of K-12 public education for several

decades (Goldin, 1999)—the developmental appropriateness of the typical middle school is still

in question (Juvonen, Kaganoff, Augustine, & Constant, 2004).

Creativity. As it is understood today, creativity includes several conceptualizations,

mostly stemming from Dewey’s (1910) and Wallas’ (1926) stages for problem solving (i.e.,

preparation, incubation, illumination, and verification) and Rhodes’ (1961) early framing of

person, process, product, and press (Beghetto & Kaufman, 2010). Most recently Glăveanu

(2013) re-imagined the Four-P framework as the Five As—actor, action, artifact, audience, and

affordances—in order to make the sociocultural dimension and context explicit and embed the

process in each component (see Table 1). Glaveanu’s Five As frame the dimensions of our own

model of creative engagement in learning at the meso-level of creativity within the culture and

context of middle school. Beghetto’s (2016) model frames the micro-level of creative learning

moments in a classroom. For this study we build from Runco, Plucker, and Lim’s (2001) work to

treat ideas as a product of creative thinking and ideational skill, potentially a criterion for

creative learning. We focus on (a) creative flexibility—the capacity to think of different types of

ideas or solutions—and (b) creative fluency—the capacity to think of many ideas or solutions.

Engagement. Conceptually, student engagement can relate to the momentary

phenomenon of time on task or motivation to learn (Finn, 1989; Wang & Eccles, 2012) as well

as the a multilayered ecology of the school experience that includes family, community, and

classroom influences (Lawson & Lawson, 2013). The field has defined engagement through a

range of interrelated cognitive, emotional, behavioral, and social aspects of learning. Fredricks et

al. (2004) distilled student engagement down into a tripartite of individual student need: (a) the


need for relatedness, (b) the need for autonomy, and (c) the need for competence. Driven by this

tripartite model, we list the factors in Table 1 that we hypothesize may influence creative

engagement in learning for students at the micro- and meso-levels in school. In this study, we

empirically explored several dimensions of engagement in our model—relevance, relatedness,

and relationships—through an iterative process of measurement refinement, factor analysis,

cross-validation, and convergent and discriminant validity.

To accomplish the aims of this present study, we used a pilot phase (Study 1) and a cross-

validation study (Study 2) to explore how the measures performed with our population of interest

and to test the robustness of the resulting models. Our research questions below target the

reliability and validity of measures of student creative ideational flexibility and fluency and

educational aspiration and relevance, relationships with peers, and relationships to teachers in

school.

1. For each measure, do the pilot sample data adequately fit a model with the factors

established in prior research? If prior models are not adequate, are there other

theoretically relevant models that fit the pilot sample data?

2. Do the data from our validation sample fit these new models? If not, through a process of

local fit-testing, item reduction, or exploratory factor re-configuration, do the data

adequately fit revised models?

3. Do different samples replicate adequate fit and composite reliability?

4. Do components of the structural configurations of the revised models demonstrate

invariance across multiple samples?

Measures

Runco Ideational Behavior Scale for Children (RIBS-C). For this study, we analyzed

the self-report RIBS-C scale. Developed as a criterion for creative behavior (Runco, Plucker, &


Lim, 2001), the RIBS-C uses a 5-point frequency of behavior scale ranging from “never” to

“almost always.” We tested the complete RIBS-C for structural validity with our spring pilot,

including all 30 items, 4 of which were contraindicative and required reverse coding when scores

were totaled. Runco, Walczyck, Acar, Cowger, and Simundson (2014) suggested that these items

target the theoretical opposite of constructs of interest, may diminish the response set patterns

(e.g. marking all responses positive), but may need to be eliminated for analyses. Past research

with the adult version of the RIBS assessments demonstrated some evidence of a two-factor

model for fluency and flexibility (Runco et al., 2001; Runco et al., 2014; Tsai, 2015). O’Neal,

Paek, & Runco (2015) published the first validity study of the RIBS-C and compared the

goodness-of-fit of different models that represented multiple theories of creativity. O’Neal et al.

retained all but five items in their two-factor model and report model fit to be “adequate” (2015,

p. 151); however, the fit statistics and reporting of local fit-testing fall short of current

recommendations (Hu & Bentler, 1999; Kline, 2016). Given that those results were not available

at the time of our pilot study, we used exploratory factor analysis to find an adequate model for

our data.

Student Engagement Instrument (SEI). The SEI is a self-report measure of

psychological, emotional, and cognitive indicators of student engagement. The SEI employs a 5-

point Likert scale ranging from “totally disagree” (1) to “totally agree” (5) with a middle term for

neutral responses (3). The authors of the SEI completed exploratory factor analyses and

convergent, concurrent, and predictive validity studies and found some evidence of adequate

robustness of the instrument (Appleton et al., 2006; Lovelace et al., 2014). For example, the 35-

item, 6-factor model (described in Table 3) reached a CFI of 0.97, a close fit by Hu and Bentler’s

(1999) criterion, but also produced a large and statistically significant Chi-square value (𝜒!=

2,780, p < .001) and an RMSEA value of .065.


Study 1

The aim of this pilot study was to test the technical adequacy for our population and

eliminate or reword items that did not function adequately. We used iterative exploratory factor

analysis to refine the measures. We analyzed the wording or items to determine potential

confusion or irrelevance and eliminated items that did not demonstrate adequate communality.

Results

RIBS-C. We identified the following factors from the retained items: (a) Future-oriented

flexibility and fluency, (b) fluency of new ideas, (c) fluency of improvement on existing ideas,

(d) flexibility, and (e) ideational self-efficacy.

SEI. Examining the content of items and factor structures in EFA, we reduced the 35-

item SEI to a 15-item, three-factor solution that appeared to represent the most salient factors for

our program of inquiry—(a) control and relevance, (b) relationships at school, and (c) school

climate—and aligned to the original factors proposed by Appleton et al. (2006).

Study 2

The aim of study 2 was to test the validity of the structural configuration of the

measurement models determined in Study 1 and the invariance of this model with multiple

different samples.

Results

As this study represents the second published use of the RIBS-C in empirical research, a

modified version of the original RIBS for adults, we examined the internal reliability and validity

of the RIBS-C scores to ensure valid and reliable use of future scores for our larger program of

study. In the last part of this section we report results from each stage of analytic procedures

described earlier.


Runco Ideational Behavior Scale for Children

In response to our first research question, we used CFA to test the five-factor model

established by EFA in the pilot study; GOF reached an SRMR = .052, CFI = .93, and RMSEA =

.057 with a statistically significant χ! value (see Table 3). Given that these results did not meet

the strict criteria for fit suggested by Hu and Bentler (1999), we concluded that the data did not

provide an entirely satisfactory fit to the model. After proceeding with fit examination, the

ideational self-efficacy factor and items proved to be problematic (e.g., weak coefficients and

residual correlations above .10) and were eliminated. In response to research question 3, we

conducted CFA with sample 2 to cross-validate the four-factor model. GOF for the four-factor

model met Hu and Bentler’s strict criteria for close fit to the data from Sample 2 (χ! (38) =

65.25, p < .05, SRMR = .040, CFI = .96, and RMSEA = .048).

In response to research question 4, we tested measurement invariance to learn about the

comparability and generalizability of the GOF statistics and model parameter estimates across

the three samples. Sample 2 served as the calibration sample in a nested series of tests of

increasing parameter constraints and sample 3 served as the final cross-validation sample. We

applied the unconstrained four-factor model to sample 3 with all parameters freely estimated. As

can be seen in Table 3, some indices for sample 3 degraded somewhat from the indices produced

by sample 2; yet, GOF still demonstrated that the model achieved a relatively close fit. To

statistically test differences between distinct parameter components of the model, the following

comparisons increased constraints in three separate steps: (1) constrained pattern coefficients

(Λ fixed), (2) factor variances and covariances (Λ,Φ fixed), (3) item residual variance

(Λ,Φ,Θ! fixed). Theses results will be published in the full paper.


Student Engagement Instrument

We followed the same procedures for the second measure of interest and will report the

results in the full paper when published.

General Discussion

Overall, results of the reduced version of the RIBS-C indicated good reliability and

structural validity within a similar factor structure to the model produced by the pilot data.

Results of the reduced version of the SEI indicated good reliability and structural validity, but

required significant model restructuring and further item reduction to achieve a close fit to the

data. The results from this study may indicate the need for further item development that can

accurately target certain factors of student engagement for our population of interest. Overall, the

majority of pattern coefficients, factor variances and covariances, and unique item residual

variance remained invariant in cross-validation with two samples.

Implications

As recognition of the importance of creativity in the classroom grows, the ability to

efficiently scan for levels of creative ideational behavior among students may have implications

on what strategies teachers consider applying to their practice. In order for the development of

ideational behaviors to become embedded in student learning, the dimensions of flexibility and

fluency each require unique strategies. And these pedagogical strategies and learning behaviors

may be highly interrelated to the dimensions of relevance, aspiration, sense of belonging, and

teacher support targeted by our revised set of engagement items.

Conclusion

The procedures we chose and the results we found provide insights that may support

further research of these multidimensional constructs as they relate to the future of learning in

schools and what the field considers as evidence of effective practices. We succeeded to reduce


the number of items needed to reliably measure several dimensions critical to our model of

creative engagement in learning. Through an iterative process with two samples from our

population of interest we were able to complete two rounds of hypothesis generation and testing.

We determined the most efficient and robust sets of indicators from available measures, reduced

testing burden for students in the future, and increased the promise for reliability and construct

validity in models of longitudinal growth. For creative engagement in learning to become a

viable model that supports improved teaching and learning in middle schools, measurement

precision of its component constructs is a prerequisite. Our study brings us one step closer to this

possibility.

References

Anderson, R. & Nelson, N. (2016). Trajectories of engagement: Who becomes less engaged in

the middle grades most rapidly? Manuscript in preparation.

Anderson, R. (2015). The makers: Creativity and entrepreneurship. In Y. Zhao (Ed.), Counting

what counts: Reframing education evaluation. Bloomington, IN: Solution Tree Press.

Appleton, J. J., Christenson, S. L., Kim, D., & Reschly, A. L. (2006). Measuring cognitive and

psychological engagement: Validation of the Student Engagement Instrument. Journal of

School Psychology, 44, 427-445.

Arbuckle, J. L. (2006). Amos 7.0 User's Guide. Chicago: SPSS.

Baer, J. (2015). The importance of domain-specific expertise in creativity. Roeper Review: A

Journal on Gifted Education, 37(3), 165-178. doi: 10.1080/02783193.2015.1047480

Beghetto, R. A. (2010). Creativity in the classroom. In J. Kaufman & R. Sternberg (Eds.), The

Cambridge handbook of creativity (pp. 447-467). New York: Cambridge University

Press.


Beghetto, R. A. (2013). Killing ideas softly?: The promise and perils of creativity in the

classroom. Charlotte, NC: IAP, Information Age Publishing, Inc.

Beghetto, R. A., & Kaufman, J. C. (2010). Nurturing creativity in the classroom. Cambridge ;

New York: Cambridge University Press.

Beghetto, R., Kaufman, J., & Baer, J. (2015). Teaching for creativity in the Common Core

classroom. New York: Teacher’s College Press.

Beghetto, R. (2016). Creative learning: A fresh look. Journal of Cognitive Education and

Psychology, 15(1), 6-23.

Beghetto, R. A. (in press). Leveraging micro-opportunities to address macroproblems: Toward

an unshakeable sense of possibility thinking. To appear in Ambrose, D. & Sternberg, R.

J. (Eds.). Creative intelligence in the 21st Century: Grappling with enormous problems

and huge opportunities.

Blinder, A. S. (2009). How many US jobs might be offshorable? World Economics, 10(2), 41-78.

Bronson, P. & Merryman, A. (2010, July 10). The creativity crisis. Newsweek. Retrieved from:

http://www.newsweek.com/creativity-crisis-74665

Cianci, A. & Klein, H. (2010). The effect of negative feedback on tension and subsequent

performance: The main and interactive effects of goal content and conscientiousness.

Journal of Applied Psychology, 95(4), 618-630.

Corbett, D., Wilson, B., & Morse, D. (2002). The arts are an “R” too. Jackson, MS: Mississippi

Arts Commission.

Csikszentmihalyi, M. (1996). Creativity: Flow and the psychology of discovery and invention.

New York: HarperCollins Publishers.

Darling-Hammond, L. (2010). The flat world and education: How America's commitment to

equity will determine our future. New York, NY: Teachers College Press.


Dewey, J. (1910). How we think. New York: Health.

Dewey, J. (1938). Experience and education. New York: Macmillan.

Every Student Succeeds Act, U.S.C. § 1177 (2015).

Florida, R. (2002). The rise of the creative class and how it's transforming work, leisure,

community and everyday life. New York: Basic Books.

Florida, R. (2011, October 6). The Creative class is alive. The Atlantic. Retrieved September 25,

2012, from http://www.theatlanticcities.com/jobs-and-economy/2011/10/creative-class-

alive/252/

Florida, R. (2012). The rise of the creative class revisited (2nd ed.). New York: Basic Books.

Fredricks, J. A., Blumenfeld, P. C., & Paris, A. H. (2004). School engagement: Potential of the

concept, state of the evidence. Review of educational research,74(1), 59-109.

Glăveanu, V. (2013). Rewriting the language of creativity: The Five A’s framework. Review of

General Psychology, 17, 69-81.

Gliem, J. A., & Gliem, R. R. (2003) Calculating, interpreting, and reporting Cronbach’s alpha

reliability coefficient for Likert-type scales. Midwest Research to Practice Conference in

Adult, Continuing, and Community Education.

Goldin, C. (1999). A brief history of education in the United States. NBER Historical Paper No.

119, Cambridge, MA.

Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis:

Conventional criteria versus new alternatives. Structural Equation Modeling, 6(1), 1–55.

IBM Corp. (2013). IBM SPSS Statistics for Windows, Version 22.0. Armonk, NY: IBM Corp.

Juvonen, J. Le, V., Kaganoff, T., Augustine, C., & Constant, L. (2004). Focus on the wonder

years: Challenges facing the American middle school. Santa Monica, CA: RAND

Corporation.


Kline, R. B. (2013). Exploratory and confirmatory factor analysis. In Y. Petscher & C.

Schatschneider (Eds.), Applied quantitative analysis in the social sciences (pp. 171-207).

New York: Routledge.

Kline, R. (2016). Principles and practice of structural equation modeling (4th ed.). New York,

NY: Guilford Press.

Kim, K. (2011). The creativity crisis: The decrease in creative thinking scores on the Torrance

Tests of Creative Thinking. Creativity Research Journal, 23(4), 285-295.

Lovelace, M. D., Reschly, A. L., Appleton, J. J., & Lutz, M. E. (2014). Concurrent and

predictive validity of the student engagement instrument. Journal of Psychoeducational

Assessment, 0734282914527548.

Lyons, I. M., & Beilock, S. L. (2012). Mathematics anxiety: Separating the math from the

anxiety. Cerebral Cortex, 22(9), 2102-2110.

MacCallum, R., Widaman, K., Zhang, S., & Hong, S. (1999). Sample size in factor analysis.

Psychological Methods, 4, 84-99.

Meeus, W., van de Schoot, R., Keijsers, L., Schwartz, S., & Branje, S., (2010). On the

progression and stability of adolescent identity formation: A five-wave longitudinal study

in early-to-middle and middle-to-late adolescence. Child Development, 81(5), 1565-1581.

Menzer, M. (2015). The arts in early childhood: Social and emotional benefits of arts

participation. Washington, DC: National Endowment for the Arts.

Muthén, L. & Muthén, B. (2012). Mplus user’s guide 7th edition. Los Angeles: CA: Muthén &

Muthén.

Papert, S. (1993). The children’s machine: Rethinking school in the age of the computer. New

York, NY: Basic Books.


Peppler, K. A., & Davis, H. J. (2010, June). Arts and learning: a review of the impact of arts and

aesthetics on learning and opportunities for further research. In Proceedings of the 9th

International Conference of the Learning Sciences-Volume 1 (pp. 1000-1007).

International Society of the Learning Sciences.

Peppler, K. (2013). New opportunities for interest-driven arts learning in a digital age. New

York, NY: Wallace Foundation.

Peppler, K., Powell, C. W., Thompson, N., & Catterall, J. (2014). Positive impact of arts

integration on student academic achievement in English language arts. The Educational

Forum, 78, 364-377.

Pink, D. (2006). A whole new mind: Why right-brainers will rule the future. New York: Penguin

Group Inc.

Preacher, K. & MacCallum, R. (2003). Repairing Tom Swift’s electric factor analysis machine.

Understanding Statistics, 2, 13-43.

Raykov, T. (2004). Behavioral scale reliability and measurement invariance evaluation using

latent variable modeling. Behavior Therapy, 35, 299-331.

Raz, G. (October 4, 2014) The source of creativity. http://www.npr.org/programs/ted-radio-

hour/351538855/the-source-of-creativity

Robinson, H. (2013). Arts Integration and the Success of Disadvantaged Students: A Research

Evaluation. Arts Education Policy Review, 114, 191-204, DOI:

10.1080/10632913.2013.826050

Rosenthal, R. & Rosnow, R. (2008). Essentials of behavioral research: Methods and data

analysis. Boston, MA: McGraw-Hill.

Runco, M., Plucker, J., & Lim, L. (2001). Development and psychometric integrity of a measure

of ideational behavior. Creativity Research Journal, 13, 393-400.


Runco, M., Walczyk, J., Acar, S., Cowger, E., Simundson, M., & Tripp, S., (2014). The

incremental validity of a short form of the Ideational Behavior Scale and usefulness of

distractor, contraindicative, and lie scales. The Journal of Creative Behavior, 48, 185-

197.

Sijtsma, K. (2009). On the use, the misuse, and the very limited usefulness of Cronbach’s

alpha. Psychometrika, 74(1), 107-120.

Spearman, C. (1904). “General intelligence,” objectively determined and measured. American

Journal of Psychology, 15, 201–293. doi:10.2307/1412107��

Stoelinga, S., Silk, Y,. Reddy, P., & Rahman, N. (2015). Turnaround arts initiative: Final

evaluation report. Washington D.C.: President’s Committee on the Arts and the

Humanities.

The Aspen Institute (Producer). (2012, September 29). Reversing the middle class jobs deficit.

[Panel Discussion] Retrieved from http://www.aspenideas.org/session/reversing-middle-

class-jobs-deficit-0

Truman, S. (2011). A generative framework for creative learning: A tool for planning creative-

collaborative tasks in the classroom. Border Crossing, 2011(1), 1-13.

Tsai, K. (2015). Assessing a Chinese version of the Runco Ideational Behavior Scale. Social

Behavior and Personality, 43(7), 1111-1122.

U.S. Census Bureau (2015). Retrieved from

http://quickfacts.census.gov/qfd/states/41/41039.html

Vygotsky, L.S (1997). Genesis of higher mental functions. In R.W. Rieber (Ed.), The collected

works of L.S Vygotsky, Vol. 4: The history of the development of the higher mental

functions. New York: Plenum. (Original work published 1960).


Vygotsky, L.S. (2004). Imagination and creativity in childhood (M. E. Sharpe, Inc., Trans.).

Journal of Russian and East European Psychology, 42, 7-97. (Original work published

1967).

Wallas, G. (1926). The Art of thought. New York: Oxford University Press.

Zhao, Y. (2012). World class learners: Educating creative and entrepreneurial students.

Thousand Oaks, CA: Corwin.


Figure 1. Our model of creative engagement in learning merges Glăveanu’s (2013) Five A’s—

Actor, Action, Artifact, Audience, and Affordances—and Beghetto’s (2016) model of creative

learning with a synthesis of student engagement, defined in the literature. This present study

focused on two components in grey: (a) intrapersonal creativity—creative ideational flexibility

and fluency—and (b) interpersonal engagement—relationships with students and teachers and

educational relevance and aspiration.


Table 1.

Detailed Elements of the Theoretical Model of Creative Engagement in Learning within

Glăveanu’s (2013) Five A’s

Elements Details and Sources

Creativity Imagination, problem-finding, risk-taking, openness, self-efficacy, non-conformity, originality, effectiveness, flexibility, fluency, self-efficacy, and growth mindset

Engagement Relevance and control of learning, excitement and flow, support and feedback from others, curiosity, aspiration, perseverance, drive, and absence of anxiety

Actor Adults and peers who govern school learning conditions and composite of experience, skills, dispositions, learning habits, and cultural orientation through which students make meaning of their world.

Action Collaboration and creation organized by the Studio Habits of Mind—reflect, engage and persist, observe, understand the world, envision, develop craft, express, and stretch and explore (Hetland, Winner, Veenema, & Sheridan, 2013).

Artifact Documented moments from the learning process, such as drafts, evidence from each stage of process, reflective insights, commentary, feedback from others, exemplars, final product or performance.

Audience Adult and peer audiences require training to provide critical support for cultivating ideas, risk-taking, constructive feedback, high expectations for effort, community, rich interpretation, and validation of personal expression.

Affordances Factors from the learning environment, available media, and prior exposure, knowledge, and opportunity to learn, classroom climate, instructional and assessment practices, classroom vernacular, opportunity to practice and demonstrate, teacher self-efficacy, and chosen medium for expression

running head: creative engagement€¦ · treat ideas as a product of creative thinking and...

Documents