Examining Learner-Content Interaction Importance and

Efficacy in Online, Self-Directed Electronic

Professional Development in Science for Elementary

Educatorsin Grades Three ─ Six

Dissertation DefenseNovember 2010

Al Byers

Research Purpose

Conduct a quantitative exploratory study to determine which features of on-demand, self-directed online professional development are of greatest import, satisfaction, and learning value from a sample of upper elementary science teachers (grades three - six).

Research Implications

It is hoped research findings will:

•Inform Instructional Designers creating online PD as to which interaction strategies of online content may be most engaging and maximize learning for elementary teachers

•Inform Education Administrators charged with selecting PD for their teachers

•Inform emerging theory related to online learner-content interaction: Anderson’s (2003) Equivalency of Interaction Theorem

4

Focus of Presentation

What content interaction strategies correlate with teacher age, experience, and preferences?

Literature Review

Discussion

Research Question and Hypotheses

Methodology

Results

Six hypotheses based on Anderson’s Equivalency of Interaction Theory (2003)& review of literature

Discusses methods, instruments, and analysis used to answer the research hypotheses

Demographics of participants and findings for each of the six hypotheses

Findings & Research Implications for Anderson’s Theory, Instructional Designers, and Education Administrators

Dissertation Findings

State of US Education, Elementary Teacher Science Content Knowledge, Scalable Self-Directed e-PD

5

Literature Review

6

• US students scores on national and international science assessments stagnantNAEP (2000, 2006); TIMSS (2007); PISA (2006)

• Elementary and Middle level Teachers’ Content Knowledge Lacking and Professional Development Inadequate to address needsBanilower et al. (2007); Dede et al. (2006); Elmore (2004); Garet et al. (2001); Loucks-Horsley (1999); US Department of Education (2009); Yoon et al. (2008)

State of Affairs in Science Education in United States: Need for Improvement in Science Instruction

MoreInfo

16

Anderson’s Equivalency of Interaction

18

Learner-Content Interaction Strategies

• Interactive Reference: Content narrative, images, animations, audio and glossary(Schaller et. al 2002, 2007)

• Hands-On Opportunities: Activities to learn content via tactile inquiries(Krall et al. 2009; Harlen et al. 2004)

• Pedagogical Implications: Application of content in classroom contexts by grade band(Asbell-Clarke 2007; Berger et al. 2008; Harlen et al. 2004;Owston et al. 2006;Russell et al. 2008 )

• Simulations: Control of relational variables, phenomenon (Schaller et al. 2002, 2007; Sherman et al. 2008)

• Personal Feedback: Provided for individual(del Valle et al. 2009; Whitaker et al. 2007; Hoskins et al. 2005)

Literature&

Examples

35

Research Questions and Hypotheses

Research Questions

Which learner-content interaction strategies of self-directed online professional development are of greatest import, satisfaction, and learning value from a sample of upper elementary science teachers (grades three - six):

– Interactive Reference– Embedded Hands-on Activities– Personal Feedback Questions– Simulations– Pedagogical Implications

Will age, years teaching experience, and learning style correlate with different content-interaction strategies?

39

Research Hypotheses

H1: Age will be positively correlated with the type of preferred content-interaction strategy desired in self-directed web-based modules(del Valle et al. 2009; Farahani, 2003; Hoskins et al. 2005; Jiang et al. 2006; Kayes 2005; Schaller et al. 2007)

H2: Years teaching experience will be positively correlated with the type of preferred content-interaction strategy desired in on-demand, self-directed web-based modules(del Valle et al. 2009; Kayes 2005)

40

Research Hypotheses

H3: Age will be positively correlated with the achievement as measured via a pre/post assessment for those thathave completed and passed an online web module(del Valle et al. 2009; Hoskins et al. 2005; Kayes 2005).

H4: Years teaching experience will be postively correlated with achievement as measured via a pre/post assessment for those that have completed and passed an online web module(del Valle et al. 2009; Kayes 2005; Russell et al. 2009)

41

Research Hypotheses

H5: Adult learners prefer an onlineinteraction type matching their learningstyle when accessing self-directed onlinePD focused on learner-contentinteraction (Lapointe et al. 2008; Rhode 2009; Schaller et al. 2002; 2007;Su et al. 2005; Farahani 2003; Harlen et al. 2004; Kolb et al. 2005; Krall et al. 2009)

1.Teachers selecting Interactive Reference most favorable will be identified with “Assimilating” learning style2.Teachers selecting Hands-On activities most favorable will be identified with “Accommodating” learning style3.Teachers selecting Pedagogical Implications most favorable will be identified with “Converging” learning style4.Teachers selecting Personal Feedback most favorable will be indentified with “Diverging” learning style5.Teachers selecting Simulations most favorable will be identified with “Converging” learning style

MoreInfo

46

Research Hypotheses

H6: Teachers completing the 10-hour web modules and passing the final assessment will demonstrate significant gains in learning.(del Valle et al. 2009; Krall et al. 2009; Sherman et al. 2008; Russell et al. 2008).

47

Methodology

48

Non-Experimental Quantitative Design:Multi-Statistical Method study

• Bivariate Pearson Product Moment Correlations for H1-H4 (age, years experience, learning achievement, interaction strategies)

• Multiple One-Way Analysis of Variance for H5 (Kolb learning preference matches content-interaction strategy)

• Paired Sample t-tests for H6 (learning outcomes between pre/post & final assessments)

• Dependent Variables:• Learning Achievement• Teacher perceptions of effectiveness for the five learner content-interaction strategies

• Independent Variables:• Age, Yrs Teaching Experience, and Learning Preference

Methodology: Study Design

MoreInfo

58

Non-probabilistic Judgment SampleBased off “intellectual strategy” and “framework” of

variables that could influence individuals’ contributions incorporating theory, availability, literature, and researcher’s practical knowledge (Marshall, 1996, p. 523; Nesbary, 2000)

708 K-12 Science Teachers in Frame:• Exact number in national database from

grades 3-6 unknown ─ privacy issues• 85 educators volunteered and met

parameters of participation• Participants from 11 different states

(CT, GA, IL, KY, MT, OH, OR, PA, TX, WA, WV)

Methodology: Sampling Frame

MoreInfo

61

• NSTA Pretest, Posttest andFinal Assessment Instruments:Measures learning achievement(Chronbach α’s between .631- .836)

• Kolb Learning Style Inventory 3.1 (2005)Determines preferred learning style.(Chronbach α’s between .73- .99)

• Learner Demographic andContent-Interaction Preference Survey:Determines learner preferences across 5 content interaction strategies for 7 science content areas (Chronbach α’s at least .91 or higher for indexes)

Methodology: Instrumentation

MoreInfo

69

• Back-end analysis of national database:Generated master excel of 708 K-12 qualified educators.

• List segmented into 7 different content areas• 100-200 personalized emails/weekend.• Sent out over 6 waves between March and May, 2010.• No one invited more than 3 times to participate.

• Once agreed, separate email URL forwarded.

• Increase Response Rate Strategies:•Personalized invitations (first name), from individual with Sr. title•Guaranteed incentive (free e-book of choice from NSTA)•Raffle opportunity to win one of 14 Apple iPod touches•Informed educators selectively chosen because of status•Invited non-respondents with updated progress of completion•Communicated importance and promise to share results (Dillman et al. 2008; Kaplowitz et al. 2004; Porter et al. 2003; Sills et al. 2002; Sue et al, 2007)

Methodology: Data Collection

MoreInfo

71

Response Rates

• 708 Eligible, 85 Responded, 102 Unique Responses•Unable to know response rate or how many taught grades 3-6.•Two of 85 did not complete survey once started (98% response

rate for eligible, self-selected participants). •Two completed 3 surveys and 15 completed 2 surveys.

•39 emails bounced (94.5% of 708 eligible for study).

• Sampling Errors:• Self-selection bias, non-response bias, non-coverage errors

not germane given not attempting to generalize findings to larger population beyond the sample in exploratory, descriptive study (Sills et al. 2002; Wright, 2005)

• Ex: non-coverage error avoided with purposive sampling, examples in survey also provided as refresher.

• Focus is on quality of content and relationshipsrevealed between variables under examination.

Methodology: Data Collection

73

Results

74

Gender• Within Study:

88% female, 12% male• Mirrors US Elementary Public Schools: (Aud et al. 2010)

84% female, 16% male

Age of Participants: Range: 27-62 Years

Results: Participant Demographics

Age Clusters Study US Public Schools

Less than 30 Years 4.88% 18.7%

30-39 Years 28.05% 26.8%

40-49 Years 40.24% 23.9%

50-59 Years 23.17% 25.4%

60+ Years 3.66% 5.2%

75

Years Teaching Experience• Largest percentage with 4-9 years

experience: 37%

• Approximately mirrors percentages reflected at National level for most categories (Aud et al. 2010)

Results: Participant Demographics

Years Teaching Experience

Study US Public Schools

0-3 Years 12.20% 17.0%

4-9 36.59% 28.0%

10-19 28.05% 27.9%

20+ 23.16% 27.0%

76

Teaching License Credentials• preK-6 Licensure Largest: 56%• Middle level certification: 28%

• Secondary Science Certificate: 9.7%

Results: Participant Demographics

Teacher Licensure Frequency Percentage

preK-6 46 56.1%

Middle Education (6-8) 23 28.0%

Secondary Science-General

5 6.10%

Secondary Science-Life Science

2 2.40%

Secondary Science-Earth/Space Science

1 1.20%

Secondary Science-Physical Science

0 0

Multiple Secondary Science Endorsements

5 6.10%

77

Grade Levels Taught• Could check multiple grade levels• 63% taught grades 3-6 (target of study)

• Largest percentage taught grade six: 21%

Results: Participant Demographics

Grade Level

Frequency PercentageTeaching Certificate

1 6 3.4%

PreK-656%

2 9 5.1%

3 20 11.2%

4 22 12.4%

5 33 18.5%

6 37 20.8% Middle Level28%

7 22 12.4%

8 29 16.3%

78

• All 85 Teachers placed into 1 of 4 categories

• 17 took survey multiple times, 15 had identical preference

Results: Learning Preferences

26 Teachers

Interactive Reference

25 Teachers

hands-on

23 Teachers

Simulations,PedagogicalImplications

11 Teachers

Personal Feedback

79

Three Interaction Strategies positively correlated with age at a significant level:(a) Simulations, (b) Personal Feedback,(c) Interactive Reference

H1 Results: Age and InteractionStrategy Correlated

Age and Content-Interaction Strategies Across Seven Science Content Areas(Table 8, p. 105)

Range of rs Overall Correlation Index

Simulations.09

(p = .33)26

(p = .03)r(102) = .2, p = .03

Hands-OnActivities

.04(p = .54)

.09(p = .34)

r(102) = .08, p = .45

Personal Feedback

.09(p = .36)

.24(p = .01)

r(102) = .20, p =.04

Interactive Reference

.10(p = .30)

.21(p = .04)

r(102) = .008, p =.02

Pedagogical Implications

-.09(p = .34)

.03(p = .76)

r(102) = .008, p = .94

Chronbach α

81

Two Interaction Strategies positively correlated with Years Teaching Experience at a significant level:(a) Personal Feedback, (b) Interactive Reference

H2 Results: Years Teaching Experience &Interaction Strategy Correlated

Chronbach α

Years Teaching Experience and Content-Interaction Strategies(Table 2, p. 107)

Interaction Strategy

Range of rs Correlation

Simulations.01

(p = .92)12

(p = .20)r (102) = .04, p = .67

Hands-OnActivities

.05(p = .54)

.14(p = .16)

r (102) = .11, p = .29

Personal Feedback

.21(p = .03)

.28(p = .005)

r (102) = .27, p = .006

Interactive Reference

.10(p = .29)

.27(p = .007)

r (102) = .22, p = .02

Pedagogical Implications

-.006(p = .95)

-.17(p = .08)

r (102) = -.08, p = .41

82

The overall correlation between age and learning achievement was not significant.

H3 Results: Age and Learning Achievement Correlated

Age, Pre- and Postassessments and Final Assessments(Table 10, p. 109)

Pretest Posttest Final Assessment

Age r(102) = .02, p = .85 r(102) = .08, p = .45 r(102) = .17, p = .08

Correlation between learning achievement assessments

Pretest–Posttest r(102) = .48, p = .001

Pretest–Final Assessment

r(102) = .22, p = .02

Posttest–Final Assessment

r(102) = .42, p = .001

83

Statistically Significant Correlation between Years Teaching Experience and Learning Achievement on the Final Assessment (which rewards certificate).

H4 Results: Year Teaching Experience andLearning Achievement Correlated

Years Experience, Pre- and Postassessments and Final Assessments(Table 10, p. 109)

Pretest Posttest Final Assessment

Years Experience

r(102) = -.07, p = .47 r(102) = .15, p = .12 r(102) = .22, p = .03

Correlation between learning achievement assessments

Pretest–Posttest r(102) = .48, p = .001

Pretest–Final Assessment

r(102) = .22, p = .02

Posttest–Final Assessment

r(102) = .42, p = .001

84

H5 Results: Content Interaction StrategyMatches Preferred Learning Style

Learner Preference

Classification

Predicted Interaction

StrategyFindings

Assimilating(n = 26)

Interactive Reference

Pedagogical Implications least preferred (M = 3.58, SD = 1.39), and at significant level, F(4,116) = 13.40, p < .001.

Accommodating(n = 25)

Hands-OnPedagogical Implications least preferred(M = 3.14, SD = 1.39), and at significant level, F(4,116) = 20.33, p < .001.

Converging(n = 23)

Simulations,Pedagogical Implications

Pedagogical Implications least preferred (M = 3.14, SD = 1.39), and at significant level, F(4,116) = 11.81, p < .001.

Diverging(n = 11)

PersonalFeedback

No significant difference between any of 5 interaction strategiesF(4,116) = .41, p < .80

(Combination Tables 11-15, pp. 113-118)

86

H6 Results: Significant Learning Gains for Teachers with Self-Directed Web Modules across 2 instruments

Paired Sample Descriptive Statistics (Table 16, p. 120)

Pair N Mean SDStandard Error

Mean

Pair 1Pretest 102 61.31 18.45 1.83

Posttest 102 82.39 7.40 .73

Pair 2Pretest 102 61.31 18.45 1.83

Final Assessment 102 79.14 12.91 1.28

Paired Sample t-Tests for Teacher Learning Gains (Table 17, p. 120)

Paired Samples N Mean SD t df p

Pretest–Posttest 102 21.09 1.81 11.63 101 .000

Pretest–Final Assessment 102 17.84 16.62 10.84 101 .000

88

Discussion

Discussion Findings

Age and Content-Interaction Strategies

•Predicted age would correlate with strategies

•Others found age affecting preferences for different types of online content (Schaller, et al. 2002, 2007) and as potential predictor of achievement (Hoskins & van Hooff, 2005).

•Three strategies significantly correlated– Simulations– Personal Feedback– Interactive Reference

•Seems to Makes Sense! Research supports value of simulations for science (Bayraktar, 2001; Cameron, 2003; Lee et al. 2004; Reed et al.,

2002; Renkl et al. 2002; Schnotz et al., 2005), and other two strategies from instructional design for Science and Online Learning (Kali et al. 2008; Narciss, 2008; Shute et al., 2008).

MoreInfo

Discussion Findings

Years Experience and Interaction Strategies

•Predicted years experience would correlate with content-interaction strategies

•Others have found work experience affecting preferences for different types of online content(del Valle et al. 2009; Kayes, 2005)

•Two strategies significantly correlated in this study– Personal Feedback– Interactive Reference

•Seems to Makes Sense! Research studying PD and teacher demographics reveal differences based on where they are in their profession (needs, motivation, time to contribute, & skill vary) (Anderson et al. 2006; del Valle et al. 2009; Mulholland et al. 2005).

Discussion Findings

Age, Years Experience correlated withLearning Achievement

•Predicted both age and years experience would correlate with learning achievement

•Age was not significantly correlated (but approached)•Years Experience was significantly correlated with Final Assessment (tied to certificate and reward).

•Research Findings Mixed. Research for self-reported learning found ceiling effect (del Valle et al. 2009; Sherman et al. 2008), or found NSD in achievement between online PD with different levels of support (Russell et al. 2009). Research espouses quality of resources, learning environment, pedagogy and context are what matters based on where teachers are in their career.(Anderson et al. 2006; del Valle et al. 2009; Mentis, 2008; Mushayikwa et al. 2009; Yoon et al. 2003).

Discussion Findings

Content-Interaction Strategy matchesPreferred Learning Style

•Prediction based on literature examining learning preferences, and science and mathematics online PD (Felder et al. 2005; Hoskins et al. 2005; Kayes 2005; Schaller et al., 2002, 2007; Asbell-Clarke et al, 2007; Harlen et al. 2007; Krall et al, 2009; Rhode, 2009).

•Partial support found: In 3 of 4 learning style categories learners’ highest rated strategy did match predicted, but not at level of significance.

•Pedagogical Implications: Least favored at significant level for 3 of 4 categories. Diverging, NSD found in preference for any of the 5 interaction strategies.

•Potential Reasons: PI provides no “interaction” espoused by Anderson, just e-textbook. Teachers experienced, not perceive need, desire sci. content.

Discussion Findings

Teacher Learning Gains

•Generalizing findings beyond sample limited given nonprobabilistic judgment sample and non-random assignment to treatment.

•Worthwhile data in context of exploratory descriptive study discerning facets of phenomenon under study

•Scores on PostAssessment and Final Assessment Significantly Higher than on PreAssessment.

•Seems to Makes Sense! Emerging consensus in K-12 science that simulations (and CAI) provide access to representations that enhance user motivation and learning (Kulik 1991; Lee, 1999; Linn et al., 2006; Lunetta et al, 2007; Songer, 2007; Varma et al, 2008). Care to avoid claims that one form of media is better than another (Head et al., 2002; Lockee et al. 2002; Mentis, 2008)

Research Implications

Instructional Designers

•Given sig. learning gains, designers may wish to review the templates as they seek to design rich interactions in self-directed web learning objects

•Templates apply the Cognitive Theory in Multimedia Learning seeking to minimize extraneous load as dual coding/processing occurs for audio/video/text/images:

• Segmenting content into chunks• Pretraining and Indexing Needs• Proximity of text, images and questions• Rich feedback from multiple choice questions

(Mayer & Moreno, 2003; Moreno & Valdez; 2005; Morrison et al., 2005; van Merrienboer & Ayers; 2005; Narciss; 2008)

MoreInfo

Research ImplicationsAdministrators

•Seek to provide teachers with level of autonomy and control over their own learning. This facilitates motivation moving from extrinsic to internally motivated and regulated as stated in Self-Determination Theory(Ryan & Deci, 2000).

•Consider need analysis and career experience of teacher workforce as one size doesn’t fit all(Anderson et al., 2006; Mulholland et al. 2005)

•Consider selecting a suite of varied PD opportunities, including learning objects, as not all need or desire learner-learner interaction(Cavanaugh et al. 2010; Lapoint et al. 2008; Su et al., 2005; O’Keefe et al, 2006; Shimic; 2008; Waight et al., 2005; Walker et. al, 2008; ).

Research ImplicationsAnderson’s Equivalency of Interaction

•The positive, significant learning outcomes across 7 science content areas lend support.

•Providing rich learner-content interaction may allow the other interactions to be offered in diminished capacities to address scale.

•Others also find significant gains in learning for self-directed web modules (Chadwick et al., 2010; Russell, 2009; Krall et al., 2009; Sherman et al., 2008)

•Many large scale efforts now looking atself-directed e-PD via web modules(Waight et al., 2005; Walker et. al, 2008; Woodbury; 2008; Cavanaugh et al. 2010).

Research ImplicationsFuture Research

•Expand analysis within a particular strategy with different teacher characteristics: prior knowledge, subject domains, grade levels.

•Examine different levels of incentives and support structures to effectively support self-directed e-PD, or the different delivery systems themselves.

•Examine how self-directed systems and features interact with different interaction strategies to increase intrinsic learner motivation in light of mobile access to web 2.0 social networks and informal learning.