Learn about research involving this online formative assessment system and hear about

how it is used in classrooms.

Worcester Polytechnic Institute Introduces

Neil Heffernan, a computer science professor about his research

using this tool.

In this Webinar you will hear from:

Matt Militello, a professor of school leadership on how school leaders can use such a system.

Courtney Mulcahy, an 8th grade teacher who uses ASSISTment in her formative assessment process

every day.Cristina Heffernan, a professional

developer who trains teachers to use the system and incorporate it

into their classrooms.And Russ Rapose, District curriculum coordinator for

Milbury Public Schools

If Old McDonald Had An Assessment…

Matthew MilitelloAssistant Professor

The Press… The PushThe Press:

• More than ever we are TOLD, MANDATED, and SANCTIONED to use data to inform education decisions.

• Specifically, assessment data. Means state tests.

The Push:

• These data are not timely, not informative for teachers; That is, these data points are autopsy data.

• TMI: Too Much Information.

In Search of the Right Data!

Central OfficeStudent/School Growth

School LeadersTeacher Effectiveness

TeachersStudent Cognition

Formative Assessment Systems (FAS)There is Demand…. Supply Followed

Multi-Million Dollar Industry- Cost usually based on per student use

Not all FAS are the SAME!

How Do Districts Select, Support, and Use FAS?

Three Districts Three FASTechnical Features Implementation

Actual Use

Framework

Framework

Fit is defined by the INTENT of the districts selection (e.g. teacher use (student-diagnostic data) or district use (student growth model data) and the PURPOSE of the assessment

system—including validity.

Framework

Readiness is defined as: (1) building capacity in regard to both knowledge of the assessment product and (2) the

motivation to use the product in practice.

Framework

Coherence is how educators used the assessment systems fit into the context and the standard operating procedures of

the district. For instance, the creation of trigger mechanisms from review of data reports.

Woodbury Schools

Good fit for Central Office,but not teachers

Teachers ill-equipped Growth data forced on

daily practice

FAS-NWEA-MAP

Can’t Get Beyond FIT!

Can’t Get Beyond FIT!

Can’t Get Beyond FIT!

Needle in a Haystack:A FAS with Real-time, Student-Level Data!

Fit! Finally, Formative Assessments for TEACHERSTechnical Features Tutoring System for Real-Time Data

and Diagnostic Data: PURPOSETeacher-Level Use- Desire For This Data: INTENT

Using Intelligent Tutoring Systems to Help Math Teachers in the

Formative Assessment Process.

Prof. Neil HeffernanWorcester Polytechnic Institute

Who Am I? Teacher for America : teacher of middle

school math and science in Baltimore City PhD in Computer Science and Cognitive

Science–Licensed to Carnegie Learning Inc.

Tenure professor at Worcester Polytechnic Grant funded research on developing

interventions

Quick Project History - 2003 Teacher are being asked to use data to inform instruction, but every minute in the classroom spent on assessment is a minute lost to instruction.

US Department of Education (2003) Funded us to build a assessment system for teachers that did not take away teacher’s time.

Lindeman, Carolina Ruiz, George Heineman,

Murali Mani, Ryung Kim and others 30 others

Definitions A diagnostic assessment is an assessment

designed to uncover any misconceptions or gaps in knowledge or skills of individual students so that teachers can base their instruction on the current state of students’ knowledge, skills, or understanding.

Formative assessment is a process used by teachers and students during instruction that provides feedback to adjust ongoing teaching and learning to improve students’ achievement of intended instructional outcomes.

Research Questions for US Dept of Ed

1. Does ASSISTments work as an assessment tool? a. Reliably predict state test performance? b. Reliably advise teachers & students on what

knowledge to focus on?

2. Does ASSISTments help enhance student learning? c. Does the System effectively teach as it

assesses? b. Can teachers use the advice of this System to

lead to higher student learning?

The ASSISTment System A web-based

assessment system, designed to collect formative assessment data on student math skills

Usage has grown as our content has grown.

Currently, thousands of students use the system.

You Just Saw One ASSISTment A question within this “tutor” is what we call an

ASSISTment . We have 2,000 such ASSISTments for 7th and 8th

grade. (66 hours of instruction) We have another 1,000 for 6th grade. We have 1000 for 9-10th grade math, There are Three other types of content.

– Teacher created questions– Mastery Learning questions (track 130 skills in 8th

grade math) – Connected Math matching units (500

ASSISTments with tutoring)

Many Different Types of Reports State Test Report Knowledge component reports (based upon 130 middle school skills)

Item Report (e.g. for small formative assessments)

Mastery Learning Report (which students have reached mastery?)

Nam

es O

bscu

red

on P

urpo

se

Report to a Teacher on Their Own State Test

Research Questions for US Dept of Ed

1. Does ASSISTments work as an assessment tool? a. Reliably predict state test performance? b. Reliably advise teachers & students on what

knowledge to focus on?

2. Does ASSISTments help enhance student learning? c. Does the System effectively teach as it

assesses? b. Can teachers use the advice of this System to

lead to higher student learning?

Report to a Teacher on Their State

Predict Student Knowledge as Measured by State Test Scores

We have state scores from 2004 to 2009 for thousands of students.– We showed our error in predicting the

state test is as low as the test itself. – We showed we can even predict their 10th

grade math score from their 8th grade data.– We have shown we can more reliably

predict their state test score if we use how much assistance they need.

Feng, M., Heffernan, N.T., & Koedinger, K.R. (2009). Addressing the assessment challenge in an Intelligent Tutoring System that tutors as it assesses. The Journal

of User Modeling and User-Adapted Interaction.  Vol 19: p243-266.

Feng, M., Beck, J,. Heffernan, N. & Koedinger, K. (2008)  Can an Intelligent Tutoring System Predict Math Proficiency as Well as a Standardized Test?

 In Baker & Beck (Eds.). Proceedings of the 1st International Conference on Education Data Mining. Montreal, Canada.  pp.107-116.

ISBN - 13: 9780615306292 

Feng, M., Heffernan, N. T., & Koedinger, K. R. (2006b). Addressing the testing challenge with a Web-based e-assessment system that tutors as it assesses. Proceedings of the

Fifteenth International World Wide Web Conference (WWW-06). New York, NY: ACM Press. ISBN:1-59593-332-9.  pp. 307-316. [Nominated for Best Student Paper]  

1. Does ASSISTments work as an assessment tool? a. Reliably predict state test performance? b. Reliably advise teachers & students on what

knowledge to focus on?

2. Does ASSISTments help enhance student learning? c. Does the System effectively teach as it

assesses? b. Can teachers use the advice of this System to

lead to higher student learning?

Research Questions for US Dept of Ed

We have shown that our transfer model is good enough to improve assessment.

Using finer-grained skill model can improve accuracy of the assessment of student knowledge.

Feng, M., Heffernan, N. T., Mani, M., & Heffernan, C. (2007). Assessing students’ performance longitudinally: Item difficulty parameter vs. skill learning tracking.  The National Council on Educational Measurement 2007 Annual Conference, Chicago.

Feng , M, Heffernan, N., Heffernan, C. & Mani, M. (2009) Using Mixed-Effects Modeling to Analyze Different Grain-Sized Skill Models. Journal of Technology, Learning, and Assessment. (Featured Article of the year!)

Pardos, Z. A., Heffernan, N. T., Anderson, B. & Heffernan, C. (2007). The effect of model granularity on student performance prediction using Bayesian networks.  The International User Modeling Conference 2007.

1. Does ASSISTments work as an assessment tool? a. Reliably predict state test performance? b. Reliably advise teachers & students on what

knowledge to focus on?

2. Does ASSISTments help enhance student learning? c. Does the System effectively teach as it

assesses? b. Can teachers use the advice of this System to

lead to higher student learning?

Research Questions for US Dept of Ed

Can teachers use the advice of this system to lead to higher student learning?

Quasi Experiment– Results: Student in schools that used

ASSISTments did better on state test than students in control schools.

– Results: Gaps were closed in achievement Recruitment for a large 60 school field trial is being organized.

$25 million dollar grant“i3” Investing in Innovation Fund grant line• If you are interested talk to me.

Stories of Formative Assessment

Online Exit Card

Mastery Learning

Book Homework

Online

Courtney Mulcahy, an 8th grade teacher who uses ASSISTment in her formative assessment process

every day.

The Formative Assessment Cycle

Objective

Lesson

Assessment

Teacher

Student

The Cycle can move through two or three iterations depending on the situation.

Objective Lesson Objective

LessonAssessmentObjectiveLesson

Assessment

Assessment

Online Exit Card

Objective

• Find the side of a triangle

using the Pythagorean

Theorem

Lesson

• Use Connected Math Book

Assessment

• Three question

Problem Set online on

ASSISTment

Assessment

• Three question

Problem Set online on

ASSISTment

Assessment

• Three question

Problem Set online on

ASSISTment

Teacher

Student

There were 43 students on this item report, not all are shown.

INCORRECTa2+b2 = c2

122+142 = x2

144 + 196 = x2

340= x2

18.44 = x

CORRECTa2+b2 = c2

122+x2 = 142

144 + x2 = 196x2 = 54x = 7.21

Online Exit Card

Objective

• Find the side of a triangle

using the Pythagorean

Theorem

Lesson

• Use Connected Math Book

Assessment

• Three question

Problem Set online on

ASSISTment

Objective

• Find the leg of a right

triangle using the

Pythagorean Theorem.

Lesson

• Discussed the different

elements of the

Pythagorean Theorem.

Mastery Learning

Objective

• Find Mean Median Mode Range

from data in a Stem and Leaf

Plot

Lesson

• Lessons in 7th Grade

Assessment

• Mastery Learning

Questions on Mean Median,

Mode and Range

from Stem and Leaf

plots

Assessment

• Mastery Learning

Questions on Mean Median,

Mode and Range

from Stem and Leaf

plots

Students work until they get 3 (or

another chosen number) right in a row. The are very involved in their

learning.

Mastery Learning

Objective

• Find Mean Median Mode Range

from data in a Stem and Leaf

Plot

Lesson

• Lessons in 7th Grade

Assessment

• Mastery Learning

Questions on Mean Median,

Mode and Range

from Stem and Leaf

plots

Objective

• Find Mean Median Mode Range

from data in a Stem and Leaf

Plot

Lesson

• Tutoring in ASSIST-ment.

Lesson

• Tutoring in ASSIST-ment.

Students gets instant

feedback and support.

Mastery Learning

Objective

• Find Mean Median Mode Range

from data in a Stem and Leaf

Plot

Lesson

• Lessons in 7th Grade

Assessment

• Mastery Learning

Questions on Mean Median,

Mode and Range

from Stem and Leaf

plots

Objective

• Find Mean Median Mode Range

from data in a Stem and Leaf

Plot

Lesson

• Tutoring in ASSIST-ment.

Assessment

• Students must work until they get 3 right in a row.

• Can give a static post test.

Assessment

• Students must work until they get 3 right in a row.

• Can give a static post

test.

Book Homework

Online

Objective

• Students will be able to

solve questions involving

exponential functions.

Lesson

• Use CMP Book

Assessment

• Homework from the

Book, with questions

online.• One question

asks about linear

equations.

Homework was Page 13 # 22 and 23 Page 16 # 34

Assessment

• Homework from the Book, with questions online.

• One question asks about linear equations.

From Connected Mathematics Project Page 16 of Growing

Growing Growing.

Assessment

• Homework from the

Book, with questions

online.• One question

asks about linear

equations.

This data informed both

teacher and students of the

need for the next objective

Book Homework

Online

Objective

• Students will be able to

solve questions involving

exponential functions.

Lesson

• Use CMP Book

Assessment

• Homework from the

Book, with questions

online.• One question

asks about linear

equations.

Objective

• While studying

exponential functions

students will be able to

answer questions

about linear relationships.

• Read what is being asked

carefully.

Lesson

• Discuss the linear

equation question in

class.

Assessment

• Next nights homework

had multiple questions

about comparing linear and

exponential functions.

From Connected

Mathematics Project Page

14 of Growing Growing Growing.

Assessment

• Next nights homework

had multiple questions

about comparing linear and

exponential functions.

Homework was Page 14 – 17 # 24 – 30 and 35 - 38

Stories of Formative Assessment

Online Exit Card

Mastery Learning

Book Homework

Online

Does It teach as It Assesses? We have done 6 different randomized controlled experiment, and ASSISTments consistently leads to statistically meaningful learning results.

Two examples 1. A homework study2. A classroom study

Comparing Traditional Paper and Pencil Homework

with Web-Based Homework

Leena Razzaq, Michael Mendicino & Neil HeffernanWorcester Polytechnic Institute & West Virginia University

Intelligent Tutoring Systems 2008

Mendicino, M., Razzaq, L. & Heffernan, N. T. (2009). Improving Learning from Homework Using Intelligent Tutoring Systems. Journal of Research on Technology in Education (JRTE). Published by the International Society For Technology in Education

(ISTE). Spring 2009 issue. 41:3 p 331-346.

Digital Divide Is Real.• Worcester is an urban district with only about

70% of homes with Internet.• Students already get weekly homework that

requires the Internet.• Worcester needs supports for students who

don’t have computers at home.– School libraries– Printable worksheet– Before school and homeroom

• These studies prepare us for a future where kids can get the benefit of computers.

Maine Lap Top Initiative Every student in the state get a lap top. Over half the districts let students take the lap top home.

State scores have gone up in English but stayed flat in Math. (Prof. David Silvernail)

Richmond Virginia (Henrico Country)

Is Homework Important?• Etta Kralovec, author of The End of Homework: How

Homework Disrupts Families, Overburdens Children, and Limits Learning

• "When work goes home, teachers have little control over who does the work. Teachers [are] unable to scaffold new knowledge for students, and [are] unaware of each student's true educational progress.”

• http://www.hno.harvard.edu/gazette/2000/09.28/homework.html

• Hattie(1999) meta-analysis across over 100 studies showed that homework has a .4 effect size.

Literature Review: Other Comparisons of PPH to Tutoring Systems

Little prior work but the best of it is at the college level

– Mastering Physics – (Warnakulasooriya & Pritchard, 2005)

– Andes (VanLehn et al., 2005) produced a mean effect size of 0.6 over paper-and-pencil homework.

74

Possible Advantages of Web Based Homework

• Better learning for students– Immediate feedback to students with

tutoring• More help for teachers

– Time saver– Common errors analyzed – Formative assessment—Teachers use the

data to guide instruction more effectively. – Homework help—Students take homework

more seriously because they know it will be graded and the grade recorded.

Possible Disadvantages • Many of these systems do not take students’

work into consideration.• Students may try to do more math in their heads. • Teacher may spend less time looking at student

work.• Cheating may be easier among students.

Experimental DesignPurpose: to determine if students can learn more by doing

their math homework with the ASSISTment System than when doing traditional paper-and-pencil homework

Control: Student did their work on paper. The next day teachers did a 10 minute review of the questions. Teachers used the same feedback the ASSISTments system gave.

Materials: Pretest and Posttest of 10 itemsHomework was also 10 questions but with different

numbers.

Topic A: Number Scene Topic B: Mixed 77

1. Which of the following is closest to the product of 397.8 * 10.3?o A) 3,000o B) 30,000o C) 400o D) 4,000

2. Which of the following shows the numbers in order from least to greatest?o A) 0.452, 0.51, 0.432o B) 0.452, 0.432, 0.51o C) 0.432, 0.51, 0.452o D) 0.432, 0.452, 0.51

4. What is 15/60 as a percent?

5. Write 10/25 as a decimal.

6. What is the value of the following expression? 7 * 6 + 3

7. Destinee has a total of 12 fish in her aquarium. Exactly 9 of the fish are goldfish. What percent of the fish in the aquarium are goldfish?

Sample “Number Sense”

Experimental Design Continued We did a repeated measures analysis where each

student participated in each condition but in different orders. This allows us to deal with the large variation in different students learning rates. We eliminate the need to model the variation between teachers.

We counter-balanced the homework assignments in a Latin square design. The order of seeing the topics were randomly assigned to different classrooms.

West Virginia- 4 math teachers

Experimental Design ContinuedDay Computer first group Paper-and-Pencil First Group

Class A Class B Class C Class D

Mon Pretest Number Sense Intro to ASSISTments

ASSISTments

Pretest MixedIntro to

ASSISTmentsASSISTments

Pretest MixPPH

Pretest Number SensePPH

Tues Post-test Number Sense

Post-test Mixed Review PPHPost-test Mix

Review PPHPost-test Number

Sense

Wed Pretest MixedPPH

Pretest Number SensePPH

Pretest Number SenseIntro to ASSISTments

ASSISTments

Pretest MixIntro to

ASSISTmentsASSISTments

Thurs Review PPHPost-test Mixed

Review PPHPost-test Number

Sense

Post-test Number Sense

Post-test Mix

80

Experimental Design Continued

To deal with the fact that not everyone had internet at home we only analyzed students that did their homework both nights in both conditions. Left us with 28 students

You can’t ask students if they internet at home

1. Its embarrassing2. They don’t know if they will be able to have access

Results• Students learned significantly in both conditions

• Students learned reliably more with ASSISTments (2.3 problems verse 1.14 problems) with 0.6 effect size (p = 0.05).

82

Results Not Due to a Few Students

Top 5 gain scores

were in the web based homework condition.

Our worst gain score is in our WBH condition.

Conclusions Students do learn significantly (.6 standard deviations more) more from using the ASSISTment System compared to traditional paper-and-pencil homework.

84

Implications• Results of this study are promising in that

kids can learn a good deal more with homework.

• Only study of its kind I am aware of in K12 that isolated the benefit of web based homework.

• Could be important to policy makers, particularly considering the popularity of one-to-one computing initiatives

• Will become be more relevant as the digital divide narrows.

Does an Intelligent Tutor lead to more learning than traditional classroom practice?

Most math classrooms give a lecture and then a period of time for practice.

We showed that kids learn a lot more when getting immediate feedback from the ITS than the “business as usual” control.

Results: Students learned more when using the computer compared to the traditional classroom.

– Razzaq, L., Mendicino, M. & Heffernan, N. (2008) Comparing classroom problem-solving with no feedback to web-based homework assistance. In Woolf, Aimeur, Nkambou and Lajoie (Eds.) Proceeding of the 9th International Conference on Intelligent Tutoring Systems. pp. 426 -437. Springer-Verlag: Berlin.

1. Does ASSISTments work as an assessment tool? a. Reliably predict state test performance? b. Reliably advise teachers & students on what

knowledge to focus on?

2. Does ASSISTments help enhance student learning? c. Does the System effectively teach as it

assesses? b. Can teachers use the advice of this System to

lead to higher student learning?

Research Questions for US Dept of Ed

Can teachers use the advice of this system to lead to higher student learning?

Quasi Experiment– Results: Student in schools that used

ASSISTments did better on state test than students in control schools.

– Results: Gaps were closed in achievement Recruitment for a large 60 school field trial is being organized.

$25 million dollar grant“i3” Investing in Innovation Fund grant line• If you are interested talk to me.

We Have Users in Over a Dozen Towns

Worcester, Auburn, Fitchburg—Classroom formative assessment

Shrewsbury—Nightly homework & Mastery Learning

Grafton—Open response explanation writing

Milbury—Benchmark Assessment every 9 weeks, teacher reports

Maine: Mastery Learning for homework

What have I not told you about? Parent Notification System How teacher and schools put in their own content.

Stick around for the next 30 minutes to learn how to– Create our own account,– Let kids log in, and– Build your own content. Cristina Heffernan

Thank you

Extra slides

Cognitive Tutor – Carnegie Learning Inc.

Replicated Field Studies

Controlled, full year classroom experiments Replicated over 3 years in urban schools In Pittsburgh

& Milwaukee

Results:50-100% better on problem solving & representation use.

15-25% better on standardized tests.

Koedinger, Anderson, Hadley, & Mark (1997). Intelligent tutoring goes to school in the big city. International Journal of Artificial Intelligence in Education, 8.

0

1 0

2 0

3 0

4 0

5 0

6 0

I o w a S A T s u b s e t P r o b l e mS o l v i n g

R e p r e s e n t -a t i o n s

T r a d i t i o n a l A l g e b r a C o u r s eC o g n i t i v e T u t o r A l g e b r a

An Intelligent Tutoring System Success Cognitive Tutor Algebra Most widely used ITS

– 1000+ schools across the country– Marketed by local spin-off company

Carnegie Learning “Exemplary Curricula” by US Dept of Ed. In

What Works Clearing House Most cited Journal of AI-ED paper

– Koedinger, Anderson, Hadley, & Mark (1997). Intelligent tutoring goes to school in the big city.

.

.

.