learning trajectory-aligned diagnostic assessments for early algebra, grades 6-8

Scaling Up Digital Design Studies | NC State University | College of Education

Seminar on formative assessment tasks in mathematicsJanuary 5-7, 2016

Weizmann Institute of Science

The Challenge of Building and Implementing Learning Trajectory-Aligned

Diagnostic Assessments for Early Algebra, Grades 6-8

Jere Confrey Joseph D. Moore Distinguished University

ProfessorNorth Carolina State University

Research funded by:

http://sudds.ced.ncsu.edu/


The Team

Jere Confrey, Project DirectorRyan Seth Jones, Learning

ScientistGarron Gianopulos,

PsychometricianBasia Coulter, Visual Designer

Yungjae Kim, Lead Software Engineer

Pedro Larios, Software EngineerDoug Ivers, Software Engineer

Meetal Shah, Graduate Student



Scaling Up Digital Design Studies:• How we work

– Research-based– Rapid Prototyping– Agile Methods– Partnerships with

Teachers and Students

– Design Research



Success of the Future• A Stanford-hosted panel of highly successful, scientific technology

entrepreneurs--Astro Teller, inventor (Google), Christina Smolke, bioengineer (Stanford), and Steve Jurvetson, venture capitalist (DFJ General Partner), reflected on what learning is essential for success in tomorrow’s technology world. They agreed on the importance of problem selection, successful work in interdisciplinary teams, outstanding communication skills, and the ability to learn quickly. They also called for learners who are able to reconstruct knowledge from first principles and to pivot as needed to adjust to new possibilities and options.

• In a nutshell, Teller advised students about the nature of their future pursuits saying, “…if you plan for static stability, you are going to be really frustrated. But if you build the skills of dynamic stability, it is going to be awesome…”

• Jurvetson summarized their views of learning up saying “Iterative learning, that is, focusing on the process learning that correlates with success, will be the increasing locus of learning.”

(http://ecorner.stanford.edu/authorMaterialInfo.html?mid=3554)


http://ecorner.stanford.edu/authorMaterialInfo.html?mid=3554


How to leverage student thinking to

strengthen instruction?

TheChallenge



The SUDDS approach• Create a digital system that:

– Represents the big ideas in a learning map– Specifies the landmarks and obstacles in learning big

ideas over time and across grades based on research on learning trajectories

– Periodically diagnoses student progress accurately in real time

– Allows teachers to address key conceptions based on student thinking patterns

– Allows teachers to flexibly create fluid groups, based on student learning profiles



• A learning trajectory connects what students bring to instruction, to a target concept, and delineates a set of landmarks and obstacles that students are likely to encounter as they move from naïve to sophisticated understandings.

A Key Concept: A Learning Trajectory



• Insert corridor picture

A Learning Trajectory Depicted



Read the following problem, and predict how students will solve it.



Score of “0” on rubric—• Circles “Bigger” or “Smaller”,• Indicates a belief that to be equal, parts must be congruent, or • No discernible or intelligible distinctions

A score of “1” indicates that they make their prediction based on “qualitative compensation”--- one piece is wider and the other is taller. (no example)



transitivity argument: shares from both splits are still ½ of the same size brownie)…

Score of “2” on rubric: Student demonstrates that results of the two strategies are equivalent:

…or shows equivalence by compensation or decomposition



PEEQ: Property of Equality of Equipartitioning

• If two congruent shapes are each split for the same number of persons, then the size of each share from one of the shapes is equal to the size of each share from the other shape, regardless of the shape of the shares.



Equipartitioning a Rectangle for 4 people• Discuss the following solution to the task “Share a rectangle

(i.e. rectangular cake) for 4 people.” Did each person receive a fair share?

• How do you know?—Justify your answer.



Learning Trajectories

• Are not simply logical deconstructions of a math concept

• Surface as a result of use of appropriate tasks, tools, and forms of discourse in classrooms

• Not just erroneous, but have roots of productive thinking

• Can be transformative, leading to new possible insights

• Sometimes have to be unearthed, sensed, and pursued



Epistemological Objects of LTs

• What is observed or heard has a meaning to the child as a means to make sense of experience, and is a form of knowledge claim; therefore from the child’s perspective, it is an epistemological object. It permits the observer to imagine how the child is interpreting the problem and what follows from that.



• creating a need for the idea• connections to prior or related ideas• misconceptions and alternative conceptions• student built representations and

coordinating representations• mental models• strategies• sets of cases• generalizations and formalizations

Types of Epistemological Objects of LTs



What is the difference between this…?



…and this?



Leveraging LTs

• What structure is needed for digital curricula?

• Can we use learning trajectories to create such a structure?



What is the SUDDS 6-8 Learning Map?● A Learning Map is a navigational system that

helps students to explore the content to be learned organized in a learner-centered way.



● Anticipate and participate in what they are going to learn

● Experience personalization, not isolation● Find and use digital resources coherently ● Receive and use diagnostic feedback and

guidance● Form flexible, just-in-time groups

With a learning map, students can:



Map Structure

Fields (4)

Regions (9)

Related Standards

Related Learning Clusters (24)

Constructs (65)

Underlying Learning Trajectories and Indicator Levels



The Related Learning Clusters within the big ideas

©2014-2015 Jere Confrey and North Caroline State University



Regions and Clusters in AlgebraRepresenting Expressions, Equations, and Inequalities

Evaluating Equations with 2 Variables and Exponents

Exploring Relations and Functions

Representing and Using Linear Functions




Representing Expressions, Equations, and Inequalities




A Related Learning Cluster (RLC)Representing Expressions, Equations, and

Inequalities

The associated U.S. Standards

The proficiency levels of the learning trajectory, listed

from bottom to top

Four constructs




• Access to the map: www.sudds.co

• Make an account


http://www.sudds.co/


What is a Digital Learning System?

Internet Links

Confrey, November, 2015



Digital Learning Systems

• Not just components• Relationship among components• Relationship among users (students, teachers,

parents, administrators, researchers)• Prediction (planning forward: anticipation)• Feedback (results to different users for different

purposes )• Analytics and experimentation



Purpose of Diagnostics

• To identify in students and classes the patterns of performance that are known from research to block progress

• To place results in a taxonomy to ease interpretation

• To make actionable information available in real time

• To guide subsequent instructional decisions• To support formative practices



Taking a Diagnostic Assessment

Students take a 20-minute

diagnostic test to explore the ideas

from the cluster and demonstrate

their understanding.



Results for Algebra Field Testing

• Describe the field-testing setting• Describe the kinds of items used for algebra• Show results for each of the four constructs in the

select RLC• Draw some conclusions and implications



Field-Testing 2015-16

• Site: High-performing district in New Jersey• Two middle schools, each with ~700 students• These data are from 7th graders (13-14 year olds)• Data are collected near the end of instruction on

units• Results are used formatively only• Current forms of items limited to multiple choice,

numeric and select all.• (this work is in its infancy→we welcome

suggestions)



Epistemological objects in Early Algebra

Epistemological objects of Learning

Trajectories Examples from the first Algebra RLC

misconceptions 2(x + 3)= 2x + 3; (x + 3)/(x + 2) = 3/2

Representations and their coordination

Number lines, set notation, intervals, expressions, equations, drawn images; coordinating equations and number line and figural images

models Solving equations as “pan balances”; variables as unknowns.

strategies Guess and check, adjust; substitution

cases

x + a = c, bx = cax + b = ca(x + b) − cax + b = cxa(x + b) + c = dx + e, etc.

outliers -x = a; 1/x = b; -x < -b; a − (x−b) = c; b – x = c

generalizations Initial amount + (rate times number of stages) = total

formalizations Term, coefficient, range, all the properties



Results from “Describing Patterns and Relations…”

Describing Patterns and Relations Using Algebraic

Expressions (C53) Results

L1: Describes simple patterns (relation or function) verbally or with pictures or diagrams

(g.6)

67% matched a verbal description using x with the correct pattern relating number of tables and people.

L2: Describes simple patterns (relation of function) using x as

a variable (g.6)

A simple growing L was shown.94% choose the correct algebraic expression.

L3: Associates coefficients, factors, and terms in

interpreting verbal or pattern-based descriptions of

expressions with conditions (g.6)

86% mistakenly said 5 + 3x + 2 + x/3 had six terms. 46% correctly identified 3 as the coefficient of the second term while 67% also agreed that it was 3x, revealing some confusion. For coefficient of x/3, 46% incorrectly called it 3, while 29% correctly called it 1/3.

L4: Describes how coefficients, factors, and terms vary with

the conditions in the situation (g.6)

For a given algebraic description of a growing pattern (arrangements of tables and chairs), 65% selected new expression that added chairs in certain positions. (They were also provided a figure of the new patterns, so could have figured it out from those.)



• 67% of students correctly found the verbal description associated with the pattern (L1). Cognitive challenge: extend pattern or explain rate of change. One more table adds two more seats.



Results from “Finding Equivalent Expressions, etc.”

Finding Equivalent Expressions and

Substituting Values (C54) Data Summary

L1: Interprets situations algebraically with simple

expressions without parentheses.

Overall the students could translate simple expressions into algebra, but performance dropped to ~80% correct for subtraction (3 less than x) and division (the quotient of x and 4).

L2: Realizes that different forms of simple expressions,

including those with parentheses, may produce

equivalent results.

80% correct: recognize equivalence to x only for simple expressions (x × 1) + 0;Performance drops to 60-70% correct for slightly more complicated expressions ( x + 7) − 7 and 4x/4. One third incorrectly chose 2(2x) − 4x as equal to x.

L3: Substitutes for x and evaluates, applying order of

operations on simple expressions, including those

with parentheses.

Perimeter problem with two stated constraints, one using expressions: 77% of the students were able to select the one correct solution. 55% of students also identified a compelling distractor meeting only the first condition. For two other distractors meeting only the second constraint, each was selected by about 15% of the students.





L4: Interprets situations algebraically or substitutes values

systematically to test whether complex expressions are

equivalent, including: combining like terms and distributing

positive factors into parentheses.

For the expression 2(x + 1.50): 73% choose the correct match;13% chose the wrong match indicating a failure to understand the meaning of the coefficient 2.

L5: Interprets situations algebraically, or simplifies and

evaluates complex expressions, including: combining like terms and distributing positive factors.

In a matching exercise at this level, students were asked to match a complicated algebraic expression with its written description. 14% got three correct; individually on those three, 47%, 60% and 34% got each individual one correct. In a similar problem, 72% identified the correct expression if two parts were expressed separately;Only 40% saw the equivalence of other expressions. In two items linked to perimeter, performance feel to around 30%.






L6: Interprets situations algebraically, or simplifies and

evaluates complex expressions, including: combining like terms

and distributing positive and negative factors.

Task: select all equivalent expressions from a list for -2(-3x − 5):92% select correctly one answer of -6x − 10; Only 67% also recognized the alternative -6x + -10.19% chose 6x − 10.

These demonstrate some weakness in multiplying with negative numbers within expressions.

Task: solve 8 − 2(3x − 1): 37% were correct;41% choosing none of the above. 21% did not distribute or added the x’s incorrectly.




Finding Equivalent Expressions and Substituting Values: Sample item-- L3 Substitutes for x and evaluates, applying order of operations on simple expressions, including those with a parentheses,

• A triangle has a perimeter of 7. Two of the sides have integer lengths equal to x and x+1. Check all that could be lengths of three sides of the triangle. (55% of students selected this

incorrect choice-adds to 7)(77% of students selected this correct answer)(14%of students selected this incorrect choice – has consecutive values)(15%of students selected this incorrect choice- has consecutive values)

1) 1,1,5

2) 2,2,3

3) 1,3,4

4) 6,7,8



The Role of Context

• An interesting question for the LTs and items is when to include context in the problems.

• Sometimes context facilitates; sometimes it adds difficulty

• Our approach to this has varied. Sometimes we see it as inherent to the LT (in patterns, in translating expressions); sometimes we make it, its own level at a complex stage to check for integration of ideas.

• Needs future experimentation to inform its proper role in diagnostics.



Results from “…Equations in One Variable…”Representing and Solving Equations in One Variable

with Justification (C55)Results

L1: Understands x stands for a number that makes the equation

true.

Given the equation, 2(x + 4) = 3x, the, students were asked to choose one of four answers that solved the equation. 80% answered the problem correctly;11% believed there was no correct answer provided;7% answered 4, which might imply a failure to distribute.

L2: Solves simple equations by inspection and guess, check, and

adjust.

Students were asked to predict what number to try next. 98% responded by suggestion a number between the two given values. For similar but more difficult equation, 3x-5 = 2 (x-5), students are told the results for x = 12, x = 5 and x = 0, and each time the differences between the two sides of the equality are smaller. 38% recommended correctly that the next number should be less than zero;19% recommended trying number between 0 and 5;38% recommended trying number between 5 and 12.





L3: Maintains balance of equation in order to solve

equations of the form x + p = q (p, q, and x are rational

numbers).

In the given problem, students are told the equation in the form of x+1.5 = total time. They are asked how to find x, the number of hours worked. 73% of them answered correctly to subtract 1.5;19% suggested adding it; 5% wanted to divide by 1.5.

L4: Maintains balance of equation in order to solve

equations of the form px = q (p, q, and x are rational numbers).

Given the formula d × v = m, with values for volume (v) and mass (m), students were asked to find density.83% correctly knew to divide by volume, but 8% produced their answer by incorrectly multiplying the two numbers. Students were asked to convert the length of a marathon into kilometers and round. 50% were correct;33% more were within one kilometer suggesting they rounded incorrectly; 7% incorrectly multiplied, getting a smaller number of kilometers.





L5: Justifies solutions to simple equations using inverses and

identity properties to maintain balance.

For the problem 124/x = 4,just under 50% could identify both steps of the solution. Nearly 20% suggested dividing both sides by 124. (This approach could work, producing 1/x = 1/31.)At this level, 65% of students thought adding the same amount to both sides was justified as an additive inverse rather than an additive axiom of equality

L6: Uses strategies to isolate x on one side of equation in two or

more operations and maintains balance to solve.

Students were asked to solve for an unknown using a representation of a pan balance. When given unknowns on one side and a representation of the equation (3x+ 6 = 12),94% got them correct. When given the figures only and a problem representing 3 x+ 7= 5x+ 1, the percentage correct dropped to 84%.

L7: Describes situations using linear equations and solves.

Students were given an isosceles triangle with a base of n and told the other two sides were congruent and one was labeled 3n+4 equal to 28: 58% solved it correctly; 32% said there was not enough information, perhaps indicating a failure to use the information about congruent sides.





L8: Justifies method for solving equations with x on one side only using properties (inverse, identity, distributivity, commutativity, and

axioms of equality).

The students were asked to order the steps to solve 5 − 5x +2(7 − x) + 9 = 0: 87% got all 4 correct.

L9: Uses strategies to eliminate x's on one side of equation,

maintains balance to solve, and justifies.

Students were given 6w = -2w + 24:89% choose adding 2w to both sides correctly and 91% identified the correct solution. Students were also given the equation x+36+ x3=x+74 ,and asked to choose productive first steps from a set of options. 36% multiplied each term by a name for one (2/2, 4/4 and 3/3) to produce a common denominator. 43% multiplied both sides by 12. 12% recognized either as a correct option. 10% chose only multiplying by 12, which is defensible as a sole answer. 35%% of the students incorrectly selected subtracting 7 from both sides.19% wanted to divide by x.21% wanted to subtract 3.





L10: Interprets meaning of one solution, multiple solutions, and

no solution for equations with x's on both sides.

Students were given the problem 12x −15 = 3(4x − 5) and are told it has an infinite number of solutions. They are given other equations which differ in one value. They are asked which of the others will have no solution. The two correct answers change either the value 15 or 5.

73% recognized the effect of changing the 5;55% saw the change of the 15;Only 7% got them both.

On the second item at this level, students were shown a solution that results in a true statement and therefore infinitely many solutions.

66% picked infinitely many.24 said there was no solution.19% thought that the expression 6 = 6 implied that 6 was the solution.14% said “rework the problem” because there must have been a mistake.



Representing and Solving Equations in One Variable: L10: Interprets meaning of one solution, multiple solutions,

and no solution for equations with x's on both sides L7 Describes situations using linear equations and solves

Student responses:• 66% correctly chose

option 4, but only 44% only chose option 4.

24% of the students said there were no solutions19% thought that the answer was 6, 14% incorrectly said a mistake was made. and the student should start over.



Results from “…Inequalities in One Variable…”

Representing and Solving Inequalities in One Variable

with Justification (C56) Results

L1: Understands that x< a means that x can be equal to a whole

range of numbers.

Students were asked to match a set of descriptions of inequalities to their symbolic display.

Only 67% of students got all of these correct. 92% correctly matched “50 less than a number is no more than 7”;Only 70% correctly matched “ 7 more than a number is no more than 50” to its symbolic representation.“50 is less than or equal to a number plus 7” was a strong distractor for 14% of the students on this item.






L2: Identifies contexts in which inequalities apply.

Students were asked to identify which of a set of solutions that completely represent x ≤ 8. The solutions included two correct answers:

32% chose one that used the infinity symbol;68% chose a graph of a number line. The incorrect solutions included a variety discrete set of values ≤ 8, chosen by 60%, 31%, and 25% of the students. 8% misread the inequality sign choosing values greater than 8.

In the second item, students were asked to identify which solution sets contain values that make an inequality true:

79% chose the complete answer as one option. Shown four correct discrete-valued options,65% identified all correct integer values, 17% chose the graph with an open circle, 30% used the infinity sign, 60% chose 8 (which was on the boundary), 46% chose the single value of 3.5.

In a third example, the inequality 5 > x:86% choose values correctly, 14% misread the direction of the relationship.






L3: Solves simple inequalities by inspection, guess, check, and

adjust to find a range of solutions.

Students were given |x| > 5, and were asked to select all numbers that made the inequality true. 85% chose 6. 62% chose -6.(23% more students chose 6 than -6)58% choose 23/4. Overall, only 42% got full credit.Given 3x − 4 ≥ 8, Only 42% chose the correct solution. 39% of incorrect responders chose x≤ 8.

L4: Maintains balance of equation in order to solve Inequalities of the form x + p = q (p, q, and x are rational numbers).

No items at this level






L5: Maintains balance of equation in order to solve Inequalities of the

form px <> q (p, q, and x are rational, including switching

inequality for negative multipliers).

Students were highly successful for b/3 ≥ 12: 96% correctly chose b>36, 91% selected b = 36. In inequalities, it may be the case that this form is easier than the x+p. For the item -y < 16, the problem requires students to multiply or divide by -1 and know to reverse the sign, unless they figure out the solution by trial and error.Only 16% answered this item correctly;63% answered incorrectly y < 16.10% chose y > 16 (a partial solution but students have to understand the item demands a complete solution.11% choose y < -16.

L6: Justifies solutions to simple inequalities using inverses and identity properties to maintain balance.


L7: Uses strategies to isolate x for inequalities with x on one side of Inequality and maintains balance to solve.







L8: Describes situations using linear inequalities and solves.

In the tee-shirt canon item, the majority of students found the correct values to solve the given equation (10x + 15 ≥ 45).

93% selected the boundary condition correctly, but 85-88% answered completely correct, suggesting difficulty with selecting the interval and not just the boundary value.

When requested to identify the correct inequality in a context of attending a fair, student performance dropped.

30% got full credit for identifying two possible inequalities—77% chose one correct option, 54% chose the other. For an item about bungee jumping, (2x + 6 + 20 < 300), 56% correctly answered the problem. 31% incorrectly gave the solution that the bungee was < 300 (giving as the solution the right side of the original equation).






L9: Justifies method for solving inequalities with x on one side only using properties (inverse, identity, distributive property, commutativity, and axioms of inequality.

When students were given a set of steps solving an inequality using properties:

89% correctly chose the distributive property, only 28% correctly chose the additive property of inequality for the second step. 58% chose the additive inverse incorrectly. Only 32% correctly chose the additive identity;63% correctly chose the division axiom of equality.

When students were asked to debug a student solution for finding x for -4(p+3) + 6 ≥ 22:

55% correctly found the error in the first step on distributing a negative number;22% mistakenly said the last step, where the inequality was reversed, represented the error.






L10: Uses strategies to eliminate x on one side of inequality, maintaining balance to solve, and justifies.

For 4x + 40 < -8x – 8: 61% solved correctly; 21% failed to correctly reverse the inequality sign.

For 3x + 1 > 5x – 3:79% correctly solved, but 18% chose either x > 4 or x < 4.



Student Difficulties with Ranges and Intervals in Inequalities

Two examples

EX. 1: which solution set contain values that make the inequality true

EX 2: which of the following represents a complete solution



Student responses:• 79% chose the complete answer as one

option. • 65% identified all correct integer values.• 17% chose the graph with an open circle.• 30% recognized the use of the infinity sign. • 60% selected the value of 8 on the

boundary.• 46% chose the single value of 3.5



Student responses:• 32%: the choice using the infinity symbol.• 68%: the number-line graph with a closed

circle. • 60%: (incorrectly) selected {…, -2, -1, 0, 1,

2, 3, 4, 5, 6. 7, 8}, (integer domain)• 31%: the single value 8.• 25%: the single value 3.5. • Only 2 of 62 students selected both correct

answers with no incorrect answers.



Overall Results for RLCSummary Table for Results for Four Constructs

Representing and Solving Equations in One Variable with Justification

Can substitute to solve showing some weakness on distributivity.

Understands the pan balance model.Need work on atypical equations with x in

denominator of a direct variation or with fractional terms.

Showed weakness applying to geometric contextNeed work on multiple or no solutions.

Representing and Solving Inequalities in One Variable with Justification

Can solve simple inequalities. Have difficulty with flipping the sign.Need work on distributivity and axioms of

inequality.Need practice on writing inequalities in applied

contexts.

Describing Patterns and Relations Using Algebraic Expressions

Can solve only simple patterns.Lack knowledge of formal terminology associated

with the structure of the expressions that support later parameterization of expressions.

Demonstrate only rudimentary understanding of how to adjust parameters in patterns and explain structural changes.

Finding Equivalent Expressions and Substituting Values

Translates easy expressions showing weakness in subtraction.

Substitutes values successfully.Recognizes equivalent expressions only in simple

cases and misses distributivity and multiplication by negatives.

Has some difficulty associating expressions with complex diagrams and contexts.



Implications of the Diagnostic Cycle

• It functions in support of a set of formative assessment practices:– Process oriented and criterion-oriented– Occurs in real time– It supports discourse– Shares responsibility with students

• Adds to formative assessment:– Personalizes results to students– Shows representativeness for whole class– Supports the creation of flexible groups– Documents progress over time



The Great Challenge

• What to do with this information?

• We will be studying this in partnership with our districts and schools next.

• Invite me back in 2-3 years???



Conclusions• Important to systematically make the results of

research on students’ learning trajectory accessible to students and teachers;

• Learning trajectories are useful ways to encapsulate empirical results on learning;

• Learning Maps can help students and teachers visualize content to be learned around big ideas and anticipate reasoning patterns;

• Diagnostic assessments tied to proficiency levels in LT can provide just in time information to guide instruction; and

• Diagnostic information can help know what to reteach, how to target information for individual needs, how to promote formative discussions, and how to form flexible groups.

• Much work remains, to translate into instructional actions.



• Thank you for your time and attention.
