a logical perspective - computer science · intelligent tutoring systems joint work with jain,...

Intelligent Tutoring Systems A Logical Perspective

Sumit Gulwani

Microsoft Research, Redmond

July 2012

• Application domain – Software Developers

– End-Users

– Education

• Specification of intent – Logic

– Examples

– Natural Language

• Search techniques – SAT/SMT solvers (Formal Methods)

– A*-style goal-directed search (AI)

– Version space algebras (Machine Learning)

1

Background: Program Synthesis

WAMBSE 2012: “Synthesis from Examples”, Gulwani.

• Interactive Feedback – Answer-scripts graded immediately.

– Students provided feedback during solution generation

• Exploration – Interactive Visualization Tools

– Simulation Tools

• Social – Students working on same problem at same time can come

together.

– Learning experience for past students can be used to benefit the learning experience for future students.

2

Value Proposition

• Various Aspects – Solution Generation

– Grading/Feedback

– Problem Generation

– Content Entry

• Various Domains – Geometry, Algebra, Physics, Chemistry, Logic

• Partnership with UW and IITK for deployment in schools

– Programming, Automata, Excel, SQL • Partnership with MIT, UPenn, UIUC for EdX/internal deployment

– Language Learning • Interest from Office Education Team

3

Intelligent Tutoring Systems




– Content Entry





4


Given a triangle XYZ, construct circle C such that C passes through X, Y, and Z.

5

Ruler/Compass based Geometry Constructions

X

Z

Y

L1 L2

N

C

PLDI 2011: Gulwani, Korthikanti, Tiwari.


Precondition:

Slope(X,Y) Slope(X,Z) Æ Slope(X,Y) Slope(Z,X)

Postcondition:

LiesOn(X,C) Æ LiesOn(Y,C) Æ LiesOn(Z,C)

Where LiesOn(X,C) ´

Distance(X,Center(C)) = Radius(C)

Formal specification of the problem

6

Geometry Program: A straight-line composition of geometry methods.

Geometry Types: Point, Line, Circle

Geometry Methods:

• Ruler(Point, Point) -> Line

• Compass(Point, Point) -> Circle

• Intersect(Circle, Circle) -> Pair of Points

• Intersect(Line, Circle) -> Pair of Points

• Intersect(Line, Line) -> Point

7

Solution is a program!


8

Example of Geometry Program

1. C1 = Compass(X,Y); 2. C2 = Compass(Y,X); 3. <P1,P2> = Intersect(C1,C2); 4. L1 = Ruler(P1,P2); 5. D1 = Compass(Z,X); 6. D2 = Compass(X,Z); 7. <R1,R2> = Intersect(D1,D2); 8. L2 = Ruler(R1,R2); 9. N = Intersect(L1,L2); 10. C = Compass(N,X);

X

Z

Y

C1 C2 P1

P2

L1

D2

D1 R1

R2

L2

N

C




– Content Entry





9


10

Classic Problem in Automata Theory Course

Solution Generation Engine

Let L be the language containing all strings over {a,b} that have the same number of occurrences of “ab” as occurrences of “ba”. Construct an automata that accepts L, or prove that L is non-regular.

“Regular”, <Automata for L>

Joint work with Cerny, Henzinger, Zufferey, Radhakrishna

11

Classic Problem in Automata Theory Course

Solution Generation Engine

Let L be the language containing all strings over {a,b} that have the same number of occurrences of “a” as occurrences of “b”. Construct an automata that accepts L, or prove that L is non-regular.

“Non-regular”, <Proof of non-regularity>




– Content Entry





12


Joint work with Jain, Singh, Sharang, Sharma, Karkare, Roy




– Content Entry





13


14

Solution Generation by Example

Joint work with Erik Andersen and Zoran Popovic




– Content Entry





15





– Content Entry





16


17

Background: PexForFun

using System;

public class Program {

public static int[] Puzzle(int[] a) {

int[] b = new int[a.Length];

int count = 0;

for(int i=a.Length; i < a.Length; i--)

{

b[count] = a[i];

count++;

}

return b;

} }

18

Buggy Program for Array Reverse

6:28::50 AM

using System;




int count = 0;

for(int i=a.Length-1; i < a.Length-1; i--)

{

b[count] = a[i];

count++;

}

return b;

} }

19


6:32::01 AM

using System;




int count = 0;


{

b[count] = a[i];

count++;

}

return b;

} }

20


6:32::32 AM

No change! Sign of Frustation?

using System;




int count = 0;

for(int i=a.Length; i <= a.Length; i--)

{

b[count] = a[i];

count++;

}

return b;

} }

21


6:33::19 AM

using System;




int count = 0;


{ Console.Writeline(i);

b[count] = a[i];

count++;

}

return b;

} }

22


6:33::55 AM

Same as initial attempt except Console.Writeline!

using System;




int count = 0;



b[count] = a[i];

count++;

}

return b;

} }

23


6:34::06 AM

No change! Sign of Frustation?

using System;




int count = 0;

for(int i=a.Length; i <= a.Length; i--)


b[count] = a[i];

count++;

}

return b;

} }

24


6:34::56 AM

The student has tried this before!

using System;




int count = 0;


{

b[count] = a[i];

count++;

}

return b;

} }

25


6:36::24 AM

Same as initial attempt!

using System;




int count = 0;


{

b[count] = a[i];

count++;

}

return b;

} }

26


6:37::39 AM

The student has tried this before!

using System;




int count = 0;

for(int i=a.Length; i > 0; i--)

{

b[count] = a[i];

count++;

}

return b;

} }

27


6:38::11 AM

Almost correct! (a[i-1] instead of a[i] in loop body)

using System;




int count = 0;

for(int i=a.Length; i >= 0; i--)

{

b[count] = a[i];

count++;

}

return b;

} }

28


6:38::44 AM

Student going in wrong direction!

using System;




int count = 0;


{

b[count] = a[i];

count++;

}

return b;

} }

29


6:39::33 AM

Back to bigger error!

using System;




int count = 0;


{

b[count] = a[i];

count++;

}

return b;

} }

30


6:39::45 AM

No change! Frustation!

using System;




int count = 0;


{

b[count] = a[i];

count++;

}

return b;

} }

31


6:40::27 AM

No change! More Frustation!!

using System;




int count = 0;


{

b[count] = a[i];

count++;

}

return b;

} }

32


6:40::57 AM

No change! Too Frustated now!!! Gives up.

Provides additional value over counterexample feedback.

• More friendly feedback. – Helpful for students who give up after several tries

(with only counterexample feedback).

• Grading – Counterexample feedback does not distinguish

between a slightly incorrect solution and one that is very far off from being correct.

33

Proposal: Semantic Grading

Array Index: v[a] -> v[{a+1, a-1}]

Increment: v++ -> { ++v, v--, --v }

Conditional: a op b -> a ops b

where ops = { <, >, <=, >=, ==, != }

Initialization: v=n -> v={n+1, n-1, 0}

Return Value: return v -> return ?v

34

Example Errors

Joint work with Rishabh Singh and Armando Solar-Lezama

Automated Grading (Array Reverse)

i = 1

i <= a.Length

--back

front <= back




– Content Entry





36


Example Problem: sec 𝑥 + cos 𝑥 sec 𝑥 − cos 𝑥 = tan2 𝑥 + sin2 𝑥

Query: 𝑇1 𝑥 ± 𝑇2(𝑥) 𝑇3 𝑥 ± 𝑇4 𝑥 = 𝑇5

2 𝑥 ± 𝑇62(𝑥)

𝑇1≠ 𝑇5

New problems generated:

csc 𝑥 + cos 𝑥 csc 𝑥 − cos 𝑥 = cot2 𝑥 + sin2 𝑥 (csc 𝑥 − sin 𝑥)(csc 𝑥 + sin 𝑥) = cot2 𝑥 + cos2 𝑥 (sec 𝑥 + sin 𝑥)(sec 𝑥 − sin 𝑥) = tan2 𝑥 + cos2 𝑥

:

37

Trigonometry Problem

AAAI 2012: Rohit Singh, Sumit Gulwani, Sriram Rajamani

38

Algebra Problem Generation

Example Problem

Query Generation

Query

Query Execution

New Problems

Query Refinement Results OK?

Refined Query

No

Yes

Similar Problems

Example Problem: lim𝑛→∞ 2𝑖2 + 𝑖 + 1

5𝑖= 5

2

𝑛

𝑖=0

Query: lim𝑛→∞ 𝐶0𝑖2 + 𝐶1𝑖 + 𝐶2

𝐶3𝑖

= 𝐶4𝐶5

𝑛

𝑖=0

C0≠ 0 ∧ gcd 𝐶0, 𝐶1, 𝐶2 = gcd 𝐶4, 𝐶5 = 1


lim𝑛→∞ 3𝑖2 + 2𝑖 + 1

7𝑖= 7

3

𝑛

𝑖=0

lim𝑛→∞ 3𝑖2 + 3𝑖 + 1

4𝑖= 4

𝑛

𝑖=0

lim𝑛→∞ 𝑖2

3𝑖= 3

2

𝑛

𝑖=0

lim𝑛→∞ 5𝑖2 + 3𝑖 + 3

6𝑖= 6

𝑛

𝑖=0

39

Limits/Series Problem

Example Problem: (csc 𝑥) (csc 𝑥 − cot 𝑥) 𝑑𝑥 = csc 𝑥 − cot 𝑥

Query: 𝑇0 𝑥 𝑇1 𝑥 ± 𝑇2 𝑥 𝑑𝑥 = 𝑇4 𝑥 ± 𝑇5(𝑥)

𝑇1≠ 𝑇2 ∧ 𝑇4 ≠ 𝑇5


(tan 𝑥) (cos 𝑥 + sec 𝑥) 𝑑𝑥 = sec 𝑥 − cos 𝑥

(sec 𝑥) (tan 𝑥 + sec 𝑥) 𝑑𝑥 = sec 𝑥 + cot 𝑥

(cot 𝑥) (sin 𝑥 + csc 𝑥) 𝑑𝑥 = sin 𝑥 − csc 𝑥 40

Integration Problem

Ex. Problem 𝑥 + 𝑦 2 𝑧𝑥 𝑧𝑦

𝑧𝑥 𝑦 + 𝑧 2 𝑥𝑦

𝑦𝑧 𝑥𝑦 𝑧 + 𝑥 2 = 2𝑥𝑦𝑧 𝑥 + 𝑦 + 𝑧 3

Query 𝐹0(𝑥, 𝑦, 𝑧) 𝐹1(𝑥, 𝑦, 𝑧) 𝐹2(𝑥, 𝑦, 𝑧)𝐹3(𝑥, 𝑦, 𝑧) 𝐹4(𝑥, 𝑦, 𝑧) 𝐹5(𝑥, 𝑦, 𝑧)𝐹6(𝑥, 𝑦, 𝑧) 𝐹7(𝑥, 𝑦, 𝑧) 𝐹8(𝑥, 𝑦, 𝑧)

= 𝐶10𝐹9(𝑥, 𝑦, 𝑧)

𝐹𝑖≔ 𝐹𝑗 𝑥 → 𝑦; 𝑦 → 𝑧; 𝑧 → 𝑥 𝑤ℎ𝑒𝑟𝑒 𝑖, 𝑗 ∈ { 4,0 , 8,4 , 5,1 ,… }


𝑦2 𝑥2 𝑦 + 𝑥 2

𝑧 + 𝑦 2 𝑧2 𝑦2

𝑧2 𝑥 + 𝑧 2 𝑥2 = 2 𝑥𝑦 + 𝑦𝑧 + 𝑧𝑥 3

𝑦𝑧 + 𝑦2 𝑥𝑦 𝑥𝑦

𝑦𝑧 𝑧𝑥 + 𝑧2 𝑦𝑧

𝑧𝑥 𝑧𝑥 𝑥𝑦 + 𝑥2 = 4𝑥2𝑦2𝑧2

41

Determinant Problem




– Content Entry





42


1. The principal characterized his pupils as _________ because they were pampered and spoiled by their indulgent parents.

2. The commentator characterized the electorate as _________ because it was unpredictable and given to constantly shifting moods.

(a) cosseted

(b) disingenuous

(c) corrosive

(d) laconic

(e) mercurial

43

Sentence Completion Problems

Joint work with Geoffrey Zweig and Vikram Rao Sudarshan




– Content Entry





44


tan 3𝑥 tan 2𝑥 tan 𝑥 = tan 3𝑥 − tan 2𝑥 − tan 𝑥

𝑦𝑧 − 𝑥2 𝑧𝑥 − 𝑦2 𝑥𝑦 − 𝑧2

𝑧𝑥 − 𝑦2 𝑥𝑦 − 𝑧2 𝑦𝑧 − 𝑥2

𝑥𝑦 − 𝑧2 𝑦𝑧 − 𝑥2 𝑧𝑥 − 𝑦2

𝐴1 sin

3 𝛼 𝐵1 sin3 𝛽 𝐶1 sin

3 𝛾𝐴2 sin 𝛼 𝐵2 sin 𝛽 𝐶2 sin 𝛾

45

Mathematical (Syntactic) Intellisense

Joint work with Alex Polozov and Sriram Rajamani

Prove (csc 𝑥 − sin 𝑥)(sec 𝑥 − cos 𝑥)(tan 𝑥 + cot 𝑥) = 1

L.H.S. = 1

sin 𝑥− sin 𝑥

1

cos 𝑥− cos 𝑥

sin 𝑥

cos 𝑥+cos 𝑥

sin 𝑥

= 1 − sin2 𝑥

sin 𝑥

1 − cos2 𝑥

cos 𝑥

sin2 𝑥 + cos2 𝑥

cos 𝑥 sin 𝑥

= cos2 𝑥

sin 𝑥

sin2 𝑥

cos 𝑥

1

cos 𝑥 sin 𝑥

= 1

46

Mathematical (Semantic) Intellisense




– Content Entry





47


CHI 2012: Cheema, Gulwani, LaViola

• Technical Perspective – Aspects: Solution+Problem Generation, Grading, Content Entry

– Domains: Math/Science, Programming, Language Learning

• Value Proposition: – Short term: Interactive Feedback, Exploration, Social

– Long Term: Ultra-intelligent computer, Model of human mind, Inter-stellar travel

• Crowdsourcing Opportunities – Solution Generation: Natural Language to Logic translation

– Problem Generation: Rate difficulty and ambiguity level

– Automated Grading: Provide natural explanations for errors

48

Conclusion

a logical perspective - computer science · intelligent tutoring systems joint work with jain,...

Documents