microsoft faculty research summit 2007 adaptive book: a platform for teaching, learning and student...

Microsoft Faculty Research Summit 2007

Adaptive Book: A Platform for teaching, learning and student

modelingAnanda Gunawardena

School of Computer ScienceCarnegie Mellon University

&Vince DeStasio

CIO and Associate Professor in ChemistryGrove City College

Pennsylvania


Outline

• Introduction• What is Adaptive Book?• Adaptive Book Authoring Tools• Just in time learning modules• Adaptive Markup Repository• Assessment of Student Reading• Pilot Results• Future Work


Adaptive Book Project

• Tablet PC is an ideal platform for electronic textbooks

• Ability to use the pen to annotate, highlight and share markups

• Online textbooks cannot be– Just the pdf versions of their printed versions

• Flexible but Limited use

• Cheap but hard to read

• Our efforts is to design the textbook of the future


Introduction• Effective teaching is not confined to the

classroom—it is successful when it enhances interest and generates thinking beyond the classroom setting.

• A great deal of customization and personalization of the content is needed to generate the interest

• Adaptive Book is a platform for customization, personalization and understanding of student behavior


Adaptive Book• A software platform developed in C# / .net

– Developed by TextCentric,Inc (CMU Spin-off)

• Research Base at Carnegie Mellon– Usability (Human Computer Interaction)– Adaptive (Machine Learning)

• Adaptive Book User Interface– Navigation, search, markup tools (highlighting,

annotating, and linking), book marking– Labeling, archiving and searching markups– Adaptive Book combines textbook content with all other

related material


Adaptive Book UI


Survey Slide 1• What do you think the students initial

reaction to Adaptive Book was?– A) They remained skeptical – B) They embraced it whole-heartedly and could

immediately see the benefit – C) They liked the idea, but it had too many

"kinks" to be immediately useful – D) All of the above – E) none of the above


Creating an Adaptive Book

• Adaptive Book is a thin client program

• Any SCORM/IMS content package can be imported into Adaptive Book

• A content authoring tool creates these content packages.

• In other words anyone can create an Adaptive Book package using their own content and/or textbook chapters

Microsoft Faculty Research Summit 2007Instructor content

Publisher content

professor

Rights and RoyaltyManagement

Adaptive Book


Adaptive Book Authoring Tool

• Uses a HTML/XML content repository to find chapters of the book and supplements

• Each chapter of the book is tagged as a Sharable Content Object (SCO)

• Uses a simple drag and drop menu to select the chapters and supplements needed to create the custom book

• Program generates the table of content and package the book as a SCO


Importing a SCO


Survey Slide 2

• Do you think combining textbook chapters with your own material is a useful way to package a course– A) Strongly Agree– B) Agree– C) Neutral– D) Disagree– E) Strongly Disagree


Markup Concept • Research shows marking up a text while reading

enhances the learning• Markup is defined as a semantically related set of

objects consists of highlights, annotations and web links.• Markups automatically generate its own metadata as

well as anyone can add other metadata to markups before saving

• Markups are stored in a searchable repository.• Search and find markups related to a certain concept

– What is the “best” markup to learn topic A?– A search algorithm specifically designed to find related markups

• Assign Rights to markups– Private, public, protected


Sample Markup


Students Sign up for Markup Services


Students can create buddy groups


Students Define their buddy groupor

Who has the access to their markups


Mark-up List


Assigning Rights to Markups


Reading a markup


Survey Slide 3• If you have the ability to markup the textbook and

related material, how would you use the feature– A) I would use it to make notes for myself– B) I would markup the text and send it to students

before class– C) I would markup the text as an answer to a frequently

asked question and share– D) I would create a repository of reusable markups that

addresses various concepts– E) I would encourage students to markup the text as

they read the book– F) I do not find this feature useful– F) Other


Just in time Learning Modules

• Markups are important learning tools• Students learn better from marked up text• Worksheets to reading the book

– This is a compromise

• Instructors can create just in time learning modules using markup tools

• Select content, highlight, place sticky notes and URL’s and package them as a markup object

• Save the markup to a customized learning objects (CLO) repository

• Others can search and find the markup, disassemble and customize to their needs


A Just-in-time learning module with highlights, and digital ink notes


More on Markups

• Markups with learning paths can be created by individual instructors

• Search and find the Markup, and import to Adaptive Book


Learning Objects Repository

• Three types of learning objects stored in the repository

• Book Chapters that can be used to build a custom book

• Individual markups packaged as IMS/SCORM with access rights

• Customized Learning Objects (CLO’s) that can be retrieved and modified using Adaptive Book


Modeling Student Behavior• One of the focuses of our research

– Do students read the book and other notes?– If so, what do they read? not pages, what

specific content?– How do they construct knowledge?

• Does the Tablet PC help students move from informal sketching to formalization?

– What can we do to capture that?


Modeling Student Behavior• If we are successful in capturing an accurate assessment

of student thinking, what can we do with that data?• Our pilots involve constructing activities that “requires” the

student to show specific things in the textbook and notes that they found useful

• Student markup is then compared to an “expert” markup• Our preliminary data show some interesting correlation

between student performance and their perception of what was important in completing the assignment

• More work is needed to accurately model student behavior• This is ongoing research


Pilot Results

• We have used Adaptive Book and markup repository with several institutions and middle schools

• Many positive Student Comments but lot needs to be done

• Interesting results on what students read

• Sample comments…


Some Student Comments• “I like the ability to find a relevant markup that can

help solve some of the programming problems”• “It is nice to receive markups from the instructor

before the lecture”• “Good thing about the Adaptive Book is that

instructor can create a markup linking textbook concepts and relevant programming examples”

• “receiving an answer to a FAQ as a markup is a very useful feature”

• “Adaptive Book search feature needs improvements”


Future Work

• More focus on the impact of Tablet PC and Adaptive Book Technology

• Making Adaptive Book a research platform for modeling student behavior

• Creating a dynamic sketch environment for conceptual understanding and automatic code generation

• Informal to formal stages of learning• Better markup analysis and classification


Ink-based Technology• Fundamental Questions

– Do ink-based technologies improve teaching and learning?• What are the differences and similarities with standard PC-

based technology in education?• How do ink-based technologies compare/compete with

paper?• Two promises

– Better document management• Electronic repositories

– Sketch recognition• Smart paper• Real time feedback on ink

• Perhaps a better platform for interacting with electronic documents– ebooks, note taking etc..


Carnegie Mellon Projects• HP Technology for Teaching Grant – 2004

– 21 HP 1100 Tablet PC’s– Other equipment

• HP Technology for Teaching Leadership grant - 2005– 42 HP 4200 Tablet PC’s– Other equipment

• Qatar Foundation Grant - 2005– 25 HP 4200 Tablet PC’s and Research funding

• Microsoft Research Grant - 2006– $150,000 funding for a 2-year impact study– With Grove City College Tablet PC program


HP Technology for Teaching Award2004

• 2004-2005 study of the integration of Tablet PC’s

• 38 Computer Science Freshman participated in the study

• Students “owned” the tablet PC for the duration of the pilot (typically one semester)– 85% of the CMU students who participated in

the pilot owned laptops– 60% of the students used “tablet PC” as their

primary machine



• Ink-based Software used– Adaptive Book Technology (CMU-TextCentric)– MS Windows Journal

• Primary activities– Reading and annotating the book– Creating and saving worksheets– Sharing worksheets using a repository

• Motivation– CMU bought 1 Tablets PC that was raffled at the end of the

semester– Those who contributed to the ink-based worksheet repository

received extra coupons for the raffle.



• Research Findings– 38 students surveyed– 60% of the students used Tablet as their primary device– We used a control group of 20 students who did not have tablet

PC’s– taught by the same professor (Don Slater), same introductory

programming course– Final analysis showed slight improvement in grades in the Tablet

group. But not statistically significant to draw conclusions– 85% of the students liked the idea of electronic note taking

• Easy organization, storage and retrieval

– 45% of the students would consider buying a tablet PC, next time they shop for a PC

• Smaller screen and performance were cited as reasons for not buying a tablet


HP Technology for Teaching Leadership Award

2005-2006• 42 HP Tablet PC’s• Target Group

– Sophomore/Junior students taking data structures and algorithm courses

– Initial survey done to select 35 students with varied criteria's• Major and GPA• Ownership of PC• General attitude towards electronic material

– Control group is 40 students who were not selected or did not want to participate

• Mac’s, unix owners etc.



2005-2006• Ink based Software tools

– Adaptive Book with Markup analysis tools– Classroom presenter– Windows journal– Data structure visualizer

• Customization of MIT Physics illustrator project

• Status of the project– Ongoing– Preliminary survey data available– Student markup analysis data available



2005-2006• Support tools

– Configured with CMU’s wireless cluster network– All Tablets belong to a virtual cluster network

• Pilot study in understanding use of mobile devices as computer cluster machines (Gates building)

• Automated configurations– Network isolation– Software access privileges

– Automated backup scripts• All related journal notes collected for analysis



2005-2006• Research Activities

– Pre-lecture markup– Post-lecture markup– Automated analysis of markups

• With an expert as well as with community• Various data mining techniques employed

• Research Question– What is the impact of pen-based technology in learning?

• Ink versus paper• More reading and markup activity• Electronic note taking

• Methods– Surveys of users and non-users of tablet technology– Analysis of pre and post lecture markup activities


Qatar Foundation Grant2005-

• 25 HP Tablet PC’s at CMU-Qatar campus in Education City• Entire CS sophomore class had the tablets• Observations

– Immediate acceptance of the technology• Only 1 out of 20 students took paper notes• Had a hard time getting the machines back at the end of the

year• Students used electronic note taking in all classes

– Classroom presenter used to create interaction• Send/receive annotations – professor-students• Great for middle east

– Adaptive Book was used to increase• Textbook reading and interaction• Reading comprehension


Qatar Foundation Grant2005-

• Informal Study Results• Potential benefits

– Teaching• Ink grading and archiving of homework• Archiving of all classwork• Pittsburgh TA’s used/uses IM tools with ink

– Learning• Use of classroom presenter tool• Ability to annotate English textbooks in Arabic• Improving reading comprehension

• Results


Microsoft Research2-year assessment study

2006-2008• Collaborative project

– Carnegie Mellon University– Grove City College

• Research subjects from Grove City College– 2400 HP 4200 Tablet PC’s

• Data from over 1200 students• Wide range of disciplines

– CS, Economics, English, Business• Software

– Adaptive Book (unlimited licenses)– DyKnow vision (600 licenses per year)


Other related Projects• Ellis School 8th Grade Geometry project

– Adaptive Book (CMU/TCI)– Geometers sketch pad (Key Press)– Classroom presenter (UW)– references


What is the future of textbooks

• Textbook is an integral component of a course– It often gives a different viewpoint with more details– Extra examples, problems and solutions– It is one place students can find answers to most of

the questions

• However, future of textbooks are threatened by online content– Wikipedia’s– Google’s


What is the future of textbooks?

• It is clear that use of online content by students are growing

• It is also clear that textbooks demand is going down– Publishers increase the price– Create extra online content and sell the book

even higher prices


Electronic Books• At Carnegie Mellon we are designing the future

textbook using Human Computer Interaction principles

• We see textbook of the future as a platform• We focus on three things

– Usability– Adaptability– Community

• Online textbooks can be extremely useful and much superior to its printed version, when all three conditions are met

• Tablet PC’s and new fonts make reading much closer to the experience of reading a printed book


Usability

• Markup tools– Highlights– Short notes– Digital Review Sheets

• Navigation and Search tools– Search by keyword– Search by concepts– Search by markups

• Multiple Views– Typical textbook view– Question view– Concept view


Adaptability

• How good is my markup?• How do I know I am reading what I am

supposed to read?• Answer: Use the markup analysis tool

– Create a markup– Submit (or synchronize with) to a website– Compare to an “expert” markup– Compare to a “community” markup

• Lots of applications– Measuring the reading comprehension


Community• Electronic textbooks are platforms.

• Find out exactly what other people are reading

• Comment about content online

• Find community markups to help study

• Classify markups– Private– Protected – public


Security of Content

• Digital Rights Management– Publishers biggest worry

• Tablet PC’s and other ink based devices can help establish the authenticity of the user– Digital ink signatures– Early research


Adaptive BookThe Thin Client

Professor

3a.Definitions

3b. Important

3c. Concepts

Mark-ups

Sticky Notes

Student

Textbooks

ResearchVPN*

StudentStudent

StudentStudent

Assignments

3. Professor mark up Content

4. Professor ask questions, make assignments.

2. Professor’s Lesson Plan

6. Students Read Content(don’t see highlighted text)

1. Content

Test

7. Students learn Content with AB

Professor monitor’ individual student’s progress.

Self-paced remediation without reprisal!

Case Studies

*Virtual Private Network

8. Students are Tested

3d. Key Words

Professoruses highlighter penfor develop learningpaths (Mark-ups),

Dean


Case Study

CMU Computer Science

17 StudentsFall Semester 2006


The Ellis Survey

7 Actual Users


The Ellis School

Survey: HP Tablet PC with Adaptive Book Thin Client 6 Students 37 Questions and 214 or 96%

Text Books19%

Markups38%

HP Tablet19%

AB Features24%


The Survey37 Questions

Ellis Student Survey1= Not Important and 5 = Very

-

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

Survey Questions

Avera

ge S

core

37. The lease important feature, the ability to change the text font size.

1. The Tablet PC is a Notebook PC and More

35. Sharing Markups

36. Using Digital Books

2. Highlight text

3. Extends use


Adaptive Book&

Tablet Math Whiz

Ananda Gunawardena

School of Computer Science

Carnegie Mellon University


Tablet MathWhiz

certified

remediate

Adaptive book

Markup

repository

pretest

professor

group


Adaptive Book• A software platform developed in C# / .net

– Developed by TextCentric,Inc (CMU Spin-off)

• Research Base at Carnegie Mellon– Usability (Human Computer Interaction)– Adaptive (Machine Learning)

• Adaptive Book User Interface– Navigation, search, markup tools (highlighting,

annotating, and linking), book marking– Labeling, archiving and searching markups– AB combines textbook content with all other related

material


Adaptive Book UI


Creating an Adaptive Book

• Adaptive Book is a thin client program• Any SCORM/IMS content package can be

imported into Adaptive Book• An authoring tool creates these content

packages. Instructors create their custom Adaptive Book

• Anyone can create an Adaptive Book package using their own content and/or textbook chapters


Importing a SCO


Markup Concept

• Markup is a semantically related set of objects consists of highlights, annotations and web links.

• Markups automatically generate its own metadata as well as anyone can add other metadata to markups before saving

• Markups are stored in a searchable repository.• Search and find markups related to a certain concept

– What is the “best” markup to learn topic A?– Google type search engine

• Assign Rights to markups– Private, public, protected


Sample Markup


Submit your Markup


Get your markup Graded


Questions: [email protected]

Thank You

mailto:[email protected]


Current Collaborators: Current Collaborators:

Andrew Owens (Cornell)Andrew Owens (Cornell)Jay Heo, Dan Horbatt & Chantelle Humphreys(CMU)Jay Heo, Dan Horbatt & Chantelle Humphreys(CMU)

Research Partially supported by: Qatar Foundation, Microsoft, HP,CMUResearch Partially supported by: Qatar Foundation, Microsoft, HP,CMU

Ananda Gunawardena, Carnegie Mellon University, CS Department John Barr , CMU-Qatar & Ithaca College


Background• My semester in Qatar• Teaching Data Structures and Algorithms• Regular Challenges in the course

– Challenging concepts– Broad curriculum– Highly Technical– Large programming assignments

• Qatar Challenges– Student readiness to accept 15-211– English reading difficulties


• Assignments– Frequent Reading and Writing assignments

• Technology Framework– Tablet PC’s– Adaptive Book

• Process– Reading Strategies– Reading Comprehension

• Implementation– Fall 2005 – Ananda Gunawardena– Fall 2006 – John Barr– Fall 2007 – Bob Monroe


• Put more emphasis on reading in the course• Reading assignments based on concepts or

problem sets• Coordinating with Reading Strategies

– Pre-Lecture(submit once before the lecture)– Post-Lecture (after lecture)

• Ask the student to create a markup• “Compare” with instructor markup

– Similarity score• Improve the markup iteratively• Meet a reading threshold


• Sample Reading Assignments– BST– Graphs– Hashing– Sorting– Heaps

• Demo– How to setup the assignment– How to submit an expert markup– How do students read in the course– Marking up what they read– Submitting so they can get a score– Iteratively improving the score


Observations

• Using reading as part of the course– English speaking abilities vary among Qatari

students – reading technical texts in English is difficult – very resistant to being forced to read– Initially didn't like their reading being

evaluated.


• Changing the perception changes about using a book

– By the end of the course good students felt that reading helped them.

– The initial reaction seems to have been partly a result of a fear that they couldn't do the reading adequately.

– By the end of the course, they realized that they had opportunities to improve their reading score, had figured out how to do the reading correctly

– realized that doing the reading helped them in other parts of the course.

– Not ALL students felt this way. – The better students, who were more careful about doing the

readings, in general had more positive responses than poor students who saw the reading assignments as chores.


• Specific challenges in reading technical material– Acceptable general reading skills– poor technical English skills.

• This makes reading time consuming.• students may read, highlight, and outline material and not really

understand it.

– Combine reading assignments with questions that force students to act on the material that they've read.

– This reinforces the material in the student's mind, forces them to think about what they've read, and helps make the book material concrete.


• The role of the instructor in encouraging students to read in technical courses

– Students in Qatar will, for the most part, not read without direct intervention of the instructor.

– They would much rather use powerpoint slides, lecture notes, and one-on-one conversations with the instructor in office hours to learn the material.

– We found that only when the reading was tested in some manner students would do the readings.


• Role of reading in technical courses

– Technical courses, especially in computer science, consist of important concepts, theories, and arcane programming details.

– It is relatively easy to convey concepts, theories, and broad programming principles in class. It is hard to discuss details unless students already have a high-level grasp of the material.

– Under these circumstances, reading plays three parts in technical courses. • students need a source where they can gain at least an overview of the material before

class. Forcing them to read the relevant material before coming to class

• In Qatar, students primarily rely on class notes and powerpoint slides. But textbooks can provide a much richer resource for review.

• The last role of reading is provide a source for the details that were not covered in class. In this sense, books provide a reference.


• How best to integrate reading assignments into CS courses

– Readings should be made part of the course

– Students must be forced to read relevant material before class. This can be done by requiring them to submit markups or by short in-class quizes. This ensures that students are somewhat familiar with the material before class begins.

– It's also useful to have students redo their reading after class. This will encourage them to look to the textbook when reviewing the material instead ofthe class lecture notes or slides.

– Require students to submit a markup before class. They could be allowed to only submit one markup, no revisions. Then after the class, they could be allowed to submit revisions. Their final grade on the markup would be some combination of the initial and final submissions.


0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

100.00%

1 2 3 4 5 6 7 8 9 10 11 12 13 14

Average Markup Score

Final Weighted Grade NOTEyou absolutely MUST accountfor the fact that the students'markup grades are somehowfactored into the f inal coursegrade. This biases theresults!

Results


0.00%

20.00%

40.00%

60.00%

80.00%

100.00%

120.00%

1 2 3 4 5 6 7 8 9 10 11 12 13 14

Average Markup Score

Total Homework Score


Average of Best Scores for Each Assignment

0

20

40

60

80

100

120

0 5 10 15

Submissions

Sc

ore Average of Best Scores

for Each Assignment


0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

100.00%

1 2 3 4 5 6 7 8 9 10 11 12 13 14

Average of Best MarkupScore on EachAssignment

Final Weighted Grade(excluding their scores forcompleting markups)


0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

100.00%

1 2 3 4 5 6 7 8 9 10 11 12 13 14

Average of Best MarkupScore on EachAssignment

Total Homework Score

microsoft faculty research summit 2007 adaptive book: a platform for teaching, learning and student...

Documents

adaptive book adaptive

adaptive book package

markups adaptive book

adaptive book ui slide

adaptive book authoring

sco slide

future slide

survey slide