annotating digital documents€¦ · annotating in the ide one of the difficulties with penmarked...

33
ANNOTATING DIGITAL DOCUMENTS L3 1

Upload: others

Post on 09-Jul-2020

0 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: ANNOTATING DIGITAL DOCUMENTS€¦ · ANNOTATING IN THE IDE One of the difficulties with Penmarked is that the markers have to go to another application to compile and execute the

ANNOTATING DIGITAL DOCUMENTS L3

1

Page 2: ANNOTATING DIGITAL DOCUMENTS€¦ · ANNOTATING IN THE IDE One of the difficulties with Penmarked is that the markers have to go to another application to compile and execute the

2

Page 3: ANNOTATING DIGITAL DOCUMENTS€¦ · ANNOTATING IN THE IDE One of the difficulties with Penmarked is that the markers have to go to another application to compile and execute the

INK ANNOTATIONS ARE DIFFERENT TO TEXT ANNOTATIONS

Ink stands out from the original

It is free form – the annotator can Emphasize – underline, highlight, asterisk Question Agree Add side notes to explain, clarify

The authorship remains with the original author

3

Page 4: ANNOTATING DIGITAL DOCUMENTS€¦ · ANNOTATING IN THE IDE One of the difficulties with Penmarked is that the markers have to go to another application to compile and execute the

WHY DO WE ANNOTATE?

Active reading Adding annotations to a document helps the reader to process the information

Communicate with others/self When developing the document collaboratively When evaluating the work (eg marking assignments)

4

Page 5: ANNOTATING DIGITAL DOCUMENTS€¦ · ANNOTATING IN THE IDE One of the difficulties with Penmarked is that the markers have to go to another application to compile and execute the

WHAT HAPPENS WHEN YOU CAN’T ANNOTATELarge study of university entrance exam

All questions with diagrams done on computer as opposed to paper student had a 10% higher chance of getting wrong.

That’s kinda scary!

5

Page 6: ANNOTATING DIGITAL DOCUMENTS€¦ · ANNOTATING IN THE IDE One of the difficulties with Penmarked is that the markers have to go to another application to compile and execute the

EXPRESSIVENESS AND USEFULNESS

Annotations are very individualisticHowever they can generally be understood by others.

Interesting studies of students choosing old –already annotated books – and looking carefully at the style of annotationsAlso students understanding pre-annotated material more quickly than the original document

6

Page 7: ANNOTATING DIGITAL DOCUMENTS€¦ · ANNOTATING IN THE IDE One of the difficulties with Penmarked is that the markers have to go to another application to compile and execute the

VISUAL MESSAGE

Looking at an annotated document in an instant you understand the position of the annotator. Are they:

Being critical?

Correcting the spelling and grammar?

Adding their own explanatory notes

7

Page 8: ANNOTATING DIGITAL DOCUMENTS€¦ · ANNOTATING IN THE IDE One of the difficulties with Penmarked is that the markers have to go to another application to compile and execute the

EXAMPLE SOFTWARE

Penmarked Assignment marking

Inking in the IDE Code review

8

Page 9: ANNOTATING DIGITAL DOCUMENTS€¦ · ANNOTATING IN THE IDE One of the difficulties with Penmarked is that the markers have to go to another application to compile and execute the

PENMARKED MOTIVATION

Electronic submission and marking of assignments is attractive to faculty and students alike

Particularly programs assignments when a digital copy of the assignment is required for marking

However providing meaningful feedback to students is more difficult

And individualised feedback is important

9

Page 10: ANNOTATING DIGITAL DOCUMENTS€¦ · ANNOTATING IN THE IDE One of the difficulties with Penmarked is that the markers have to go to another application to compile and execute the

DESIGN BRIEF

Annotation of the script

Score recording

Work practice support Who wants to ‘waste time’ marking?

10

Page 11: ANNOTATING DIGITAL DOCUMENTS€¦ · ANNOTATING IN THE IDE One of the difficulties with Penmarked is that the markers have to go to another application to compile and execute the

IMPLEMENTATION

11

Page 12: ANNOTATING DIGITAL DOCUMENTS€¦ · ANNOTATING IN THE IDE One of the difficulties with Penmarked is that the markers have to go to another application to compile and execute the

ANNOTATION PANE

Ink anywhere

Erase

Multiple files

A technical nightmare! Two layers Lots of Windows API calls

12

Page 13: ANNOTATING DIGITAL DOCUMENTS€¦ · ANNOTATING IN THE IDE One of the difficulties with Penmarked is that the markers have to go to another application to compile and execute the

SCORING PANE

Marking rubric

Numeric scores

Entered in score box with pen and recognized using Microsoft recognition engine with numeric factoid feature also range checked Or through the keyboard

13

Page 14: ANNOTATING DIGITAL DOCUMENTS€¦ · ANNOTATING IN THE IDE One of the difficulties with Penmarked is that the markers have to go to another application to compile and execute the

WORK PRACTICES SUPPORT

Pre-marking

Set up of marking rubric, directories, file type/name filters

Collection of assignments from a dropbox or directory structure

During

All assignments listed in student list pane Can be marked as opened and complete

Automatic unpackaging of zip files

Direct link to student’s folder for executing program

On completion

Exporting of marks as xml file

Conversion of assignments to pdf files (includes mark rubric and annotated files)

Email return of assignment to student

14

Page 15: ANNOTATING DIGITAL DOCUMENTS€¦ · ANNOTATING IN THE IDE One of the difficulties with Penmarked is that the markers have to go to another application to compile and execute the

EVALUATION 1

Iterative informal during developmentUsability testing with think aloud Talking while marking a program is challenging A few interaction problems pen/keyboard input ‘do you want to save’ dialogue Recognition required correct formation of digits Using the pen to interact with non-tablet programs (the students assignments) is slow!

Focus group Student list and workflow support invaluable Accurate writing required for score recognition

15

Page 16: ANNOTATING DIGITAL DOCUMENTS€¦ · ANNOTATING IN THE IDE One of the difficulties with Penmarked is that the markers have to go to another application to compile and execute the

EVALUATION 2One large programming class ~ 180 students

Six markers

Three assignments increasing and size and complexity

Three treatments Paper ‘Database’ Penmarked

Balanced treatment Each student had one assignment marked by each process Each marker used each process Rotated so student – marker different for each assignment

Page 17: ANNOTATING DIGITAL DOCUMENTS€¦ · ANNOTATING IN THE IDE One of the difficulties with Penmarked is that the markers have to go to another application to compile and execute the

MARKING TIME

0

5

10

15

20

25

30

Assignment 1 Assignment 2 Assignment 3

PaperDatabasePenmarked

minu

tes

Page 18: ANNOTATING DIGITAL DOCUMENTS€¦ · ANNOTATING IN THE IDE One of the difficulties with Penmarked is that the markers have to go to another application to compile and execute the

NUMBER OF ANNOTATIONS

0

100

200

300

400

500

600

Assignment 1 Assignment 2 Assignment 3

PaperDatabasePenmarked

Tota

l num

ber o

f ann

otat

ions

Page 19: ANNOTATING DIGITAL DOCUMENTS€¦ · ANNOTATING IN THE IDE One of the difficulties with Penmarked is that the markers have to go to another application to compile and execute the

STUDENT SATISFACTION – PROCESS

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Paper Database Penmarked Paper Database Penmarked

I found the submission easy I found the return of work easy

Disagree

Neutral

Agree

They did not like paper!

Voluntary survey 42 respondents

Submission and return of work

Page 20: ANNOTATING DIGITAL DOCUMENTS€¦ · ANNOTATING IN THE IDE One of the difficulties with Penmarked is that the markers have to go to another application to compile and execute the

STUDENT OPINION AID TO LEARNING

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Paper Database Penmarked Paper Database Penmarked

I understood where I had lost or gained marks The feedback helped my learning

Rarely

Some-times

Mostly

Correlation between number of comments and students’ opinion of aid to

learning.

(statistically significant)

Page 21: ANNOTATING DIGITAL DOCUMENTS€¦ · ANNOTATING IN THE IDE One of the difficulties with Penmarked is that the markers have to go to another application to compile and execute the

OVERALL PREFERENCEOverall preference for Penmarked

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Paper Database Penmarked

I prefer to get my assignments marked this way

Not at all

A bit

Completely

Page 22: ANNOTATING DIGITAL DOCUMENTS€¦ · ANNOTATING IN THE IDE One of the difficulties with Penmarked is that the markers have to go to another application to compile and execute the

WHAT ELSE WE COULD DO WITH DIGITAL INKIDEs

Web

Page 23: ANNOTATING DIGITAL DOCUMENTS€¦ · ANNOTATING IN THE IDE One of the difficulties with Penmarked is that the markers have to go to another application to compile and execute the

ANNOTATING IN THE IDE

One of the difficulties with Penmarked is that the markers have to go to another application to compile and execute the program

Three generations of prototypes in IDE RCA is an annotation addin for Visual Studio 2006 (Richard Priest) CodeAnnotator addin for Eclipse (~ 2007) (Emily Chen) VSInk addin for Visual Studio 2010+ (Craig Sutherland)

23

Page 24: ANNOTATING DIGITAL DOCUMENTS€¦ · ANNOTATING IN THE IDE One of the difficulties with Penmarked is that the markers have to go to another application to compile and execute the

RCA IN VISUAL STUDIO (2006)

Technically

Transparent overlay Another technical nightmare!

Each annotation consists of: A linker – line or circle Annotation – group text or diagram A severity indicator

24

Page 25: ANNOTATING DIGITAL DOCUMENTS€¦ · ANNOTATING IN THE IDE One of the difficulties with Penmarked is that the markers have to go to another application to compile and execute the

CODEANNOTATOR (2007)

Same idea but trying to use Eclipse.

Added some navigation and character recognition

Pretty much the same problem!

No extensibility points from the code windows!

25

Page 26: ANNOTATING DIGITAL DOCUMENTS€¦ · ANNOTATING IN THE IDE One of the difficulties with Penmarked is that the markers have to go to another application to compile and execute the

VSINK(VISUAL STUDIO 2010)

This one actually works properly

26

Page 27: ANNOTATING DIGITAL DOCUMENTS€¦ · ANNOTATING IN THE IDE One of the difficulties with Penmarked is that the markers have to go to another application to compile and execute the

WHAT DO PROGRAMMERS DO?

We (Craig Sutherland) did a paper based study. They spread the code out around a desk Kinda make the classes into an architecture diagram

They will annotate code if they can.

The annotation may be highlights, notes, connectors

Other note pages

Sutherland, C. J., et al. (2015). An Observational Study of How Experienced Programmers Annotate Program Code. Interact: in press.

27

Page 28: ANNOTATING DIGITAL DOCUMENTS€¦ · ANNOTATING IN THE IDE One of the difficulties with Penmarked is that the markers have to go to another application to compile and execute the

BUT WHAT ABOUT DYNAMIC DOCUMENTS

All the prior examples are of static documents (pdf). There is no functionality to support changes in the underlying document.

And of course documents change.

Lets have a play with what word has

28

Page 29: ANNOTATING DIGITAL DOCUMENTS€¦ · ANNOTATING IN THE IDE One of the difficulties with Penmarked is that the markers have to go to another application to compile and execute the

DYNAMIC ANNOTATIONS

4 Jul, 2014 29

Original

Unchanged* Split

Split – v. join

Whole stretch*

Split – h. join Split & stretch

Page 30: ANNOTATING DIGITAL DOCUMENTS€¦ · ANNOTATING IN THE IDE One of the difficulties with Penmarked is that the markers have to go to another application to compile and execute the

DYNAMIC ANNOTATIONS

4 Jul, 2014 30

Original

Unchanged* Whole stretch* Split

Split – v. join Split – h. join Split & stretch

Page 31: ANNOTATING DIGITAL DOCUMENTS€¦ · ANNOTATING IN THE IDE One of the difficulties with Penmarked is that the markers have to go to another application to compile and execute the

DISCUSSION

Why is it so difficult? There is a fundamental digital divide between text and images Trying to combine the two and keep them consistent is really difficult Knowing what to do with ink when the underlying document changes its problematic Many standard windows do not easily support non-text

31

Page 32: ANNOTATING DIGITAL DOCUMENTS€¦ · ANNOTATING IN THE IDE One of the difficulties with Penmarked is that the markers have to go to another application to compile and execute the

RELATED WORKPaperless environments Sellen and Harper (2002)

Ink interaction Plimmer and Apperley (2003) Freeform - UI sketching environment

Jarrett and Su (2003)

Annotation Marshall (1997) Shipman, Price et al (2003) Wolfe (2000)

Recognition Tablet OS

Marking software Heinrich and Lawn (2004)

32

Page 33: ANNOTATING DIGITAL DOCUMENTS€¦ · ANNOTATING IN THE IDE One of the difficulties with Penmarked is that the markers have to go to another application to compile and execute the

FINALLYGoing paperless requires Providing informal inking Reliable recognition (sometimes)

and Attending to work practices which are more varied than we first imagine!

33