the intelligent user interface of the livescribe echo pen

I.U.I. of Livescribe Echo Pen 1

Running Head: THE INTELLIGENT USER INTERFACE OF THE LIVESCRIBE ECHO PEN

The Intelligent User Interface of the Livescribe Echo PenArmen J. Chakmakjian


Bentley UniversityAbstract

This paper studies the intelligent user interface of the Livescribe Echo Pen. The research is limited to the pen and pad system itself and does not address the design and use desktop software and offline storage except to note that it exists. The paper describes the intelligent user interface of the pen including common operations and audio and written recording synchronization. A comparison to an aggregation of separate technologies, which includes human memory, is done. The paper also examines the efficacy and value of combining the various technologies and the underlying intelligence. The combination of written and audio capture allows for several interesting use cases to be created that might not have been broached by the previous generation of separate technologies. Security and privacy concerns that are inherent with the audio recording technology are addressed. The paper evaluates the underlying intelligence and concludes that it is an extremely compelling technology. This conclusion comes with the caveat that the dotted-paper technology and the associated gesture language limit the feature set and in fact leaves much of the underlying intelligence unused.


THE INTELLIGENT USER INTERFACE OF THE LIVESCRIBE ECHO PENThe Livescribe Pulse Pen was introduced in 2008 and the Livescribe Echo Pen followed

in 2010. The writing capture system was based on the handwriting capture technology invented by the members of the Anoto AB team, through many antecedent technologies and patents. The Anoto technology allowed pen strokes to be captured on dot paper. Livescribe, the company, was born when LeapFrog married the Anoto pen and paper technology with audio recording and playback technology. Livescribe combined the two technologies to allow for the ability to simultaneously record handwriting and the concurrent sound track in the vicinity of the user. The purpose of this paper is to study the user interface of the Echo Pen and the underlying intelligence. That intelligence allows several interesting features to be manifest. These are:

Synchronized handwriting and audio playback for any page of dot paper Location (pen stroke specific) recording playback anywhere on the page Addition of new pen strokes while replay and synchronization to existing audio

The intelligence to be investigated is how the pen and paper system interacts with the user. The system being augmented of course is part of the discussion. That system is a combination of existing 19th century ballpoint ink pen and paper technology, a companion audio recording system playback technology and human memory.

The deeper purpose of this handwriting capture technology was defined after the technology was invented. On the Anoto Group website, they ascribe the idea to the inventor Christer Fåhræus who envisioned in 1996 “a high tech pen that could get the paperwork done more efficiently”. (Anoto AB) The combination of synchronized audio and handwriting make it invaluable for anyone who is attempting to record what they feel is important in written form and still having the ability to go back and edit or redact their notes and easily share what they have witnessed with others

The ability of the Livescribe Echo pen to synchronize written text and the concurrent audio stream has many applications. The most obvious example is a student taking notes during a lecture with the associated synchronized audio, replaying it later, and possibly updating the written notes while replaying. Another example would be for field methods in user research. The pad and pen with the addition of audio would allow the review of notes taken and concurrent audio at any point to make sure that the research point was correct. One interesting application is anywhere where a paper form needs to be filled in by a user. An excellent example of this is the QuickBase Digital Pen solution. (Wang, 2009) QuickBase is an online database with forms. This example shows that the web form could be replicated on Anoto paper. In a customer service setting, the written responses on the Anoto paper form could be used to populate the database when the pen was docked with the computer.

While the inventors have created a compelling system, one that extends the human capability of record and recall, there seems to be an underlying immaturity to the user interface and subsequent underutilization of the intelligence. One of the main challenges that the Livescribe pen faces is that it is a bridge technology between paper and electronics. The pen and paper must be used together and (at the time of this analysis) the pen has not been used with alternative media in place of the paper.

This paper will address the underlying technologies, the digital pen user interface and intelligence and will present a comparison of the Livescribe Echo pen with previous technologies. Once these have been studied, the paper then presents an intelligent user interface evaluation with recommendations for extending the existing feature set using that intelligence and addresses privacy and security concerns.


The Underlying TechnologiesThe underlying writing technology is described in US Patent 6548768 (Petterson & Edso,

2003), in which an optical device is used to determine the location of that device on that paper and can read an encoded paper surface. The specific technology here (Figure 1) is that the encoded paper has a fixed grid of dots at each location. The placement of any dot or combination of dots in any way different from the nominal pattern allows the location on the paper to be

determined. Combining this with the invention of a pen with a camera in it (there are numerous patents in this area and to pick the specific one that describes the pen itself is immaterial to the discussion) one can capture pen strokes and record the pen stroke location on the page.

Figure 1: Patented position coding pattern of the Anoto technology


Figure 2 is part of the patent filings for the concept of the Leapfrog pen that in time became the capture and processing basis for the Livescribe Pen. In that patent filing (Marggraff & Chisolm, 2008) we see how the synchronization of audio and video are done. In that patent filing, the inventors describe a system in which a user writes on an encoded medium and can record and replay a concurrent audio stream. The key to their patent is the fact that when the pen detects that it is recording the written content on that medium, and the user requests that audio be simultaneously recorded, that the two recordings can be synchronized for later replay. Replay is requested from the device by controls on the medium that the optical subsystem can recognize as commands to control the recorded audio. (Marggraff & Chisolm, 2008) The recording replayed on the pen does not block the optical circuit from capturing input, so the synchronized record can allow for the optical record to be updated at the point of replay. The ability to add information to the existing recording using the pen and encrypted paper opens up all sorts of interesting human-computer interaction as will be discussed later in this paper.

The Digital Pen User Interface and IntelligenceThe Leapfrog patent (Marggraff & Chisolm, 2008) does allude to user interaction with

the device beyond stoke and audio recording and audio playback that is controlled by graphical elements on the page. Figure 3 shows that at the bottom of the Livescribe paper there is a set of controls not dissimilar to icons found on any audio device. Starting from left there is a menu control, audio recording controls (record, pause, and stop), audio position skip forward and back,

Figure 2. Block Diagram of the Leapfrog Pen with synchronized audio and handwriting capture


bookmarks that can be added to the audio recording, an audio position control (ranging from 0 to 100% on the page), playback speed and volume controls including muting the pen. Inside of most of the notebooks, on the inside hard cover, are several other controls that are accessible including a calculator, setting time and date, checking pen memory and battery state et cetera. Tapping on any one of these controls allows the user to invoke that command. Playback is controlled by tapping on a location on the page near any handwriting, and the nearest audio point is found and started.

One of the most interesting features of the pen is that when the user is playing back a recording and would like to annotate the page further, the handwriting is added in at that point in the recording. This is particularly useful when replaying on the desktop software since the full real time set of notes are shown as the recording is played back. The desktop software while an important part of the system is not a part of this analysis.

Figure 3. Digital Paper used in conjunction with the Livescribe pen


The pen allows some level of interaction by certain strokes on the page that it interprets as commands. For example, the pen can do simple calculations if they are written in a particular form and the display on the pen will show the user the result on the LCD. This means that the pen’s intelligence is not only recording the pen strokes, but also is able to invoke simple handwriting recognition and conversion that can translate those pen strokes into input for the calculator software module.

Figure 4. Echo pen physical user interfacesThe user can invoke the menu in the LCD display shown in Figure 4 by drawing the

control on the bottom anywhere on the page. At any point if the user would like to invoke the menu system on the LCD display, can create the similar cross, called the Nav Plus, that is available on the bottom left, tap in the center of it and begin invoking commands.

The inside covers of most of the standard notebook types also include calculators and other desktop functions as can be seen in Figure 5. From here one can set and retrieve the time and date, perform calculations, do keyboard like input to an embedded application (including 3rd party applications like a spell checker, translator, dictionary or a game). The ability to point and click on functions that dissuade the user from employing the corresponding physical device is evidently included as a convenience feature.


Figure 5. Back cover of a full sized Livescribe notebook to illustrated functions available As the paper ecosystem has evolved from simple college ruled notebooks, other types of

office paper products have been introduced. Sticky-notes similar to the 3M yellow stickies are available, and are encoded. On those types of paper, since the full set of controls are not available (given the limited real estate) a set of “shortcut” buttons have been made available which can be assigned to execute certain functions. For example, the on page “calc” function which is invoke by a horizontal stroke and then the word calc, interprets the immediately subsequently written equation and displays the result on the LCD. The “calc” function could be invoked by hitting one of the shortcut buttons.

Also the system gives the user audio feedback as they traverse the menu system using speech, evidently only at the first level of menu or in response. To most other functions that are invoked, it simply emits a short tone to let the user know that the display has been updated.

Comparison of Echo Pen with previous technologies


The Livescribe Echo Pen and special notebook replace a series of technologies that could have been brought together to accomplish the goal of recording both written and audio input. A student would buy a standard notebook and ballpoint pen, and use a recording device (it could be a smartphone) and begin to record, for example, an MBA lecture. If the student wanted to synchronize the written and audio record of the event, they would have to look down at the recording timer and write in the recording time on the pad at the time of the initial recording. In this model (see Figure 6), the student could later replay the audio recording while continuing to

update the notebook with the concepts that were missed, or just annotate the written record. When in this later session if they had not recorded the time-stamps, they could use their memory (in Figure 6: the cloud) while replaying the audio recording and attempt to recall and write down the approximate times that corresponded. The student might also want to write down the name of the recording that they chose in the notebook for later retrieval. They could attempt to just remember which recording corresponded to which pages of notes. This relatively complex operation would work as efficiently as the student’s process and memory allow.

The addition of a scanner would allow the student to store the pages of the notebook on some sort of offline storage and they might store the recordings in the same location. If the student wanted to subsequently annotate the notes, they could do updates using an image or document editor that allowed adding annotations. However, those annotations would not be recorded in the notebook, and the likelihood is that the physical and digital records would be out of sync with each other.

In the case of the previously described on-board interpreter and calculator, a human could write the equation down, enter it into a calculator and then record the result back in the page. In this case the human intervention is translating the equation into a set of keystrokes that the

Figure 6. Set of technologies replaced or enhanced by the echo pen and paper


calculator understands; Thus in Figure 6, the user would have to know Reverse Polish Notation in order to do the calculation.

This whole system of technologies requires that human intervention and in particular human memory tie all the pieces together to have an accurate recording and playback of the lecture. Most people, except for the most obsessive compulsive, would probably not put in the amount of effort this required. Finally this system did not afford any disaster recovery. If student did not scan the notebook, and the notebook were destroyed (by a simple coffee accident for example) there might be only an audio recording of the event saved. In this event, the student might play the recording over again and borrow notes from fellow students to reconstruct their original notes from the lecture.

Intelligent User Interface EvaluationThe collection of technologies and intelligence present a compelling replacement for the

previous generation of disconnected tools. This combination also presents a powerful extension of the original technologies (one of which is human memory) by providing an auditory record that corresponds to the written record done by the user. So a basic question is whether the use of an intelligent user interface is justified in this case. In order to answer that question one must evaluate whether the underlying technology is actually intelligent. Since it reacts to human behavior and can enhance human capabilities it is. Its technology is not (discounting the crude games one can play on the LCD) a typical computer interface, however, like many software sites. This is a physical tool in the form of its antecedents but interacts with both the user and the medium on which it leaves its record. Since it can also interpret handwriting and react to it, for example, doing calculations and with added applications, definitions and translations, it is an incredibly intelligent device.

One area where the user interface and the underlying intelligence seem to create a divergent set of functions is that the playback feature for a recording is simply tapping on the page near a word. Any audio that was recorded at that point forward begins to play. This on its own is an excellent piece of intelligence since it is directly augmenting the human memory record of the event, and in fact is replacing it. However, in order to start a recording while writing, a similar stroke in place is not used. The user, in mid-pen stroke, decides to record a session and must lift the pen to the controls at the bottom (Figure 4) and tap on the record control. The user then returns to the point where the writing record was stopped and can continue writing. It seems an awkward motion for a common operation. In this case a simple double tap at the point of writing would have worked. In fact, the user interface uses the double tap for other commands so there doesn’t seem to be a barrier for the motion to be detected.

Another area that the pen’s intelligence is wasted is turning itself on and off. One of the most common mistakes for the user would be to forget to turn on the pen before writing. If they forget, the written record is now on the dot page but has not been recorded by the pen. There should be some way, using a gyroscope and pressure sensitization to just turn the pen on. In fact if the user wants to retrace the unrecorded notes, they must fully rewrite over the existing notes on the paper. This seems like an area in which the underlying intelligence of the pen is left unused.

A third area for improvement is the gesture controls on the main display. For example, the user may desire to do a simple mathematical calculation. The gesture to do this on paper is to make a horizontal stroke back and forth over itself once. The pen responds “Command?” in the LCD with an associated beep. The user can write in “calc” over the line. This gesture is followed by a response by the pen in its disply “calc” and an audible beep. At that point a user can write


“1+1=” and see that that equation has been interpreted and is now on the display. The display delays for a short period and then beeps, and shows “1+1=2”. This is an excellent use of the intelligence of the pen. Similar to the “calc” function, the user can initiate several of the menu commands including setting up a send of the page at the next sync to a destination (like a Facebook feed). The odd thing here is that as in the first case, not all commands are available by interpretation; The most egregiously missing one is “record”. Replay seems to be available, but not record.

The three areas listed so far are somewhere in between feature requests and quirks. The main functional constraint of the pen is its dependence on the dotted paper. The difficulty here is that there isn’t an obvious way to extend this technology to a blank piece of paper. There is a feature of the desktop software to “declare” a notebook and print pages of the dotted paper and use them. However, the user cannot take a scrap of paper write something on it and record it. There is a workaround provided by Livescribe. If no paper is available, the user can record a session by holding down the on/off button for a second. This puts the pen in audio recording mode. An unattached recording session can be recorded, and then replayed later when a Livescribe notebook is available and the user can annotate the recording by writing something that will appear on the desktop in one of the user open notebooks. While this is a laudable workaround, it causes the user to relive the session in order to remember what note would have been written at the time if the notebook were available.

A better use of the intelligence might have been to do something in which the pen could somehow detect that a line was written on a blank piece of paper. The pen would record the grains and natural terrains of the paper as the lines were written. The user would then be required to start writing near the beginning of the originally drawn line (maybe in revision 1, write directly on the line itself so that the midsection of the letters cross the line). When returning to the desktop a new “note” type could be created in a “notebook” of this “note” type. Just as in real life, these slips of paper “notes” would have a single use and would not be able to take advantage of annotation. One could see synchronizing the original note taking session with sound however.

The penultimate area that needs to be address is non-paper versions of notebook pages. With this technology, it is possible that an exact duplicate of the notebook page could be rendered on a handheld device or a computer screen eliminating the use of physical paper altogether. Rather than write with the ink tip on the page, using Bluetooth or some other connection technology, the user could ask for a “page” of the notebook, write something that is immediately rendered on the virtual page and an audio recording could be simultaneously. There is a likelihood that this proposed technology has already gotten many patents associated and Livescribe may be avoiding the area altogether.

Finally, one area that is completely unaddressed by the current technology stack is a concurrent photographic record. A small camera is already embedded in the device for near area focus. One could see that a fixed focus camera could be embedded at the other end of the pen for distance photography. For example, a note-taker who was observing something, taking notes and recording a session, might want to embed a picture in the written record. The user could gesture to the pen (maybe by drawing a box on the notebook page) and then snap off a picture with the pen, and then return to the page and continue taking notes. The uploaded session would place the photo in the box that was drawn on the notebook page. This could be extended awkwardly to motion video recording. However, taking notes and recording a video stream would be too odd a gesture with only the pen, since unlike a video recording device, the pen would be moving while recording the motion video. This would detract from the visual record.


Privacy and Ethical IssuesThe user of a pen that records handwriting doesn’t present privacy issue on its own. To

some extent writing things down except in the most extreme cases isn’t viewed as obtrusive in modern society, nor is sketching something. However, secret audio recording with the device may brush up against existing laws and societal norms. In fact the Livescribe website addresses this fact in their 2010 FAQ:

Similar to using cameras, cell phones, digital voice recorders and other consumer electronic devices, the owners have a responsibility to behave ethically and demonstrate common courtesy when it comes to personal privacy. (Livescribe, Inc, 2010)Federal law in this area is somewhat opaque, while several states have specific limitations

on audio recording. Specifically, Federal law under 18 U.S.C. 2511(2)(d) allows for one party assent in recording of conversations. Twelve states have stricter two party assent laws, including Massachusetts (Harvard Univerity Berkman Center for Internet & Society, 2011). Regardless of the laws in this area, ethical concerns should be considered by the use of the Livescribe pen. In particular, the audio recording capability can be invoked without the pen being used against a notebook. The action in which someone could give implied consent to be recorded is the fact that notes are being taken, but without that gesture, the notetaker is treading on thin ethical ice. The question here is should the underlying intelligence be extended to request the user to get consent for the recording. That seems like an unworkable and possibly distracting feature to implement.

ConclusionThe intelligent user interface of the Livescribe Echo pen was studied and presented in this

paper. The pen and the associated system do achieve what the inventor of the pen stroke capture system wanted, namely to make paperwork more efficient and in particular note taking in this particular form of paperwork. (Anoto AB) The addition of the audio recording and handwriting synchronization by the Leapfrog team (Marggraff & Chisolm, 2008) enhanced the underlying intelligence of the device and in turn made it an extension of the human using it which is the basis of intelligent agency. This paper addressed the underlying technology stack and the intelligent user interface of the device and found it a compelling replacement to the hodgepodge of technologies that would (and often were) used by anyone who wanted to take notes and record the audio of a session, lecture or interview. The answer to the question “was it worth it” is found to be a hearty yes.

The gesture portion of the user interface was examined with respect to improvements that could be made. The conclusion is that there were more natural gestures that could have been used in line to control the pen such as starting and stopping recording without lifting the pen from the current writing location to hit an on-page control. One serious limitation of the pen is in the underlying technology itself in which the pen can only understand the paper with the dot pattern on it, and is a common pen otherwise. In fact the current model does not allow for using virtual paper, such as a form of the dot paper being rendered on a tablet device.

Privacy concerns were also examined and addressed. The pen does introduce privacy concerns with the marriage of the audio recording capability. These concerns need to be squarely addressed by the user and it was not felt that the intelligence of the pen should be extended to protect others.


ReferencesAnoto AB. (n.d.). History. Retrieved March 12, 2012, from Anoto:

http://www.anoto.com/history-2.aspxChapman, C. N., Lahav, M., & Burgess, S. (2009). Digital Pen: Four Rounds of Ethnographic

and Field Research. Proceedings of the 42nd Hawaii International Conference on System Sciences - 2009 (pp. 1-10). IEEE.

Eva-Lotta, C. (2011, October 7). Anoto History Since 1995 (PDF). Retrieved March 4, 2012, from Anoto: http://www.anoto.com/filearchive/1/11425/Anoto_history.pdf

Harvard Univerity Berkman Center for Internet & Society. (2011, 10 25). Recording Phone Calls and Conversations. Retrieved 04 04, 2012, from Citizen Media Law Project: http://www.citmedialaw.org/legal-guide/recording-phone-calls-and-conversations

Hollstron, M., Lynggaard, S., Zander, J., & Sandstrom, O. (2008). Patent No. 20080296074. US.Livescribe, Inc. (2010, 07 19). Livescribe FAQ. Retrieved 04 04, 2012, from Livescribe:

http://www.livescribe.com/en-us/presskit/assets/Livescribe_FAQ_071910.pdfMarggraff, J., & Chisolm, A. (2008). Patent No. 7453447B2. US.Petterson, M. P., & Edso, T. (2003). Patent No. 6548768. USA.Pettersson, M.-p., & Björklund, A. (2006). Patent No. 7999798. US.Schreiner, K. (2008, Nov/Dec). Uniting the Paper and Digital Worlds. IEEE Computer Graphics

and Applications , pp. 6-10.Wang, G. (2009, October 26). QuickBase Digital Pen Solution. Retrieved April 16, 2012, from

Intuit Labs: https://intuitlabs.com/experiments/quickbase-digital-pen-solution

the intelligent user interface of the livescribe echo pen

Documents