pc dinners, mr. java and counter intelligence:...

PC Dinners, Mr. Java and CounterPC Dinners, Mr. Java and CounterIntelligence: Prototyping SmartIntelligence: Prototyping SmartAppliances for the KitchenAppliances for the Kitchen

Joseph Kaye, Niko Matsakis, MatthewGray, Andy Wheeler, Michael Hawley

jofish | niko | mkgray | ajwheele |mike @ media.mit.edu

MIT Media LaboratoryPersonal Information Architecture /Counter Intelligence

1 November 1999;Revised 14 February, 2000

Abstract: We look at the effect ofcapillary networks and introduction ofuniversal digital IDs on previouslyunconnected objects. We havedeveloped three prototype intelligentappliances: a microwave, a coffeemachine and a kitchen counter, and lookat technical and human interfaceconsiderations that arise from connecteddevices.

1. Introduction

We have developed a series of Internet-connected kitchen appliances whichallow the user to interact with themachine in an augmented but simplemanner. We chose to concentrate ourresearch on the kitchen, which hasbecome the social and communicationcenter of the home. A popular topic forresearch has been the living room of thefuture, which is an essentially passiveenvironment. In the living room, you sitin front of your television surroundedby your stereo system and absorb. Inthe kitchen, objects and informationflow in and out, and are created andabsorbed by the users.

We have created three devices, each ofwhich addresses a different aspect ofinteraction and connectivity. The PCDinners project explores a barcode-based microwave that modifies cookingtimes and personality, depending on thefood being cooked. Mr. Java is a coffeemachine that recognises users and givesthem the coffee and the information thatthey want. Finally, Counter Intelligenceis a sketch of a smart kitchen counterthat helps users prepare a recipe.

2. Trends & Concepts

We see these trends shaping the waythat we interact and deal with productson an everyday basis. Each one of theprojects presented is attempts to exploreramifications and user considerations ina real-world scenario.

2.1. Mass Customization

This century has seen a great deal ofwork in the fields of mass production,statistical sampling and demographics.With an increasing awareness ofquestions of privacy, and an increasingreluctance to provide personalinformation, mass customizationprovides an incentive or advantage forthe customer or user –- a change fromthe general tendency of corporatedemographic gathering benefiting thecompany rather than the customer.

2.2. Adaptive Behavior

We expect to see appliances modifyingtheir own behaviour based on theirusage, perhaps around times of highuse. For example, if a fridge knows it isopened frequently between eight andeight thirty in the morning, then itmight decide not to attempt to coolduring that time as the energy would bewasted, and wait until eight thirty toturn on the compressor. It is a simplestep from this point to addcommunication such that, for example, avending machine can inform its ownerof the optimal time of day forrestocking.

This type of self awareness does havesome current use in maintenanceapplications, such as office copiers.However, such information is generallyused by the company for maintainance,rather than actively modifying themachine’s behavior.

2.3. Task-appropriate internet

We feel that successful integration of theinternet into an appliance doesn’t comefrom simply dropping a desktopcomputer into an object. The optimalway to connect one’s oven to theinternet is not necessarily to install atouch screen into the door. Instead, theobject should be to prioritize theinformation flow into the oven itself:recipes and cooking times areimportant, whereas the ability to checkyour email is significantly less so.

2.4. Shared input & output devices

As the kitchen itself becomes treatedmore as a system than as a collection ofindividual objects, the ability to shareinput and output devices amoungintelligent appliances becomes key.Developing an oven with voicerecognition is difficult and expensive,but developing an oven that can becontrolled by the kitchen, and thekitchen’s voice recognition system, is auseful goal. It is thus fundamental toany standards effort thatinteroperability of controls and input befunctional considerations across theboard.

2.5. Intelligent appliances should beeasier and better

Current appliances are dumb. Theirinterfaces are burdensome, requirelearning and rarely mean that theappliance works as well as it could. Anintelligent and aware appliance willknow what to do and when. If awashing machine knows what clothesare in it, it’s no longer necessary to tellthe machine anything; putting the

clothes in and closing the door shouldbe sufficient. At the time of writing,Merloni Electrodomesticci andElectrolux have announced network-connected appliances; we expect otherappliances manufacturers to follow.

2.6. Current networking paradigmscannot scale

Thirty years ago, the kind of networkingthat we all currently have on our deskswas developed. We are currently seeingmoves by various entities to move thesame thing on the same scale into thehome, to wire fridges and dishwashers.However, we feel that the outer limits ofthe next wave is not wired fridges anddishwashers; it’s digitallycommunicating Barbies, coffee cups andpaintbrushes. These ‘capillarynetworks’, connected intermittentlywith low bandwidth, low cost andprimarily wireless networking are thefuture.

The current system for connecting adevice to a network simply doesn’t scalewhen tens or hundreds of objects in aroom or house are connected. New,dynamic, scaleable and activelyreconfiguring networking paradigms,along the lines of Hyphos1 and others,are necessary for such networks. Selforganization of such networks is afundamental part of this vision, ascurrent network configurationtechniques cannot scale. We recognizethere are many technical issues relatedto this change in networking that we donot feel qualified to comment upon, butfeel that the situation itself is clear.

2.7. Ubiquitous Unique IDs

We predict – even assume, in many ofour scenarios – that all products soldwill have a digital ID. The barcode canbe seen as a proof of concept, but,although the cost of printing isnegligible, there is a cost in physical realestate, and most importantly a cost inconvenience, in that the barcode canonly be read when it can be seen.Another proof of concept is the MobilSpeedpass system, which has beenextremely popular, and actually addedrevenue as users search out Mobile gasstations in return for convenience.

Most importantly, these digital Ids willbe unique. Currently, UPC barcodeslack serial numbers: on the simplesthome-use level, one can of peas lookslike another can of peas, making it foryour kitchen to know how many youhave in stock. The added cost ofimplementing a serial number inaddition to an identification number is anegligible additional cost. However,the cumulative advantages to consumer,retailer, manufacturer and recycler arehuge.

3. PC Dinners

PC Dinners is a microwave with abarcode scanner that reads the UPCcode on pre-packaged food and cooks itfor the appropriate length of time. Inaddition, it ‘acquires’ the personality ofthe food. For example, when heatingfrozen Danishes, the ‘Danish Chef’ voiceadvises the user to put the Danish in themicrowave, and warns it may be hotwhen the cooking is finished. PCDinners was developed during Fall1996, and while it is now far fromvisionary technology, it laid thegroundwork for much of the KitchenSync vision.

3.1. UPC Meets URL

It is clear that automatic identification offood is a key issue in a technologicallyaugmented kitchen, and we haveconsidered a number of ways ofaddressing this problem. One of ourmore unlikely suggestions, using lasersto print barcodes on food, wasindependently implemented in thespring of 1999 by David Small2, using anindustrial laser cutter to print diagrams,instructions and barcodes on foods.

Such proposals aside, however, a largeclass of foods (prepackaged, ready-to-eat items in particular) already had UPCsymbols as part of the packaging andthese could be used for identificationpurposes.

A key realization was that anidentification tag, be it barcode or RFIDtag, need not be merely an identity, butcan be a URL-like pointer to an infiniteamount of information. A resource thatcurrently doesn’t exist for public accessis a universal UPC to URL database.The concept that a product has anelectronic identity in addition to itsphysical structure is something thecorporate world is only now realising,and is some years from implementing.Such types of publicly accessibledatabases do exist for some information,such as track information on CDs3, butthe exception is the rule.

In a more sensible and sense-ablekitchen, we see these digital identitieswill be able to react with smarterappliances in the environment. Like theNutcracker or Pinocchio, packages,bottles, utensils, and food will tell theirstories. A combination of ingredients

could be more like a cast of characters,coaching you through a recipe.

3.2. Barcodes

Our decision to use barcodes wasmotivated by several factors. Theimmediacy of their availability and lowcost was a primary element in thisdecision. Some of our later work,particularly the Counter Intelligenceproject, has revolved around a scenariodependent on packaging whichincorporated RFID tags as a replacementor augmentation for barcodes. Weexpect RFID or similar tags to supplantbarcodes in the near future, but suchdevelopment is conditional on bothtechnical improvements in reliablecollision-detection algorithms andvolume production leading toinexpensive tags.

It also became quickly apparent that aviable solution for product identificationwould require a serial number field todistinguish different iterations of thesame product. The current format ofUPC codes doesn’t provide for a serialnumber, and thus our implementationdidn’t include this as a possibility. Alogical step is an expansion of theexisting UPC code to include areasonably sized serial number for suchuse. Defining such standards iscurrently being undertaken by theDistributed Intelligent Systems Centerat MIT in conjunction with the UniformCode Council, the World Wide WebConsortium and the InternationalStandards Organization.4

3.3. Implementation

PC Dinners is based on a conventionaloff-the-shelf microwave, augmentedwith a bar code reader, a PIC-based IRXboard within the microwave for controlpurposes, and a computer with custom-written software.

During the course of cooking theproduct, the computer plays a pre-selected sound file. For thedemonstration, a piece of music waschosen in keeping with the ‘personality’of the food, but there is an obviouspossibility for advertising in acommercial context. When cooking iscomplete the computer, using theappropriate personality, thanks the user,advises removal of the product from themicrowave and warns them that it maybe hot.

The use of different personalities fordifferent foods is a deliberately trivialapplication of a nonetheless usefulconcept. The idea that appliancesshould react differently with differentobjects has a number of uses: a washingmachine that warns you when you put ared sock in with your white shirts, forexample.

3.4. Expansion possibilities

PC Dinners functioned only on a verylimited number of products, and eachproduct required custom-recordedinstructions. However, each instructionwas stored remotely and a pointer to itwas stored as a URL with a predefinedformat using the UPC code. Forexample, a product with UPC code12345.67890 would have the

introduction sound file recorded by theappropriate personality athttp://pcdinners.media.mit.edu/12345.67890/introduction.wav.

It would therefore have been possible tointroduce a new product, with itsassociated new barcode, and PC Dinnerswould know the appropriate URL tolook at for associated information.Essentially, PC Dinners could downloadinformation about how to cook aparticular product directly from theweb.

3.5. Modifications

Steve Gray, a graduate student,continued to develop PC Dinners,renamed as Microchef, incorporating aweight sensor and a touch screen formore extensive control and moreelaborate user interface, allowing theuser to modify preset times, cookdifferent quantities of an item, and cookdifferent weights of an item. Thisaddressed many of the limitations of theoriginal PC Dinners design whilemaintaining the fundamentalrelationship between the tagged orbarcoded package and associatedinformation.

3.6. Analysis

We feel that PC Dinners remains acuriosity and a thought experiment,rather than a potential product, due toits inability to heat unbarcoded foods,and the only demo-appropriate changesin personality, which would becomeinsufferable on day-to-day use.However, we feel the underlying

concept is sound, and note that at theJanuary 2000 Consumer ElectronicsShow in Las Vegas, Samsung released abarcode-enabled microwave ovendeveloped in conjunction with Kit Yamat Rutgers.

4. Mr. Java

4.1. Introduction

Mr. Java is a smartcoffee machine.When you put yourmug under thespout, the machinerecognises it andgreets you, makesyou the coffee thatyou want and playsyour choice of newsor information. Forexample, whenKaye, a British expatriate, puts his cupunder the spout, the machine’s LCDscreen lights up with ‘Hi Jofish’, itmakes a double tall latte, and plays thelatest news from London over thespeakers. There is a delay betweendisplaying the greeting and starting tomake the coffee to enable users to pick adifferent selection from their usualwhile maintaining their standardpreference.

Furthermore, the software updates thedatabase with information about thetime and type of coffee selected. Thisalso occurs when the machine is used inthe normal manner with an untagged

cup. Mr. Java generates graphs of usagedisplayed on a nearby monitor.

It is interesting to note that it appearsthat Mr. Java serves more coffee onTuesday than any other day of theweek, and that Thursday is the leastpopular. It is possible that this dataactually means that Mr. Java was leastlikely to be functioning (and thusgathering data) on a Thursday, but thisis as mystifying.

Mr. Java was initially created in the fallof 1997, exhibited at Comdex that yearat EDS’s booth, and ran intermittentlyfor approximately two years at theMedia Lab.

4.2. Technical Details

Mr. Java is a prototype intelligentappliance: a high-end Acortocappuccino machine augmented with anetworked computer and a RFID tagsensors.

The Acorto 2000s automatic coffeemachine has a serial port built in to themotherboard for use by service

engineers for diagnostic and update use.We used this port for software control.

The serial port connects through an RJ-11 jack to a controlling computer. Thecomputer itself is a low-end machinerunning Windows 95 and, appropriatelyenough, Java. The only othermodification to the coffee machine is theaddition of a waterproofed RFID tagreader, with a single cable providingpower and communication also routedthrough the ceiling to the computer.Mugs for use with Mr. Java weremodified by attaching RFID tags to theirbases with hot glue.

Once registered with the system, amanual process, users were able tomodify their preferences through theirpassword-protected web page. Whenpresented with a tag that was not in thesystem, the user was instructed throughthe small LCD screen on the coffeemachine to select a drink. When theuser pressed a button that drink wasstored as their default. To change thepersonal greeting displayed on the LCDscreen, to change their audio feedpreferences, or to permanently changetheir coffee choice it was necessary tomanually log on to the website withtheir username and password assignedwhen the cup was first set up.

4.3. Identification

It was apparent after the creation of PCDinners, that the next step was thepersonalization of the experience to theuser. To grossly generalize, current userinterfaces are rarely designed with theintent of providing each individual with

an individualized experience. We feelthat as identification technologybecomes more widespread, this will be acommon trend – it’s already commonfor ATMs to greet you in your ‘home’language, for example.

It was not initially clear what systemwould be appropriate for recognisingthe user in this context. We do notpredict that our choice is necessarilyvalid for any other application, but wewanted a robust technology that couldbe easily implemented in a naturalmanner: swiping a card, for example,we felt would make the user interfaceharder rather than easier to use. Weconsidered a variety of technologies forthis task.

4.3.1. IR Badges

An earlier project by our group hadinvolved the use of IR-emitting nametags that continuously broadcast an IRsignature unique to each tag. However,the problems inherent in continuouslywearing a nametag (absent from theLab’s culture, unlike some otherinstitutions), the need for batteryreplacement, and, above, all the lack ofcontext, made us decide against IRrecognition. By lack of context, wemean the absence of a verb associatedwith the noun of identity. An IR-badgerecognition system doesn’t distinguishbetween the action of standing in frontof the coffee machine and the action ofstanding in front the coffee machine andwanting a coffee.

4.3.2. Biometric ID

We considered a variety of biometricmeans of identification, includingfingerprint identification and facerecognition. Both are technologicallyentirely capable of performing the taskwe needed. However, we felt therewould be considerable social oppositionto such personally intrusive technology.

4.3.3. Barcodes

Barcodes printed or stuck on the cupwere considered as a possibility.However, the vicinity of the spout of acoffee machine involves a considerableamount of debris, primarily drops offoaming milk and espresso, and abarcode, not to mention the reader,could quickly become obscured andunreadable.

4.3.4. Touch memory

We also briefly considered touchmemory technologies. This remains apossibility for an alternate paradigmwhere the user carries identification,such as on a keyring, much as the MobilSpeedpass system uses RFID. However,we felt this would impact the eleganceof the interface and make it less likely tobe used. Installation of the touchmemory device on the cup seemeddifficult in view of the comparativelybulky packaging.

4.3.5. RFID

RFID had several advantages for us.Due to the creation of the Accessmousepads (an RFID-enabledconnection to the Internet), there were

both expertise and materials alreadyavailable in the lab. The tags in questionwere about an inch in diameter and thethickness of a penny, which meant theycould be unobtrusively attached to thebottom of the cups. Having a systemthat would supplement rather thanreplace the current paper-cup-and-press-the-button interface wasnecessary. The tags’ robustness wasadequate for the high temperatures,liquids, and relatively causticconditions, although several attempts atwaterproofing were necessary beforethe same could be said about the reader.The tagged cups were found to bedishwasher safe, although they were notsuitable for microwaving.

We felt that identifying the cup ratherthan the user allowed for a relativelyrobust identification, and had otheradvantages, such as the ability forsomeone else to get you your coffee.

Furthermore, the tag reader could beplaced directly under the spout, solvingthe problem of context: the only contextin which the user would place their tagthere would be when they wanted acoffee. Read range through thealuminium grate under the spout wasapproximately three inches, so thechanges of an accidental read were slim.

4.4. Data tracking

The decision was made from an earlystage not to track individual usage data.The nature of the RFID system we usedmade it impossible to have a trulyprivacy-conscious system. Some usersexpressed a desire for an untraceable

system, with coffee and newspreferences stored locally on the cup.We recognize this issue, but had settledon tags with a hardwired ID that wasautomatically read by the tag reader.

However, we did make it possible forusers to track their personal coffeeconsumption, by making it possible forusers to specify a custom URL for theirsound preference. At present, this isjust a response to curiosity on the behalfof the user. However, as suchtechnology becomes more ubiquitous,there are tangible health and nutritionalbenefits of being able to keep track of apatient’s inputs, of which this is onesmall part.

Aggregate data, however, was tracked,and graphs of the result were displayedas a screen saver on the computer aswell as being web-accessible. Althoughthis ability is comparatively common inhigh-end devices, we were surprised tolearn how rare this is in beverage orvending machines. We think this abilityof a machine to track its own usageappears to be a fundamental part offuture appliance design.

4.5. Applications & Analysis

There are some comparatively simplereal-world applications for Mr. Java’stechnology. There is potential for rapidtransactions with Speedpass-like tags,held on the owner’s keyring.Alternatively, the user could purchase a“fifty-cup” coffee cup, a travel mug withfifty prepaid cups of coffee. However,we feel the primary application for thisretail technology is outside of the realmof coffee, and should been seen as anexercise in user interaction andcustomization.

We found that the cup interface wassuccessful, and feel it raises aninteresting question of having multiplevery application-specific means ofidentification, rather than a universaldigital identity system, which has muchmore serious privacy concerns.

5. Counter Intelligence

5.1. Kitchens & Counters

Over the latter half of the twentiethcentury, the kitchen has moved from theback of the house to the center of thehome. At the center of the kitchen itselfis the kitchen counter, where food iscleaned, prepared, and frequently eaten.Counter Intelligence is a sketch of aprototype intelligent counter.

It is a flat surface with a digital scaleflush with the surface on the sidenearest the cook. RFID tag readers aretaped to the underside of the surface,and a touchscreen with built in speakerssits on one corner. Hidden beneath thecounter is a computer with a twelve-serial port interface card.

Users select a recipe from a listdisplayed on the touchscreen. Thescreen displayed an ingredient list, andas tagged jars of ingredients wereplaced on the counter, the item wouldbe checked off.

5.2. Counter Intelligence I

Ingredients assembled, the next screenwould instruct the user to pour in threecups of chocolate chips. At this time,the chocolate chips would be in a taggedjar on top of an RFID tag reader, whichwas continuously sending the tag IDback to the computer. When the RFIDreader no longer transmitted a tag ID,the computer assumed the jar had beenlifted by the cook and would display thecurrent weight and volume of chocolate

chips, reading the scale through theserial port, and updating as chips werepoured into the bowl.

The system would then look for thesame to happen with the nextingredient. The screen would theninstruct the cook to place the bowl in themicrowave and it would microwave theingredients for thirty seconds.


Counter Intelligence refers to twosequential design. Our initial version,with software produced by NikoMatsakis and Andy Wheeler, ran underWin95 and used the touchscreen for allinput and output.

An RFID tag reader was placed underthe surface of the scale, with an initialplan to read tags on mixing bowls. Thisturned out to be not practical as wefound that microwaved tags frequentlyexplode loudly in a flash of orange lightand emit a surprisingly large amount ofsmoke. We thus note for the record thatcurrent RFID technology does notmicrowave elegantly.

5.4. Counter Intelligence II

Our second version of CounterIntelligence added voice output, and are-write of the entire softwareinfrastructure to run under Linux. On-screen choices were confirmed with pre-recorded voice output, and as the userpoured items such as flour, the voicewould advise “One cup…two cups… alittle bit more…Stop!”

We informally studied potential users’reactions to the voice output as opposedto a purely screen-based output, andsaw some very mixed responses. Manyfelt that voice output had a great deal ofpotential to be useful, and liked theconcept of having their kitchen talk tothem. By contrast, some hated the idea,and did not want their kitchen talking tothem under any circumstances. We feelthere is more potential for voiceinteraction in the home than in theconventional office environment.

Everything in the system was made aHive device. Hive5 is a Java extensionallowing for interaction andconnectivity of multiple objects easilyacross a network. Previous efforts aspart of the ‘Things That Think’ projecthad relied upon one-off methods tocombine sensors and computers withembedded computation. Hive providesa logical and extensible method of doingso. This process is more fully describedin Matthew Gray’s S.M. thesis.6

5.5. CI: analysis

It is important to observe that we do notconsider Counter Intelligence to be amodel for future kitchen interaction, butrather a sketch incorporatingtechnologies that could be useful. Thedifficulties in determining a logical datastructure for recipes rapidly becameapparent, and we devoted considerableeffort to this question. It was alsoapparent to us from the beginning that atouchscreen is not the ideal input devicefor a kitchen, but it did provide a simplemethod for input without using thekeyboard or mouse of conventional

computing systems. We feel that acontext-aware voice recognition systemis a key interface for such interaction-intensive systems in the future.

The Counter Intelligence system hadsome interesting problems. Atouchscreen is not a perfect interface fora kitchen, where one’s hands arefrequently covered with flour or butter,although it may well have a role to play:keyboards and mice are even lessappropriate. Voice output becomesmore reasonable when a betterinfrastructure is in place to determinecontext: if the phone rings in the middleof baking, the last thing you want is avoice repeating “Just a bit more… Just abit more…” The tagging technology wewere using allowed only one tag in afield, with circular fields 3” in diameter,necessitating careful placing ofingredients on market spots.Ingredients had to be manually tagged;however, we feel that RFID tagging willundoubtedly be as common as barcodesare today.

Acknowledgments

The above research would not havebeen possible with significantcontributions from many individuals.In particular, the researchers would liketo thank Kyle Anderson, Chris Connorand Acorto Inc. for the coffee machineand support, Jim Young, Marc D’Arcyand EDS, Robert Poor, Kristin Hall forcreative and administrative support,and multiple undergraduate researchersincluding Anthony Johnson, Sandy

Choi, and Rhett Creighton, and thePersonal Information ArchitectureGroup, particularly Adam Smith andWendy Ju. Funding came initially fromthe Things That Think Consortium atthe MIT Media Lab, and the project laterdeveloped into and was funded by theCounter Intelligence Special InterestGroup. ReferencesReferences

1 Poor, Robert. Hyphos: A Self-Organizing, Wireless Network.Master’s Thesis, MIT Media Lab. June1997.

2 Small, David. Covered in Newsweek,5 July 1999.

3 http://www.cddb.com

4 Distributed Intelligent Systems Center,Massachusetts Institute of Technology.http://disc.mit.edu

5

Nelson Minar, Matthew Gray, OliverRoup, Raffi Krikorian, and Pattie Maes.Hive: Distributed Agents forNetworking Things. Proceedings ofASA/MA'99, the First InternationalSymposium on Agent Systems andApplications and Third InternationalSymposium on Mobile Agents. August1999.

6 Gray, Matthew. Infrastructure for anIntelligent Kitchen. Master’s Thesis,SM, MIT Media Lab. May 1999.