COMMUNICATIONS OF THE ACM April 2006/Vol. 49, No. 4 69

FIND THAT PHOTO!Interface Strategies to Annotate,Browse, and Share

As digital photos become thestandard media for personal photo taking,supporting users to explore those photosbecomes a vital goal. Dominant strategiesthat have emerged involve innovative userinterfaces that support annotation, browsing,and sharing that add up to rich support forexploratory search. Successful retrieval isbased largely on attaching appropriate anno-tations to each image and collection sinceautomated image content analysis is still lim-ited. Therefore, innovative techniques, novelhardware, and social strategies have been pro-posed. Interactive visualization to select andview dozens or hundreds of photos extractedfrom tens of thousands has become a popularstrategy. And since the goal of photo search isto support sharing, storytelling, and remi-niscing, experiments with new collaborativestrategies are being examined.

While digital photographic databases andretrieval systems have been in use for manyyears, these systems were typically designed for

professionals in museums, libraries, advertis-ing, and journalism, to name a few specíálitíes.Such systems employed a cadre of financiallymotivated individuals to hand-annotate thepictures with metadata such as keywords,dates, and locations, often using fixed vocabu-laries, to support traditional search techniques.By contrast, consumers typically put littleeffort into photo annotation; they are morefocused on exploratory search and serendipi-tous discovery of photos with a strongeremphasis on entertainment. This leads to avery different set of requirements for personalphoto use where ease of annotation, supportfor exploratory browsing, and convenientsharing is crucial.

Annotate. In textual exploratory search,users can enter key phrases from a docu-ment to retrieve similar content. But forimages, retrieval based on contentthrough automated analysis is oftenlimited to some forms of shape analysis(such as finding the presence of faces inan image) and color matching to find sunrisesor determine whether an image was takeninside or outside.

To support effective exploratory search onphotos, appropriate annotations must be asso-

By Ben Shneiderman, Benjamin B. Bederson, and Steven M. Drucker

70 April 2006/Vol. 49, No. 4 COMMUNICATIONS OF THE ACM

ciated with the images either by the camera or byusers of the images, such as the photographer orpotentially a larger community of users. Cameras areincreasingly recording information about the photo-graph including time and date stamps, tilt sensors fororientation, light levels, focal distances, and evenglobal position. Barcodes, RFID tags, or other label-ing methods could enable a higher percentage of pho-tos to be annotated automatically.

Many interfaces enable manualannotation of photographs by “painting” keywords[3] or dragging and dropping names onto images.Commercial tools such as Adobe PhotoShop Albummake tags drag-able onto photo borders. Other toolsperform temporal clustering to create a more man-ageable set of photo groups [1]. As with many tasks,manual annotation can be improved by designinginterfaces that support faster and easier annotation as

well as making the futurebenefits more apparent.Automatic and manualannotations are valuable insupporting both searchingand browsing.

Browse. Users browsefor fun and to find a spe-cific photograph. Theymay be looking for photosof their grandfather, theirhike down the GrandCanyon, or a friend’s wed-ding. They also may belooking for a great photo toaccompany a story of a sun-rise hike or memorablebaseball game.

Clearly, if the photo col-lection has been extensivelyannotated, techniques suchas faceted search (seeHearst’s article in this sec-tion) can help users filterdown a collection and showpotential targets for brows-

ing. User-controlled visualization of photos groupedby date, location, or annotation can greatly facilitatebrowsing and increase enjoyment [4]. Different lay-outs of photos can exploit this metadata to help peo-ple find desired photos and discover new ones. Inparticular, geo-tagging of photos and interfaces, likeWWMX, allows people to find all those photographsassociated with a particular area (see Figure 1).

Chronological displays work well for dates as well,but large numbers of photos can be overwhelming, sogroups of photos can be clustered by date and represen-tative photos can be manually or automatically chosenfor each cluster [1, 2]. These representative photosagain help to provide landmarks in order for users tolocate photos from particular events. Interfaces such asPhotoMesa use powerful filtering tools, plus flexiblegrouping and rapid zooming, to enable users to explorethousands of photos fluidly (see Figure 2).

Figure 1. The WorldWide MediaExchange (WWMX) interfaceshowing map and calendarviews along with images aspublished in ACM Multimedia2003; wwmx.org.

Consumers typically put little effort into photo annotation; they are more focused onexploratory search and serendipitous discoveryof photos with a stronger emphasis on entertainment.

Share. Sharing photosby email, instant messag-ing, Web sites, and cellphones is a growing suc-cess story. When users

select photos and make them available to others, theyseem to be willing to invest more effort in annota-tion. Also by making them public, they invite othersto comment and add annotations. More elaboratestory-generating tools invite users to provideslideshow sequences with text captions and audionarration.

Recent innovations in social experiences on theWeb have sought to encourage annotation by increas-ing satisfaction and making the benefits immediatelyapparent. A game-like approach to image annotationgets players to cooperate with anonymous, remotelylocated partners in assigning keywords for pho-tographs [5]. This surprisingly addictive game hassucceeded in labeling over 10 million images as ofAugust 2005 (since its introduction in 2003). Othercommunities, such as Flickr, allow users to share andannotate images on a Web site using tags. These“folksonomies” have now gone past photos to Webpages and blogs as well (such as technorati anddeli.cio.us).

The trend toward annotating, browsing, and sharingyour photos via Web sites such as Flickr, Ofoto, andShutterfly is perhaps one of the biggest changes enabledby the transformation from analog to digital photogra-phy. Photos no longer sit unattended in shoeboxesstored in attics, but are available for ready viewing byfriends and family distributed around the world.

SUMMARY

A combination of annota-tion, browsing, and sharingof photos can support thespecial exploratory searchneeds of personal digitalphoto users by gettingaround the fact that directsearch of image content con-tinues to be beyond the capa-bilities of current systems.

The special needs ofamateur digital photogra-phers are pushing the photoindustry to support userswith their desired activities.Social networking, in com-bination with innovativeuser interfaces and visualiza-tion, is just beginning tosupport everyday photogra-

phers. However, we see significant work remaining,especially in metadata standardization to help userscope with their rapidly growing and increasingly val-ued collections.

References1. Graham, A., Garcia-Molina, H., Paepcke, A., and Winograd, T. Time as

essence for photo browsing through personal digital libraries. In Pro-ceedings of the 2nd ACM/IEEE-CS Joint Conference on Digital Libraries(2002). ACM Press, NY, 326–335.

2. Huynh, D., Drucker, S., Baudisch, P., and Wong, C. Time Quilt: Scal-ing up zoomable photo browsers for large, unstructured photo collec-tions. In Proceedings of the SIGCHI Conference on Human Factors inComputing Systems (2005). ACM Press, NY, 1937–1940.

3. Kuchinsky, A., Pering, C., Creech, M., Freeze, D., Serra, B., and Gwiz-dka, J. FotoFile: A consumer multimedia organization and retrieval sys-tem. In Proceedings of the SIGCHI Conference on Human Factors inComputing Systems (1999). ACM Press, NY, 496–503.

4. Kustanowitz, J. and Shneiderman, B. Meaningful presentations of photolibraries: Rationale and applications of bi-level radial quantum layouts.In Proceedings of the 5th ACM/IEEE-CS Joint Conference on DigitalLibraries (2005). ACM Press, NY, 188–196.

5. van Ahn, L. and Dabbish, L. Labeling images with a computer game. InProceedings of the SIGCHI Conference on Human Factors in ComputingSystems (2004). ACM Press, NY, 319–326.

Ben Shneiderman (ben@cs.umd.edu) is a professor and thefounding director of the Human-Computer Interaction Lab, Computer Science Department, at the University of Maryland, CollegePark, MD. Benjamin B. Bederson (bederson@cs.umd.edu) is an associateprofessor and director of the Human-Computer Interaction Lab, Com-puter Science Department, at the University of Maryland, CollegePark, MD.Steven M. Drucker (sdrucker@microsoft.com) is lead researcher ofthe Next Media Research Group, Microsoft Research, Redmond, WA.

© 2006 ACM 0001-0782/06/0400 $5.00

c

COMMUNICATIONS OF THE ACM April 2006/Vol. 49, No. 4 71

Figure 2. PhotoMesa showing 114photos in six groups in a singleview with integrated annotationand search tools as published inACM UIST 2001; (courtesy ofwww.photomesa.com).