Photography by Alan AdamsIllustrations by Steve Salerno

In your hand, imagine a lump of clay, an earthy, supple material that children love to squeeze betweentheir fingers,potters fire into bowls and artists shape into sculptures.This simple clay comprises hundreds ofthousands of microscopic particles.Now, imagine that each particle contains a computer that enables all of them to move together

and form representations of objects and even people in three-dimensional space. This is claytron-ics, an amazing new technology for the display of information, being created at Carnegie MellonUniversity.

When the technology is perfected, the images of people and other objects that you see repre-sented today inside a television will take forms in the space that you occupy. Moreover, these com-pletely realistic representations will interact with you as if they were the actual object, animal orperson. You could touch this image, talk to it, manipulate it—treat it the way you would a real-lifeobject.Video cameras would capture the original images and then transmit them in real time to beconverted into the 3-D representation, which takes shape from the millions of catoms in a claytron-ic “ensemble.” That representation would be essentially indistinguishable from the original object.

Claytronics is the future of information technology, conceived by Seth Goldstein and ToddMowry, computer scientists at CMU’s School of Computer Science. They created the idea for theproject at a Grand Challenge workshop of the Computer Research Association in June 2002. CMU

Reality

By Joseph Plummer

OCTOBER 2007 PITTSBURGH 81PITTSBURGH OCTOBER 200780

Carnegie Mellon University andIntel are at work to create 3-Drepresentations of people and objectsthrough the magic of claytronics.

has partnered with microchip manufacturer Intel, which is supporting the research and where Intel senior researcher JasonCampbell is managing the project.

Goldstein, Mowry and Campbell lead a 20-member team of faculty, researchers, staff and students whose goal is to create theelectronic specks—called claytronic atoms or “catoms”—that would enable images now seen inside a video screen to take a tangi-ble form in front of you. As conceived in an advanced stage of development, perhaps a few decades away, this radical new conceptin display technology will change the way you handle just about everything presented to you by a computer.

Each catom will carry the equivalent of the data-storage-potential capability of a 1980s-vintage personal computer, and the com-bined computing power of hundreds of thousands of catoms will make possible the real-life qualities of the representations.

Here’s a striking example of how it would work: Suppose you are unable to attend an important event thousands of miles away.You could send real-time images of yourself across the Internet to a claytronic device—essentially a pile of catoms. There, the tinyrobotic devices would assemble a 3-D version of you that would interact with real people at the event.

Your sensory-rich 3-D replica would be capable oftwo-way communication. Moreover, your distantparticipation would be much more realistic thanattending through televised video-conferencing.People with whom you want to meet could alsosend claytronic versions of themselves to you toclose the loop in your conference room and to maketransactions among catom-based and atom-basedindividuals seem completely real. As colleaguesinteract with your claytronic presence in their space,you would conduct the business of the meeting

with their claytronic representatives in your space.Reproductions from catoms would have a look, feel and movement

that will evolve with the technology toward greater and greater realism.However, the surface characteristics of these reproductions will have nounderlying anatomical or physical structure. The catoms will assemble asurface upon which features will be painted electronically, similar to theway that video displays today use individual pixels to contribute tinydots of color and shadow to paint a much larger moving image.

Electrostatic forces will hold the catoms together in the form on whichthe image will be “painted” and, if you were to penetrate the object witha knife, you would find the object to be hollow inside. If you turn off the

〈 〉Back Row: M. Emre Karagozler (kneeling),Casey Helfrich, Michael P. Weller,

Gary Fedder; Middle Row:Burak Aksak (leaning against the wall),

Michael Ashley-Rolman,Stanislav Funiak, Brian Kirby;

Front Row: Todd Mowry, Jason Campbell and Seth Copen Goldstein.

Reality

OCTOBER 2007 PITTSBURGH 83PITTSBURGH OCTOBER 200782

juice that powers the pile of catoms, thecatoms return to a formless pile—readyfor another charge and the softwareinstructions they need to form anotherobject.

How did this concept come about?Goldstein is particularly interested in thestudy and creation of nanotechnology,which involves very small-scale devices.He saw the potential for nanotechnolo-gy to build some of the world’s tiniestrobots into a new electronic mediumthat could create 3-D representations ofalmost any object.

“We wanted to come up with a projectthat would be equivalent to the chal-lenges faced in the days when peoplecame up with the idea of sending a per-son to the moon,” Mowry says. Mowryenvisions a world in which claytronic con-ferencing would reduce much of theneed for business travel. Claytronics alsowould support an entirely new realm ofinteractivity in games and entertainmentand even overcome the physical dis-tances separating families and friends.

“We already play tricks with sight andsound,” Mowry says. “Claytronics will letus play tricks with space and make it fea-sible to work and play in locations that wedon’t actually occupy.”

Does this human fax machine soundlike science fiction? The horseless car-riage probably did, too. Computersalready have the capacity to handle theheavy lifting of information processingthat will be required to form reproduc-tions of real people. You can see this pro-

cessing capacity in the realism of com-puter-created characters in movies suchas Shrek, Happy Feet and Polar Express.

Needed now is the hardware—a manu-facturable version of the tiny catom thatcan carry and respond to microelectronicdata—and the software engineering todevelop programs that will enable such avast number of tiny computing devices towork together.

“I consider claytronics to be the mostprominent instance of our strategic focuson ’big intellectual bets,’” says RandallBryant, dean of CMU’s School ofComputer Science. “Looking back overthe 50 years of computer science at CMU,we can see that the reason we’re the topcomputer-science school today is that a

number of visionary people were able tosee the potential for computer technolo-gy far past the horizon most people wereviewing. It’s important to have far-reach-ing projects like claytronics, which forceus to think about what we could do withcomputer technology once it reaches thepoint that a computer can be the size of agrain of sand.”

Bryant goes on to say that CMU hasbeen at the forefront of key conceptualinsights that have shaped the world ofcomputing, including machines thatexhibit intelligent behavior and recog-nize spoken and written natural lan-guages. The claytronics project, he adds,also reflects the kind of thinking in theuniversity that conceived the assembly of

large numbers of commodity micro-processors into supercomputers. Most ofall, he believes that claytronics presentsthe type of scientific and engineeringchallenge that will keep CMU’s class-rooms filled with the world’s brighteststudents for decades.

Imagine this potential use of claytron-ics: Cameras capture the image andmotion of a doctor in a clinic and trans-mit his or her sensory-rich reproductionto a private home, where there is aclaytronic device to reproduce theactions of the doctor. The 3-D reproduc-tion of the doctor making this claytronic“house call” can then help a person inthe house who is having chest pain—orother unusual symptom. As the real doc-

tor in the clinic monitors the replica’severy move and guides it from a distance,the claytronic reproduction can take vitalsigns and provide assistance to the ailingperson, even before a live crew of emer-gency medical technicians arrives in anambulance. The patient is taken to a hos-pital, where the physician determines heneeds a heart transplant.

Today, a surgeon prepares to oper-ate by studying flat-screen images of apatient's anatomy. However, claytron-ics could provide a 3-D replica of aperson’s internal organs assembledfrom body scans. Long before makingthe first incision, the doctor couldreview a replica of the tissue anddetermine the best approach for a surgery. 〈

〉“Claytronics willlet us play trickswith space andmake it feasibleto work and playin locations that

we don’t actually occupy.”

—Todd Mowry

Emre Karagozler (left) and Brian Kirby working in the claytronics lab. Reality

OCTOBER 2007 PITTSBURGH 85PITTSBURGH OCTOBER 200784

While the real doctor in Pittsburgh, aworld-renowned specialist, is treating apatient here, he or she could pay a visitvia transmission of a claytronic represen-tation over the Internet to consult withcolleagues on the condition of anotherpatient in a hospital room in Japan.

Catoms will also find many uses forbuilding amazingly flexible electronicgadgets. Today’s palm device for com-puting, communicating and personalorganizing could be transformed into anadaptable, fluid design. Flip it open tohandle a call. Stretch it wide to provide afinger-friendly keyboard. Shape it longlike a calendar to post an upcomingappointment.

Architects would throw away theirbalsa wood and construction paper anddesign from micro-electronic models. Anudge of the fender on a new autodesign could alter the vehicle’s shape,color or other features. Because everyclaytronic model would be transmissible,design teams could work from remotelocations with a version of the samemodel synchronizing in real time with anoriginal.

The CMU-Intel researchers write abouta machine that would transmit full-dimension facsimiles of inanimateobjects. Paleontologists could use thisportable 3-D fax machine from a remotelocation to transmit the replica of afreshly unearthed fossil to academicpeers elsewhere. Quickly sharing a tan-gible, virtually identical, fully dimen-sioned copy of a rare artifact with aworldwide community of experts wouldhasten the creation of knowledge. Justas the 2-D fax machine forced a revolu-tion in letter delivery, a reasonablypriced, easy-to-handle 3-D fax could shiftthe roadway under package delivery.

Intel’s Jason Campbell, one of theauthors of the paper on the 3-D faxmachine, says that claytronic devices willrequire software language that instructshundreds of thousands of catoms to workwith a level of virtually instinctive cooper-ation. Like bees or ants searching for foodand finding their way around every obsta-cle, computer devices in such numbersneed to operate from instructions thatdefine broad goals for their motion ratherthan giving detailed, top-down instruc-tions describing every step, which is thenorm for today’s machine languages.Designing such software is the greatestchallenge the team faces.

“If we can really do it,” says Campbell,“then this type of computing will radical-ly change the way we use computers andshift our relationship to computing acrossthe board.”

While human replicas offer vivid exam-ples of what claytronics might become,Goldstein, a former software entrepre-neur who returned to graduate school atBerkeley to become a computer scientist,says the manufacturing of tiny particles ofprogrammable matter will open a vastfrontier in the design and use of comput-

ers.“Imagine having millions of tiny com-puters working together in one smallspace,”he says.Because the catoms wouldfunction both independently and cooper-atively, they could be programmed tomodel activities in small spaces that inreal life now occur literally spread outacross the planet. ”You could have amodel of the Internet in front of you onyour desk,” he says. Or you could begin tobuild much better simulations of organicprocesses. “Catoms in large numberscould be used for many things, even tosimulate the rich connectivity of neuronsinside a brain,” he says.

These dust-sized particles could giveeveryone the means to turn an idea intoa 3-D object as quickly and easily as wedownload documents. In principle, itwould close the gap between the freshidea and its finished product.

Using this dynamic material to buildthings, creative minds could design con-sumer applications that cover the spec-trum from serious to silly. Imagine alounge chair that reconfigures as a writ-ing table at the command of the user.Where earpieces now stick out from thefaces of people on cell calls, a designer ofclaytronic phones might simulate ear-lobes, with skin-toned extensions for thetalking end that match and enhance jab-bering jawbones.

A culture accustomed to electronicillusions in sight and sound would awak-en to the tricks of electronic touch andthe 3-D excitement of home entertain-ment that plays the Super Bowl on yourdining-room table. You might rent Yo-YoMa in replica to play cello in the cornerduring a romantic candlelight dinner youshare with the representation of a newfriend in Spain whom you’ve met on theInternet. All sorts of arrangements arepossible when a claytronic double canstand in.

You might invite the neighbors intoyour greatly enhanced Steelers room forthe scaleable 3-D experience that almost

beats season tickets at Heinz Field. Inaddition, makers of claytronic furniturecould provide people who live in smallapartments or homes with an all-pur-pose furniture piece that can reformfrom worktable to love seat to breakfasttable.

To turn this vision into reality, the CMU-Intel team needs to create the sub-mil-limeter-sized catom that will compute,move, communicate and carry power.The making of the first batch of 1-mil-limeter catoms promises to be the equiv-alent of the Big Bang for the claytronicsuniverse. For the first time ever, electron-ic information would form objects andinteract with users of the informationwithout the need for a “box”—such as avideo display—to contain the message.

Silicon-chip makers today can placethe processing power of a vintage early-1980s personal computer—the IBMPC-AT, for example—in the space of apunctuation mark. When that comput-ing power is manufactured inside oneperiod-sized sphere, claytronic devicesmade from millions of catoms will har-ness the multiplied effect of millions ofpersonal-computers working togetherin very small spaces. How quickly willthese wonders materialize? The CMU-Intel researchers developing thisremarkable technology are moving asfast as they can—and making notableprogress. Building on the work of agraduate student on the team, MichaelDeRosa, the team has offered one possi-ble solution for one of the project’sbiggest software riddles—how toinstruct millions of catoms to shape thesurfaces of claytronic ensembles intorecognizable forms. Other complicatedsoftware-programming issues, whichpose the most difficult challenges fac-ing claytronics, are also yielding topromising solutions.

Early results have challenged theskepticism of those who believeclaytronics is impossible. Still, like televi-

sion and other electronic media,claytronic appliances no doubt willrequire decades to evolve to peak per-formance.

The capacity for claytronic shape shift-ing, for example, might appear early insmart antennas that alter shape to cap-ture broadcast frequencies. Such deviceswould have a revolutionary impact onthe cost of telecommunication, whichtoday depends on antennas that pick upbandwidth with less efficiency than ashortstop can pick up grounders.

Some may wonder, “Would there besecurity problems—such as identitytheft—associated with this new medi-um?” Because a claytronic device wouldrepresent the characteristics of a persondirectly from gestures and expressions ofthe individual in real time, says Goldstein,it would be almost impossible to captureidentity without the person’s knowledgeor cooperation.Would there be a crediblethreat from another scenario? Could apile of formless catoms, resting like anidle computer, somehow be activated bya remote intruder and instructed to gath-er private information by rifling throughpapers on a desk? Or could those catomsbe instructed by a virus to take anotherform that would enable it to do some-thing worse?

These scenarios are conceivable toGoldstein. However, he says, as a concernfor computer security, it resembles issuesthat already exist today and that peopleare working diligently to resolve.Solutions for computer crimes in theirclaytronics versions will come with the

development of strategies to controlsimilar concerns with two-dimensionalcomputing—improved authenticationof the users of remote systems andgreater security built into the systemsthemselves so that control remains in thehands of legitimate users.

One thing is clear: Just as sight andsound virtually come alive through com-puting, claytronics promises to bring thesense of touch into a realm that only yes-terday was pure science fiction.

“This is an exciting technology,”

Goldstein says. “It has great potential totransform the experience of computingand to enable people to experience inthree dimensions images and represen-tations of information that they nowreceive in two dimensions. Just as the tel-evision has been a work-in-progress formore than a half-century, there are goingto be many steps in the development ofnew versions of claytronics devicesbefore we see human-looking represen-tations of people present for remote tele-conferences. Someday, although theywon’t have intelligence or behavior thatis independent of the original, I’m con-vinced claytronics will reproduce imagesof individuals that will have the look andfeel of a real person—with a shirt on aclaytronic image that appears to ripple inthe wind and a hand that feels like realskin extended in greeting.” PM

Joseph Plummer is a freelance writer andwriting coach who works with academicand other writers. He lives in Churchill. Youcan reach him at wordfisher@comcast.net.

〈〉

〈 〉You might rent

Yo-Yo Ma in replicato play cello in the

corner during a romantic candlelight

dinner.“Claytronics will reproduce images of

individuals that will have the look and feel of

a real person—with a hand that feels like real

skin extended in greeting.” —Seth Goldstein

Reality

OCTOBER 2007 PITTSBURGH 87PITTSBURGH OCTOBER 200786