Shape Display Shader Language(SDSL): A New Programming Modelfor Shape Changing Displays

Christian WeichelLancaster University, UKc.weichel@lancaster.ac.uk

John HardyLancaster University, UKjohn@highwire-dtc.com

Jason AlexanderLancaster University, UKj.alexander@lancaster.ac.uk

Figure 1: The SDSL IDE. (a) The arrangement, actuation andpixel shader editors. (b) The realtime program inspector. (c)Program running in the simulator. (d) Uniform control panel.

Permission to make digital or hard copies of part or all of this work for personalor classroom use is granted without fee provided that copies are not made ordistributed for profit or commercial advantage and that copies bear this noticeand the full citation on the first page. Copyrights for third-party componentsof this work must be honored. For all other uses, contact the Owner/Author.Copyright is held by the owner/author(s).

CHI’15 Extended Abstracts, Apr 18-23, 2015, Seoul, Republic of KoreaACM 978-1-4503-3146-3/15/04.http://dx.doi.org/10.1145/2702613.2732727

AbstractShape-changing displays’ dynamic physical affordanceshave inspired a range of novel hardware designs to supportnew types of interaction. Despite rapid technologicalprogress, the community lacks a common programmingmodel for developing applications for these visually andphysically-dynamic display surfaces. This results incomplex, hardware-specific, custom-code that requiressignificant development effort and prevents researchersfrom easily building on and sharing their applicationsacross hardware platforms. As a first attempt to addressthese issues we introduce SDSL, a Shape-Display ShaderLanguage for easily programming shape-changing displaysin a hardware-independent manner. We introduce the(graphics-derived) pipeline model of SDSL, anopen-source implementation that includes a compiler,runtime, IDE, debugger, and simulator, and showdemonstrator applications running on two shape-changinghardware setups.

Author KeywordsShape-Changing Displays; Shader Programming

ACM Classification KeywordsH.5.2. [Information Interfaces and Presentation (e.g.HCI)]: Miscellaneous

IntroductionRecent interest in shape-changing displays—visual outputsurfaces that can physically deform—has intensified, withmany point-designs showcasing a variety of features,technologies, and applications [1, 5, 10]. To control thedisplay and actuation, each deployment requires a custom,hardware-dependent software architecture. This softwarerequires significant programming effort and is ‘unsharable’across devices. The community lacks a commonprogramming model, and vocabulary, for easily creatinghardware-independent, portable applications forshape-changing displays.

Figure 2: (left) wave animationrunning in the simulator and onTilt Display, (right) the samewave animation running ondifferent hardware.

As a first approach to overcome these issues, we introducethe Shape Display Shader Language (SDSL): aprogrammable shader pipeline inspired by modern graphicspipelines [3], for developing content for shape-changingdisplays (motion design). SDSL is a platform-agnostic,hardware-independent pipeline that unifies contentdevelopment by providing a common programmingvocabulary and environment. The SDSL pipeline supportsscalable, flexible, and more controllable applicationscompared to custom-built, hardware-specific solutions.Using mature shader concepts, SDSL allows simple codeto achieve complex display, actuation, and interactionscenarios. We provide an initial SDSL implementation forz-actuated displays. SDSL supports our vision of acommunity who can easily design and share shape-changing applications across different hardware platforms.

This paper contributes: 1) The concept of applyinggraphics shaders to shape-changing displays; 2) Aprogramming language and IDE, including a simulator anddebugger, that support the SDSL pipeline; 3) Ademonstration of the hardware agnostic runtime andSDSL flexibility using two different shape-changing

displays; 4) An open-source implementation of the SDSLcompiler, runtime, and IDE.

Related WorkShape-changing displays encompass a diverse range ofphysically-dynamic visual output surfaces. These devicescan be described by their topological form,transformation, and interaction [8]; their materiality orsurface properties [2]; or by their geometric shape [10].SDSL represents shape-change using geometric modelingof surface features.

SDSL provides programming support for single-coordinatespace shape-change. This allows it support many of theprototypes present in the literature: z-actuation [5, 7],tilting [1], and bending [10]. Due to the adaptable natureof the pipeline, material properties such as stiffnesscontrol [4] could also be included.

In modern computer graphics, programmable shaderpipelines [3] enable a high degree of control over therendering process. Shaders are written inhardware-independent languages (e.g., GLSL [9] orHLSL [6]) that allow developers to focus on contentpresentation. Existing graphics shader pipelines can bere-purposed for shape-changing displays [5] to leverage theparallelism of graphics hardware. However, this requires alot of custom code and is hardware specific. Previouswork has also applied shader pipelines to non-graphicdomains. OpenFab [11] is a programmable pipeline formulti-material 3D printing. Much like SDSL, OpenFabdefines it’s own shader types and programming language.

SDSLUsing a shader-based programming model forshape-changing displays eases development, and makes

code scalable and portable across devices. The modularand flexible pipeline (Figure 3) is programmed in a customshader language (similar to the well known GLSL or HLSLlanguages). We provide a linear-actuation focused SDSL,including a runtime environment and IDE with aninteractive inspector and display simulator.

Pipeline

SD

SL A

rran

gem

ent

Sha

der

Act

uati

onS

hade

rP

ixel

Sha

der

actu

atio

n

visu

al o

utpu

t

geom

etry

Act

uato

rA

rran

gem

ent

(Dri

ver)

SD

SL

App

licat

ions

and

Par

amet

ers

(App

licat

ion

Cod

e)

Har

dwar

eO

utpu

t(D

rive

r)

Uni

form

s/S

ampl

ers

Arr

ange

men

tD

escr

ipti

onC

ontr

olS

igna

ls

Inpu

t

Figure 3: The SDSL pipeline,including arrangement shader,actuation shader, and pixelshader; all encapsulated by ahardware driver.

The SDSL pipeline (Figure 3) abstracts actuatorplacement, spatial configuration, and visual configuration.The pipeline consists of a series of programmable stages,each addressing a specific aspect of content generation. Itadopts a “frame”-based approach: as new frames of inputarrive (for either visual or physical output, or both), thepipeline executes the programmable stages and generatesappropriate output.

A hardware driver is responsible for describing thehardware configuration to the pipeline: the actuatorarrangement and its visual capabilities. The actuatorarrangement is then processed by the arrangement shaderand forwarded to the actuation shader which computeshow the display will change it’s shape. Finally, the pixelshader computes the visual output of the display. Allshaders can have uniforms (custom parameters). Oneparticularly important type of uniform is the sampler thatprovides access to external data sources in the form of anarray of 1/2/3/4-dimensional vectors (e.g., a Kinect depthstream or touch input).

Hardware DriverThe driver encapsulates all hardware-specific details,making SDSL code portable across differentshape-displays. It is implemented once per display andsupports three main tasks:

1. Provide the device profile, specifying the visualresolution per actuator and optionally the maximaland minimal update speed, and the actuators’ rangeof motion. These values configure the pipeline(number of pixel shader instances per actuator) andvalidate if the program can be correctly executed onthe given display.

2. Provide the actuator arrangement. The physicalarrangement of actuators can be dynamic; thehardware driver captures the instantaneousarrangement and provides it as input to thearrangement shader.

3. Implement the computed height and color byappropriately driving the actuators and visualdisplay.

If the hardware supports features beyond linear actuation,the driver can supply additional information via customuniforms. For example, if the actuators can sense theirrotation, this data could be made available to SDSLprograms as a 1D texture sampler (linear float array).

Applications and RuntimeSDSL programs seldom stand alone, but generally are partof a bigger application (much like graphics shaders).Applications can use the SDSL runtime to execute SDSLprograms. The runtime supports real-time uniformconfiguration, so that applications can provide data to thepipeline. For example, to support interactivity, theencapsulating application can use the runtime to supplyuser input to a shader program.

Arrangement ShaderThe optional arrangement shader transforms the physicalactuator coordinates and normals into a more convenient

coordinate space e.g., a low-density actuator grid definedin millimeters to a depth camera image defined in pixels(Figure 4, top). This shader can also be used to supportnon-planar actuator arrangements, e.g. unwrapping asphere onto a planar texture (Figure 4, bottom).

Figure 4: Different geometryshader applications. (left)mapping a physical actuatorarrangement to the pixel space ofa camera image. (right)Unwrapping a spherical actuatorarrangement to a flat texturespace.

Actuation ShaderThis stage computes the physical state of each actuator.The computation is based on the actuator’s location,normal vector, and custom application parameters. Theseshaders typically use external data sources throughsamplers. Our implementation currently assumes a linearactuation model (see Discussion on how this can beextended) that outputs a single value: the target heightfor each actuator.

Pixel ShaderThe pixel shader computes visual output on aper-actuator basis. The displayed colors can be based onpredefined imagery, mathematical models, a sampler orcomputed in real-time.

The visual resolution of shape-changing displays is oftenhigher than their physical resolution (e.g. high-resolutiondisplays attached to actuators [1], or projection on theactuated surface [5]). Pixel shaders support this resolutionmismatch. For example, if a projection overlay is used, thehardware driver would specify that each actuator has aprojected surface of N ×M px. The pixel shader computesan RGBA (red, green, blue, alpha) color for each pixelwithin an actuator, based on the actuator’s coordinates inthe overall arrangement and the pixels coordinates withinthe actuator. This enables developers to apply complexsub-actuator detail across the entire display.

LanguageSDSL is similar to existing shader languages [6, 9]. Itfollows the well known C-style syntax (including customfunctions and block-scoping), and supports many“intrinsic” instructions familiar to graphics programmers.We support native vector and matrix arithmetic, as well asuniform declarations and samplers (externally providedarrays of 1/2/3/4-dimensional vectors). The language istype-safe and supports basic polymorphic functions.

SDSL code compiles to an intermediary representation,called Shape-Display Shader Execution code (SDSE).SDSE is completely linked, has all references resolved, andtypes checked. Using SDSE we can introducecompile-time optimizations and static program analysis.The latter could be used to check if a program can bedisplayed correctly given a certain device profile.

Integrated Development Environment (IDE)To ease the development of shape-changing applicationswe provide a web-based Integrated DevelopmentEnvironment (IDE) for SDSL (Figure 1). This IDE isstructured around an in-browser 3D simulation of theshape-changing display, enabling developers to test theirprograms without running the risk of damagingexperimental hardware.

The syntax-highlighting code-editors (Figure 1, left)highlight syntax errors. The input and output of eachshader stage can be inspected in real-time to easedebugging. Runtime errors and uniform property modifiers(Figure 1, right) allow the developer to experiment withparameters. All uniforms can be manipulated in real-timeand texture sampler data can be provided usingWebSockets. The arrangement and device profile can beconfigured within the IDE or provided via WebSockets.

ImplementationSDSL is implemented in Ruby, including the IDE, usingthe Opal Ruby-to-Javascript compiler. SDSL code isparsed using the Treetop parser generator, compiled toSDSE using custom code, and uses the Ruby built-instring expansion to turn SDSE into Ruby code. A runtimeenvironment, again written in Ruby, supports theexecution of the SDSL pipeline. Using Opal, we can usethis infrastructure in a web-browser.

Example Application: Depth Data as InputDepth data, as provided by a depth camera (e.g., theMicrosoft Kinect) or as computed from the height profileof terrain, is often used as shape display input. This depthdata has to be mapped to the individual actuators. SDSLmakes this task straightforward: we implement anarrangement shader that maps the physical hardwarearrangement, first to the unity, then to the properresolution space (Figure 4, left). The arrangement andpixel shader then sample the depth (and color stream)and transform it into actuator height and pixel color.

ARRANGEMENTACTUATIONPIXEL

1234567

uniformvec2dataRes;

voidmain(){

vec2inUnitySpace=iFragCoord.xy/iResolution.xy;

vec2coord=mat2(dataRes.x,0.0,0.0,dataRes.y)*inUnitySpace;

floatidx=floor(coord.x+coord.y*640);

sd_FragCoord=vec3(coord,idx);

}

1234

uniformsampler1DdepthData;

voidmain(){

sd_FragHeight=depthData[iFragCoord.z]/255.0;

}

1234

uniformsampler3DcolorData;

voidmain(){

sd_PixelColor=vec4(colorData[iFragCoord.z]/255.0,1.0

}

Figure 5: Example Application:using depth data. The geometryshader maps the actuatorarrangement to the Kinect space,the actuation shader uses thedepth values and the pixel shadermaps the image colors.

DiscussionWe presented a first step towards hardware independentshape-changing displays. In the following we discuss howthe concept of SDSL can be generalized and what itslimitations are.

Supporting User-Input/InteractivityThe SDSL pipeline supports user-input at every stagethrough uniforms. For example, touch input on thedisplay surface could be provided as 3D vector of thetouch coordinates in space. More complex interaction issupported through samplers as demonstrated by the”Depth Data as Input” example.

Non-linear actuation modelsOur pipeline can be adapted to other types ofshape-change by extending the shader stages to outputcustom data structures, rather than pre-defined datatypes. The computer graphics community already usessuch an approach, but applying it generically toshape-changing displays requires greater understanding ofthe requirements. If a display differs significantly from thelinear actuation model we currently focus on, one couldalter the pipeline itself. We implemented SDSL so thatit’s easy to implement custom pipelines. By open-sourcingSDSL, we hope to encourage the community to contributenew shaders that fulfill their needs.

Adapting to HardwareHardware is abstracted via drivers which provide theactuator arrangement and receive the computed actuatorand pixel data. How the driver bridges between thehardware and pipeline, depends on the hardware itself andthe runtime environment used to execute the SDSL code.Our Ruby based runtime environment supports TCP andWebSocket based I/O, so that the hardware driver islanguage agnostic.

PerformanceOur SDSL implementation is able to drive all testedapplications and shape-changing hardware in real-time.Pipeline execution time will increase depending on thecomputational complexity of the programs for each shaderstage and linearly with spatial and visual resolution. Whileour Ruby-based runtime implementation tradesperformance for flexibility, the compiler was designed withextensibility in mind, so that the runtime can beimplemented in a better performing language.

LimitationsAs in graphics, there are scenarios ill-suited to ashader-based development model, e.g. user-inputvalidation. Shape-display programming scenarios however,are centered around the parallel control of many actuatorsand thus well suited to a shader-based approach.Developing in a fixed programming model comes at thecost of flexibility. SDSL enforces fixed pipeline stages toenable cross-hardware interoperability. The pipeline canbe changed at the cost of hardware interoperability.

ConclusionResearch into shape-changing displays continues toinvestigate new materials and hardware; driving the needfor hardware-independent programming models. Wepresent SDSL as an initial approach—inspired by maturegraphics pipeline concepts—to foster discussion whilstsimplifying shape-change motion design. We hope thatour open-source implementation, including a simulatorand IDE, will allow the community to share applicationsand inspire new ways of thinking about programmingshape-changing displays.

References[1] Alexander, J., Lucero, A., and Subramanian, S. Tilt

Displays: Designing Display Surfaces with Multi-axisTilting and Actuation. MobileHCI ’12, ACM (2012),161–170.

[2] Coelho, M., and Zigelbaum, J. Shape-ChangingInterfaces. Personal and Ubiquitous Computing 15, 2(2011), 161–173.

[3] Cook, R. L. Shade trees. SIGGRAPH ’84, ACM(New York, NY, USA, 1984), 223–231.

[4] Follmer, S., Leithinger, D., Olwal, A., Cheng, N., andIshii, H. Jamming User Interfaces: ProgrammableParticle Stiffness and Sensing for Malleable and

Shape-changing Devices. UIST ’12, ACM (2012),519–528.

[5] Follmer, S., Leithinger, D., Olwal, A., Hogge, A., andIshii, H. inFORM: Dynamic Physical Affordances andConstraints Through Shape and Object Actuation.UIST ’13, ACM (2013), 417–426.

[6] Gray, K., and Corporation, M. Microsoft DirectX 9programmable graphics pipeline. Developer Series.Microsoft Press, 2003.

[7] Poupyrev, I., Nashida, T., and Okabe, M. Actuationand tangible user interfaces: The vaucanson duck,robots, and shape displays. TEI ’07, ACM (2007),205–212.

[8] Rasmussen, M. K., Pedersen, E. W., Petersen,M. G., and Hornbæk, K. Shape-Changing Interfaces:A Review of the Design Space and Open ResearchQuestions. CHI ’12, ACM (2012), 735–744.

[9] Rost, R., and Kessenich, J. OpenGL ShadingLanguage. Graphics programming. Addison-Wesley,2006.

[10] Roudaut, A., Karnik, A., Lochtefeld, M., andSubramanian, S. Morphees: Toward High ”ShapeResolution” in Self-actuated Flexible Mobile Devices.CHI ’13, ACM (2013), 593–602.

[11] Vidimce, K., Wang, S.-P., Ragan-Kelley, J., andMatusik, W. Openfab: A programmable pipeline formulti-material fabrication. ACM Trans. Graph. 32, 4(July 2013), 136:1–136:12.