1

Lecture 17:Development of and for

Devices

Jake Wobbrock

05-830Advanced User Interface Software

2

Not a Proper Toolkit Lecture

This lecture might have been about UI toolkits for devices

There’s just one problem There really aren’t any

At least, not like we think of toolkits for desktop software

Instead we will talk about “things that help you develop for devices.” And other interesting device issues…

3

Overview

Introduce work in multiple semi-related areas of software development of and for devices Palm OS Pocket PC / Windows CE .NET Compact Framework J2ME Waba BREW WAP and WML DirectX

4

Overview cont.

Highlight HCI research projects with and for devices Phidgets

Rapid prototyping for physical devices (physical widgets)

Pebbles Project Interoperation of handhelds and PCs

Handheld Web browsing (various efforts) WebThumb, Power Browser, The Gateway, M-

Links

5

Motivation: Handhelds

Cell phones and PDAs have seen huge growth in use over recent years. “Last year [2002] there was one cell phone sold

for every 15 human beings on the planet … Worldwide mobile phone unit sales totaled 423.4M in 2002, a 6% increase from 2001.”

“From 1999 to 2000 the number of PDAs sold nearly doubled, from some 3.6M units to 6.9M… By 2004, the number of units is expected to top 33.5M.”

Source: Biz Journals

6

Motivation cont.: Gaming

Nintendo GameBoy NTT DoCoMo Inc. handhelds running

Java Others by Qualcomm, L.M. Ericsson,

Motorola, Siemens … Convergence: Games for phones and

PDAs, not only custom handhelds

Source: CNN

7

The Punch Line

PDAs, phones, other handhelds experiencing great growth in use

Development community needs to have good tools and toolkits

What is available today? Sadly, not much (at least compared to

what’s available for the desktop)

8

Palm OS by PalmSource Acer AlphaSmart Fossil Garmin Handspring HandEra Kyocera Palm Samsung Sony Symbol

Palm OSVisit http://www.palmsource.com/products/

9

Palm OS Facts

As of January 2002: 16M Palm-powered PDAs shipped (to date) 7000 applications 150,000 registered developers

Growing at 1000 per month

Source: Palm OS Programming, O’Reilly

10

Palm OS Development Two main ways

CodeWarrior for Palm OS PRC-Tools (GCC)

Other ways Sun KVM and J2ME IBM VisualAge Micro Edition (Windows or Linux) Jump (use Java, compiles to Motorola 68K) Waba (subset of Java, stripped-down VM) PocketStudio (Pascal-based, aimed at Delphi users) Pocket C (develop on the handheld for the handheld) Assembler SDK (ASDK) – write directly for Motorola

68K Pocket Smalltalk (includes garbage-collecting VM)

11

CodeWarrior for Palm OS

Special line of Metrowerks CodeWarrior releases for Palm OS development

Three key components Editor Constructor Palm OS Emulator (POSE)

Write code in C/C++ Downloaded to Palm via HotSync™

12

Editor

13

Constructor

14

Emulator (POSE)

Runs on the desktop Emulates from a

“very low level” Designed to simulate

speed despite running on a desktop

Useful for debugging Not useful for

interactions requiring pen-dexterity

15

Technical Details

Palm OS apps center around event cascade

Catch events by defining handlers and handling them But not nearly as much framework as

Visual Basic or MFC… more like Win32 Often involves large switch statements Single-threaded, single-process

16

The Event Cascade

17

CodeWarrior Summary Generally the best option Originally for Macintosh, then for Windows! BUT…

Constructor is often not as WYSIWYG or Direct Manipulation as you’d like (e.g., text fields, scroll bars)

It lacks options for all programmatically-settable options that affect visual appearance of controls (e.g., button borders)

Emulator doesn’t work exactly like the device Sometimes this matters, often this doesn’t

18

PRC-Tools (GCC)

Alternate means for Palm OS development Developed before CodeWarrior ran on

Windows by the Free Software Foundation Collection of tools for Unix or Windows

Gnu C Compiler (GCC) for Motorola 68K Build-PRC (combine binaries into *.prc file) Gnu Debugger (GDB) PilRC (compiles resource files)

Falch IDE combines these with a UI

19

Pocket PC? Windows CE?

Pocket PC Not considered (strictly-speaking) a PDA

Or at least not marketed that way Meant to be a full-fledged PC but pocket-sized Refers to a device (platform)

Also to a set of standard applications common on all Pocket PC devices, esp. Pocket PC Shell

Windows CE (WinCE) Refers to just the OS that runs on Pocket PCs Has large number of capabilities (components) for

the Pocket PC developer to choose from Current version is WinCE .NET

20

Pocket PCs vs. Palm OS Devices

“Basic PDAs allow you to store and retrieve addresses and phone numbers, maintain a calendar, and create to-do lists and notes. More sophisticated PDAs can run word processing, spreadsheet, money manager, games and electronic book reading programs and also provide email and Internet access.”

Source: Microsoft

21

Pocket PCs

Toshiba Compaq iPaq HP Jornada Audiovox T-Mobile Dell Axim ViewSonic Razor Zayo Cassio Cassiopeia NEC MobilePro

22

Pocket PC Facts

Projected 2003 sales between 14M – 17M devices Compare to PC sales of 135M – 138M Pocket PC projected to catch up by 2008!

Sales hit 10M after just 10 months on the market Introduced April 2000

In May 2001, held 26% of market. Rest held by Palm OS devices.

Source: C|Net

23

The Shrunken Desktop

Pocket Word Pocket Excel Pocket Outlook Pocket IE Pocket Windows Media Player … ??

24

Pocket PC Development

eMbedded Visual Tools 3.0 (VC++, VB) Largely just like Windows programming

Can even use MFC for VC++ Compiles for many different processors

25

Challenge

Problem: Portability Not all Pocket PCs utilize the same WinCE

components So not all Pocket PC apps will run on all Pocket

PCs Despite same underlying OS!

Example: The menu bar at the bottom is specific to Pocket PCs, not WinCE

Created and passed to the Pocket PC shell

Developers often have to change much UI code for different Pocket PC platforms Example: Handheld PC (H/PC) is wide and short,

so tall dialogs have to be re-laid-out

26

Resource Editor

27

Pocket PC Summary Powerful palmtop computers More complicated than Palm OS Devices

(e.g., can be multi-threaded) Less integrated vertically than Palm OS

devices Complicates development (e.g., UI

components not often supported on all Pocket PC devices)

With more power and capability comes more complexity and complication Affects the toolkits for development on such

devices

28

.NET Compact Framework

Subset of .NET framework for desktops Not PDA-specific but targeted to any “smart

devices”: PDAs, mobile phones, set-top boxes, automobiles, etc.

Delivered as extensions to Visual Studio .NET called “Smart Device Programmability”

Tools and programming model are same as for desktop .NET platform Easy transition from .NET desktop development to

.NET compact framework development

29

.NET Compact Framework cont.

Argument is that there are currently too many mobile CPUs and OSes, so development is difficult True – but we know the Microsoft answer

.NET would allow integration of devices across multiple platforms, networks, and programming languages A vision behind .NET in general

Compact Framework is a subset of all .NET capabilities Reduces footprint on resource-strained devices

30

J2ME – Java 2 Micro Edition

Uses subset of Java Virtual Machine for smaller footprint (i.e., KVM) Highly optimized runtime environment Not PDA-specific: smart cards, pagers,

mobile phones, set-top boxes, vehicle telematics systems (e.g., OnStar system)

Set of APIs defined by the Java Community Process Program™ Includes user interface, security,

networking protocols, and more

31

J2ME Architecture

J2ME Architecture defines… Configurations: A virtual machine and a

minimal set of class libraries For Palm OS, configuration is “Connected

Limited Device Configuration” (CLDC) Profiles: Higher-level APIs that further

define the application, user interface, and device-specific properties

For Palm OS, profiles are PDA and “Mobile Information Device Profile” (MIDP)

32

J2ME Architecture cont.From http://java.sun.com/j2me/j2me-ds.pdf

33

Waba

Subset of Java Stripped-down VM: small and fairly fast No longs, doubles, exceptions, or threads Open source: http://www.wabasoft.com/

Waba defines… A language (strict subset of Java) A virtual machine (Waba VM) A class file format (strict subset of Java

bytecode) A set of foundation classes

34

BREW

Qualcomm’s Binary Runtime Environment for Wireless (BREW) Like J2ME, provides application execution

environment Goes farther! Provides business model for

certifying, downloading, and charging premium content

With J2ME, this is left entirely to the marketers and developers to figure out for themselves

Based on C++ (so it is fast, and small binaries)

35

WAP and WML WAP (Wireless Application Protocol)

Provides “flat” Web content on handhelds Requires constant connectivity by the device Lacks a rich interactive experience for end-users

WML (Wireless Markup Language) Specifies content and user interface for WAP-

delivered information Based on XML: so describes data, not just

presentation of data WMLScript

WAP:WML:WMLScript :: HTTP:HTML:JavaScript Provides only minimal interactivity

36

WAP and WML cont.

Four things specified by WML Text and image presentation and layout,

including a variety of formatting commands Deck/card organizational metaphors for

information (similar to HyperCard) Inter-card navigation and linking Card parameterization and state management

The fate of WAP Hasn’t been the success its advocates hoped New push to make possible highly interactive

content J2ME, BREW, .netCF, Waba, but these all send a VM and

increase footprint

37

Game Development: DirectX

DirectX: Advanced suite of multimedia APIs DirectDraw, Direct3D, DirectSound,

DirectMusic, DirectInput, DirectPlay, DirectShow…

Allows accessing of special hardware capabilities (e.g., graphics and sound cards) without requiring hardware-specific code

DirectX exists for Pocket PC Handheld gaming projected to be a big new

market opportunity Again, similar to programming for desktop

38

Transition from (Mainly) Industrial Tools to

Research Initiatives for and with Devices

39

Phidgets (Greenberg et al.)

Phidgets are “physical widgets” Play the same role in physical UIs as

widgets do in graphical UIs Package IO Hide implementation details, provide

abstraction Have additional characteristics

Require a connection manager A link between the software and hardware

parts A simulation mode in software only

40

Phidget Architecture Physical Device

Packaged physical unit to be used by the designer. Most phidgets are built around a circuit board.

Wire Protocol Communication protocol between device and host computer. Not visible to end-

programmers.

Phidget Manager COM object that has event-based API for end-programmers for connection management.

Phidget-specific COM objects Created by the Phidget Manager whenever a device is seen. They correspond 1:1 to

physical devices.

IGlabPhidget interface Generic interface which all phidgets have. End-programmers use it for basic identification

and capabilities.

Phidget-specific interface Capabilities specific to a particular phidget. Extends the basic IGlabPhidget interface.

Phidget ActiveX controls Wraps up the phidget code and provides an on-screen user interface and the ability to

simulate the phidget as an control.

41

Pebbles (Myers et al.) Research project exploring the

interoperation of handhelds and PCs Handhelds can be an aspect of the user,

PCs can be a fixture of the environment Users may move from environment to

environment, PCs may stay behind Handhelds can control other devices too

Personal Universal Controller Handhelds can be used to control a PC by

people with motor impairments

42

A Current Research Issue

Handheld User Interfaces for Web Browsing

Challenges Constrained screen-space No mouse cursor (e.g., no link roll-overs) Pages designed on a desktop for display

on a desktop Scrolling more difficult

Graphic refresh can be slow, scrolling blurry

43

Handheld Web Browsing The Gateway (MacKay)

Pages displayed as thumbnails expand as certain parts of them are interacted with (CHI 2003).

WebThumb (Wobbrock et al.) Exploration of interaction techniques for better

handheld browsing (UIST 2002). M-Links (Trevor et al.)

Uses modes to separate action from navigation on mobile phone Web pages (UIST 2001).

Power Browser (Buyukkokten et al.) Generates summary views of handheld web

content for display (CHI 2000).

44

Conclusions

Toolkits for development for devices are far from Amulet-level sophistication

Many processors and devices exist – complicates development and toolkit design

Numerous standards and niche technologies Minimizing footprint on resource-constrained

devices is a key constraint HCI research can tackle issues beyond the

specifics of device hardware and software e.g., interaction techniques for PC/PDA

interoperability e.g., interaction techniques for browsing