[chi2016] gaussmarbles: spherical magnetic tangibles for interacting with portable physical...

GaussMarbles: Spherical Magnetic Tangibles for Interacting with Portable Physical Constraints Han-Chih Kuo, Rong-Hao Liang, Long-Fei Lin, Bing-Yu Chen National Taiwan University

Interactive Surface Constructive Assembly Token+Constraint

Token+Constraint Systems for Tangible Interaction with Digital Information (Ullmer et al. TOCHI ‘05)

Tangible User Interfaces

Interactive Surface Constructive Assembly Token+Constraint

MagGetz (Hwang et al. UIST ’13)

Tangible Magnetic Appcessories (Bianchi et al. TEI ’13)

Tangible Remote Controllers for Wall-size Displays (Jansen et al. CHI ’12)

Tracking Token+Constraint Interactions on Portable PlatformsToken + Strict Constraint

Tracking Ball+Constraint Interactions on Stationary PlatformsBall + Loose ConstraintinFORM (Follmer et al. UIST ’13)

Ball + Constraint Interactions on Portable Platforms

Tracking Ball+Constraint Interactions on Stationary Displays

camera camera

magnetic-field camera

�

�

SURWRW\SH�� ZH� SULQWHG� WKH� IL[WXUHV� RXW� RI� SODVWLF� XVLQJ� D�'LPHQVLRQ� (OLWH� �'� SULQWHU�� :KLOH� SULQWHG� SODVWLF� ZDV�VDWLVIDFWRU\� IRU� SURWRW\SLQJ� WKH� IL[WXUHV�� ZH� EHOLHYH� WKDW�VWURQJHU�SODVWLF�RU�HYHQ�DOXPLQXP�LV�SUHIHUUHG��7R�SURYLGH�DGGLWLRQDO�UREXVWQHVV�WR�EUHDNLQJ��IOH[LEOH�KLQJHV�VLPLODU�WR�WKRVH�XVHG�LQ�H\H�JODVVHV�FRXOG�DOORZ�3RUWLFR¶V�DUPV�WR�EH�EHQW� RXWZDUGV�� 6XFK� D� GHVLJQ� LV� SDUWLFXODUO\� LPSRUWDQW�ZKHQ� 3RUWLFR� LV� XVHG� RQ� WKH� IORRU�� RU� XVHG� E\� FKLOGUHQ�SOD\LQJ� D� JDPH�� H�J��� RXU� 3HQDOW\� 6KRRWRXW� DSSOLFDWLRQ��GHVFULEHG�EHORZ��� )RU� RXU� SURWRW\SH��ZH� XVHG� D� ����'HOO�/DWLWXGH�;7� FRQYHUWLEOH� WDEOHW� WKDW� VXSSRUWV� ERWK� SHQ� DQG�ILQJHU�PXOWL�WRXFK� LQSXW��:H� DOVR� XVHG� D� SDLU� RI�/RJLWHFK�:HEFDP� 3UR� ����� FDPHUDV� H[WUDFWHG� IURP� WKHLU� RULJLQDO�KRXVLQJV��$V�VKRZQ�LQ�)LJXUH����HDFK�FDPHUD�LV�PRXQWHG�DW�WKH� HQG� RI� D� IROGDEOH� DUP� DWWDFKHG� WR� HDFK� VLGH� RI� WKH�VFUHHQ�� :H� GHVLJQHG� RXU� V\VWHP� VXFK� WKDW� ZKHQ� IROGHG�GRZQ��WKH�DUPV�DQG�FDPHUDV�DUH�IOXVK�DJDLQVW�WKH�WDEOHW�DQG�GR�QRW�LQWHUIHUH�ZLWK�QRUPDO�XVH���:KHQ�WKH�DUPV�DUH�UDLVHG��WKH�FDPHUDV�DUH�EHQW�GRZQ�XVLQJ�D�VLQJOH�URWDWLRQ�KLQJH�WR�DOORZ�WKHP�WR�VHH�WKH�WDEOHW�VFUHHQ�DQG� WKH� WDEOHWRS�� )LJXUH� �D� VKRZV� DQ� LOOXVWUDWLRQ� RI� WKH�XQLILHG� DUHD� YLHZDEOH� E\� WKH� WZR� FDPHUDV�� )LJXUH� �E�LOOXVWUDWHV�WKH�V\VWHP�UHDFWLQJ�WR�REMHFWV�RQ�DQG�DURXQG�WKH�VFUHHQ�� )LQDOO\�� WR� DYRLG� WKH� V\VWHP� FRQIXVLQJ� GLJLWDO�HOHPHQWV� GUDZQ� RQ� WKH� VFUHHQ� ZLWK� SK\VLFDO� REMHFWV�� ZH�DSSOLHG� OLQHDU�SRODUL]LQJ� ILOWHUV� WR� WKH� VFUHHQ�DQG�FDPHUDV���1RWH� WKDW� ZKLOH� /&'V� DUH� OLQHDUO\� SRODUL]HG�� LQ� PDQ\�WDEOHWV� WKH� WRXFK�VHQVLWLYH� HOHPHQW� GLIIXVHV� WKH� OLJKW��UHTXLULQJ�DQRWKHU�SRODUL]LQJ�OD\HU����&RPSXWHU�9LVLRQ�6\VWHP�:H� LPSOHPHQWHG� D� FRPSXWHU� YLVLRQ� V\VWHP� WR� DOORZ�DSSOLFDWLRQV� WR� VXSSRUW� LQWHUDFWLRQV� ZLWK� REMHFWV� RQ� WKH�WDEOHW� DQG� WKH� VXUIDFH� DURXQG� LW�� 7KH� YLVLRQ� V\VWHP� ZDV�ZULWWHQ�SUHGRPLQDQWO\� LQ�3\WKRQ�DQG�XVHV�,QWHO¶V�2SHQ&9�OLEUDU\��:H�QRZ�GHVFULEH� WKH�YLVLRQ�V\VWHP�LQ�VRPH�GHWDLO�WR�HQDEOH�UHDGHUV�WR�UHSOLFDWH�RXU�V\VWHP��$V� VKRZQ� LQ� )LJXUH� �� �QH[W� SDJH��� WKH� YLVLRQ� V\VWHP�ZDV�LPSOHPHQWHG� LQ� D� WKUHDGHG�� KLHUDUFKLFDO� VWUXFWXUH�� $W� WKH�FRUH�RI�WKH�YLVLRQ�V\VWHP�LV�WKH�&DPHUD�0RGXOH��ZKLFK�LV�UHVSRQVLEOH� IRU� UHWULHYLQJ� IUDPHV� IURP� D� VLQJOH� FDPHUD��

SHUIRUPLQJ� DQ� RSWLRQDO� EDFNJURXQG� VXEWUDFWLRQ� VWHS�� DQG�JDWKHULQJ� GHWHFWHG� REMHFWV� IURP� GLIIHUHQW� 9LHZ� 0RGXOHV��(DFK� 9LHZ� 0RGXOH� LV� UHVSRQVLEOH� IRU� PDQLSXODWLQJ� UDZ�FDPHUD� IUDPHV� WR� SURGXFH� D� VSHFLILF� VLPXODWHG� YLHZ�� DQG�FRQWDLQV�D� VHW�RI�REMHFW� UHFRJQLWLRQ�FODVVLILHUV� WKDW�RSHUDWH�RQ�WKH�VLPXODWHG�YLHZ��7KHVH�YLHZV�DUH�XVHIXO�VLQFH��DV�FDQ�EH� VHHQ� LQ� )LJXUH� �D�� LQ� RUGHU� IRU� WKH� FDPHUD� WR� VHH� WKH�WDEOHW� VFUHHQ� DQG� WKH� VXUURXQGLQJ� VXUIDFH� �DQG� VWLOO� EH�IROGHG�GRZQ�ZKHQ�QRW� LQ�XVH��� WKH�FDPHUDV�PXVW�YLHZ�WKH�ZRUOG� IURP� D� YHU\� REOLTXH� DQJOH�� 2XU� V\VWHP� XVHV� WZR�&DPHUD� 0RGXOHV�� RQH� IRU� HDFK� FDPHUD�� %RWK� &DPHUD�0RGXOHV� DQG� DOO�9LHZ�0RGXOHV� DUH� WKUHDGHG� WR� DOORZ� WKH�YLVLRQ�V\VWHP�HOHPHQWV�WR�RSHUDWH�FRQFXUUHQWO\��$�VLQJOH�SDVV�RI�WKH�YLVLRQ�V\VWHP�FRQVLVWV�RI�WKHVH�VWHSV����� *UDEELQJ� FDPHUD� IUDPHV�� (DFK� &DPHUD� 0RGXOH� LV�FRQQHFWHG�WR�D�VLQJOH�FDPHUD�GHYLFH��8SRQ�UHTXHVW�IURP�WKH�3HUFHSWLRQ� 0DQDJHU�� D� &DPHUD� 0RGXOH� UHWULHYHV� D� QHZ�IUDPH� IURP� WKH� FDPHUD� DQG� SDVVHV� LW� WR� HDFK� RI� LWV� 9LHZ�0RGXOHV����� 2SWLRQDO� EDFNJURXQG�VXEWUDFWLRQ�� 3RUWLFR� SURYLGHV� D�EDFNJURXQG� VXEWUDFWLRQ� FDSDELOLW\� DQG� RIIHUV� LW� DV� DQ�RSWLRQDO� VWHS� LQ� WKH� SHUFHSWLRQ� SURFHVV� SULRU� WR� YLHZ�JHQHUDWLRQ� DQG� REMHFW� UHFRJQLWLRQ�� :H� XVH� D� *DXVVLDQ�0L[WXUH�0RGHOV�DSSURDFK�IRU�EDFNJURXQG�VXEWUDFWLRQ�>���@��ZLWK�D�GHGLFDWHG�PRGHO� IRU�HDFK�FDPHUD��:KHQ�DSSOLHG��D�EDFNJURXQG�IRUHJURXQG�PDVN� LV�SURGXFHG�DQG� LV�SDVVHG� WR�HDFK�9LHZ�0RGXOH�DORQJ�ZLWK�WKH�UDZ�FDPHUD�IUDPH����� 3URGXFLQJ� VLPXODWHG� FDPHUD� YLHZV�� 8VLQJ� FDOLEUDWLRQ�KRPRJUDSK\� FRPSXWHG� GXULQJ� V\VWHP� VHWXS�� D� 9LHZ�0RGXOH�FDQ�SURGXFH�RQH�RI�D�VHW�RI�VLPXODWHG�YLHZV� WR�EH�XVHG� IRU� REMHFW� GHWHFWLRQ�� (DFK� &DPHUD� 0RGXOH� DOVR�FRQWDLQV� D� 5DZ� 9LHZ� 0RGXOH� UHVSRQVLEOH� IRU� REMHFW�UHFRJQLWLRQ�GRQH�RQ�WKH�UDZ�FDPHUD�YLHZ��VHH�)LJXUH��D�E���:H� KDYH� LPSOHPHQWHG� VXSSRUW� IRU� D� QXPEHU� RI� VLPXODWHG�YLHZV��2XU�SURRI�RI�FRQFHSW�DSSOLFDWLRQV��GHVFULEHG�EHORZ��PDNH�XVH�RI�WKH�IROORZLQJ�WZR�VLPXODWHG�YLHZV��7KH�6FUHHQ�DQG�%H]HO�9LHZ��)LJXUH��F��SURGXFHV�D�UHFWLILHG�YLHZ�RI�WKH�WDEOHW� VFUHHQ� DQG�EH]HO� DQG� LV� XVHIXO� IRU� SHUIRUPLQJ�PRUH�SUHFLVH�GHWHFWLRQ�RI�REMHFWV�SODFHG�RQ�WKH�WDEOHW��7KH�7DEOH�9LHZ� �)LJXUH� �G�H�� SURGXFHV� D� UHFWLILHG�YLHZ�RI� WKH� WDEOHW�

� �)LJXUH����3RUWLFR�V\VWHP�LOOXVWUDWLRQV��7KH�XQLILHG�DUHD�YLHZDEOH�E\�WKH�WZR�FDPHUDV��OHIW���DQG�WKH�V\VWHP�UHDFWLQJ�WR�REMHFW�RQ�DQG�DURXQG�WKH�WDEOHW��ULJKW���

Paper Session: Tangible UIST’11, October 16–19, 2011, Santa Barbara, CA, USA

349

Better Form Factors

Vision-based Object Tracking Technologies

GaussSense (Liang et al. UIST ’12)

Portico(Avrahami et al. UIST ’11)

Challenges of Magnetic TrackingOne of the bipolar magnetic field may disappear while rolling

Solution: Magnetic Regular PolyhedronExpanding magnetic fields in equal dihedral angles to make the bipolar magnetic fields visible at any direction

Solution: Magnetic Regular PolyhedronExpanding magnetic fields in equal dihedral angles to make the bipolar magnetic fields visible at any direction

Explorative Study

different sizesdifferent faces

Form factors vs. tracking performances

Form factors vs. tracking performancesExperimental Apparatus

servo motor

stabilizeranalog Hall-sensor grid

Form factors vs. tracking performancesExperimental Apparatus

servo motor

stabilizeranalog Hall-sensor grid

Form factors vs. tracking performancesExperimental Apparatus

single magnet 16mm-radius

6-face 13.5,16,18.5,21mm-radius

4, 6, 8, 12-face 16mm-radius

servo motor

stabilizeranalog Hall-sensor grid

Form factors vs. tracking performancesExperimental Apparatus

8 (units) × 5 (positions) × 4 (hover heights) × 10 (angles) × 100 (samples) = 160,000 bitmaps of magnetic fields

single magnet 16mm-radius

6-face 13.5,16,18.5,21mm-radius

4, 6, 8, 12-face 16mm-radius

Form factors vs. tracking performancesData Processing

Bipolar-blob contours North-blob contours South-blob contours

3 types of contour

8 (units) × 5 (positions) × 4 (hover heights) × 10 (angles) × 100 (samples) = 160,000 bitmaps of magnetic fields

(max Intensity M and blob Area A)

Form factors vs. tracking performancesData Processing

Bipolar-blob contours North-blob contours

1. Centroid of contours

2. Centroid of pixels (in all contours)

3. Centroid of mass (in all contours)

3 types of contour x

8 (units) × 5 (positions) × 4 (hover heights) × 10 (angles) × 100 (samples) = 160,000 bitmaps of magnetic fields

(max Intensity M and blob Area A) 3 types of centroid

South-blob contours

9 Distributions of centroids

Form factors vs. tracking performancesData Processing

Bipolar-blob contours North-blob contours

1. Centroid of contours

2. Centroid of pixels (in all contours)

3. Centroid of mass (in all contours)

3 types of contour3 types of centroidx d

8 (units) × 5 (positions) × 4 (hover heights) × 10 (angles) × 100 (samples) = 160,000 bitmaps of magnetic fields

(max Intensity M and blob Area A) =

South-blob contours

9 Distributions of centroids

Form factors vs. tracking performancesData Processing

Bipolar-blob contours North-blob contours

1. Centroid of contours

2. Centroid of pixels (in all contours)

3. Centroid of mass (in all contours)

d

mean(d),std(d): measured dispersion

8 (units) × 5 (positions) × 4 (hover heights) × 10 (angles) × 100 (samples) = 160,000 bitmaps of magnetic fields

3 types of contour3 types of centroidx(max Intensity M and blob Area A) =

South-blob contours

0

1

2

3

4

5

6

1M P4 6M 8M 12M

IN-OC IN-OP IN-OM IS-OC IS-OP IS-OM IB-OC IB-OP IB-OMOc-IN Op-IN Om-IN Oc-IS Op-IS Om-IS Oc-IB Op-IB Om- IBN

/ A

N /

A

N /

A

0

1

2

3

4

56

(mm)

dist

ance

(d)

1. Bi-polar centroid of mass is the most stable feature for xy-plane tracking.

single magnet 16mm-radius

4-face 16mm-radius

6-face 16mm-radius

8-face 16mm-radius

12-face 16mm-radius

Summary of Findings

Summary of Findings

0

1

2

3

4

5

6

1M P4 6M 8M 12M

IN-OC IN-OP IN-OM IS-OC IS-OP IS-OM IB-OC IB-OP IB-OMOc-IN Op-IN Om-IN Oc-IS Op-IS Om-IS Oc-IB Op-IB Om- IBN

/ A

N /

A

N /

A

0

1

2

3

4

56

(mm)

dist

ance

(d)

1. Bi-polar centroid of mass is the most stable feature for xy-plane tracking.

single magnet 16mm-radius

4-face 16mm-radius

6-face 16mm-radius

8-face 16mm-radius

12-face 16mm-radius

2. Polyhedrons support z-axis tracking by using the product of south-blob area and intensity (AS × MS), and a single magnet does not.

3 mm

6 mm

9 mm

0

50

100

150

200

0

50

100

150

200

0

50

100

150

200

0

50000

100000

150000

0

50000

100000

150000

0

50000

100000

150000

(gau

ss)

200

150

100

50

0

150000

100000

50000

0

(gau

ss x

mm

2 )

a b4-face 6-face 8-face 12-face

Summary of Findings

012345678

1 2 3 4

��1 ��2 ��3 ��4

h = 12 mm

h = 9 mm

h = 6 mm

h = 3 mm

012345678

1 2 3 4

��1 ��2 ��3 ��4mm

h = 3 mm h = 12 mmh = 9 mmh = 6 mm

Om- IB24

0

2

4

0

6

8

202020

(mm) (mm)

dist

ance

(d)

dist

ance

(d)

a b

3. More faces yield greater accuracy, but accuracy drops as sensing distance increases.

(cont’d)

4face 6face 8face 12face

012345678

1 2 3 4

��1 ��2 ��3 ��4

h = 12 mm

h = 9 mm

h = 6 mm

h = 3 mm

012345678

1 2 3 4

��1 ��2 ��3 ��4mm

h = 3 mm h = 12 mmh = 9 mmh = 6 mm

Om- IB24

0

2

4

0

6

8

202020

(mm) (mm)

dist

ance

(d)

dist

ance

(d)

a b

3. More faces yield greater accuracy, but accuracy drops as sensing distance increases.

Summary of Findings

4face 6face 8face 12face 4. Smaller polyhedrons yield slightly greater accuracy.

012345678

1 2 3 4

��1 ��2 ��3 ��4

h = 12 mm

h = 9 mm

h = 6 mm

h = 3 mm

012345678

1 2 3 4

��1 ��2 ��3 ��4mm

h = 3 mm h = 12 mmh = 9 mmh = 6 mm

Om- IB24

0

2

4

0

6

8

202020

(mm) (mm)

dist

ance

(d)

dist

ance

(d)

a b

13.5mm radius

16mm radius

19.5mm radius

21mm radius

(cont’d)

Designing Ball + Constraint Interactions on Portable PlatformsApplications

Interacting with Constraints on a DisplayUsing Physical Constraint to Manipulate a Ball

Interacting with Constraints on a DisplayEmbodied Gestures

Interacting with Constraints on a DisplayBall+Constraint Interactions on Handheld Displays

Interacting with Constraints on a Display

conductive rubber

Touch interactions on a magnetic sphere

Interacting with Constraints on a DisplayClay-made territory as a continuous constraint

Interacting with Constraints on a DisplayAround-Device Interactions

Interacting with a Constraint Nearby a DisplayUsing Physical Constraints to Bridge Digital- and Real-world Experiences

Interacting with a Constraint Nearby a DisplayUsing Physical Constraints to Bridge Digital- and Real-world Experiences

Interacting with a Constraint Distant from a Display

Velcro strap

High-friction Materials as Physical Constraints

Interacting with a Constraint Distant from a Display

Velcro strap

High-friction Materials as Physical Constraints

GaussMarbles: Spherical Magnetic Tangibles for Interacting with Portable Physical Constraints Han-Chih Kuo, Rong-Hao Liang, Long-Fei Lin, Bing-Yu Chen National Taiwan University

Thanks! Questions?

The proposed magnetic regular polyhedron design enables • stable 3D tracking by an analog Hall-sensor grid

without priori knowledge • the explorations of ball+constraint interaction

on portable platforms

Conclusion