IOWA STATE UNIVERSITYIOWA STATE UNIVERSITYDepartment of Electrical and Computer EngineeringDepartment of Electrical and Computer Engineering

CComputer Control of omputer Control of Theater Performance Theater Performance

ElectronicsElectronics

Clients:Clients:Co-Motion Dance CompanyCo-Motion Dance Company

Iowa State DanceIowa State Dance

SD May06-SD May06-1818

Team Members: Faculty Advisors:

Amanda Farniok Dr. Julie Dickerson

Sheng Ly Prof. Gerald Sheble*

Alex Sills Dr. David Stephenson

Tarun Bhatia* *Only on project Fall 2005

April 26, April 26, 20062006

Presentation OutlinePresentation Outline

Project OverviewProject Overview Research Research

ActivitiesActivities Hardware Hardware

ConfigurationConfiguration Software Software

DevelopmentDevelopment ImplementationImplementation

Resources and Resources and SchedulingScheduling

CommercializatioCommercializationn

Future WorkFuture Work Lessons LearnedLessons Learned Closing RemarksClosing Remarks QuestionsQuestions

DefinitionsDefinitions

Isadora Isadora - Isadora is a graphic programming - Isadora is a graphic programming environment that provides interactive control over environment that provides interactive control over digital media, with special emphasis on the real-time digital media, with special emphasis on the real-time manipulation of digital video.manipulation of digital video.

SDK SDK - Software development kit- Software development kit ActorActor – Module for Isadora software developed as an – Module for Isadora software developed as an

end product.end product. SceneScene – A collection of user grouped actors in Isadora. – A collection of user grouped actors in Isadora. TinyOSTinyOS - TinyOS operating system designed for the - TinyOS operating system designed for the

Crossbow transceiversCrossbow transceivers nesCnesC - an extension to the C programming language - an extension to the C programming language

designed to embody the structuring concepts and designed to embody the structuring concepts and execution model of TinyOS.execution model of TinyOS.

OSC OSC – Open Sound Control– Open Sound Control

AcknowledgementAcknowledgement

Janice Baker – Iowa State DanceJanice Baker – Iowa State Dance Valerie Williams – Co-Motion Dance Valerie Williams – Co-Motion Dance

CompanyCompany Dr. Julie Dickerson – faculty advisor Dr. Julie Dickerson – faculty advisor Dr. Gerald Sheble – faculty advisorDr. Gerald Sheble – faculty advisor Dr. David Stephenson – emeritus facultyDr. David Stephenson – emeritus faculty Jason Boyd – ECpE departmental Jason Boyd – ECpE departmental

techniciantechnician Daji Qiao – associate professorDaji Qiao – associate professor Wen-Chieh – HCI 572 studentWen-Chieh – HCI 572 student

Problem StatementProblem Statement

Control dancer environment without the Control dancer environment without the help of stagehands or technicianshelp of stagehands or technicians

Need dancer-operated sensor pack Need dancer-operated sensor pack which sends input to be received by a which sends input to be received by a computer program called Isadora computer program called Isadora

Isadora can be programmed to control Isadora can be programmed to control lighting, video projection, audio, and lighting, video projection, audio, and other characteristics of the dancer’s other characteristics of the dancer’s stage environmentstage environment

Similar commercial technologies are not Similar commercial technologies are not able to differentiate between the inputsable to differentiate between the inputs

Solution-approach Solution-approach StatementStatement

Crossbow Technology Components: Crossbow Technology Components: Sensor Board, Transceivers, Serial Sensor Board, Transceivers, Serial Interface Board Interface Board

Flex sensors (variable resistors) used to Flex sensors (variable resistors) used to monitor movementmonitor movement

Used nesC code in TinyOS (Crossbow Used nesC code in TinyOS (Crossbow program) to program components and program) to program components and forward dataforward data

Used C code to write serial-to-OSC relayUsed C code to write serial-to-OSC relay

Operating Operating EnvironmentEnvironment Theatrical dance settings: Betty Toman Theatrical dance settings: Betty Toman

Dance Studio in Forker Hall, Fischer Dance Studio in Forker Hall, Fischer Theater, Stephens Auditorium, Ames City Theater, Stephens Auditorium, Ames City AuditoriumAuditorium

Possibility of ambient electromagnetic Possibility of ambient electromagnetic noise caused by the multitude of noise caused by the multitude of electrical equipment in the immediate electrical equipment in the immediate vicinityvicinity

Substantial abuse caused by the motion Substantial abuse caused by the motion of the dancers, their impact with the floor of the dancers, their impact with the floor or wall, and the sweat and heat that their or wall, and the sweat and heat that their bodies produce during performancebodies produce during performance

Intended Users and Intended Users and UsesUsesUsersUsers Members of Iowa State Members of Iowa State

DanceDance Members of Co-Motion Members of Co-Motion

Dance CompanyDance Company Directors and colleagues Directors and colleagues

of the companiesof the companies

UsesUses Practices, performances, Practices, performances,

and productions of the and productions of the usersusers

Usable in any situation Usable in any situation analogous to the analogous to the conditions of these conditions of these groups’ activitiesgroups’ activities

Transmitter is within the Transmitter is within the requested range of 60 requested range of 60 feet from the receiverfeet from the receiver

Impacts inflicted on the Impacts inflicted on the transmitter and sensor transmitter and sensor by the dancers are by the dancers are minorminor

Assumptions and Assumptions and LimitationsLimitationsAssumptionsAssumptions Flex sensor data Flex sensor data

transmission onlytransmission only No clear line of sight No clear line of sight

from the transmitter to from the transmitter to the receiverthe receiver

Receiver and computer, Receiver and computer, as well as the visual as well as the visual aids, will run on power aids, will run on power supplied from an supplied from an external sourceexternal source

Sufficient power to Sufficient power to operate for at least the operate for at least the duration of an ISU/Co-duration of an ISU/Co-Motion dance productionMotion dance production

LimitationsLimitations End-product will be only End-product will be only

one transmitter/receiver one transmitter/receiver pair but will allow for pair but will allow for expansion for up to four expansion for up to four transmitterstransmitters

Small and compactly Small and compactly packaged to allow for packaged to allow for full range of motion by full range of motion by the userthe user

Distance from the Distance from the transmitter to the transmitter to the receiver will be at most receiver will be at most 60 feet60 feet

End-Product and End-Product and DeliverablesDeliverables

Wireless transmitter with four Wireless transmitter with four sensorssensors

Receiver connected through Receiver connected through Serial/USB adapter to computerSerial/USB adapter to computer

Sensor watcher written to Sensor watcher written to communicate between TinyOS and communicate between TinyOS and IsadoraIsadora

User Manual for client to recreate User Manual for client to recreate and use full functionality of systemand use full functionality of system

Data Flow - Data Flow - FunctionalityFunctionality

Hardware Flow Diagram

Flex Sensor, Analog Voltage Signal

MDA300 Analog Input Board

MICAz Transmitter Mote

MICAz Receiver Mote

MIB510 Serial Interface

2.4 GHz Wireless

Transmission

HA

RD

WA

RE

CO

NN

EC

TE

D T

O C

OM

PU

TE

R

HA

RD

WA

RE

WO

RN

BY

DA

NC

ER

Previous Research Previous Research ProjectProject VrJuggler was previously used on a research VrJuggler was previously used on a research

project to manipulate input from sensors project to manipulate input from sensors and display 3D video on projectors.and display 3D video on projectors.

Certain aspects of previous research were Certain aspects of previous research were taken into considerationtaken into consideration– Flex sensorsFlex sensors– Crossbow Technology Inc.Crossbow Technology Inc.

Present Present AccomplishmentsAccomplishments Transmitter/receiver tested for range Transmitter/receiver tested for range

and use with sensors, being used in and use with sensors, being used in theatertheater

Hardware programmed through nesC Hardware programmed through nesC for correspondence between for correspondence between transmitter and computertransmitter and computer

Packaging purchased, installedPackaging purchased, installed Able to get sensor readings into Able to get sensor readings into

IsadoraIsadora

Future TasksFuture Tasks

Complete User ManualComplete User Manual Purchase Crossbow system for Purchase Crossbow system for

client’s possessionclient’s possession

Hardware Design - Hardware Design - ResearchResearch Analog audio wireless Analog audio wireless

transmission methodtransmission method– Problems: Modulation/compression Problems: Modulation/compression

issues, capacity/channel issues, issues, capacity/channel issues, possible interference problemspossible interference problems

Hardware Design - Hardware Design - ResearchResearch Industrial SCADA-type wireless Industrial SCADA-type wireless

transmission methodtransmission method– Problems: Bulky packaging, insufficient Problems: Bulky packaging, insufficient

channels/capacity, possible software channels/capacity, possible software (Isadora) compatibility issues(Isadora) compatibility issues

Hardware Design - Hardware Design - XBowXBow Crossbow “Mote” Transmission Crossbow “Mote” Transmission

MethodMethod– Reasons for choice: Compact sizing, Reasons for choice: Compact sizing,

2.4GHz band avoids interference 2.4GHz band avoids interference and allows for multiple transmitters, and allows for multiple transmitters, TinyOS accompanying software TinyOS accompanying software suited for project goalssuited for project goals

– Possible problems: TinyOS/Isadora Possible problems: TinyOS/Isadora communication, line of sight issuescommunication, line of sight issues

Hardware Design - Hardware Design - XBowXBow

Analog Output CircuitAnalog Output Circuit

Crossbow motes accept analog inputsCrossbow motes accept analog inputs Flex sensor variable resistors used in Flex sensor variable resistors used in

a simple analog circuit to give 1.31 - a simple analog circuit to give 1.31 - 1.95 Volt output to mote board1.95 Volt output to mote board

Circuit Diagram - Four Circuit Diagram - Four Sensor OutputSensor Output

MDA300 Sensor BoardMDA300 Sensor Board

Accepts analog inputs via screw terminalsAccepts analog inputs via screw terminals Provides excitation voltagesProvides excitation voltages Converts analog to digital signal for wireless Converts analog to digital signal for wireless

transmissiontransmission

MICAz TransceiversMICAz Transceivers

Connects to sensor board and serial Connects to sensor board and serial interface with 51-pin connectorinterface with 51-pin connector

Transmits on 2.4 GHz Zigbee compliant Transmits on 2.4 GHz Zigbee compliant band (good range, multiple channels, band (good range, multiple channels, minimal interference problems)minimal interference problems)

Powers sensor board, simple AA battery Powers sensor board, simple AA battery power supplypower supply

MIB510 Serial MIB510 Serial InterfaceInterface

Receives signal from transceiver and Receives signal from transceiver and transmits to computer through TinyOS transmits to computer through TinyOS softwaresoftware

Powered by external AC adapterPowered by external AC adapter Through TinyOS, programs/configures Through TinyOS, programs/configures

transceivers and sensor boardtransceivers and sensor board

Hardware Software Hardware Software IntegrationIntegration TinyOS is a platform bridging the TinyOS is a platform bridging the

hardware serial input using nesChardware serial input using nesC– Provides Libraries for parsing sensor Provides Libraries for parsing sensor

datadata– Environment for the Programming the Environment for the Programming the

Wireless Motes.Wireless Motes. Program for receiving wireless data Program for receiving wireless data

and putting on the serial port and putting on the serial port provide by Xbowprovide by Xbow

Wireless Transmitter Wireless Transmitter ProgramProgram Developed in nesCDeveloped in nesC

Isadora Module Isadora Module DevelopmentDevelopment Provides overall design needed Provides overall design needed

for working Isadora Plugin/Actor.for working Isadora Plugin/Actor. Provides data structures and Provides data structures and

functions that are needed to functions that are needed to implement.implement.

General Flow of the General Flow of the FunctionsFunctions

Open Sound Control Open Sound Control SolutionSolution

– C backend solutionC backend solution

Isadora Actor MockupIsadora Actor Mockup

Our Actor Mockup

Actor Link Connection

Putting the Pieces Putting the Pieces TogetherTogetherImplementationImplementation Integration between hardware and software Integration between hardware and software

using the research and documentationusing the research and documentation Implementing in theater with components Implementing in theater with components

on loan from the Department of Electrical on loan from the Department of Electrical EngineeringEngineering

Testing Procedures and Testing Procedures and ResultsResults

Measuring flex sensor voltageMeasuring flex sensor voltage– 1.31 to 1.95 volts1.31 to 1.95 volts

Distance between transmitter/receiver Distance between transmitter/receiver and line of sight issuesand line of sight issues– Work up to 60 feet and without line of Work up to 60 feet and without line of

sightsight– Reliability issues when in enclosed cornersReliability issues when in enclosed corners

Packaging integrityPackaging integrity Currently beta testing by client for use Currently beta testing by client for use

in a performancein a performance

ResourcesResources

ScheduleSchedule

Project EvaluationProject Evaluation

Research of Hardware and SoftwareResearch of Hardware and Software– 100% Complete100% Complete– Using findings to implement systemUsing findings to implement system

TestingTesting– 95% Complete95% Complete– Hardware works with softwareHardware works with software

ImplementationImplementation– 98% Complete98% Complete– Software designed and loaded onto hardware, currently Software designed and loaded onto hardware, currently

in usein use DocumentationDocumentation

– 95% Complete95% Complete– All formal reports completeAll formal reports complete– User Manual under revisionUser Manual under revision

CommercializationCommercialization

Theatrical companies readily share Theatrical companies readily share technological findingstechnological findings

Share findings with those that shared Share findings with those that shared information with usinformation with us

Consult Crossbow Technology before Consult Crossbow Technology before any package sales could be doneany package sales could be done

Selling Price: $1,000Selling Price: $1,000 Market: High demand in performing Market: High demand in performing

arts in near futurearts in near future

Future Future RecommendationsRecommendations Switch batteries from AA to 3 V coinSwitch batteries from AA to 3 V coin Custom design casing for compact Custom design casing for compact

yet safe designyet safe design Connectors for between sensors and Connectors for between sensors and

wires and stability of sensor jointwires and stability of sensor joint Expand to six sensors per Expand to six sensors per

transmittertransmitter Expand to four transmittersExpand to four transmitters Use different sensor inputUse different sensor input

Lessons LearnedLessons LearnedSuccessesSuccesses

Group teamwork and Group teamwork and communicationcommunication

Group supervision by facultyGroup supervision by faculty Attainable trial hardwareAttainable trial hardware

Lessons LearnedLessons LearnedSetbacksSetbacks

Loss of team memberLoss of team member Original circuit design not Original circuit design not

applicableapplicable Developing software before Developing software before

hardware was decidedhardware was decided

Lessons LearnedLessons LearnedExperience GainedExperience Gained

nesC ProgrammingnesC Programming RF communicationRF communication MultitaskingMultitasking Client relationsClient relations Communicating different Communicating different

expertiseexpertise Delegating responsibilitiesDelegating responsibilities

Lessons LearnedLessons LearnedIf we did it all again…If we did it all again…

Hardware team more involved Hardware team more involved with software developmentwith software development

More research on software cross-More research on software cross-platform developmentplatform development

Risk and Risk Risk and Risk ManagementManagement Procurement of MaterialsProcurement of Materials

– Hardware borrowed from department for testing Hardware borrowed from department for testing and implementation before purchaseand implementation before purchase

Loss of team memberLoss of team member– Encountered and overcome by members gathering Encountered and overcome by members gathering

completed progress and taking on more taskscompleted progress and taking on more tasks– Involved other resources – HCI 572 Design and Involved other resources – HCI 572 Design and

Evaluation of HCIEvaluation of HCI Data LossData Loss

– All documentation easily transferable between All documentation easily transferable between group membersgroup members

Physical DamagePhysical Damage– Kept components in tackle box for safe keepingKept components in tackle box for safe keeping

Closing RemarksClosing Remarks

Successfully provided means for Successfully provided means for dancers to control own dancers to control own environmentenvironment

Human computer interactionHuman computer interaction Leading other companies in Leading other companies in

theatrical technologytheatrical technology Expansion is possibleExpansion is possible

QuestionsQuestions