music everywhere project half presentation
TRANSCRIPT
PowerPoint Presentation
Halves Presentation Fall 2016AR-Assisted Piano Learning
AR-assisted piano learning using a MIDI-enabled piano with a MS HoloLens.
MusicianMusic Everywhere Team
Seth Glickman Byunghwan Lee Shantanu DasFu Yen HsiaoProgrammerProgrammerUI Designer
Faculty advisors.
Pitch Project InspirationTo explore how Augmented Reality technology can be used to assist in the learning of a musical instrument.
Surveys and Outreach
ReadingmusicRhythm ortimingTwo-hand coordination
Student ResponsesTeacher Responses37 Students11 TeachersWillingness to use technology in lessonsBarriers to successful piano learning27students volunteered to playtestHighest 3 responses
72.7%54.5%
Big PictureBest use of the AR tools we developed
AR Projection Interface
?QuartersPiano and HoloLens Connection
WebRTCWhat have we learned. Show the picture shows the problem
Quarter Semester Feedback
Top notes from faculty walkthroughs and sitdowns
What style(s) of music will you be working in? Is reading music required?Choose an area to investigate that augmented reality is well-suited to address.Experiment with animated hand as instructional device.Focus on precision in keyboard overlays.
Quarter Semester Feedback
Top notes from faculty walkthroughs and sitdowns
What style(s) of music will you be working in? Is reading music required?Choose an area to investigate that augmented reality is well-suited to address.Experiment with animated hand as instructional device.Focus on precision in keyboard overlays.
Quarter Semester Feedback
Top notes from faculty walkthroughs and sitdowns
What style(s) of music will you be working in? Is reading music required?Choose an area to investigate that augmented reality is well-suited to address.Experiment with animated hand as instructional device.Focus on precision in keyboard overlays.
Quarter Semester Feedback
Top notes from faculty walkthroughs and sitdowns
What style(s) of music will you be working in? Is reading music required?Choose an area to investigate that augmented reality is well-suited to address.Experiment with animated hand as instructional device.Focus on precision in keyboard overlays.
Quarter Semester Feedback
Top notes from faculty walkthroughs and sitdowns
What style(s) of music will you be working in? Is reading music required?Choose an area to investigate that augmented reality is well-suited to address.Experiment with animated hand as instructional device.Focus on precision in keyboard overlays.
Big Picture
Remote LearningBest use of the AR tools we developed
AR Projection Interface
?Artistic ExperienceScore ReadingImprovisationImprovisationScore ReadingQuartersPiano and HoloLens Connection
WebRTCWhat have we learned. Show the picture shows the problem
HoloLens and Music Improvisation
UI design informs users which notes will sound good, and change as the song progresses.Animated band provides music for users to play along with (and solos to imitate.
Research: Improvisation as Music InstructionHow do musicians who dont read music learn to play?The parallel paths of score readers and those who play by earLearning a to speak a musical language
Research on supplemental improvisation instruction included in music curriculum:
Can Improvisation be taught? A Call for free improvisation in our schools-Hickey, M.Northwestern UniversityUnveiling the Mysteries of Musical Spontaneity-Campbell, PMENC: National Association for Music Education
Developing Musicianship Through Improvisation-Azzara, CPh.D
Target Audience
Subject Matter Experts
Jack KurutzMavis PanAmaury MoralesPittsburgh Music AcademyBrooklyn College, Afro-Latin Jazz OrchestraCarnegie Mellon UniversityAdvisorReviewerReviewerTeaches basic improvisation with his private piano students, Jack is interested in having his 30+ students try an AR prototypeClassical and jazz instructor, performer and recording artist willing to be a sounding board for improvisation concepts
ConsultantWayne OquinThe Juilliard SchoolClassical piano instructor interested in contributing to improvisation model(s) for classical repertoireClassical composer and pianist will assist with high-level concepts and teaching strategiesMOST INVOLVEMENTLEAST INVOLVEMENT
Tina Blaine, Jack, Amaury, Mavis, Roger Powell, Dannenberg?
Related WorkPiano instruction
Music engagementAcademic AR piano studyLED-lighted keyboard hardware
Guitar instruction gameAR piano prototype
How different are you from these? AR + Real piano methodology. List each ones problem and goal. How you guys work differently in AR field?
Learning Objectives
Introduce skills of improvisation in various musical styles.Instruct improvisation techniques, phrasing and expression.Teach and reinforce two hand coordination.
Target Audience:Students 14+Intended as a supplement to standard piano lessons or to be used by a former piano student who wants to gain instruction and experience in improvisation.
4 styles
Development
Make lesson design process more flexible.
Development: Lesson Builder
We buildt lesson builder which can designer edit the lessone withouth coding.it provides flexibility and control of the structure for the lesson designerIt frees up programmers availablity to focus instead on building new features and new nodes.This is basic usage of lesson buidler
Development: Lesson Builder
SibeliusExample: Score NodeImport, parse and draw musical scores in Unity.
One of the example as score node,Score node functions loading score data from composition program to unity.
Development: Lesson Builder
Create node structure for all features.
Like score node, Each feature is a node inside of lesson builder.Right side of picture is one of our lesson.By using this, we achived more iterative and efficeint lesson design process
Development: Band System
Guitar PlayerBass Player
Drummer(Robert Johnson)(Ray Brown)(Albert Heath)
Blues Style Stage in 1950s
In order to enhance the improv experience, we design virtual band characters based on real musicians who will be playing with users. The band provide the sense of participation to users with responsive animations and representative musical theming. Take Blues band for an example, we got a cool guitar player, bass player and drummer in decent suits on a stage in a blues bar in 50s. Now, we are developing more music bands: Rock, Jazz and Classical in order to complete our music improv journey.
Development: Band SystemBass Player
(Kim Deal)Drummer
(Carter Beauford)Guitar Player
(Nigel Tufnel)Singer(David Lee Roth)
This is the rock style virtual band. More colorful, wild and complex.
Development: Band SystemPiano AccompanimentPiano ImprovisationPlaying With Band
It provides the experience which make user feel like playing with bandWe implemented central clock from which to sync all aspects of animation, sound and musical score in the project. Based on setted bpm or time scaleFor example , the drummer has sound clip and score data so it gonna hit drum right timing with musicThis can make more juicy experience to user.
Development: AR Piano Key Overlays
Place overlays to prevent players from looking down.Overlays should occupy minimal space in FOV to make space for the score and band.
We recognized very early in our development that an accurate and easily understandable overlays will be a very crucial for our project.Therefore we built a overlay interface where we could integrate new overlays very easily.UI feedback of the player - know when to play what to play - all filtered through AR - through a field of view when limited
UI: Overlay Iteration and Testing
Key overlay UI system plays a critical role in the improv experience and also an important design component in this project. How to guide users to hit the right keys with limited FOV efficiently without always looking down which is not the proper way to play piano and interact with the virtual band on the top of the piano. We hope them to look up
UI: Overlay Iteration and Testing
Key Overlay: Quarter / FullNo key overlay: Oval with line / TriangleLookup: Floating keys / Mirrored Keys
Last, we design floating keys hovering in the air and a mirrored keyboard in order to encourage users to look up.
22.2%25.9%
14.8%14.8%2.7%7.4%11.1%Playtest Results
53.8%38.5%
7.7%
Piano ExperienceOverlay PreferenceBeginnerIntermediate/Advanced
OtherTriangleMirrorFull KeyHalf KeyQuarter KeyFloating KeyOval with Lines
Out of the overlays we tested , the playtesters could identify the notes with high accuracy for all the overlay options. However, they preferred the mirror and the triangle overlay. Therefore from the results we can say that there were 2 favourites.
Development: Finger Assistance
Guest unfamiliar with the piano will often not play with proper fingers.
From our playtesting we observed, that Guest unfamiliar with the piano will often not play with proper fingers.
One of the benefits of using AR above other systems, is the ability to show hand projections for specific purposes.We can remedy the problem by showing a virtual hand that demonstrates finger technique and proper form
Development: AR Hand Display
Whats Next
Plan
ImprovisationImprovisationQuarters
Beyond the basic toolset
WebRTC
Stage 1 ImprovisationBand: BluesInstruction: Dom7 scale minor pentatonic scale Triads: I, IV, V Phrasing
WebRTC
Stage 1 ImprovisationBand: BluesInstruction: Dom7 scale minor pentatonic scale Triads: I, IV, V PhrasingStage 2 ImprovisationBand: RockReview: Dom7 scale minor pentatonic scale Triads: I, IV, V PhrasingAdd:Major scale Natural minor scale Triads: ii, iii, vi, vii Seventh: V7, vii7 Articulation Basic rhythms
Stage 3 ImprovisationBand: Jazz IntroductionReview:Major scale Natural minor scale Triads: ii, iii, vi, vii Sevenths: V7, vii7 Articulation Basic rhythmsAdd: ChromaticismModes Ear Training Musical Context Chart Reading Intermediate rhythms
Stage 4 ImprovisationEnsemble: ClassicalContinue: Ear Training Musical Context Chart Reading Intermediate rhythmsAdd: Sevenths: V of IV, V of V, Temporary tonicization Non-chord tones Arpeggios Chord function Tonic Dominant Sub-dominant Styles of classical improvisation
WebRTC
Project Scope
Level 5: Hands together
Level 4: Alternating hands
Level 3: Right hand improvisation
Level 2: Left hand accompaniment
Level 1: Improvisation basics
Showcase:Rock improvisation
Showcase:Jazz improvisation
BLUES
ROCK
JAZZ
CLASSICAL
Showcase:Classical improvisation
Technical Support
References, Feedback and Advice
Steve Audia, Jon Underwood, Bryan Maher, David Purta
Eric Brown, Anthony Daniels, Tina Blaine, Jessica Hammer, John Dessler, Scott Stevens, Jesse Schell, Michael Christel, Brenda Harger, Shirley Yee, Chris Klug, Tom Corbett, Heather Kelley, Richard Hilleman, Robyn Miller, Roger Powell, Jay Schneider, Jesse Stiles, Mavis Pan, Amaury Morales, Jack Kurutz, Daniel Cohen, Julian Korzeniosky, Drew DavidsonSpecial thanks to...
Early Prototyping