Media and Emotion Detection

In Ur FaceABC R+D: DEMO

2R+D 2016 2

CARS EMOTION DETECTION

In this rapid prototype, we looked beyond voice interactions and focused on other kinds of user feedback that connected cars could track in the future. We created a simple simulation that primarily focuses detecting a driver’s emotions and level of engagement as they listen to an ABC podcast.

Along with seven emotion metrics and audience engagement, the demo also includes two appearance metrics, current audience location, fake traffic data, and a voice commenting feature that uses Natural Language Processing (NLP). All of these are captured in real-time.

Car Context:

Anna, is driving her connected car to work early morning. The traffic is slowly moving, she is listening to the ‘Should you eat your pet?’ episode of the ABC’s Short and Curly podcast. The advanced artificial intelligence entity in a form of virtual assistant (VA) installed in her car notices her mood change from neutral to interest, increased heart rate, and eye dilation.

Relevance for content makers

> how will the ABC gain access to this kind of data? Will the tech giants be the middle man, gathering all the data around user comments and behaviours? If so, partnerships will be key.

> privacy - how much will our audiences be willing to allow us to use this kind of personal data? We’ll need to work out how to gauge the level of ‘personalisation’ people want in their experiences with ABC content and give them control.

> how might mood and engagement data inform the kinds of content experiences we offer audiences? e.g. if you could know how people were feeling at certain moments as they are listening, could this influence how you shape your program? How could this data help the ABC offer more personalised media experiences for drivers on their morning commute?

> what kinds of editorial considerations come into play with all of this? e.g. how do you know if the emotions detected at certain points are specifically in response to the content and not something else, like the traffic?

What is this?

R+D 2016 3

CARS EMOTION DETECTION

2016 202620212017 20182011

Learning to listen

We allow people to talk to us, basic commands and simple questions

Seamless, natural conversations

You can converse with your virtual assistant, who now feels more like a friend or companion

Anticipating audience needs

We can facilitate highly personal journeys and more conversational interactions with our content

Building understanding and a relationship

We begin to learn about user context and preferences, starting to recognise patterns

Speech recognition improves

Coupled with natural language understanding communication opens up from basic commands

Q&A with a podcast via a virtual assistant

NIGEL

Interactive podcast with voice comments

SHORT & CURLYVoice

activated news headlines

SOUND DEMO

Interactive audio explainers

IN FIVE YEARSVoice and emotion detection

IN UR FACE

ABC NewsbotFacebook

TRIAL

Interactive audio explainers

IN TEN YEARS

R+D 2016 4

CARS EMOTION DETECTION

Advanced CUIs might use more than language

ABC can use existing technology to start catering for basic voice commands and speech recognition - i.e. not conversational interactions.

Learning to listen

Anticipating audience needs and facilitating highly personal journeys, more conversational interactions

By this point, users will expect virtual assistants to dynamically link ABC content segments together on the fly, according to their needs.

Virtual assistants will listen to and understand us in much more advanced ways

Audiences will increasingly feel as though they can use more natural language to engage with ABC.

Next, we need to increase our understanding of more complex voice interactions so we can better gauge audience needs.

We also need to make ABC content machine readable - e.g. through automatic transcription.

Then we can serve audiences the content and services they ask for, and offer them richer, more personal experiences.

Building understanding

Seamless, natural conversations with your virtual assistant, who now feels more like a friend or companion

Virtual assistants will automatically generate content segments, and quickly summarise stories and information.

They will initiate conversations with us (i.e. it won’t just be us asking them questions).

Audiences will be able to have take very personalised journeys through our content.

Play demo

R+D 2016 6

CARS EMOTION DETECTION

Audience EngagementDashboard for producersImagine having access to a tool that allowed you to easily gauge how your audience was feeling and how engaged they were whilst listening to your content?

This concept of an “engagement dashboard” shows just how nuanced our audience data could be in the future. Features include - tracking mood, attention levels over time, voice comments, user location, user submitted photos, voice responses to polls.

R+D 2016 7

CARS EMOTION DETECTION

What technologies enable this type of experience?NOW

Emotion Recognition - machine’s ability to understand the facial expressions and visual cues of an individual to identify human emotion. In this demo, we used Affectiva’s Emotion recognition to detect real-time emotions from the audience.

Geo-Location - ability to identify a person’s real-world geographic location. In this demo, we used the Google Maps geolocation feature. For privacy reasons, users would have to allow their device access their GPS location for this to work.

Speech-to-text Recognizer - machine’s ability to transcribe audio into written words. In this demo, we used Google’s Web Speech API.

Natural Language Processing - machine’s ability to extract meaning from natural language. In this demo, we used IBM’s AlchemyAPI to extract keywords, topics and sentiment for the audience’s comment.

THE FUTURE

Examples of the kinds of additional data that will be able to be captured inside future connected cars include:

Face Recognition - machine’s ability to identify and verify a person from an image or a video frame.

Biometrics - machine’s ability to identify and verify a person from measurable physical characteristics such as retinal scan, facial features, voice, and fingerprints.

Real-time Traffic, Weather

Live-streaming my emotions? That is so intrusive! I know so much of our lives, that data is easily collectible, but at the same time, I don't

like having a camera looking at me.Dave, 23

AUDIENCE FEEDBACK

R+D 2016 9

CARS EMOTION DETECTION

We discussed thisconcept with content makersHow do ABC radio makers and reporters feel about technologies that allow them to collect data or comments directly from the audience?

We invited 12 content makers from news, radio and regional to try a demo that used facial recognition software to track their emotion while listening to a podcast, and feed that data to a dashboard for analysis by producers. In addition, the tool could capture voice comments and make them available as text or audio.

This is really interesting because we're doing comedy, the most subjective of things. The reaction we get is from a few vocal people and this might be a way of proving in real time that a lot of people like it... Comedy is entirely subjective. To predict what an audience is going to find funny is the holy grail of commissioning. This would be a great feedback report on 'should we commission this again?'

TomProducer

R+D 2016 10

CARS EMOTION DETECTION

“If people would do this, it’s a god send for radio makers”

People who produce both live radio and podcasts for the ABC have very little audience feedback to guide their decisions - relying largely on a vocal minority on social media.

Direct access to audience responses, whether comments or emotion data - while a bit scary, would give insight and help them make decisions about what to commission or produce.

“It’s more raw material - and fewer barriers”

The idea of ‘farming’ voice comments either as audio or as text to be read to use in a live broadcast is an exciting possibility.

Direct access - literally talking back to the radio - might help remove some of the barriers to audience feedback that currently exist.

It could also give a new dimension to audio content as it would allow people to replicate online community behaviour more seamlessly within a listening experience.

In the next couple of years it's going to get pretty good at recognising tones and things like that. And I don't know how far off the facial recognition stuff is, but that coupled with what is being said, you'll have a much better idea of whether they're confused or interested or bored.

Key reactions from ABC content makers

StephenReporter

R+D 2016 11

CARS EMOTION DETECTION

This is really cool, but it’s highly inaccurate.

People enjoyed playing with the facial-recognition software. They had fun trying it out. We heard many exclamations like ‘It’s cool!’ or ‘Interesting’.

Many people questioned the accuracy of the data, and the appropriateness of the ABC interpreting this data, especially in the context of a car. What if I’m responding in surprise to a cat running out of the road in-front of me?

Many said they’d like to see how the technology develops, and to test it out again once it improves.

It’s a bit creepy, but Facebook is spying on us all anyway!

There were wildly different reactions to the emotion detection feature. A couple of participants weren’t at all bothered by privacy implications, while others felt that data related to one’s face is highly personal, even if no image is captured or stored. Interestingly, the willingness to freely share this type of data with ABC did not correlate those aged under 25.

However, detecting location and other personal data such as gender and age, people were much more relaxed. They knew that tech giants like Google and Facebook were already capturing and using this information and so why not release it to the ABC as well?

How did audience respond?We showed the In Ur Face demo to 16 audience members and walked them through its various features and what the prototype aimed to demonstrate to the ABC. We sought to gauge their reaction, and asked them to describe how willing they would be to share their reactions, responses and comments with the ABC.

What value will I get out of it?

Most participants could clearly see how these types of datasets might be useful to ABC content makers. However, what was unclear to participants is how giving their data to the ABC could translate as an improved or more engaging experience with ABC’s content. People felt that the value proposition would need to be clearly articulated and demonstrated by the ABC before they’d sign up.

Fortunately, ABC is still in a league of media brands that people trust. People expect ABC to collect, and interpret personal data in a responsible and ethical way.

R+D 2016 12

CARS EMOTION DETECTION

Emotion data collection. Is it too creepy for audiences?16 user testing participants were asked to mark on the graph how they felt about the voice commenting feature.

You are listening to an ABC reporter interview a politician on an issue you feel strongly about. How willing are you to allow to software track your face; measuring your facial expressions and reporting your emotional responses back to the ABC so they can gauge public sentiment in reaction to the interview?

No way

Sure thing!

Live-streaming my emotions? That is so intrusive! I know so much of our lives, that data is easily collectible, but at the same time, I don't like having a camera looking at me.

DuncanUser testing participant

This is really cool! It would be an in-built feature of the car, so I wouldn't care. It's just there all the time.

AnnaUser testing participant

To have someone watching me feel a bit funny, uncomfortable. I'm not normally so sensitive about this stuff, but I feel like my time in my car is personal. If I choose to pick my nose, or turn and chat to the kids, that's what I choose to do! I know it will all be super secure and confidential, but it doesn't sit well with me.

PipUser testing participant

I'd feel self-conscious to begin with. I guess the fact that it's data about my facial expression, rather than capturing my image, that's a little more reassuring.

MattUser testing participant

I don't think there's a problem with taking that information so you can gauge how receptive people are, or how they have reacted to the content that you're broadcasting. It's all metadata. So long as you're not taking a photo of me, that's fine.

SantoUser testing participant

R+D 2016 13

CARS EMOTION DETECTION

Voice commenting. Would audiences do it?

I wouldn't use this, but I think it's a good idea because I know a lot of people want to be heard, but I am not the kind of person to voice my opinion.

LaceyUser testing participant

I think it's an interesting idea to record a voice reflection. I listen to a podcast called Modern Love and they're always asking for people to submit stories, but it's too much hard work to head the website and record...but I would totally do this.

EilyUser testing participant

Sometimes with Hack I feel like screaming at the radio when you get some guy just calling up and I think 'What are you saying?!' I guess that could be interesting [to be able to leave a voice comment], because you can't call when driving. But at the same time, if it's made public, would I regret it later? Like a tweet.

DuncanUser testing participant

I think I'm too lazy to give an opinion! I'd rather give a statistic [vote in poll] because that would come up with a measurement [in relation to other people other people] but with comments, you could just argue on and on! Comments might not make a difference to political outcome, but a poll percentage might.

MingUser testing participant

I am interested in trying this out, but on social media, I don't express my opinions that often. I tend to hold back on expressing my opinion to a wider audience because I don't want to sound stupid. I don't want to be caught up with the drama of keyboard warriors!

DanUser testing participant

16 user testing participants were asked to mark on the graph how they felt about the voice commenting feature.

You are listening to an ABC reporter interview a politician on an issue you feel strongly about. How likely would you be to ‘talk back to the radio’ and instantly submit a voice recording of your comment that might be used by the media or a service?

No way

Sure thing!

If this [emotion detection] was to be a thing, I would need to know what I get out of it. What's the incentive? I would want to

know what it's for.

Dan, 22

AUDIENCE FEEDBACK

R+D 2016 15

CARS INTERACTIVE PODCAST - SHORT & CURLY

Questions or comments?

Astrid Scott

Priscilla Davies

NicolaasEarnshaw

Jo Szczepanska

Amy Nelson

JanelleHerrera

Anne Lin

Charlie Szasz

Email ABC R&D to get in touch.