bbc iplayer: bigger, better, faster (and shorter)
DESCRIPTION
As presented in Salford at develop:BBC, 11/11/2014. A shorter version of the presentation given at the AWSUKUG, with but with a section added where I talk about the historical context of the project.TRANSCRIPT
BBC Media Services
Rachel Evans
bigger, better, faster
@rvedotrc
(and shorter)
BBC iPlayer: bigger, better, faster A year ago, BBC iPlayer could have died - but it didn’t. Instead, we built a bigger, better, faster iPlayer that provides a foundation for the future. Find out how this was achieved, what part AWS plays in iPlayer’s success, and what’s next for BBC online media.”
On October 1st, 2013, iPlayer didn’t die
“On October 1st 2013, iPlayer didn’t die. But it could have. The reason iPlayer is still alive is Video Factory, and Amazon Web Services play a big part in Video Factory’s success. My name’s Rachel Evans. I’m a Principal Software Engineer in BBC Media Services. We created Video Factory.”
“For the next 20 minutes or so, I’d like to tell you some of the Video Factory story. How it came to exist; what it is; how we made it; and a glimpse into Video Factory’s future. And of course, what part Amazon plays in the whole story.”
What is BBC Media Services?
(full name: BBC Future Media Platform Media Services)
Video Factory was created by BBC Media Services.
Who are we? Here’s our mission statement:
“Publish all BBC AV media produced for IP platforms”
“AV” means audio and video, includes radio and TV Includes iPlayer, iPlayer Radio, News, Sport Both live and on-demand
✓✓✓
⬅ ☹
The shiny new system
2008 - Hosted in-house; used contracted 3rd party services for storage and transcode.
Limited capacity. Bad architecture. Bad engineering. Ageing badly. Didn’t scale well. This system did not have a long-term future.
The legacy system
2008 - Hosted in-house; used contracted 3rd party services for storage and transcode.
Limited capacity. Bad architecture. Bad engineering. Ageing badly. Didn’t scale well. This system did not have a long-term future.
Why 1st October 2013?
In May 2012, the BBC decided not to renew that 3rd party contract. It was to be allowed to lapse, at the end of September 2013. So this system, and therefore iPlayer, will die. “By the time the London 2012 Olympics was out of the way, we had just over 12 months to build a replacement.”
Why the cloud?
Elasticity etc
June 2012: First steps into the cloud
“At this point, there is no BBC Amazon account yet. There are no BBC FM projects in the cloud. We’re taking a step into the unknown.” We choose Amazon. June 2012: AWS accounts created.
Starting small, thinking big
- First BBC FM project in the cloud: Sky (no Cosmos)
- Second BBC FM project in the cloud: Video Factory (early Cosmos)
- prove we can transcode video in the cloud Both of these things are key to what we’re going to need for Video Factory.
The origin of Video Factory
“By this point you might be wondering: What does Video Factory actually do?”
Video Factory in a nutshell
Two things drive Video Factory
(source video + metadata => transcode)
source video+
programme data
programme data: - what programme is this; - where and when is it broadcast; - which platforms do we publication rights for
source video+
programme data
transcode, distribute, and publish
=
programme data: - what programme is this; - where and when is it broadcast; - which platforms do we publication rights for
PrerecordedLive
Transcode
Distribute
Publish
Once we’ve got the video, the rest of the chain is the same.
“So let’s talk about live. Most iPlayer content is stuff that’s been on TV, so if we can capture and publish that, then iPlayer won’t die. So, we built what I like to think of as …”
The world’s largest public-service video recorder
“… which we call ‘Mezz-to-VOD’, and here’s a quick overview of how it works.”
Mezz-to-VOD:
“In a secure location somewhere is part of the BBC’s TV broadcast chain.” Playout; Thomson Video Networks; RTP video over multicast UDP; includes timecodes. Capture and chunk. Mustn’t miss a single packet. Upload chunks to S3. click - Playout; broadcast end SNS message; find relevant chunks and join click - Transcode with trim; distribute; publish. click - talk about inaccurate trims and “Resilient broadcast-grade system”.
Mezz-to-VOD:
“In a secure location somewhere is part of the BBC’s TV broadcast chain.” Playout; Thomson Video Networks; RTP video over multicast UDP; includes timecodes. Capture and chunk. Mustn’t miss a single packet. Upload chunks to S3. click - Playout; broadcast end SNS message; find relevant chunks and join click - Transcode with trim; distribute; publish. click - talk about inaccurate trims and “Resilient broadcast-grade system”.
Mezz-to-VOD:
“In a secure location somewhere is part of the BBC’s TV broadcast chain.” Playout; Thomson Video Networks; RTP video over multicast UDP; includes timecodes. Capture and chunk. Mustn’t miss a single packet. Upload chunks to S3. click - Playout; broadcast end SNS message; find relevant chunks and join click - Transcode with trim; distribute; publish. click - talk about inaccurate trims and “Resilient broadcast-grade system”.
Mezz-to-VOD:
Broadcast-grade resilient system
Mezz-to-VOD:
“In a secure location somewhere is part of the BBC’s TV broadcast chain.” Playout; Thomson Video Networks; RTP video over multicast UDP; includes timecodes. Capture and chunk. Mustn’t miss a single packet. Upload chunks to S3. click - Playout; broadcast end SNS message; find relevant chunks and join click - Transcode with trim; distribute; publish. click - talk about inaccurate trims and “Resilient broadcast-grade system”.
What bits of AWS do we use?
For legal reasons, it’s all in the EU, hence eu-west-1. EC2 compute, VPC, ELB, Autoscaling S3, SQS, SNS, SimpleDB Cloudwatch, Cloudformation
What bits of AWS do we use?
(nothing too exciting, actually)
For legal reasons, it’s all in the EU, hence eu-west-1. EC2 compute, VPC, ELB, Autoscaling S3, SQS, SNS, SimpleDB Cloudwatch, Cloudformation
but here’s the fun part…
video is big
SD video
mpeg-ts / avc 720 x 576 9.4Mbps 25fps / mpeg audio 256Kbps
SD video1.3MB/sec/channel
mpeg-ts / avc 720 x 576 9.4Mbps 25fps / mpeg audio 256Kbps
SD video1.3MB/sec/channel
= 109 GB/day/channel
mpeg-ts / avc 720 x 576 9.4Mbps 25fps / mpeg audio 256Kbps
SD video1.3MB/sec/channel
= 109 GB/day/channel
x 21 channels
mpeg-ts / avc 720 x 576 9.4Mbps 25fps / mpeg audio 256Kbps
SD video1.3MB/sec/channel
= 109 GB/day/channel
x 21 channels
= 2.3 TB/day
mpeg-ts / avc 720 x 576 9.4Mbps 25fps / mpeg audio 256Kbps
HD video
mpeg-ts / avc 1920 x 1080 ~38Mbps 25fps / mpeg audio 256Kbps
HD video4.2MB/sec/channel
mpeg-ts / avc 1920 x 1080 ~38Mbps 25fps / mpeg audio 256Kbps
HD video4.2MB/sec/channel
= 365 GB/day/channel
mpeg-ts / avc 1920 x 1080 ~38Mbps 25fps / mpeg audio 256Kbps
HD video4.2MB/sec/channel
= 365 GB/day/channel
x 8 channels
mpeg-ts / avc 1920 x 1080 ~38Mbps 25fps / mpeg audio 256Kbps
HD video4.2MB/sec/channel
= 365 GB/day/channel
x 8 channels
= 2.9 TB/day
mpeg-ts / avc 1920 x 1080 ~38Mbps 25fps / mpeg audio 256Kbps
+ 2.9 TB/day
2.3 TB/day
5.2 TB/day
5.2 TB/day/copy
x 2 locations
“each channel is captured in 2 physical locations” “at each location we capture 2 copies"
5.2 TB/day/copy
x 2 locations
x 2 copies
“each channel is captured in 2 physical locations” “at each location we capture 2 copies"
21TB per day
“… for a total of 21TB per day. (maybe mention “bigger” now? 40hrs=>250hrs, 4TB=>20TB) Handling this much data wouldn’t have been possible on our previous platform, but with Amazon Web Services, Video Factory is able to handle this much data, all day, every day.”
The origin of Video Factory
The world’s biggest public service video recorder
“I’d like to talk now about the practices and tooling we have in place that make this possible.”
Continuous Integration
Continuous Integrationcommit ↓
build and unit tests ↓
component tests ↓
build into a Cosmos release ↓
deploy to int via Cosmos
Small components Loosely coupled
Lots of small deployments Microservices
Design for failure
use Chaos Monkey lose your fear of failure
Continuous Delivery
(describe CD first…) “Building Video Factory on Amazon, using Continuous Delivery was a success; and to a large extent this formed the basis of what then became the BBC Continuous Delivery pilot, that just recently ended.”
Deployment weekly averages
(total for 10 weeks, divided by 10)
int: test: live:
Deployment weekly averages
(total for 10 weeks, divided by 10)
int: test: live:
120
Deployment weekly averages
(total for 10 weeks, divided by 10)
int: test: live:
12025
Deployment weekly averages
(total for 10 weeks, divided by 10)
int: test: live:
1202521
0
1
2
3
4
5
Monday Tuesday Wednesday Thursday Friday Saturday Sunday
Live deployments by day of week (total for 10 weeks, divided by 10)
3.5 deployment days per week vs 5 days per week No live deployments on Sunday :-) System as a whole is more reliable (“better”)
“So, with Mezz-to-VOD in place, iPlayer is saved. We record whatever was broadcast on TV, and we publish it. So now at the click of a button you can enjoy world-class content like this…”
We’re saved!
“So, with Mezz-to-VOD in place, iPlayer is saved. We record whatever was broadcast on TV, and we publish it. So now at the click of a button you can enjoy world-class content like this…”
Examples of things that appear on TV, that we don’t want to publish on iPlayer.
Examples of things that appear on TV, that we don’t want to publish on iPlayer.
Channel logo
Examples of things that appear on TV, that we don’t want to publish on iPlayer.
Channel logoSubtitles indicator
Examples of things that appear on TV, that we don’t want to publish on iPlayer.
Fade between channel ident and programme
Channel logoSubtitles indicator
Examples of things that appear on TV, that we don’t want to publish on iPlayer.
Examples of things that appear on TV, that we don’t want to publish on iPlayer.
Channel logo
Examples of things that appear on TV, that we don’t want to publish on iPlayer.
Channel logo
Credit squeeze
Examples of things that appear on TV, that we don’t want to publish on iPlayer.
Continuityvoice-over
Channel logo
Credit squeeze
Examples of things that appear on TV, that we don’t want to publish on iPlayer.
File-Based Delivery
What is FBD - e.g. EastEnders Better than M2V: * cleaner signal * higher bit rate * available sooner Build up an archive
The Video Factory Archive
size of archive (“hours” figure is an estimate)
38453 files
The Video Factory Archive
size of archive (“hours” figure is an estimate)
38453 files20902 hours
The Video Factory Archive
size of archive (“hours” figure is an estimate)
38453 files20902 hours
599 TB(Data accurate as of 2.30pm 11/11/2014)
The Video Factory Archive
size of archive (“hours” figure is an estimate)
Simulcast (“Watch Live”)
Bring the handling of live IP audio and video in-house Build on the success of Video Factory VOD Mostly not in the cloud, so only a very brief overview…
Simulcast
Only the packager is in the cloud Explain what HLS (Apple) is Explain what HDS (Adobe) and Smooth (Microsoft) are Provides rewind window
Simulcast
Only the packager is in the cloud Explain what HLS (Apple) is Explain what HDS (Adobe) and Smooth (Microsoft) are Provides rewind window
Simulcast
Only the packager is in the cloud Explain what HLS (Apple) is Explain what HDS (Adobe) and Smooth (Microsoft) are Provides rewind window
Simulcast
Only the packager is in the cloud Explain what HLS (Apple) is Explain what HDS (Adobe) and Smooth (Microsoft) are Provides rewind window
Simulcast
Only the packager is in the cloud Explain what HLS (Apple) is Explain what HDS (Adobe) and Smooth (Microsoft) are Provides rewind window
Simulcast
Only the packager is in the cloud Explain what HLS (Apple) is Explain what HDS (Adobe) and Smooth (Microsoft) are Provides rewind window
Converging Live and
On-Demand
Here’s the latter half of the Simulcast chain.
And here’s the Mezz-to-VOD chain, fed from exactly the same video feed. But we’re transcoding again, and always late. We can do better…
L2V is triggered by the same event as Mezz-to-VOD. L2V is an order of magnitude faster. The 6.30pm news: ideal: 7pm; old system: maybe 11pm, midnight; with mezz-to-vod: ~ 8pm; with live-to-vod: ~ 7.05pm
However L2V only does some formats (albeit they’re important ones), and doesn’t trim accurately. So we deliberately allow L2V and M2V to run in parallel; L2V will win, M2V is better.
The origin of Video Factory
The world’s biggest public service video recorder
Live and Live-to-VOD
“At BBC Media Services we have to handle audio - that is, radio - as well as video, so we’re creating Audio Factory.”
Audio Factory
“How does Audio Factory work?” (click twice to reveal obvious description)
Audio Factory
like Video Factory,
“How does Audio Factory work?” (click twice to reveal obvious description)
Audio Factory
like Video Factory,but without the pictures
“How does Audio Factory work?” (click twice to reveal obvious description)
“Publish all BBC AV media produced for IP platforms”
“So as well as video on demand; there’s the simulcast chain providing live video; and live-to-VOD bridging the two together, so that programmes are playable as soon as possible. Audio - including BBC Radio - can be handled almost identically to video,
… and so that the system works together.
“Publish all BBC AV media produced for IP platforms”
“so at last we can handle audio and video, live and ondemand, all in-house, using a consistent, proven set of technologies that work together.”
“It took us just over a year to build the basic video-on-demand features of Video Factory that we had to build, to prevent iPlayer from dying. It was a completely new solution: new architecture, new code, new platform. We chose the cloud because it was more flexible, more reliable, more scalable. We chose Amazon because it was a mature cloud platform that provided the right technical and support services that we needed.”
MMXIV
“It took us just over a year to build the basic video-on-demand features of Video Factory that we had to build, to prevent iPlayer from dying. It was a completely new solution: new architecture, new code, new platform. We chose the cloud because it was more flexible, more reliable, more scalable. We chose Amazon because it was a mature cloud platform that provided the right technical and support services that we needed.”
MMXIV
“We moved to Continuous Integration and Continuous Delivery with a Microservices architecture, so that the benefits of higher quality and faster turnaround could be enjoyed by everyone - by our engineers, by the product stakeholders, by the licence-fee-paying audience. Building Mezz-to-VOD to avoid killing iPlayer was just the beginning. I’m very excited about seeing the future of Video Factory unfold on Amazon, and I hope you enjoy using iPlayer even more now that you’ve heard the story behind it. Thank you.”