PBS on Amazon
Jon Brendsel
Vice President, Products
Iconic Brands
A little bit about PBS
• General Audience, PBS Kids, PBS Education• We are not NPR!• Member organization
– 165 dues paying stations
• Dues are used for…– License broadcast and digital content– Build distribution infrastructure
• Producers are not owned by PBS– Station sponsored/owned (e.g. Frontline)– Independent (Children’s Television Workshop)
#1 in Unique VisitorsAmong major networks
PBS sites 8555
ABC.com 5977
CBS.com 5338
NBC.com 5266
PBS.org (no Kids)4566
FOX.com 3073
CWTV.com 1120
0 2,000 4,000 6,000 8,000 10,000
PBS sites
ABC.com
CBS.com
NBC.com
PBS.org (no Kids)
FOX.com
CWTV.com
(comScore MediaMetrix, July 2011)
Fun With Numbers
• Facebook Fans: ~ 1M
• Unique visitors: 18M/mo
• Visits: 37M/mo
• Page views: 367M / mo
• Video streams: 93M/mo
• Mobile Apps
– 27% of hours watched, 11% of streams
Three Years AgoLife before Amazon
• Traditional Big Iron infrastructure– Ping Power and Pipe managed by PBS IT
– Applications (Apache and higher) managed by PBS Developers
– No DR
– Fixed bandwidth
• Shared Hosting Environment– Traditional three-tier architecture
– Little isolation between 1200+ applications / producers
– Limited freedom to innovate
• No real product development capability– Small team of “web technologists”
– Projects built by outside design build firms
– No product lifecycle management
• No shared services
Three Years AgoLife was Painful
• Routine outages, lots of downtime– Multiple single points of failure– Perpetual state of emergency
• No application/producer isolation– One mistake impacts everyone– Producers hated the restrictions
• Long lead times to get new hardware• Limited long term product planning
– Ready-Fire-Aim
• Poor morale– Lack of confidence in technology organization
TodayAll New Applications / Producers -> Ec2
• Hosted producer sites
• PBS.org National Portal
• Video distribution platform
• UUA (Registration and Profiles)
• Bento (NG templated publishing system)
• TV Schedules
• Other: tools, PBS Education, PBS Kids, dev, staging, etc.
A Birds Eye View
• ~270 EC2 machines
• 75 ELBs
• 260 Elastic IPs
• 5 AWS accounts
• 52 RDS databases
• 608 EBS volumes
• many S3 buckets
• 19 COVE Cloudfront distributions
Technology Stack
• Linux Centos 5.2 (32 and 64-bit)
• Python/Perl/PHP
• Webserver: Apache
• DB: Mysql (RDS and hosted)
• Caching: memcached, SOLR
• File storage: S3, EBS
• Job management: SQS, celery
• Transcoding: Zencoder service
• Configuration management: Rightscale
Rightscale
• Manage 5 AWS accounts
• Performance monitoring and alerting (collectd)
• Configuration management of 5 shared templates
• Easy provisioning of most AWS services
• Authentication and authorization for ~30 employees
Hosted Producer Sites
Hosted Producer Sites
• Dedicated VPS-style hosting
• 43 machines (mostly m1.small)
• LAMP templates
• Nightly backups (EBS or S3)
• Isolation from other producers / systems
• VPS style SLA
• Provides freedom to innovate
PBS Video for iPadLaunched Nov ‘10
PBSKids Video for iPadLaunched April ‘11
PBS Video for iPhoneLaunched Jan ‘11
Local/National Video Portals
video.pbs.orgNext Generation Video Platform
• CDN
– Limelight (static http)
– Cloudfront (rtmp, HTTP pseudo streaming (Apple HLS)
• Presentation Layer
– Manually scaled array (4 medium)
– ELB load balancing
– S3 application server template
• Caching Layer (memcached)
– Memcached
video.pbs.orgNext Generation Video Platform (cont.)
• Service Layer ( REST API)– Manually scaled array (m1.medium)
– ELB load balancing
– S3 application server template
• DB layer– RDS Master (m1.medium)
– RDS Slaves (4 x db.xlarge)
– HAProxy load balancing
PBS Video for MobileiPhone and iPad
• Compiled Apps– PBS General Audience iPad and iPhone Apps– PBS Kids iPad App (iPhone on the way)
• Video API• VTS (Video Transcoding Service)
– EC2 instance m1.small– API wrapper around Zencoder– Transcoding from MP4 to Apple HLS for iOS distribution– Application hosted on PBS App Server template (Ec2) with RDS
DB
• File storage in in S3• Distribution via Cloudfront CDN
Video Transcoding Service
• Transcoding for multiple distribution endpoints– Ingest: Mezzanine quality (5 mpbs)– Outputs:
• Current: Apple HLS for iOS distribution• Future: web output, + partners (iTunes, Amazon, etc.)
• API wrapper around Zencoder• Application:
– 1 App ServerEC2 instance with RDS DB
• File storage in in S3• Distribution via Cloudfront CDN
Zencoder
• Encoding needs– Today: Mobile only
– Soon: All web outputs
• Powerful API
• Fast encoding times
• Based on FFMPEG
• Strong support for Apple iOS HLS (HTTP Live Streaming)
• Reliable support for Amazon S3 storage
Development / QA / Staging
• Mirrored environment and topology
• Load testing
• Demos & training
• On demand
• Bamboo-based continuous integration
Takeaways & Lessons Learned
• Vastly improved uptime
• Infrastructure: from an obstacle to an asset
• DevOps
– Build applications around infrastructure
– Design for failure
• Easy can become expensive
– PBS has proliferating infrastructure
– Now need to invest in efficiency