media cloud services mcs approach to re-architecting dmr 02-july-2013

Media Cloud Services


MCS Approach to Re-architecting DMR

02-July-2013



Agenda

• General Problem Areas and Solutions• Other Areas of Improvement• Summation of Approach



General Problem Areas

• Code Base• Scalability• Performance



Problem: Code Base

• It’s just too complex – “it’s a bit like a slinky – super flexible but easy to get tangled up in”

• Complex functionality tailored to small portion of user base has led to feature bloat and unwieldy code base

• There are too many solutions, it’s hard for new developers to navigate

• Common utilities is too common – results in too many dependencies

• Too many objects, too granular• There’s a ton of business logic stuck in Oracle stored procedures• Build, deployment and config has become onerous



Solution: Code Base

• Rewrite service layer leveraging the “wisdom” accumulated in DMR

• Add analytics to track feature usage, periodically evaluate cost/benefit of maintaining seldom-used features

• Expose services as simple, REST-ful API• Allow ZERO business logic in the persistence

store/data layer• Test Driven approach to facilitate quality and

understandability



Problem: Scalability

• Ability to handle processing spikes was limited• Significant processing (including search) is

handled by Oracle which is expensive to scale



Solution: Scalability

• Continue to leverage auto-scaling, refine rules• Leverage AWS spot market where appropriate

to reduce costs• Migrate from Oracle to Mongo



Problem: Performance

• Search sucks• Reliance on pre-calculated data introduces lag

time for users (used because of security model complexity)

• Insufficient usage of caching• Granularity of service layer requires multiple

calls to get many things done• Spinning up new EC2 instances is too slow



Solution: Performance

• CloudSearch• Simplify security model to eliminate need for

pre-calc’d data• Leverage memcached at service layers• Design service layer for “purpose” instead of

“possibility”• Migrate CPS workers w/high file I/O to linux



Other Areas of ImprovementTopic Fix

Ingest Process Flexibility Enable ingest requests to specify the services they need (ie. Glacier, Proxy Creation, etc.)

Export Process scalability/reliability Migrate to SWF framework

Proxy creation process Centralize transcoding services, move to Linux machines



Summary of Approach

• Replace Oracle with Mongo• Rewrite DMR against Mongo• Ensure simple, REST-ful API is available to

enable partners to build to their needs• Simplify security model• Implement CloudSearch and caching layer



Mediabox Auth Admin Splunk Aspera

DMR Auth CPS Audit Notification

Workflow Replay CPS Datama

rt AsperaMkProx

y

Oracle MongoDB MS SQL

S3 CloudFront

ElastiCache EC2 VPC

Hotfolders

Master SuiteUIs

APIs

Workers

Data

Infrastructure(AWS)

Splunk

Route 53

Continued Enhancements

New

Some enhancement

sSome enhancements made in past 12 months

Completely new

Area of focus for more enhancements

Legend

Significant enhancements made in past 12 monthsSignificant

enhancements

Areas of Focus going forward in GREEN

media cloud services mcs approach to re-architecting dmr 02-july-2013

Documents