serverless architecture

Managing Digital Assets in a

Serverless Architecture

Chris Elleman – Manager, Solutions Architect (Media Practice)

Erik Åhlin – Founder and CEO, Vidispine

March 2016

We see lots of customers at different points on the spectrum

File Based Management Asset Management

Challenges of Managing Content

Integrated services for ingestion, annotation, cataloguing, storage, retrieval and

distribution of digital assets (photos, video, audio, documents)

CMS DAM MAM

DocumentsWeb Content

PhotosCreativesModels

VideoAudio

WordpressDrupalCenshareEscenic

Adobe CQOpenText CHPT3 Media/Wazee

VidispineDalet Examples

Creation

Distribution

Edit

Repurpose

Review

Approve

Asset Management Definition

• Gap: Reconciling Legacy vs. Born-In-The-

Cloud

– Large enterprises are challenged with

keeping legacy systems and processes in-

house

– Need to share/collaborate is strong LegacyDAM

S3 Glacier

DAM DAM DAM

Asset Management Gaps (1)

• Solution: Federate DAM solutionsEdge DAMs + Core DAMs

Deep Storage on AWS

• Gap: Reconciling Content Gravity

– For heavy-duty DAM/MAM

requirements the volume and mass of

the generated content makes file-

based workflows very painful

• Solution: co-locating the

creative/edit/render/review process

with the data

– Amazon AppStream

– GPU Enabled VDI (AWS Workspaces,

Teradici)

– Others: browser based tooling (edits,

modeling, correction)

S3(ex: Avere,SoftNAS)

DAM

VirtualCreativeApps

Asset Management Gaps (2)

Online

Storage

Catalog

Management

Proxy &

Transcode

Processing

Search | Collaboration

Services

Process Management | Workflow Services

Nearline

Storage

Offline

Storage

Import / Export

Services

Tools Adapter

IngestServices

Security | Rights Management Services

CDN

File Transfer

WAFS

Architecture Concepts – Digital Asset Management

Online

Storage

Catalog

Management

Proxy &

Transcode

Processing


Services


Nearline

Storage

Offline

Storage

Import / Export

Services

Tools Adapter

IngestServices


CDN

File Transfer

WAFS

Standard S3-IA Glacier

S3 Glacier

Architecture Concepts – with AWS Services

Online

Storage

Catalog

Management

Proxy &

Transcode

Processing


Services


Nearline

Storage

Offline

Storage

Import / Export

Services

Tools Adapter

IngestServices


CDN

File Transfer

WAFS


S3 Glacier

Cognito Identity & Access Management


Online

Storage

Catalog

Management

Proxy &

Transcode

Processing


Services


Nearline

Storage

Offline

Storage

Import / Export

Services

Tools Adapter

IngestServices


CDN

File Transfer

WAFS


S3 Glacier


DynamoDB


Online

Storage

Catalog

Management

Proxy &

Transcode

Processing


Services


Nearline

Storage

Offline

Storage

Import / Export

Services

Tools Adapter

IngestServices


CDN

File Transfer

WAFS


S3 Glacier


DynamoDB

Elastic Transcoder & Elemental


Online

Storage

Catalog

Management

Proxy &

Transcode

Processing


Services


Nearline

Storage

Offline

Storage

Import / Export

Services

Tools Adapter

IngestServices


CDN

File Transfer

WAFS


S3 Glacier


DynamoDB


API GatewayLambda


Online

Storage

Catalog

Management

Proxy &

Transcode

Processing


Services


Nearline

Storage

Offline

Storage

Import / Export

Services

Tools Adapter

IngestServices


CDN

File Transfer

WAFS


S3 Glacier


DynamoDB


API GatewayLambda

ElasticSearch


Online

Storage

Catalog

Management

Proxy &

Transcode

Processing


Services


Nearline

Storage

Offline

Storage

Import / Export

Services

Tools Adapter

IngestServices


CDN

File Transfer

WAFS


S3 Glacier


DynamoDB


API GatewayLambda

ElasticSearch

AWS Import/Export Snowball


Ingest

Bucket

CloudFormation

Template

DynamoDB

Cognito

IAM

role

Lambda

MetaData

Extract

Asset Bucket

multimedia

User

Index

HTML5

Static Site

Elastic

Transcoder

IAM

ElasticSearch

API Gateway

Lambda

App Logic

SimpleDAM – Serverless Architecture

GlacierArchive

JS SDK

Meta Data

Searches

Ingest

Bucket

Multimedia

file

1. S3 Upload

2. S3 Event

Lambda

Function

4. Transcode Job

Lambda

Sub-Function

(metaDataExtract)

Elastic

Transcoder

3. Extract MetaData 5. Build source XML

Lambda

Sub-Function

(metaTransform)

+

DynamoDB

6. ETL Job

format data for DynamoDB

7. Create

DynamoDB record

Asset

Bucket

9. Copy Asset Into Asset Bucket

ElasticSearch

8. Index Content

SNS

Topic

10. On Success or Failure,

Remove Ingest Object

Lambda

Function

SimpleDAM – Video Ingest Process

Key Decision for a DAM – Buy or Build?

Buy• Lack of free development

resource or skills

• Exiting on premise commercial relationship

• DAM capabilities aren’t a market differentiator

• Broad set of requirements

• Fast to set-up and provision

Build• Development resource and skills

available in-house

• Lack of cap’ex budget for licensing

• DAM capabilities will be a market differentiator

• Narrow or specialist requirements

• Start with an MVP and iterate with the needs of the business

©Amazon.com, Inc. and its affiliates. All rights reserved.

Thank you

Vidispine -

Cloud Native Content Management for

Developers and Media Professionals

Erik Åhlin, CEO, co-founder

Vidispine in 15s – the problem we solve

Broadcasters and media companies are looking for

• Incorporate cloud and cloud services in a pace the budget

allows. An evolution rather than a revolution.

• Complete, managed and real elasticity for video content to

control cost but still being agile

• Repurposing content to a much higher degree

• Fast turn-around story-telling with video and images on all

platforms

• Flexibility in business models and technology choices

Vidispine API-based Content Management

PaaS

Fully featured back-end for any media application

Key areas

• Metadata

• Multi-format management

• Performance & Scalability

• Cloud native architecture (componentized and distributed)

• Support for Multi-application

VidiXplore – The Gravity Point for your content

”Easy to Like” user experience

Support for core media management tasks

Extensible

Component in a ”bigger picture”

SaaS (private or public)

Leverage Vidispine Content Management PaaS

Combine on-prem & cloud - start your

migration1. Place Vidispine Server Agent (VSA) next to existing on-prem

storage

2. Create proxy, analyze metadata, connect to MAM/DAM locally

3. Keep link between ’cloud-side’ asset and ’on-prem’ asset

4. Start execute your cloud strategy by doing what cloud does best

5. Migrate infrastructure in your pace means you can focus on

business

VidiXplore – The Gravity Point for your content

VIDISPINE PaaS with APIs

Transcoder

& QCVDA

Sto

rage

On-Prem

Infrastructure

Vidispine

Server

Agent

Cloud Services

VSAs

and/or Biz

Destinations

Real World Example #1Global Premium Content Delivery

Spread out workforce and clients

Cost efficient and adaptive workflows

VidiXplore at Premium Content Delivery Leader

VidiXplore with Custom Panels

for managing audio tracks,

mezzanine files, delivery points,

etc

Amazon

EC2

Amazon

CloudFront

Amazon

S3 w SSE

hosting

8 TB proxies

65 GB

thumbnails

Amazon

VPC

Amazon

Route 53

On-Prem

VSAs

connected

to local

storage

Vidispine

APIs for

integration

to workflow

engine and

more

Data & Metadata to Vidispine PaaS

Proxies direct to S3 for performance gain

Amazon

RDS

Migrate and Scale

By knowing your content you are ready to scale

Spend wisely - let cloud do what cloud do best

• Workflows, common user interface for spread-out workforce

• Distribute and share ready content

• Store/archive and ”second copy”

• Burst-out transcoding and compute

Integrate and leverage popular cloud services

• Dropbox, Slack, Box.com, WeVideo, CloudConvert, Metadata

harvesting

Migrate, Scale, Adapt also Business Processes, Functions,

Real World Example #2Partner Company

Alfred – the Dev Ops Butler

Partner Company: DSB

Using Vidispine dockerized

Scaling, Logging, Action

Packs

AWS Services used

• EC2 Container Service

• Docker

• Node.Js

• MongoDB

Real World Example #3Leading US-based Media Giant

Quickly prototyped a complete DAM workflow (Clip Library)

Validated the technology stack on very low budget

Prototyping on AWS Marketplace

Leading US-based Media Giant

10+ MAMs and DAMs already

Some workflows are fit and ready for cloud

Overview

• Total POC went 6 weeks – total of 4 developers.

• Total assets added (5 thousand).

• Used the bare minimum box that Vidispine AMI Trial would run

on (m3.medium)

Prototyping on AWS Marketplace

Next steps

• Run Vidispine as HA, where SOLR runs in another EC2 and Postgres

runs in RDS

• Performance testing with hundreds of thousands of assets

Resources used

• AWS EC2 (website PoC ran here)

• AWS RDS (Postgres DB)

• AWS S3 (hot folder in/out)

• Vidispine Developer Edition AMI

• Bitbucket (PoC code)

• Jenkins (for deploying to EC2)

• MEAN stack(code stack)

What’s next?

Editing

• Some very promising prototypes using AWS WorkSpaces

Expert systems based on machine learning

• Systems adapting to the task at hand rather than how it was

originally designed

Large-scale metadata harvesting

True ubiquitous processing and storing

• What is a file, what is a computer?

• What to we really want computers to do for us and how?

Amazon

WorkSpaces

Amazon Machine

Learning

AWS

Lambda