denis leconte€¦ · calypso update over the past few months:-vfx pipeline-tape-based...
TRANSCRIPT
Denis Leconte
Calypso UpdateCalypso Update
Over the past few months:
- VFX Pipeline
- Tape-based pipeline
- LTO5 migration
- Infrastructure upgrades
5/5/2011Production Backbone / Calypso Update
5/5/2011Production Backbone / Calypso Update
OverviewOverview
The Smurfs Movie
- First project to use Calypso
- Started mid-2010
- VFX heavy
- 2D to 3D Conversion
- Currently in DI
“The 2D Smurfs conform is going amazingly well. “
Jay Warren, Project Manager
5/5/2011Production Backbone / Calypso Update
The ChallengeThe Challenge
Multi-vendor VFX Delivery
- Needed to centralize deliveries
- Needed to validate them
- Delivery specs are not always respected
- Ability to fix simple problems
- Delivering “clean” material to Editorial
The Smurfs Delivery Spec
- Name shots starting at 1001 on FFOA- If possible, at a slate at the head (frame 1000)- Header timecode should start at 01:00:00:00 on slate (01:00:00:01 on FFOA)- Directory structure shot/resolution/shot.number.dpx- Follow the shot naming conventions as requested by production
045CD_050_x00001
% of vendors delivering shots just like this?
5/5/2011Production Backbone / Calypso Update
Solution: Replacing this:Solution: Replacing this:
DPX Files QT or MXFs
EDL
2 Separate paths – possibility for 2 Separate paths – possibility for disconnectdisconnect
5/5/2011Production Backbone / Calypso Update
……With this:With this:
DPX Files
EDL
QT or MXFs
Single Path: Single Path: No disconnect!No disconnect!
5/5/2011Production Backbone / Calypso Update
VFX Dailies: Validation toolsVFX Dailies: Validation tools
Run on every incoming VFX shot
Performs a technical check
Validation pipeline
- Sequence checker: are frames in a continuous sequence (otherwise: reject shot)- File checker: are files legitimate DPX/TIFF/EXR files (otherwise: reject shot)- Naming convention verification: does the directory structure and shot naming follow conventions?- Number convention validation- Header timecode validation
5/5/2011Production Backbone / Calypso Update
VFX Dailies: Correction toolsVFX Dailies: Correction tools
Run on shots that have correctable noncompliance errors
Correction pipeline
- Directory reorganization- Header timecode rewriting- Renumbering/Padding change
5/5/2011Production Backbone / Calypso Update
VFX Dailies: Putting it togetherVFX Dailies: Putting it together
Essential requirements
- Unobtrusive
- Automated as much as possible
- Try and reject as few shots as possible (fix as much as one can)
- Detailed notifications to Production
Automated system
- Frequent schedule runs- Propagates data from Isilon to BB Staging directory- Validates- Notifies of rejects- Moves data to BB Asset storage area- Fixes what can be fixed- Registers assets- Generates Quicktimes/ALEs/MXFs- Pushes Editorial material back to Isilon- Notifies all concerned parties of new assets- Separate emails to BB admin for exceptions
5/5/2011Production Backbone / Calypso Update
Data Flow and Isilon IntegrationData Flow and Isilon Integration
5/5/2011Production Backbone / Calypso Update
In Progress: VFX InspectorIn Progress: VFX Inspector
5/5/2011Production Backbone / Calypso Update
5/5/2011Production Backbone / Calypso Update
Recent DevelopmentsRecent Developments
Asset Request Processing
- On a SAN
- On a tape robot
- Potentially more than one of each
- All tracked in Calypso
- Asset selection is made for shortest time to delivery
Retrieve it
5/5/2011Production Backbone / Calypso Update
Asset request processingAsset request processing
The assets for a request can be in a number of places:
- On a SAN
- On a tape robot
- Potentially more than one of each
- All tracked in Calypso
- Asset selection is made for shortest time to delivery
Retrieve it
5/5/2011Production Backbone / Calypso Update
Cache managementCache management
System to keep track of:
- Unique request count per asset
- Asset age in cache
Automated cache expiration, based on target cache size, targeting least used/older assets.
5/5/2011Production Backbone / Calypso Update
LTO migrationLTO migration
- All LTO4 content migrated to LTO5.
-1,775 tapes
- Done with no impact on production archival/retrieval
- Scheduling system written to optimize robot use
- Efficient use of operator time (only 45 hours for the complete migration)
Tape operations priorities
(Implemented using Rush and TSM)
5/5/2011Production Backbone / Calypso Update
Additional Tape Archive Additional Tape Archive DevelopmentsDevelopmentsTape scheduling expert system
- Retrievals often involved several operations on the same tape
- System tries to minimize tape contention
- Retrieval sequencing- Using multiple sets
Throughput improvement
- From 50MB/s (per drive) to over 100MB/s
Compression Staging
-To compress/decompress media backups on the fly
LTFS
-Set up LTFS archive/retrieve capability
-Used for 21 Jump Street
5/5/2011Production Backbone / Calypso Update
5/5/2011Production Backbone / Calypso Update
Asset “Heat Map”Asset “Heat Map”
Asset activity is tracked
- What assets were pulled
- How many pulls
Can prepare a list of everything that was used
- And everything that was not
- We are ready for EDL-based archival (of only what was used)
5/5/2011Production Backbone / Calypso Update
5/5/2011Production Backbone / Calypso Update
REDRED
RED files ingested natively
- Asset metadata extracted and part of Calypso registration
Media pulls
- R3D, DPX, RMD, Metadata
- Hybrid REDline + Nuke workflow for best image quality
5/5/2011Production Backbone / Calypso Update
ARRIRAWARRIRAW
Currently: Metadata only
- Working on licensing of ARRIRAW format and SDK
- Header documented in-house
- Asset registration with metadata
More to come…
5/5/2011Production Backbone / Calypso Update
CODEX CompressedCODEX Compressed
Codex can record compressed
- Native handling of compressed files
- Metadata aware
- Ability to transcode on the fly to DPX files
- Bandwidth/Storage savings
- 3:1 compression tested and satisfactory
5/5/2011Production Backbone / Calypso Update
5/5/2011Production Backbone / Calypso Update
Improved Editorial/Conform Improved Editorial/Conform FeaturesFeatures- Interactive feedback on EDL request
- On-the-fly EDL correction- EDL modification/rewrite
- More sophisticated “Shopping Cart”
- Persistence- Customizable processing
parameters
5/5/2011Production Backbone / Calypso Update
More Camera/Workflow SDKsMore Camera/Workflow SDKs
-Plug-in architecture
-Make it as simple as possible to add a new system
5/5/2011Production Backbone / Calypso Update
-Sony F65
-Sony F3 (MPEG2 within MPEG4 wrapper)
-3D F35/F3 workflow
Dailies Validation/AssistDailies Validation/Assist
- System for thorough dailies/footage metadata review
- Naming conventions- Timecode- Directory organization
- Necessary for Backbone ingest- Naming conventions- Timecode- Directory organization
5/5/2011Production Backbone / Calypso Update
- Plug-in architecture here too- Dailies organization tend to be
very show-specific- Some common elements,
many differences
CompressionCompression
5/5/2011Production Backbone / Calypso Update
- Aiming to make compression a near-transparent process
- Implemented as a standard “data move” plug-in
- Also available to/from tape
- JPEG2000-based- In the process of licensing
Kakadu- Additional enhancements
added for metadata handling
- Do be deployed mid-may 2011- Study of optimal compression
rates/options under way
Web Services Web Services
-SOAPify targeted Calypso services
- Query for new assets- Query for update assets- Query for updated asset
metadata- Asset updates- Metadata update- Asset deletion - Asset export- Title search
5/5/2011Production Backbone / Calypso Update
-MBC/EAGL integration- Web service based- Also including processing
services
Denis Leconte