Windows Vista Video Windows Vista Video Pipeline Architecture Pipeline Architecture And ImplementationAnd Implementation

Glenn EvansGlenn EvansTechnical LeadTechnical LeadDMD/WAVEDMD/WAVEMicrosoft CorporationMicrosoft Corporation

Session OverviewSession Overview

Video pipelineVideo pipelineOverviewOverview

New components for Windows VistaNew components for Windows Vista

Windows Vista enhancementsWindows Vista enhancementsDirectX Video Acceleration 2.0 (DXVA2)DirectX Video Acceleration 2.0 (DXVA2)

Software componentsSoftware components

Display devicesDisplay devices

ImprovementsImprovementsPerformance and qualityPerformance and quality

Extensibility and integrationExtensibility and integration

Future directionsFuture directions

Call to actionCall to action

Video Pipeline StagesVideo Pipeline Stages

Decoding

Capture Processing

Processing

Encoding

OutputControls

Storage

Desktop

Display 1

Display NOutputControls

ComponentComponent

DeviceDevice

New Windows Vista ComponentsNew Windows Vista Components

Video ProcessingDecode

DMMDDC/CI OPM

MFT

Decoders Transforms

EVR D3DManager

WDDMCapture

MediaSources

D3DDXGI

DXVA2

AVStream

APIs/DDIs

Plugins

App/Pipeline

OS Component

Application and Pipeline (Media Foundation, DShow, …)

UVC

MediaSinks

Example Playback PipelineExample Playback Pipeline

EVR

D3DManager

MediaSource

SW DecoderMFT

DWM

Desktop

Application

Pipeline

APIs/DDIs

Plugins

App/Pipeline

OS Component

GPU assisted

DecoderMFT

XformMFT

Video ProcessingDecode

DMMDDC/CIOPM

D3DDXGI

DXVA2

DXVA2DXVA2

What is itWhat is itDirectX Video Acceleration 2.0 (DXVA2.0)DirectX Video Acceleration 2.0 (DXVA2.0)

APIs for decoding and processingAPIs for decoding and processing

Separate from renderersSeparate from renderers

Enhanced enumeration, object creationEnhanced enumeration, object creation

APIs for D3D device managementAPIs for D3D device management

Many opportunities for IHVs to differentiate Many opportunities for IHVs to differentiate Power optimizationsPower optimizations

Processing quality and featuresProcessing quality and features

Parallel processing hardwareParallel processing hardware

Bandwidth optimizationsBandwidth optimizations

DXVA2 – DecodingDXVA2 – Decoding

Simplified enumeration (caps Simplified enumeration (caps versus profiles)versus profiles)

Used by decoder directlyUsed by decoder directly

Renderers take uncompressed buffersRenderers take uncompressed buffers

Support for many codecsSupport for many codecsWMV, VC-1, MPEG2, H.264, and moreWMV, VC-1, MPEG2, H.264, and more

ExtensibleExtensible

DXVA2 – Video ProcessingDXVA2 – Video Processing

Single step conversion, scaling Single step conversion, scaling and compositionand composition

Vendors can optimizeVendors can optimize

HQ output to match output deviceHQ output to match output device

DeinterlacingDeinterlacingStream metadata given to rendererStream metadata given to renderer

More input formatsMore input formatsRGB inputsRGB inputs

Video ProcessingVideo Processing

Colorspace conversionsColorspace conversionsEnhanced input/output colorspace controlEnhanced input/output colorspace control

High quality output controlHigh quality output controlGamma, colorspace, Y’CbCr and fp output, Gamma, colorspace, Y’CbCr and fp output, ‘studio RGB’‘studio RGB’

Output filtering and ProcAmpOutput filtering and ProcAmp

Extensive linear processing and progressive Extensive linear processing and progressive RGB output supportRGB output support

RenderersRenderers

Enhanced video renderEnhanced video renderNew render supersedes OvMixer, VMR(7), and VMR9New render supersedes OvMixer, VMR(7), and VMR9

Video mixing, output timing schedulingVideo mixing, output timing scheduling

Many enhancementsMany enhancementsComposites to output – non-square pixel, colorspace supportComposites to output – non-square pixel, colorspace support

Pluggable mixer and presentersPluggable mixer and presenters

Pull basedPull based

Advanced presenter – synchronized with monitorAdvanced presenter – synchronized with monitor

Automatically handles output modeAutomatically handles output modeTear free windowed output, DWM support, fullscreen supportTear free windowed output, DWM support, fullscreen support

Glitch resilience – MMCSS, deep queuing, timeline mappingGlitch resilience – MMCSS, deep queuing, timeline mapping

Application integrationApplication integrationEVR available in Media Foundation and DShowEVR available in Media Foundation and DShow

Stand-alone Mixer MFTStand-alone Mixer MFT

Not dependent on DXVA decodingNot dependent on DXVA decoding

TransformsTransforms

Media Foundation PipelineMedia Foundation PipelineGlitch resilience, content protectionGlitch resilience, content protection

Media Foundation Transform (MFT) infrastructureMedia Foundation Transform (MFT) infrastructureSimple to use plug-in modelSimple to use plug-in model

Addresses issues with DMOsAddresses issues with DMOs

Supports DXVA and D3D processingSupports DXVA and D3D processing

MFTs useable outside of pipeline MFTs useable outside of pipeline

D3D ManagerD3D ManagerManaged pipeline components use of D3D devicesManaged pipeline components use of D3D devices

CaptureCapture

AVStreamAVStreamWDDM capture extensionWDDM capture extension

Integration with WDDM and DXVA2 pipelineIntegration with WDDM and DXVA2 pipeline

Robust queuing, simplifies capture driverRobust queuing, simplifies capture driver

Supports analog, BDA, multi-tunersSupports analog, BDA, multi-tuners

A/V sync lockA/V sync lock

Multi-stream (A/V) capture devices (e.g., tuners)Multi-stream (A/V) capture devices (e.g., tuners)

Selective suspend on USBSelective suspend on USB

USB Video Class Driver (UVC)USB Video Class Driver (UVC)Still image 2 supportStill image 2 support

Colorspace supportColorspace support

DDC/CIDDC/CI

Monitor configuration control via DDC/CIMonitor configuration control via DDC/CIDisplay Data Channel Command InterfaceDisplay Data Channel Command Interface

Bi-directional communication and control Bi-directional communication and control of external devicesof external devices

Remote adjustmentsRemote adjustmentsBrightness, contrast, positioning, color temp Brightness, contrast, positioning, color temp

Low level/high level APIsLow level/high level APIs

ExtensibleExtensible

Implement DDC/CI for improved Implement DDC/CI for improved connectivity and usabilityconnectivity and usability

PVP-OPM, PVP-UABPVP-OPM, PVP-UAB

PVP-OPMPVP-OPMProtected Video Path Output Protected Video Path Output Protection ManagerProtection Manager

Analog protection controlAnalog protection controlBeyond COPP, VideoParametersBeyond COPP, VideoParameters

Digital protection controlDigital protection control

Output device topology discovery Output device topology discovery (clone mode)(clone mode)

PVP-UABPVP-UABProtected Video Path – Protected Video Path – User Accessible BusUser Accessible Bus

Display SupportDisplay Support

Improved TV mode supportImproved TV mode supportNative OS support for TV modes and resolutionsNative OS support for TV modes and resolutions

p24, p48, i60, i50, +p24, p48, i60, i50, +

1080i, 720p, +1080i, 720p, +

Clone modeClone mode

Drivers can populate modes and users can overrideDrivers can populate modes and users can override

Better EDID supportBetter EDID support

Presentation feedback APIs (DX9L)Presentation feedback APIs (DX9L)History of presented framesHistory of presented frames

Tighter A/V sync, glitch detectionTighter A/V sync, glitch detection

scRGB framebuffer supportscRGB framebuffer supportHQ output, including 10 bit Y’CbCrHQ output, including 10 bit Y’CbCr

Video ImprovementsVideo Improvements

Areas of improvements for video Areas of improvements for video in Windows Vistain Windows Vista

PerformancePerformance

QualityQuality

ExtensibilityExtensibility

IntegrationIntegration

Connectivity – input and outputConnectivity – input and output

PerformancePerformance

CaptureCaptureHigh glitch resilience on uncompressed captureHigh glitch resilience on uncompressed capture

Results available for direct GPU processing (VP, encode,…)Results available for direct GPU processing (VP, encode,…)

DecodersDecodersWork queued ahead of HW (lock/discard semantics)Work queued ahead of HW (lock/discard semantics)

Decoder no longer watches HW completionsDecoder no longer watches HW completions

HW/driver throttledHW/driver throttled

Video processingVideo processingMultiple frames queued for processingMultiple frames queued for processing

Pipelined into display queue – no stallsPipelined into display queue – no stalls

Driver can influence queueDriver can influence queue

Parallelizable with D3D, decoding – use it!Parallelizable with D3D, decoding – use it!

Video bandwidth reductionVideo bandwidth reduction

PerformancePerformance

DisplayDisplayFrames decoded, processed, queued Frames decoded, processed, queued in advance of presentationin advance of presentation

Renderers schedule/queue aheadRenderers schedule/queue ahead

Hybrid planar Y’CbCr formatsHybrid planar Y’CbCr formatsNV11, NV12, P208, Y416…NV11, NV12, P208, Y416…

Improved cache usage, fewer DMA engines Improved cache usage, fewer DMA engines needed,…needed,…

Fewer conversions/temporaries in pipelineFewer conversions/temporaries in pipeline

Simplifies apps, drivers, HW supportSimplifies apps, drivers, HW support

QualityQuality

Glitch resilienceGlitch resilienceImproved queuing and latency controlImproved queuing and latency control

HW should implement Vsync reporting, deep queuesHW should implement Vsync reporting, deep queues

Multimedia class schedulerMultimedia class scheduler

FidelityFidelitySingle step composition to native device characteristicsSingle step composition to native device characteristics

HQ video processing and scalingHQ video processing and scaling

Renderers now monitor awareRenderers now monitor aware

End to end pipeline support for color spaces and overscanEnd to end pipeline support for color spaces and overscanCapture, decoders, transforms should implement itCapture, decoders, transforms should implement it

Optimized for output deviceOptimized for output device

Chroma handling reduces conversionsChroma handling reduces conversions

Lossless Y’CbCr <-> RGB conversions and interlace supportLossless Y’CbCr <-> RGB conversions and interlace support

High precision fp scRGB supportHigh precision fp scRGB support

IntegrationIntegration

CaptureCaptureUncompressed output can pass directly to GPU for DXVA processingUncompressed output can pass directly to GPU for DXVA processing

TransformsTransformsMFTs – write once, use everywhere (MF, DShow, custom apps)MFTs – write once, use everywhere (MF, DShow, custom apps)

Support for HW acceleration (DXVA/D3D)Support for HW acceleration (DXVA/D3D)

EVR presenter designEVR presenter designSame presenter plug-in useable in both MF and DShowSame presenter plug-in useable in both MF and DShow

Surface formatsSurface formatsStandardized Y’CbCr formatsStandardized Y’CbCr formats

Fewer conversions and copiesFewer conversions and copies

Simplifies capture, transforms, video output formatsSimplifies capture, transforms, video output formats

Chroma sampling info preserved – avoid resamplingChroma sampling info preserved – avoid resampling

High precision scRGB (fp16) surfacesHigh precision scRGB (fp16) surfacesEasy mixing of text, video, photos, etc.Easy mixing of text, video, photos, etc.

Leverage color calibration for videoLeverage color calibration for video

Future DirectionsFuture Directions

Expand video processing featuresExpand video processing features

Improve performance, usability and qualityImprove performance, usability and quality

Future DirectionsFuture Directions

Video analysisVideo analysisEncode accelerationEncode acceleration

Motion estimation, DCT,…Motion estimation, DCT,…

Inverse telecine support (IVT)Inverse telecine support (IVT)

Programmable video processingProgrammable video processingAdvanced de-interlacing and frame rate conversionAdvanced de-interlacing and frame rate conversion

Extended output device supportExtended output device supportTVs completely integrated into operating systemTVs completely integrated into operating system

Cross-process pipelining of videoCross-process pipelining of video

Efficient streaming of data between CPUs and GPUsEfficient streaming of data between CPUs and GPUs

Resource management and load balancingResource management and load balancingCPU multi-cores, 3-DCPU multi-cores, 3-D

Video decoding and processingVideo decoding and processing

Call To ActionCall To Action

Take advantage of new features in Windows VistaTake advantage of new features in Windows Vista

Graphics IHVsGraphics IHVsPipelining of decoding, processing operationsPipelining of decoding, processing operations

Parallel hardware to improve concurrencyParallel hardware to improve concurrency

DXVA2 is power management and quality API/DDIDXVA2 is power management and quality API/DDIHigher power density decode, VP, encode, deinterlaceHigher power density decode, VP, encode, deinterlace

Accurate vsync and deep queuing supportAccurate vsync and deep queuing support

Video processingVideo processingColorspace support, multi-tap, gamma correct scalingColorspace support, multi-tap, gamma correct scaling

10 bit and scRGB buffer support10 bit and scRGB buffer support

WDDM capture driversWDDM capture drivers

ISVsISVsStart the transition to MF, MFTs and EVRStart the transition to MF, MFTs and EVR

Additional ResourcesAdditional Resources

Web ResourcesWeb ResourcesSpecifications Specifications http://msdn.microsoft.com/library/default.asp?url=/library/en-us/Dhttp://msdn.microsoft.com/library/default.asp?url=/library/en-us/Display_d/hh/Display_d/UserModeDisplayDriver_Guide_9205cb11isplay_d/hh/Display_d/UserModeDisplayDriver_Guide_9205cb11-857b-4ea0-b0d0-a4fb5bdb3e83.xml.asp-857b-4ea0-b0d0-a4fb5bdb3e83.xml.asp

Related SessionsRelated SessionsWindows Vista Output Content ProtectionWindows Vista Output Content Protection

DXVA 2.0: A New Hardware Video Acceleration DXVA 2.0: A New Hardware Video Acceleration Pipeline For Windows VistaPipeline For Windows Vista

Questions and commentsQuestions and comments askDXVA @ microsoft.comaskDXVA @ microsoft.com

© 2006 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.

The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation.

MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.