dali virtual machine wei tsang ooi (with brian, sugata, tibor, steve, haye, matthew, dmitriy)
Post on 19-Dec-2015
218 views
TRANSCRIPT
![Page 1: Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d3e5503460f94a16c02/html5/thumbnails/1.jpg)
Dali Virtual Machine
Wei Tsang Ooi(With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)
![Page 2: Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d3e5503460f94a16c02/html5/thumbnails/2.jpg)
Motivations
We know how to encode, transmit and store multimedia data today
People start looking into ways to process multimedia data
![Page 3: Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d3e5503460f94a16c02/html5/thumbnails/3.jpg)
Video Editing
Concat, cross-fade, overlay, cut and paste
![Page 4: Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d3e5503460f94a16c02/html5/thumbnails/4.jpg)
Gateway
Convert from one format to anotherChange compression rate
transcoder(format A) format Binput video output video
![Page 5: Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d3e5503460f94a16c02/html5/thumbnails/5.jpg)
DBMS
“ Find all movies in the database that contain this image “
![Page 6: Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d3e5503460f94a16c02/html5/thumbnails/6.jpg)
Set-top Boxes
Subtitling and close captioningMixing of incoming movies
![Page 7: Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d3e5503460f94a16c02/html5/thumbnails/7.jpg)
Lecture Browser
Match video to slideSwitch between video streams
![Page 8: Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d3e5503460f94a16c02/html5/thumbnails/8.jpg)
Current Solutions
Black box C code (mpeg_play) Hard to reuse/adapt/break apart Unpredictable performance
Standard library (PPM, IJG) Least common denominator (RGB for
all)Video frames --> RGB --> gray
![Page 9: Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d3e5503460f94a16c02/html5/thumbnails/9.jpg)
History
Rivl - high level scripting language for multimedia processing
Still suffers from “black box” problem
![Page 10: Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d3e5503460f94a16c02/html5/thumbnails/10.jpg)
Experience
Real cost of processing Inherently complex operations
Projective transform
Layered operationsVideo decoding
Simple operations but lots of dataCopy
![Page 11: Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d3e5503460f94a16c02/html5/thumbnails/11.jpg)
Dali Architecture
Dali Code
Dali Compiler
DVM Code
Dali Virtual Machine
User Expert User
MMX TriMediaC
![Page 12: Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d3e5503460f94a16c02/html5/thumbnails/12.jpg)
Today’s Talk
DVM Code
Dali Virtual Machine
![Page 13: Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d3e5503460f94a16c02/html5/thumbnails/13.jpg)
DVM Code : Design Goal
Small number of composable abstractions
Simple, predictable opcodesHigh performanceTarget for compilerEasy to extend
![Page 14: Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d3e5503460f94a16c02/html5/thumbnails/14.jpg)
Abstractions
Byte imageBitstreamMPEGJPEGBit masksPCM
![Page 15: Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d3e5503460f94a16c02/html5/thumbnails/15.jpg)
Abstractions
Byte imageBitstreamMPEGJPEGBit masksPCM
![Page 16: Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d3e5503460f94a16c02/html5/thumbnails/16.jpg)
Byte Image
Two dimensional array of byteCan be either physical or virtual
physicalimage virtual
image
![Page 17: Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d3e5503460f94a16c02/html5/thumbnails/17.jpg)
Byte Image Can Represent
Gray scale imageRGB imageYUV imageAlpha channel
![Page 18: Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d3e5503460f94a16c02/html5/thumbnails/18.jpg)
Example Code : Fractal
![Page 19: Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d3e5503460f94a16c02/html5/thumbnails/19.jpg)
smaller = byte_new (neww, newh);byte_shrink_2x2(image,smaller);
image
smaller
neww
newh
![Page 20: Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d3e5503460f94a16c02/html5/thumbnails/20.jpg)
image
target = byte_clip(image,0,0,dx,2*dy);byte_set(target,0);target = byte_clip(image,3*dx,0,dx,2*dy);byte_set(target,0);
targetdx
2dy
![Page 21: Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d3e5503460f94a16c02/html5/thumbnails/21.jpg)
target = byte_clip(image, dx, 0, neww, newh);byte_copy(smaller, target);
target
image
smaller
newh
neww
dx
![Page 22: Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d3e5503460f94a16c02/html5/thumbnails/22.jpg)
target = byte_clip(image, 2*dx, 2*dy, neww, newh);byte_copy(smaller, target);target = byte_clip(image, 0, 2*dy, neww, newh);byte_copy(smaller, target);
image
smaller
newwnewh
dx
2dy
![Page 23: Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d3e5503460f94a16c02/html5/thumbnails/23.jpg)
DVM CodeDx = 0.25 * byte_width(image);dy = 0.25 * byte_height(image);neww = 0.5 * byte_width(image);newh = 0.5 * byte_height(image);smaller = byte_new (neww,newh);for (i = 0; i < n; i++) { byte_shrink_2x2(image,smaller);
Target = byte_clip(image,0,0,dx,2*dy);byte_set(target,0);target = byte_clip(image,3*dx,0,dx,2*dy);byte_set(target,0);target = byte_clip(image, dx, 0, neww, newh);
byte_copy(smaller, target); target = byte_clip(image, 2*dx, 2*dy, neww, newh); byte_copy(smaller, target); target = byte_clip(image, 0, 2*dy, neww, newh); byte_copy(smaller, target);}
![Page 24: Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d3e5503460f94a16c02/html5/thumbnails/24.jpg)
General Dali Strategies
Specific instruction byte_shrink_2x2, byte_shrink_4x4, etc
Explicit memory allocation byte_new
Reduce data byte_clip
Composable abstraction byte image
![Page 25: Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d3e5503460f94a16c02/html5/thumbnails/25.jpg)
Performance
Dali VM Rivl
Sparc 20 0.5 s 8.8 s
P2 266 0.3 s 4.0 s
run fractal with n = 4 on 800x600 gray scale image
about 21 frames/sec for 320x240 video
![Page 26: Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d3e5503460f94a16c02/html5/thumbnails/26.jpg)
Abstractions
Byte imageBitstreamMPEGJPEGBit masksPCM
![Page 27: Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d3e5503460f94a16c02/html5/thumbnails/27.jpg)
MPEG
Getting important and pervasiveComplex formatComplex code (mpeg_play)Most decoders provides
GetNextFrame() interfaceRandom access, direct transfer
difficult
![Page 28: Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d3e5503460f94a16c02/html5/thumbnails/28.jpg)
Abstraction for MPEG Video
. . .seqhdr
gophdr gop
gophdr gop
seqend
![Page 29: Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d3e5503460f94a16c02/html5/thumbnails/29.jpg)
Abstraction for MPEG Video
. . .seqhdr
gophdr gop
gophdr gop
seqend
. . .pichdr
pichdrpic pic
![Page 30: Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d3e5503460f94a16c02/html5/thumbnails/30.jpg)
DVM Code Examples
Skip to the n-th frame
for (i = 0; i < n; i++) { mpeg_pic_hdr_find(bs); mpeg_pic_hdr_skip(bs);
}
![Page 31: Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d3e5503460f94a16c02/html5/thumbnails/31.jpg)
DVM Code Examples
Skip to the n-th frame
for (i = 0; i < n; i++) { mpeg_pic_hdr_find(bs); mpeg_pic_hdr_skip(bs);
}
![Page 32: Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d3e5503460f94a16c02/html5/thumbnails/32.jpg)
DVM Code Examples
Skip to the n-th frame
for (i = 0; i < n; i++) { mpeg_pic_hdr_find(bs); mpeg_pic_hdr_skip(bs);
}
![Page 33: Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d3e5503460f94a16c02/html5/thumbnails/33.jpg)
DVM Code Examples
Skip to the n-th frame
for (i = 0; i < n; i++) { mpeg_pic_hdr_find(bs); mpeg_pic_hdr_skip(bs);
}
![Page 34: Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d3e5503460f94a16c02/html5/thumbnails/34.jpg)
inbs1 inbs2
outbs
hdr hdr
hdr hdr
gop
gop
gop
gop
Concat 2 Video Sequences
![Page 35: Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d3e5503460f94a16c02/html5/thumbnails/35.jpg)
DVM Code
Concat two sequences
While not end_of(inbs1) { gop_hdr_dump(inbs1, outbs); gop_dump(inbs1, outbs);}While not end_of(inbs2) {
gop_hdr_dump(inbs2, outbs); gop_dump(inbs2, outbs);}
![Page 36: Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d3e5503460f94a16c02/html5/thumbnails/36.jpg)
Performance Analysis
Decode MPEG sequences to PPMNo outputRun on 3 different sequences
![Page 37: Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d3e5503460f94a16c02/html5/thumbnails/37.jpg)
Performance
Rate (frame/sec)
Mpeg PlaySparc 20
Dali VMSparc 20
psycho(160x120)
27.3 27.9
bus(320x240)
8.8 9.9
tennis(320x240)
7.8 9.1
![Page 38: Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d3e5503460f94a16c02/html5/thumbnails/38.jpg)
Performance
Rate (frame/sec)
Mpeg PlaySparc 20
Dali VMSparc 20
Dali VMP2 266
psycho(160x120)
27.3 27.9 112.1
bus(320x240)
8.8 9.9 36.0
tennis(320x240)
7.8 9.1 32.6
![Page 39: Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d3e5503460f94a16c02/html5/thumbnails/39.jpg)
MPEG Strategies
Expose structure Gop header, gop instead of just frames
Break up layer Parse, decode, color space conversions
![Page 40: Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d3e5503460f94a16c02/html5/thumbnails/40.jpg)
Show Time !
Decode a MPEG sequenceFor each frame scale it to “stamp”
sizeCreate a contact sheet
![Page 41: Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d3e5503460f94a16c02/html5/thumbnails/41.jpg)
Abstractions
Byte imageBitstreamMPEGJPEGBit masksPCM
![Page 42: Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d3e5503460f94a16c02/html5/thumbnails/42.jpg)
What Is Bitstream ?
Skip to the n-th frame
for (i = 0; i < n; i++) { mpeg_pic_hdr_find(bs); mpeg_pic_hdr_skip(bs);
}
![Page 43: Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d3e5503460f94a16c02/html5/thumbnails/43.jpg)
Bitstream
parser1011001
chunk of memory
![Page 44: Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d3e5503460f94a16c02/html5/thumbnails/44.jpg)
Mode of Operations
File i/ostatic memory
fread/fwrite
parser
![Page 45: Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d3e5503460f94a16c02/html5/thumbnails/45.jpg)
Mode of Operations
Networkstatic memory
socket
parser
![Page 46: Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d3e5503460f94a16c02/html5/thumbnails/46.jpg)
Mode of Operations
Memory map
parser
entire file
![Page 47: Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d3e5503460f94a16c02/html5/thumbnails/47.jpg)
inbs videovideo videoaudio audio audio
mpeg system stream
![Page 48: Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d3e5503460f94a16c02/html5/thumbnails/48.jpg)
filter
videobs
inbs videovideo videoaudio audio audio
mpeg system stream
![Page 49: Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d3e5503460f94a16c02/html5/thumbnails/49.jpg)
filter
videobs
inbs videovideo videoaudio audio audio
parser
mpeg system stream
![Page 50: Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d3e5503460f94a16c02/html5/thumbnails/50.jpg)
Bitstream Strategies
Predictable performance Explicit I/O
![Page 51: Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d3e5503460f94a16c02/html5/thumbnails/51.jpg)
Recap: Current Solutions
Black box C code (mpeg_play) Hard to reuse/adapt/break apart Unpredictable performance
![Page 52: Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d3e5503460f94a16c02/html5/thumbnails/52.jpg)
Recap: Cost of Processing
Inherently complex operations Layered operationsSimple operations but lots of data
![Page 53: Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d3e5503460f94a16c02/html5/thumbnails/53.jpg)
How We Solve the Problems
Difficult to reuse/adapt/break apart Minimum set of abstractions Abstractions are simple Simple reusable opcodes
Unpredictable performance Explicit I/O Explicit memory allocations
![Page 54: Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d3e5503460f94a16c02/html5/thumbnails/54.jpg)
How We Solve the Problems
Inherently complex operations Special case Clipping and masking
Layered operations Expose structure
Simple operations but lots of data Special case Clipping and masking
![Page 55: Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d3e5503460f94a16c02/html5/thumbnails/55.jpg)
Extending with new formats : JPEG, WAV, AVI etc
TriMedia chip implementation
Application : lecture browser
Work in Progress
![Page 56: Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d3e5503460f94a16c02/html5/thumbnails/56.jpg)
Future Work
Multithreading
MMX implementationCompiler