technology & ventures - research laboratory of …€¦ · · 2011-08-03technology &...

Technology & Ventures

Early Days

• Acquisition• Processing• Application

• Signal Acquisition• Signal Processing• Signal ProcessingApplication & Ventures

Signal AcquisitionWilliam Blackwell, MIT Lincoln Laboratory

Signal ProcessingHenrique Malvar, Microsoft Research

Signal Processing Application & VenturesJeffrey Bernstein, Analog DevicesBrian Hinman, Oak Investment Partners

StaelinFest-1WJB 7/18/2011

Dave Staelin’s Influence on Microwave & Hyperspectral

Infrared Remote SensingBill Blackwell (Staelin Ph.D., 2002)

StaelinFest

18 July 2011


Dave's Gift (one of many):Elegant Thinking about Complex Problems

First slide of course notes fromMIT EECS 6.661: Receivers, Antennas, and Signals


• A New Generation of Satellite Sensors– Hyperspectral IR spectrometers: From RGB to 1000’s of “colors”– Opportunities for new signal processing methods

• A New Generation of Remote Sensing Systems– NASA Aqua (Dave’s methods now used operationally)– NPOESS/JPSS (Dave’s methods to be used operationally)– DWSS (Dave’s methods to be used operationally)

• A Look Forward and A Look Back

Principal Components


Atmospheric Remote Sensing: Measurement Scenario


Passive Microwave Sensing of Precipitation

35 km

45 k

m

Image credit: Vince Leslie (Staelin Ph.D., 2004)


Atmospheric Transmission at Microwave Wavelengths

The frequency dependence of atmospheric absorption allows different altitudes to be sensed by spacing channels along absorption lines

“Sounding” channels


Synergistic Use of MW+IR:Infrared Provides High Spatial Resolution

Typical infrared sensors provide 15km horizontal and 1km vertical resolution


Passive Microwave Measurements Provide Low Spatial Resolution, but Penetrate Clouds

Typical microwave sensors provide 35km horizontal and 3km vertical resolution

Dave’s contributions span both the spectral and spatial domains


Innovation in Algorithm Development: Neural Network Retrievals

AIRS measures upwelling thermal emission in 2378 spectral bands from 4-15 µmSounding is accomplished using absorption near CO2 lines

AMSU measures upwelling thermal emission in 20 spectral bands near 60 and 183 GHzSounding is accomplished using absorption near O2 (60-GHz) and H2O (183-GHz) lines

50 k

mSCC/NN Algorithm:

• 100X faster than state-of-art• Improved accuracy and precision• Enables rapid processing of

multi-year climate records


US Next-Generation Weather Satellite ProgramOctober 25, 2011 Launch

NPP spacecraft photo courtesy of Ball Aerospace, Boulder, CO.

“Dave’s Instrument”


• FUN times: He loves his work and it is contagious– Many great summer parties

• ALWAYS a champion for his students• A man of many accomplishments

– #1 (by far) was marrying Ellen

• "Life Lessons" (of which there were many)– Management– Psychology– Politics– Art of engineering

• A lasting legacy– Educator– Innovator– Mentor– Friend

Reflections on Working with Dave

Blackwell wedding, Aug 2000

Jack and Ann Barrett

StaelinFestSignal Processing

Henrique (Rico) MalvarChief Scientist

Microsoft Research

Contents• Signal Processing is very broad

– So we’ll zoom into “Lapped Transforms”– Dave pioneered the research and coined the name

Lapped Orthogonal Transform

• Contents– LT Basics– Applications– Example: Image Compression with JPEG XR

(Motivated by Dave’s question: “Rico, tell me more about JPEG XR”)

StaelinFest – MIT, July 18, 2011 Signal Processing 2

3

Block processing

• Signal is reconstructed as a linearcombination of basis functions

• Can lead to discontinuities across blockboundaries (blocking artifacts, as in JPEG)

ExtractBlock

DirectOrthogonalTransform

InverseOrthogonalTransform

AppendBlock

I nputSignal

OutputSignal

X xT= Px

~ ~x X= P

~XCoding, filtering, etc.

StaelinFest – MIT, July 18, 2011 Signal Processing

Lapped transforms• Basis functions have tails beyond block boundaries

– Linear combinations of overlapping functions such as

– generate smooth signals, without blocking artifacts:


5

Basis functions

StaelinFest – MIT, July 18, 2011Signal Processing

Discrete Cosine Transform( DCT )

basis functions end abruptly at block boundaries

Lapped Orthogonal Transform( LOT )

basis functions decay smoothly to zero beyond block boundaries

LOTs: properties• Functions can be orthogonal

and lapped orthogonal(Staelin, Cassereau, ’85)– W is one-block overlap operator

• Fast-computable via DCT (Staelin, Malvar, ’86)


T

T

==

=

P P IP W P I

0 IW

0 0

Z

E

ODCT

E

ODCT

-- -

input block2-N samples

N/2 even-symmetricLOT coefficients

N/2 odd-symmetricLOT coefficients

LOTs: design evolution• Linear-phase, DCT-based (LOT ’85)

– Best for images• DCT-IV based, good filter banks (MLT ’88)

– Best for audio• Extended multi-block overlap (ELT ’92)

– Best for modems / digital communication• Biorthogonal, hierarchical (LBT, HLBT ’99)

– Best for images, with multi-resolution• Integer-reversible (PTC ’02, JPEG XR ’08)

– Best for image compression


Applications of LTs• Audio and speech format standards (MLT)

– Audio: MP3, AAC, WMA, PAC, …– Wideband speech: G.722.1, CELT

• DSL modems (ELT)– Multitone modulation w/ large # of subbands

• Image compression (LBT)– PTC image coder (used in some Xbox games)– JPEG XR – ITU-T/ISO standard, successor to JPEG


Lapped Transforms inImage Compression

JPEG XR


Original LBT construction

x(0) X(0)

x(1) X(2)

X(1)

X(3)

to previous block

to next block

X(0)

X(2)

X(1)

X(3)

x(2)

x(3)

x(0) X(0)

x(1) X(2)

DCT

X(1)

X(3)

x(2)

x(3)

Zangle = /8

a = (direct)2

1/2a = (inverse)

DCT

c

c


New LBT with pre/post-filters


x(0) X(0)x(1) X(2)

DCT-like

X(1)X(3)

to previous block

to next block

x(2)x(3)

New Length-4LBT, Direct

pre-overlap

y(0)Y(0)y(1)Y(2)

Y(1)Y(3)

y(2)y(3)

IDCT-like

post-overlapfrom previous block

from previous block

New Length-4LBT, Inverse

codi

ng

overlappre-filter

overlappre-filter

overlappost-filter

overlappost-filter

Reversible & non-separable 2D LBTs

Stage 1

Stage 2

Forward transform shown, inverse is similar

Step 1.1 – HT applied to corners

Step 1.2 – HT applied to centers

Step 1.3 – HT applied to edges

Step 1.4 – HT applied to edges

Step 2.1 – HT for even-even basis

Step 2.2 – oddT for even-odd basis

Step 2.3 – oddT for odd-even basis

Step 2.4 – odd-oddT for odd-odd basis

A C B D

a b c d

1−

1−

1−

1−

1−2

1− 21

A B C D

1−

1−

21

21− 2

1

21−16

3

163−

83−

83

1−

163

163−

a b c d

a b c d

A B C D

1−2

1

21− 2

1 1−

21−8

3

43−

1−

1−8

3


Basic JPEG XR architecture

13

Reversible color

conversionTiling

Reversible HLBTTransform

Quantization, Entropy code

0110010101…

Packetization

YCoCgcolor space


Key aspects of JPEG XR• Supports 8, 16, 24, 32 bpc & 32-bit floats• Decoder spec 100% integer arithmetic• Lossless & lossy coding in same codec• Compression ≈ JPEG 2000, ≈ 2x JPEG• Complexity ≈ 1.3x JPEG ≈ JPEG 2000 / 3• Single-company IP, licensed royalty-free• ITU-T & ISO Standard• Commercial apps: few cameras, WP 7, IE 9, …


15

Image Coding – JPEG 1


16

Image Coding – JPEG XR 1


17

Image Coding – JPEG 2


18

Image Coding – JPEG XR 2


Thanks

Q & A


Dave Staelin: Our Professor, Mentor, Business Partner and Friend

Brian Hinman and Jeff Bernstein

Two Maryland Boys Going to MIT

Professor Staelin’s Research Group, Circa 1983

Video Coding Research

Morphological Image Coding

Ali Thesis, “Morphological Coding of Images,” August 1984

The Hybrid MC/DCT Coding Algorithm

Jain and Jain: Displacement Measurement and Interframe Image Coding,IEEE Transactions on Communications, COM-29, No. 12, December 1981

The Short-Space Fourier Transform (SSFT)

Bernstein Thesis, “Properties and Applications of the Short-Space FourierTransform,” May 1984

SSFT Still Image Compression

Presented, ICASSP, March 1984

SSFT versus DCT in Hybrid Video Coder

Bernstein Thesis, “Properties and Applications of the Short-Space FourierTransform,” May 1984

Lapped Orthogonal Transforms

• Finite extent basis functions:Encoding of images based on a lapped orthogonal transform, Cassereau, de Jager and Staelin, IEEE Transactions on Communications, February 1989

• Optimally derived LOT:The LOT: Transform Coding Without Blocking Effects, Malvar and Staelin, IEEE Transactions on ASSP, April 1989

Visualizing Motion Estimation

Hinman Thesis, “Theory and Applications of Image Motion Estimation,” May 1984

Early Concepts in Block Motion Estimation

STEP 1:

Picking a good initial motion vector based upon motion vectors from surrounding blocks


STEP 2:

Finding the motion vector through steepest decent motion estimation

Resolution Preserving Frame Interpolation


Let’s Start a Company

Rejection from Venture Capitalists

“You should stop playing part-time businessman and your two students should find jobs at Raytheon to get some experience.”

Robert G. BarrettBattery VenturesJuly 1984

Dave Said We Should Go Public Instead

Dave Gathers the PicTel MIT Team

• Dave Staelin – Chairman• Norm Gaut – Director and later CEO• Brian Hinman – V.P. Research, then V.P. Engineering• Jeff Bernstein – Principal Engineer• Rico Malvar – Video Designer and later V.P. Research• Mike Dertouzos – Consultant• Greg Papadopoulos – Processor Architect• Richard Soley – Microcode Development Environment

David H. Staelin, Chairman of PicTel

18 Months From IPO to Prototype

Celebrating the C-2000 codec operating, May 1986

Our First Product: C-2000 Video Codec

• Video Resolution: 256 x 240• Frame Rate: 15fps• Transmission Rate: 56kbps – 224kbps• Processors: 7 microcoded array processors• Total Compute Power: 200 MOPS• Power Consumption: 1,350 Watts• Weight: 200 lbs

Desktop Video Circa 1986

We were a little ahead of our time…

The Use of ASICs: C-3000 Video Codec

• Video Resolution: 256 x 240• Frame Rate: 15fps• Transmission Rate: 56kbps – 384kbps• Processor: LSI Logic Gate Arrays• Total Compute: 240 MOPS• Power Consumption: 420 Watts• Weight: 70 lbs

First Integrated Roll-About: V-3100

• Integrated PTZ Camera• Picture-in-Picture Video Display• Still Graphics Sub-channel• Integrated Audio with Acoustic Echo Cancellation• Control Panel with Far-end Camera Control

Fame and Fortune Follows

Jeff Bernstein and Norman Gaut, Fortune Magazine, October 1992

Lessons Learned from Dave

Great research can be done with limited resourcesState-of-the-art lab not required

Create a team of brilliant people with a diverse set of backgrounds

Think it through carefully…Then jump in all the way

Use your skills and energy to help others

technology & ventures - research laboratory of …€¦ · · 2011-08-03technology &...

Documents