Brushing Up on Upconversion

Bruce JacobsTwin Cities Public Television

2

At Stake

Millions of hours of SD contentNew consumer HD displaysAt best, 480i becomes 720pAt worst 360i becomes 1080p!

UC

720

360i

1920

1080p

3

Brush Up

Upconverter Theory- Digitize- Deinterlace- Scale

Upconverter Practice- Software- Hardware

4

ReferencesLarry Thorpe: "The HDTV Camcorder and the March to

Marketplace Reality" SMPTE Journal, March 1998

Jed Deame: "Motion Compensated De-Interlacing: The Key to the Digital Video Transition"Presented at the SMPTE 141st Technical Conference, November 19-22, 1999

Charles Poynton: "Digital Video and HDTV: Algorithms and Interfaces"Morgan Kaufmann, 2003

Digitize & Scale

6

Spatial Frequency

7

Spatial Frequency

8

Spatial Frequency

9

Spatial Frequency

10

Tip #1

To successfully reconstruct a digitized signal, we must sample at more than twice the highest input frequency – the "Nyquist Rate".

11

Four Sample Examples

At Nyquist Ratef(s) = 2 * f(max)

Below Nyquist Ratef(s) = 1.3 * f(max)

Double Nyquist Ratef(s) = 4 * f(max)

Just Above Nyquist Ratef(s) = 2.3 * f(max)(Rec 601 = 13.5/5.75=2.3)

12

At Nyquist Rate

13

Pan Right ⅛λ

14

Pan Right ¼λ

15

Tip #2

In a camera, direct pixel mapping isn't good enough, despite the "PC" experience.

- See Tip #1

16

Below Nyquist Rate

Alias = 1/3 f !

17

Pan Right ⅛λ

18

Pan Right ¼λ

Alias Moved ¾λ Right!

19

Tip #3

Aliasing is bad enough in audio where it is "stationary". In video, aliasing also moves, the wrong distance, often in the wrong direction, resulting in obvious and annoying artifacts.

20

Pre-Filter

f(s)Aliasing

Lost Resolution

½ f(s)

21

Tip #4

Avoid aliasing by using a steep LP filter before sampling.

- Steep filters are impossible with optics- Camera designers say some aliasing is better than degraded resolution

Avoid resolution loss by sampling at far more than double highest desired frequency.

- Requires extra pixels and higher bit rate

22

Oversampling

f(s1)

½ f(s1) ½ f(s2)

Pre-Filter Post-Filter & Sub-Sample

23

Tip #5

Use Oversampling 1. Increase sample frequency2. Increase pre-filter pass-band3. Add steep digital LP post-filter4. Subsample to original rate

- Legacy technique in SD CCD cameras- Improves resolution- Reduces aliasing- No increase in bit rate- Does increase camera noise

24

Double Nyquist Rate

25

Pan Right ⅛λ

26

Pan Right ¼λ

27

Just Above Nyquist Rate

28

Pan Right ⅛λ

29

Pan Right ¼λ

Alias Moved ≈2λ Right!

30

Tip #6

Even with adequate sampling, mapping the output directly to perfect digital display does not produce perfect results!

31

Reconstruction Filter

f(s)Aliasing

Lost Resolution

½ f(s)

Pre-Filter

32

Tip #7

To reconstruct the signal, a low pass filter is needed, to remove the sample frequency image.

- Too steep results in ringing- An analog filter for analog output- Can be digital, if a higher frequency is used!

33

Reconstruction LPF

Filter Magic

34

Tip #8

Using "reconstruction filters" in both dimensions will upconvert the image!

- Doing this well (using many "taps") requires perhaps thousands of multiplications for every pixel, in every color channel, in every frame, all in real time!

Deinterlace

The interlace problem:- Simply using fields might leave only 180 V lines!

36

Deinterlace

1.1

60 fields per second from source

even

odd

1.2

2.1

2.2

3.1

3.2

4.1

4.2

5.1

5.2

? ? ? ? ? ? ? ? ??

60 Frames per second needed for upconversion

37

3/2 Pulldown

1

24 Fps

1.1

60 fps "3""3" "2""2"

even

odd

1.2

1.1

2.2

2.1

3.2

3.1

3.2

4.1

4.2

32 4

38

Tip #9

If the 60i content came from 24p, weave matching fields back into frames, and ignore the duplicative fields.

39

Pulldown Removal

1

24 Fps

1.1

60 fps

even

odd

1.2

1.1

2.2

2.1

3.2

3.1

3.2

4.1

4.2

32 4

1 1 2 2 3 3 3 4 41

60 Fps

40

Tip #10

If the two fields are well correlated when there is no motion, weave them back into a frame and repeat it.

=

41

Tip #11

If the two fields are completely uncorrelated, bob between fields with half the vertical resolution for each.

≠

42

Tip #12

Adapt by using weave in regions of correlation, and bob between fields where not correlated.

≠

=

43

Tip #13

Search for motion vectors, and compensate where possible.

Practice

45

DemonstrationSoftware

- Final Cut Pro- Apple Compressor

Embedded Hardware- AJA Kona- Panasonic AJ-HD1400 DVCProHD VTR- Sony HDW-M2000 HDCam VTR

Stand-Alone Hardware- Leitch X75- Miranda XVP-811i- Snell and Wilcox Quasar PH.C- Teranex Xantus