Slide 1

From SD to HDImproving Video Sequences Through Super-Resolution Michael EladComputer-Science Department The Technion - Israel al Workshop - November 10, 2011

finSpeaker: Matan ProtterHave You Seen My HD?1In this Talk we shall describe a fascinating technology called Super-Resolutionwhich suggests ways to take (very) poor quality images and fuse them to high-quality imageFew Interesting Facts: " "? Not here!This field is ~25 years oldIsraeli scientists play a major role in itDespite its appeal, there were no industrial applications until recently !Speaker: Matan ProtterHave You Seen My HD?2AgendaIntroductionBasic Concepts: Image, Pixel, ResolutionClassic Super-ResolutionHow does it Work?Problems and (Many) LimitationsNew Era of Super-ResolutionChanging the Basic ConceptNew ResultsSummarySpeaker: Matan ProtterHave You Seen My HD?3AgendaIntroductionBasic Concepts: Image, Pixel, ResolutionClassic Super-ResolutionHow does it Work?Problems and (Many) LimitationsNew Era of Super-ResolutionChanging the Basic ConceptNew ResultsSummarySpeaker: Matan ProtterHave You Seen My HD?4What is an Image?

Speaker: Matan ProtterHave You Seen My HD?5What is an Image?The camera counts the number of photons

Speaker: Matan ProtterHave You Seen My HD?6What is an Image?

To the computer, an image is nothing but a table of numbers

Each one of these color points is called aPixel = Picture ElementThis is how we see an imageSpeaker: Matan ProtterHave You Seen My HD?7ResolutionThis image is 18 by 25 pixels

Speaker: Matan ProtterHave You Seen My HD?8Resolution

This image is 36 by 50 pixelsSpeaker: Matan ProtterHave You Seen My HD?9ResolutionThis image is 72 by 100 pixels

Speaker: Matan ProtterHave You Seen My HD?10ResolutionThis image is 144 by 200 pixels

Speaker: Matan ProtterHave You Seen My HD?11ResolutionThis image is 288 by 400 pixels

Resolution: the size of the smallest detail that can be seen in the imageSpeaker: Matan ProtterHave You Seen My HD?12

Color Colors can be created by a mixture of Red , Green and Blue

Speaker: Matan ProtterHave You Seen My HD?13Color Images Each pixel counts the number of photons in each color

RGBSpeaker: Matan ProtterHave You Seen My HD?14Back to Resolution Screens are composed of pixels as well

What happens when the image has a different number of pixels from the screen?Speaker: Matan ProtterHave You Seen My HD?15Resolution DisparitiesCase 1 (easy!): Image has more pixels than screenOption 1: Crop

Speaker: Matan ProtterHave You Seen My HD?16Resolution Disparities

Case 1 (easy!): Image has more pixels than screenOption 2: ShrinkSpeaker: Matan ProtterHave You Seen My HD?17Resolution DisparitiesCase 2 (problem!): screen has more pixels than image

This is a common problem with HD screens nowadaysScreen typical size:Height = 1080Width = 1920Video typical size:Height = 480 Width = 640Speaker: Matan ProtterHave You Seen My HD?18Why Isnt Everything HD in the First Place?Various reasons:Old material: Sequences shot before HD cameras (HD came to the market ~3 years ago )Cost: HD video technology is too expensive (but this is fast-changing):HD camera and equiment is more expensiveHD is more expensive to broadcast: More pixels wider pipesHD requires more storage spaceBottom Line: much of the available video material is of inadequate resolution for display on HD screens.Comment: The above problem is relevant to other needs (security, military, medical, entertainment, )Speaker: Matan ProtterHave You Seen My HD?19Resolution DisparitiesCase 2 (problem!): screen has more pixels than image

Option 1: Put image in the middle of the screenSpeaker: Matan ProtterHave You Seen My HD?20Resolution DisparitiesCase 2 (problem!): screen has more pixels than image

Option 2 : Interpolation - Guess pixel values between known pixels

Speaker: Matan ProtterHave You Seen My HD?21Interpolation MethodsInterpolation possibilities:Nearest neighborBilinear interpolationBicubic interpolation There are also edge-adaptive techniques that give more sharpnessNone of the above lead to a true increase in the optical (true) resolution

Spatially invariant methodsMagnify 4:1Speaker: Matan ProtterHave You Seen My HD?22Resolution DisparitiesCase 2 (problem!) : screen has more pixels than image

Desired imageSpeaker: Matan ProtterHave You Seen My HD?23IntroductionBasic Concepts: Image, Pixel, ResolutionClassic Super-ResolutionHow does it Work?Problems and (Many) LimitationsNew Era of Super-ResolutionChanging the Basic ConceptNew ResultsSummaryAgenda

Joint work with Arie Feuer Jacob Hel-Or

Peyman Milanfar Sina FarsiuSpeaker: Matan ProtterHave You Seen My HD?24Our ObjectiveGiven: A set of degraded (warped, blurred, decimated, noised) images:Required: Fusion of the given images into a higher resolution image/s

Speaker: Matan ProtterHave You Seen My HD?25Intuition

DFor a given band-limited image, the Nyquist sampling theorem states that if a uniform sampling is fine enough (D), perfect reconstruction is possibleDSpeaker: Matan ProtterHave You Seen My HD?26IntuitionDue to our limited camera resolution, we sample using an insufficient 2D grid

2D2DSpeaker: Matan ProtterHave You Seen My HD?27Intuition

However, we are allowed to take a second picture and so, shifting the camera slightly to the right we obtain 2D2DSpeaker: Matan ProtterHave You Seen My HD?28Intuition

Similarly, by shifting down we get a third image2D2DSpeaker: Matan ProtterHave You Seen My HD?29Intuition

And finally, by shifting down and to the right we get the fourth image2D2DSpeaker: Matan ProtterHave You Seen My HD?30Intuition

It is trivial to see that interlacing the four images, we get that the desired resolution is obtained, and thus perfect reconstruction is guaranteedThis is Super-Resolution in its simplest formSpeaker: Matan ProtterHave You Seen My HD?31Super-Resolution: The Simplest Example

Speaker: Matan ProtterHave You Seen My HD?32

Super-Resolution: The Simplest Example

Speaker: Matan ProtterHave You Seen My HD?33Intuition Complicating the Story

In the previous example we counted on exact movement of the camera by D in each direction

What if the camera displacement is uncontrolled?Speaker: Matan ProtterHave You Seen My HD?34Intuition Complicating the Story

It turns out that there is a sampling theorem due to Yen (1956) and Papulis (1977) for this case, guaranteeing perfect reconstruction for periodic uniform sampling if the sampling density is high enoughSpeaker: Matan ProtterHave You Seen My HD?35Intuition Complicating the Story

In the previous examples we restricted the camera to move horizontally/vertically parallel to the photograph object. What if the camera rotates? Gets closer to the object (zoom)?Speaker: Matan ProtterHave You Seen My HD?36Intuition Complicating the Story

There is no sampling theorem covering this caseSpeaker: Matan ProtterHave You Seen My HD?37The Problem is Actually More DifficultProblems: Sampling is not a point operation there is a blurMotion may include perspective warp, local motion (moving objects), etc.Samples may be noisy any reconstruction process must take that into accountSolution: Change the treatment: Forget about the sampling interpretation, and replace it with an inverse problem point of viewA consequence: We need to estimate the motion between the different images in a very accurate (sub-pixel accuracy) way

Speaker: Matan ProtterHave You Seen My HD?38Inverse Problem? High-ResolutionImageHHBlur 1N F 1FN Geometric WarpDD1NDecimationAdditive NoiseLow-ResolutionImagesGiven the low-res. imagesReverse the processFind the most-suitable super-res. image

Elad & Feuer (1997)Speaker: Matan ProtterHave You Seen My HD?39Evolution of Super-Resolution1987 Peleg, Keren & Schweitzer345

# of papers1950-1980 Early theory1991 Irani & PelegSpeaker: Matan ProtterHave You Seen My HD?40Evolution of Super-Resolution1950-1980 Early theory1987 Peleg, Keren & Schweitzer1992 Ur & Gross1997 Elad & Feuer2005 Elad, Farsiu & Milanfar2007 # of papers3456217001991 Irani & PelegSpeaker: Matan ProtterHave You Seen My HD?41We Rely on Motion EstimationIf we dont know how the camera moved, we must figure this out from the low-resolution images themselves

All pixels moved exactly 1/3 of a pixel to the rightGlobal motionSpeaker: Matan ProtterHave You Seen My HD?42Motion Estimation Another Example

All pixels moved 2 pixels to the right and 1 pixel upSpeaker: Matan ProtterHave You Seen My HD?43Motion Estimation: Global versus LocalWhat about here?For SR, we need the accurate motion of every pixel

Speaker: Matan ProtterHave You Seen My HD?44Motion Estimation Basic TechniqueComputing the motion for a specific pixelWe look for the pixel with the most similar surroundings

Speaker: Matan ProtterHave You Seen My HD?45Motion Estimation Basic TechniqueComputing the motion for a specific pixelWe look for the pixel with the most similar surroundings

Speaker: Matan ProtterHave You Seen My HD?46SR with Motion Estimation

??????Speaker: Matan ProtterHave You Seen My HD?47SR with Motion Estimation

Speaker: Matan ProtterHave You Seen My HD?48SR With Motion Estimation Results (Scanner) Elad & Hel-Or (1999)16 images, ratio 1:2 in each axis

Taken from one of the given imagesTaken from the reconstructed resultSpeaker: Matan ProtterHave You Seen My HD?49SR With Motion Estimation - Results

Farsiu, Elad & Milanfar (2004)

Speaker: Matan ProtterHave You Seen My HD?50SR With Motion Estimation - ResultsFarsiu, Elad & Milanfar (2004)

Speaker: Matan ProtterHave You Seen My HD?51SR With Motion Estimation - ResultsFarsiu, Elad & Milanfar (2004)

40 images ratio 1:4

30 images ratio 1:4

Speaker: Matan ProtterHave You Seen My HD?52SR With Motion Estimation - ResultsFarsiu, Elad & Milanfar (2005)

Speaker: Matan ProtterHave You Seen My HD?53SR With Motion EstimationHas been practiced for about 20 years, since ~1987 and till 2007 along the above linesThis process requires knowing the motion very accuratelyVery difficult to obtain when motion is not globalOutput result is HORRIBLE when estimation makes mistakesTherefore, SR has been limited to movies with simple motion This may explain why we have not seen this technology coming into play in the industrySpeaker: Matan ProtterHave You Seen My HD?54SR Doesnt Work Here OriginalInterpolationSuper-Resolution with motion estimation

Speaker: Matan ProtterHave You Seen My HD?55And a Snapshot Super-Resolution with motion estimation

Is there no hope for sequences with complicated (local) motion?True high-qualitySpeaker: Matan ProtterHave You Seen My HD?56IntroductionBasic Concepts: Image, Pixel, ResolutionClassic Super-ResolutionHow does it Work?Problems and (Many) LimitationsNew Era of Super-ResolutionChanging the Basic ConceptNew ResultsSummaryAgendaJoint work with Matan Protter

Speaker: Matan ProtterHave You Seen My HD?57Principles of the New SR ApproachThe solution is based on:Probabilistic motion estimationUsing selfmotion that exploits spatial redundancyLocal processingOur mathematical formulation (for those who care):

This formulation leads to a family of algorithms. We will discuss the simplest of them

Speaker: Matan ProtterHave You Seen My HD?58The Problem

Speaker: Matan ProtterHave You Seen My HD?59Probabilistic Motion EstimationWhere does this pixel go?

Speaker: Matan ProtterHave You Seen My HD?60

Probabilistic Motion EstimationWhere does this pixel go?Speaker: Matan ProtterHave You Seen My HD?61

Probabilistic Motion EstimationWhere does this pixel go?

Speaker: Matan ProtterHave You Seen My HD?62Probabilistic Motion EstimationWhere does this pixel go?

In probabilistic motion estimation, all motions are possibleSome are just more likely than othersSpeaker: Matan ProtterHave You Seen My HD?63Exploiting Self-MotionWhat about similar patches within the same image?

When seeking the relevant matches, we consider matches within the same image just as well Thus, even one image can boost itself to SR if it contains self-similaritiesSpeaker: Matan ProtterHave You Seen My HD?64Computing The ProbabilitiesComputing probabilities for a specific pixelWe use similarity of surroundings to measure the probability

Probability: low

Probability: high

Probability: zeroProbability: zeroProbability: lowSpeaker: Matan ProtterHave You Seen My HD?65SR with Probabilistic Motion Estimation

Speaker: Matan ProtterHave You Seen My HD?66SR with Probabilistic Motion Estimation

Result Pixel Value:132

Speaker: Matan ProtterHave You Seen My HD?67

SR with Probabilistic Motion Estimation

Speaker: Matan ProtterHave You Seen My HD?68SR with Probabilistic Motion Estimation

Speaker: Matan ProtterHave You Seen My HD?69Results: Miss AmericaOriginal Sequence (Ground Truth)Lanczos InterpolationAlgorithm ResultInput Sequence (30 Frames) Created from original high-res. sequence using 3x3 uniform blur, 3:1 decimation, and noise with std = 2

Speaker: Matan ProtterHave You Seen My HD?70Results: ForemanLanczos InterpolationAlgorithm ResultInput Sequence (30 Frames) Original Sequence (Ground Truth)

Speaker: Matan ProtterHave You Seen My HD?71Results: Foreman

Classic Super-ResolutionNew Super-ResolutionSpeaker: Matan ProtterHave You Seen My HD?72Results: SalesmanLanczos InterpolationAlgorithm ResultInput Sequence (30 Frames) Original Sequence (Ground Truth)

Speaker: Matan ProtterHave You Seen My HD?73Results: SuzieLanczos InterpolationAlgorithm ResultInput Sequence (30 Frames) Original Sequence (Ground Truth)

Speaker: Matan ProtterHave You Seen My HD?74Results : SD to HD

Speaker: Matan ProtterHave You Seen My HD?75IntroductionBasic Concepts: Image, Pixel, ResolutionClassic Super-ResolutionHow does it Work?Problems and (Many) LimitationsNew Era of Super-ResolutionChanging the Basic ConceptNew ResultsSummaryAgendaSpeaker: Matan ProtterHave You Seen My HD?76Evolution of Super-Resolution1950-1980 Early theory1987 Peleg, Keren & Schweitzer1992 Ur & Gross1997 Elad & Feuer2005 Elad, Farsiu & Milanfar2008 Protter & Elad7011000?2007 1991 Irani & Peleg# of papers3456217002009 Other solutions (Kimmel, Milanfar)Speaker: Matan ProtterHave You Seen My HD?77Limitations in The Proposed ApproachTwo main limitation:The improvement is not always largeEspecially if the movie has been played around with (e.g. deep compression)It does not look worse, thoughComputation time is hugeWe are still far from improving movies as they are played (real-time processing) by factor 20:1 (with GPU implementation)

Speaker: Matan ProtterHave You Seen My HD?78SummaryProblem: displaying low-quality sequences on high-quality screensSolution: Super-Resolution can improve the quality of the sequenceLimitation: SR requires accurate motion estimation; only possible in few sequencesBreakthrough: Introducing probabilistic motion estimation into Super-Resolution

Implications: No limitation on type of sequenceTypically much improved qualityCan be applied to other tasksMany attempts to improve/extend/compete with this approachSpeaker: Matan ProtterHave You Seen My HD?79Thank You For Inviting Me

Questions?

Speaker: Matan ProtterHave You Seen My HD?80Developing a Solution (In General) A common method for developing image processing algorithms:

We look for the image with the highest gradeChallenges: Forming the expression to grasp the results quality, and Optimize the expression with respect to the unknown image

Grading SystemA Mathematical ExpressionGradeImage

Speaker: Matan ProtterHave You Seen My HD?81Principles of the SolutionThe solution is based on:Probabilistic motion estimationUsing selfmotion that exploits spatial redundancyLocal processingOur mathematical formulation:

This leads to a family of algorithms. We will show the simplest of them

Speaker: Matan ProtterHave You Seen My HD?82