video object tracking and replacement for post tv production
DESCRIPTION
Video Object Tracking and Replacement for Post TV Production. LYU0303 Final Year Project Fall 2003. Outline. Project Introduction Basic parts of the purposed system Working principles of individual parts Future Work Q&A. Introduction. - PowerPoint PPT PresentationTRANSCRIPT
Video Object Tracking Video Object Tracking and Replacement for and Replacement for
Post TV ProductionPost TV Production
LYU0303 Final Year ProjectLYU0303 Final Year Project
Fall 2003Fall 2003
OutlineOutline
Project IntroductionProject Introduction Basic parts of the purposed systemBasic parts of the purposed system Working principles of individual partsWorking principles of individual parts Future WorkFuture Work Q&AQ&A
IntroductionIntroduction
Post-TV production software changes the Post-TV production software changes the content of the original video clips.content of the original video clips.
Extensively used in video-making Extensively used in video-making industries.industries.
Why changing the content of a video?Why changing the content of a video? Reducing video production costReducing video production cost Performing dangerous actionsPerforming dangerous actions Producing effects those are impossible in realityProducing effects those are impossible in reality
Difficulties to be overcomeDifficulties to be overcome Things in video can be treated individually Things in video can be treated individually
called “video objects”.called “video objects”. Computers cannot perform object detection Computers cannot perform object detection
directly because…directly because… Image is processed byte-by-byteImage is processed byte-by-byte Without pre-knowledge about the video objects to be detectedWithout pre-knowledge about the video objects to be detected Result is definite, no fuzzy logic.Result is definite, no fuzzy logic.
Though computers cannot perform object Though computers cannot perform object detection directly, it can be programmed to detection directly, it can be programmed to work indirectly.work indirectly.
Basic parts of the purposed systemBasic parts of the purposed system
Simple bitmap reader/writerSimple bitmap reader/writer RGB/HSV converterRGB/HSV converter Edge detectorEdge detector Edge equation finderEdge equation finder Equation processorEquation processor Translation detectorTranslation detector Texture mapperTexture mapper
RGB/HSV converterRGB/HSV converter
Human eyes are more sensitive to the Human eyes are more sensitive to the brightness rather than the true color brightness rather than the true color components of an object.components of an object.
More reasonable to convert the More reasonable to convert the representation of colors into HSV (Hue, representation of colors into HSV (Hue, Saturation and Value (brightness)) model.Saturation and Value (brightness)) model.
After processing, convert back to RGB and After processing, convert back to RGB and save to disk.save to disk.
RGB/HSV converterRGB/HSV converter
RGB to HSVRGB to HSV HSV to RGBHSV to RGB
Edge detectorEdge detector
Usually, a sharp change in hue, saturation or Usually, a sharp change in hue, saturation or brightness means that there exist a boundary brightness means that there exist a boundary line.line.
HSV: (0,0,0)
HSV: (0,255,255)
Edge equation finderEdge equation finder
Derives mathematical facts out of the edge Derives mathematical facts out of the edge points.points.
Works with voting algorithm of Hough Works with voting algorithm of Hough Transform.Transform.
Automatically adjusts tolerance value to Automatically adjusts tolerance value to minimize the effect of noise points.minimize the effect of noise points.
This helps when the edge is not completely This helps when the edge is not completely straight or blurred.straight or blurred.
Edge equation finderEdge equation finder
0
4590
(x1,y1)
Angle in degree Frequency
0 1
45 3
90 1
135
135 1
Desired linear equation in point-slope form:
)1(*45tan1 xxyy
Equation processorEquation processor
After finding out the equation, constraints After finding out the equation, constraints can be applied in order to remove can be applied in order to remove redundant equations, get shadows or redundant equations, get shadows or detect occultation.detect occultation.
Find out the corner points for translation Find out the corner points for translation detector and texture mapper.detector and texture mapper.
Translation detectorTranslation detector
A simple object motion tracker.A simple object motion tracker. Collects the data of the first key frame to Collects the data of the first key frame to
accelerate the process of the remaining accelerate the process of the remaining video frames.video frames.
Can be beneficial if the video segment is Can be beneficial if the video segment is long and the scene seldom changes.long and the scene seldom changes.
Texture MapperTexture Mapper
A graphics design process in which a 2-D A graphics design process in which a 2-D surface, called a surface, called a texture maptexture map, is "wrapped , is "wrapped around" a 3-D object. around" a 3-D object.
The 3-D object acquires a surface texture The 3-D object acquires a surface texture similar to the texture map.similar to the texture map.
Texture MapperTexture Mapper
Mapping
Original position of pixel
New position of pixel
Texture MapperTexture Mapper
Every polygon is assigned 2 sets of Every polygon is assigned 2 sets of coordinatescoordinates
Image coordinates Image coordinates (r, c)(r, c) : location of pixel : location of pixel in the imagein the image
Texture coordinates Texture coordinates (u, v)(u, v) : location in : location in texture image which contains color texture image which contains color information for image coordinatesinformation for image coordinates
Texture MapperTexture Mapper
Mapping functionsMapping functions map texture map texture coordinates to image coordinates or vice coordinates to image coordinates or vice versa.versa.
They are usually determined by image They are usually determined by image points whose texture coordinates are points whose texture coordinates are given explicitly.given explicitly.
Texture MapperTexture Mapper
(r1, c1)
(u1, v1)
(r2, c2)
(u2, v2)
(r4, c4)
(u4, v4)
(r3, c3)
(u3, v3)
(u1, v1) (u2, v2)
(u3, v3)(u4, v4)
Texture MapperTexture Mapper
Scan conversionScan conversion: the process of scanning : the process of scanning all the pixels and perform the necessary all the pixels and perform the necessary calculation.calculation.
Forward mappingForward mapping maps from the texture maps from the texture space to image spacespace to image space
Inverse mapping Inverse mapping maps from the image maps from the image space to texture spacespace to texture space
Scan conversion with forward Scan conversion with forward mappingmapping
Algorithm: Algorithm: for u = umin to umaxfor u = umin to umax for v = vmin to vmixfor v = vmin to vmix r = R(u,v)r = R(u,v) c = C(u,v)c = C(u,v) copy pixel at source (u,v) copy pixel at source (u,v) to destination (r,c)to destination (r,c)
Scan conversion with forward Scan conversion with forward mappingmapping
Advantage:Advantage:Easy to compute as long as the forward Easy to compute as long as the forward mapping function is known.mapping function is known.
DisadvantageDisadvantagePixel-to-pixel mapping is not 1-1.Pixel-to-pixel mapping is not 1-1.Holes may appear.Holes may appear.Can result in aliasing.Can result in aliasing.
Scan conversion with forward Scan conversion with forward mappingmapping
Scan conversion with inverse Scan conversion with inverse mappingmapping
Algorithm: Algorithm: for (r,c) = polygon pixelfor (r,c) = polygon pixel u = TEXR(r,c)u = TEXR(r,c) v = TEXC(r,c)v = TEXC(r,c) copy pixel at source (u,v) copy pixel at source (u,v) to destination (r,c)to destination (r,c)
Scan conversion with inverse Scan conversion with inverse mappingmapping
Advantage: Advantage:
Every destination pixel is filled (no holes).Every destination pixel is filled (no holes).
Allow easy incorporation of pre-filtering & resampling Allow easy incorporation of pre-filtering & resampling operations to prevent aliasingoperations to prevent aliasing
Scan conversion with inverse Scan conversion with inverse mappingmapping
Take advantage of Take advantage of Scanline Polygon Fill AlgorScanline Polygon Fill Algorithmithm
For a row scan, maintain a list of scanline / polFor a row scan, maintain a list of scanline / polygon intersections.ygon intersections.
Intersection at scanline r+1 efficiently computeIntersection at scanline r+1 efficiently computed from row r.d from row r.
Scanline yk+1
Scanline yk{xk, yk}
{xk+1, yk+1}
Scan conversion with inverse Scan conversion with inverse mappingmapping
Coordinates at a non-boundary level are compCoordinates at a non-boundary level are computed by linearly interpolating (u,v) coordinates uted by linearly interpolating (u,v) coordinates of bounding pixels on the scanline.of bounding pixels on the scanline.
Scanline yk+1
Scanline yk{xk, yk}
{xk+1, yk+1}
Scan conversion with inverse Scan conversion with inverse mappingmapping
Suppose (ri,ci) maps to (ui,vi), i = 1,…, 5Suppose (ri,ci) maps to (ui,vi), i = 1,…, 5 (r4,c4) = s (r1,c1) + (1-s) (r3,c3) { s is known }(r4,c4) = s (r1,c1) + (1-s) (r3,c3) { s is known } (u4,v4) = s(u1,v1) + (1-s)(u3,v3) {u4,v4 are known}(u4,v4) = s(u1,v1) + (1-s)(u3,v3) {u4,v4 are known} Similarly, (u5, v5) can be found.Similarly, (u5, v5) can be found. t = (c-c4)/(c5-c4)t = (c-c4)/(c5-c4) (r,c) = t*(u5,v5) + (1-t)*(u4,v4)(r,c) = t*(u5,v5) + (1-t)*(u4,v4)
Scanline yk(r4, c4)
(r1, c1)
(r3, c3) (r2, c2)
(r5, c5)
(r, c)
image
Basic 2D linear mappingBasic 2D linear mapping
Scaling & TranslationScaling & Translationu = ar + du = ar + dv = bc + ev = bc + eupright rectangle upright rectangle uprig upright squareht square
Euclidean mappingEuclidean mappingu = (cosu = (cos)r – (sin)r – (sin)c + d)c + dv = (sinv = (sin)r + (cos)r + (cos)c + e)c + erotated unit square rotated unit square upr upright squareight square
Basic 2D linear mappingBasic 2D linear mapping
Similarity mappingSimilarity mappingu = s(cosu = s(cos)r – s(sin)r – s(sin)c + d)c + dv = s(sinv = s(sin))r + s(cosr + s(cos)c + )c + eerotated square rotated square upright u upright unit squarenit square
Affine mappingAffine mappingu = f(cosu = f(cos)r – g(sin)r – g(sin)c + d)c + dv = h(sinv = h(sin)r + i(cos)r + i(cos)c + e)c + erotated rectangle rotated rectangle uprigh upright unit squaret unit square
DEMO !
Basic 2D linear mappingBasic 2D linear mapping
Projective mappingProjective mappingThe most general 2D linear mapThe most general 2D linear mapSquare Square arbitrary quadrangle ! arbitrary quadrangle !
u = (a11r+a12c+a13) / (a31r+a32c+1)u = (a11r+a12c+a13) / (a31r+a32c+1) v = (a21r+a22c+a23) / (a31r+a32c+1)v = (a21r+a22c+a23) / (a31r+a32c+1)
The 8 variables a11,a12, … , a32 have to be The 8 variables a11,a12, … , a32 have to be found out.found out.
Basic 2D linear mappingBasic 2D linear mapping
We have a system of 8 equations solving 8 We have a system of 8 equations solving 8 unknowns.unknowns.
(x1,y1)
Future WorkFuture Work
Mapping cansMapping cans Speed optimizationSpeed optimization Movie manipulationMovie manipulation Use of 3D markersUse of 3D markers
Q & AQ & A
See the foot notes.