![Page 1: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocument.in/reader035/viewer/2022062816/56815980550346895dc6be3f/html5/thumbnails/1.jpg)
Video Object Tracking Video Object Tracking and Replacement for and Replacement for
Post TV ProductionPost TV Production
LYU0303 Final Year ProjectLYU0303 Final Year ProjectFall 2003Fall 2003
![Page 2: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocument.in/reader035/viewer/2022062816/56815980550346895dc6be3f/html5/thumbnails/2.jpg)
OutlineOutline
Project IntroductionProject Introduction Basic parts of the purposed systemBasic parts of the purposed system Working principles of individual partsWorking principles of individual parts Future WorkFuture Work Q&AQ&A
![Page 3: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocument.in/reader035/viewer/2022062816/56815980550346895dc6be3f/html5/thumbnails/3.jpg)
IntroductionIntroduction
Post-TV production software changes the Post-TV production software changes the content of the original video clips.content of the original video clips.
Extensively used in video-making Extensively used in video-making industries.industries.
Why changing the content of a video?Why changing the content of a video? Reducing video production costReducing video production cost Performing dangerous actionsPerforming dangerous actions Producing effects those are impossible in realityProducing effects those are impossible in reality
![Page 4: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocument.in/reader035/viewer/2022062816/56815980550346895dc6be3f/html5/thumbnails/4.jpg)
Difficulties to be overcomeDifficulties to be overcome Things in video can be treated individually Things in video can be treated individually
called “video objects”.called “video objects”. Computers cannot perform object detection Computers cannot perform object detection
directly because…directly because… Image is processed byte-by-byteImage is processed byte-by-byte Without prior knowledge about the video objects to be detectedWithout prior knowledge about the video objects to be detected Result is definite, no fuzzy logic.Result is definite, no fuzzy logic.
Though computers cannot perform object Though computers cannot perform object detection directly, it can be programmed to detection directly, it can be programmed to work indirectly.work indirectly.
![Page 5: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocument.in/reader035/viewer/2022062816/56815980550346895dc6be3f/html5/thumbnails/5.jpg)
Basic parts of the purposed systemBasic parts of the purposed system
Simple bitmap reader/writerSimple bitmap reader/writer RGB/HSV converterRGB/HSV converter Edge detector with smootherEdge detector with smoother Edge equation finderEdge equation finder Equation processorEquation processor Translation detectorTranslation detector Texture mapperTexture mapper
![Page 6: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocument.in/reader035/viewer/2022062816/56815980550346895dc6be3f/html5/thumbnails/6.jpg)
RGB/HSV converterRGB/HSV converter
Human eyes are more sensitive to the Human eyes are more sensitive to the brightness rather than the true color brightness rather than the true color components of an object.components of an object.
More reasonable to convert the More reasonable to convert the representation of colors into HSV (Hue, representation of colors into HSV (Hue, Saturation and Value (brightness)) model.Saturation and Value (brightness)) model.
After processing, convert back to RGB and After processing, convert back to RGB and save to disk.save to disk.
![Page 7: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocument.in/reader035/viewer/2022062816/56815980550346895dc6be3f/html5/thumbnails/7.jpg)
RGB/HSV converterRGB/HSV converter
RGB to HSVRGB to HSV HSV to RGBHSV to RGB
![Page 8: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocument.in/reader035/viewer/2022062816/56815980550346895dc6be3f/html5/thumbnails/8.jpg)
Basic parts of the purposed systemBasic parts of the purposed system
Simple bitmap reader/writerSimple bitmap reader/writer RGB/HSV converterRGB/HSV converter Edge detector with smootherEdge detector with smoother Edge equation finderEdge equation finder Equation processorEquation processor Translation detectorTranslation detector Texture mapperTexture mapper
![Page 9: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocument.in/reader035/viewer/2022062816/56815980550346895dc6be3f/html5/thumbnails/9.jpg)
Edge detectorEdge detector Usually, a sharp change in hue, saturation or Usually, a sharp change in hue, saturation or
brightness means that there exist a boundary brightness means that there exist a boundary line.line.
HSV: (0,0,0)
HSV: (0,255,255)
![Page 10: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocument.in/reader035/viewer/2022062816/56815980550346895dc6be3f/html5/thumbnails/10.jpg)
Edge detectorEdge detectorBefore edge highlightingAfter edge highlighting
![Page 11: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocument.in/reader035/viewer/2022062816/56815980550346895dc6be3f/html5/thumbnails/11.jpg)
Smoother for Edge detectorSmoother for Edge detector
Sometimes noise will affect the edge Sometimes noise will affect the edge detection result of low resolution images.detection result of low resolution images.
Include an image smoother to remove Include an image smoother to remove large noise points in the image.large noise points in the image.
In some cases performing a smoothing will In some cases performing a smoothing will greatly enhance the performance of edge greatly enhance the performance of edge detection due to the decrease in “fake” detection due to the decrease in “fake” edge points.edge points.
![Page 12: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocument.in/reader035/viewer/2022062816/56815980550346895dc6be3f/html5/thumbnails/12.jpg)
Smoother for Edge detectorSmoother for Edge detector
![Page 13: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocument.in/reader035/viewer/2022062816/56815980550346895dc6be3f/html5/thumbnails/13.jpg)
Basic parts of the purposed systemBasic parts of the purposed system
Simple bitmap reader/writerSimple bitmap reader/writer RGB/HSV converterRGB/HSV converter Edge detector with smootherEdge detector with smoother Edge equation finderEdge equation finder Equation processorEquation processor Translation detectorTranslation detector Texture mapperTexture mapper
![Page 14: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocument.in/reader035/viewer/2022062816/56815980550346895dc6be3f/html5/thumbnails/14.jpg)
Edge equation finderEdge equation finder
Derives mathematical facts out of the edge Derives mathematical facts out of the edge points.points.
Works with voting algorithm of Hough Works with voting algorithm of Hough Transform.Transform.
Automatically adjusts tolerance value to Automatically adjusts tolerance value to minimize the effect of noise points.minimize the effect of noise points.
This helps when the edge is not completely This helps when the edge is not completely straight or blurred.straight or blurred.
![Page 15: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocument.in/reader035/viewer/2022062816/56815980550346895dc6be3f/html5/thumbnails/15.jpg)
Edge equation finderEdge equation finder
0
4590
(x1,y1)
Angle in degree Frequency0 1
45 3
90 1
135
135 1
Desired linear equation in point-slope form:
)1(*45tan1 xxyy
![Page 16: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocument.in/reader035/viewer/2022062816/56815980550346895dc6be3f/html5/thumbnails/16.jpg)
Edge equation finderEdge equation finder
![Page 17: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocument.in/reader035/viewer/2022062816/56815980550346895dc6be3f/html5/thumbnails/17.jpg)
Basic parts of the purposed systemBasic parts of the purposed system
Simple bitmap reader/writerSimple bitmap reader/writer RGB/HSV converterRGB/HSV converter Edge detector with smootherEdge detector with smoother Edge equation finderEdge equation finder Equation processorEquation processor Translation detectorTranslation detector Texture mapperTexture mapper
![Page 18: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocument.in/reader035/viewer/2022062816/56815980550346895dc6be3f/html5/thumbnails/18.jpg)
Equation processorEquation processor
After finding out the equation, constraints After finding out the equation, constraints can be applied in order to remove can be applied in order to remove redundant equations, get shadows or redundant equations, get shadows or detect occultation.detect occultation.
Find out the corner points for translation Find out the corner points for translation detector and texture mapper.detector and texture mapper.
![Page 19: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocument.in/reader035/viewer/2022062816/56815980550346895dc6be3f/html5/thumbnails/19.jpg)
Equation processorEquation processorBefore edge findingAfter edge and equation findingAfter extra equation removal
![Page 20: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocument.in/reader035/viewer/2022062816/56815980550346895dc6be3f/html5/thumbnails/20.jpg)
Equation processorEquation processor The following criteria are currently adopted The following criteria are currently adopted
in the equation processor:in the equation processor: Distance between the equationsDistance between the equations Angle between the equationsAngle between the equations Whether the equation intersects the object or Whether the equation intersects the object or
not.not. Since equation processor is a potential Since equation processor is a potential
burden to the system, it may be replaced burden to the system, it may be replaced in the future by improving other parts.in the future by improving other parts.
![Page 21: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocument.in/reader035/viewer/2022062816/56815980550346895dc6be3f/html5/thumbnails/21.jpg)
Basic parts of the purposed systemBasic parts of the purposed system
Simple bitmap reader/writerSimple bitmap reader/writer RGB/HSV converterRGB/HSV converter Edge detector with smootherEdge detector with smoother Edge equation finderEdge equation finder Equation processorEquation processor Translation detectorTranslation detector Texture mapperTexture mapper
![Page 22: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocument.in/reader035/viewer/2022062816/56815980550346895dc6be3f/html5/thumbnails/22.jpg)
Translation detectorTranslation detector
A simple object motion tracker.A simple object motion tracker. Collects the data of the first key frame to Collects the data of the first key frame to
accelerate the process of the remaining accelerate the process of the remaining video frames.video frames.
Can be beneficial if the video segment is Can be beneficial if the video segment is long and the scene seldom changes.long and the scene seldom changes.
![Page 23: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocument.in/reader035/viewer/2022062816/56815980550346895dc6be3f/html5/thumbnails/23.jpg)
Translation detectorTranslation detector
Records the approximate location of the Records the approximate location of the object in the key frame.object in the key frame.
When processing the following video When processing the following video frames, just scan within a certain boundary frames, just scan within a certain boundary near the recorded location.near the recorded location.
Will improve its efficiency later.Will improve its efficiency later.
![Page 24: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocument.in/reader035/viewer/2022062816/56815980550346895dc6be3f/html5/thumbnails/24.jpg)
Basic parts of the purposed systemBasic parts of the purposed system
Simple bitmap reader/writerSimple bitmap reader/writer RGB/HSV converterRGB/HSV converter Edge detectorEdge detector Edge equation finderEdge equation finder Equation processorEquation processor Texture mapperTexture mapper
![Page 25: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocument.in/reader035/viewer/2022062816/56815980550346895dc6be3f/html5/thumbnails/25.jpg)
Texture MapperTexture Mapper
A graphics design technique used to wrap A graphics design technique used to wrap a surface of a 3-D object with a a surface of a 3-D object with a texture texture mapmap
The 3-D object acquires a The 3-D object acquires a surface texturesurface texture similar to the texture map.similar to the texture map.
ColorsColors, , brightnessbrightness valuesvalues or or altitudesaltitudes
![Page 26: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocument.in/reader035/viewer/2022062816/56815980550346895dc6be3f/html5/thumbnails/26.jpg)
Texture MapperTexture Mapper
Mapping
(color)
Texture map Image
![Page 27: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocument.in/reader035/viewer/2022062816/56815980550346895dc6be3f/html5/thumbnails/27.jpg)
Texture MapperTexture Mapper
Definition of terms:Definition of terms: Image coordinates Image coordinates (r, c)(r, c) : location of pixel in : location of pixel in
the the imageimage Texture coordinates Texture coordinates (u, v)(u, v) : location in : location in texture texture
mapmap which contains color information for which contains color information for image coordinatesimage coordinates
Mapping functionMapping function : determines : determines howhow texture texture coordinates are mapped to image coordinates coordinates are mapped to image coordinates or vice versa. or vice versa.
e.g. linear scan-line interpolatione.g. linear scan-line interpolation
![Page 28: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocument.in/reader035/viewer/2022062816/56815980550346895dc6be3f/html5/thumbnails/28.jpg)
Texture MapperTexture Mapper
Mapping
function
(u,v)(r,c)
Texture coordinates
Image coordinates
![Page 29: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocument.in/reader035/viewer/2022062816/56815980550346895dc6be3f/html5/thumbnails/29.jpg)
Texture MapperTexture Mapper
Definition of termsDefinition of terms Forward mappingForward mapping maps from the maps from the texture texture
spacespace to to image spaceimage space Inverse mapping Inverse mapping maps from the maps from the image spaceimage space
to to texture spacetexture space Scan-line conversionScan-line conversion: an area-filling technique : an area-filling technique
processing a surface processing a surface line by lineline by line. It can be . It can be applied with forward/inverse mapping.applied with forward/inverse mapping.
![Page 30: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocument.in/reader035/viewer/2022062816/56815980550346895dc6be3f/html5/thumbnails/30.jpg)
Texture MapperTexture Mapper
Forward mapping
Inverse mapping
![Page 31: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocument.in/reader035/viewer/2022062816/56815980550346895dc6be3f/html5/thumbnails/31.jpg)
Texture MapperTexture Mapper
Texture scanning
Image scanning
More important
Scan-line conversion
![Page 32: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocument.in/reader035/viewer/2022062816/56815980550346895dc6be3f/html5/thumbnails/32.jpg)
triangletriangle//parallelogramparallelogram scanning scanning line by lineline by line, from top to bottom, from top to bottom process process eacheach pixel on every line pixel on every line
Scan-line conversionScan-line conversion
Scanline yk
Scanline yk+1
scanning order
for every line
![Page 33: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocument.in/reader035/viewer/2022062816/56815980550346895dc6be3f/html5/thumbnails/33.jpg)
Scan-line conversionScan-line conversion
For a row scan, maintain a list of scanline / polFor a row scan, maintain a list of scanline / polygon intersections.ygon intersections.
Intersection at scanline r+1 efficiently computeIntersection at scanline r+1 efficiently computed from scanline r.d from scanline r.
Scanline yk
Scanline yk+1
Coordinates differ by Δx and Δy
![Page 34: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocument.in/reader035/viewer/2022062816/56815980550346895dc6be3f/html5/thumbnails/34.jpg)
Scan-line conversionScan-line conversion
quadrilateral quadrilateral trianglestriangles or or parallelogramsparallelograms scan each scan each sub-polygonsub-polygon special case: only special case: only 22 sub-polygons sub-polygons
12
3
![Page 35: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocument.in/reader035/viewer/2022062816/56815980550346895dc6be3f/html5/thumbnails/35.jpg)
Scan-line conversion with forward Scan-line conversion with forward mappingmapping
AlgorithmAlgorithm
texture scanning
forward mapping functions
r = R(u,v)r = R(u,v)c = C(u,v)c = C(u,v)
copy pixel at source (u,v) copy pixel at source (u,v) to destination (r,c)to destination (r,c)
for u = umin to umaxfor u = umin to umax for v = vmin to vmixfor v = vmin to vmix
![Page 36: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocument.in/reader035/viewer/2022062816/56815980550346895dc6be3f/html5/thumbnails/36.jpg)
Scan-line conversion with inverse Scan-line conversion with inverse mappingmapping
texture scanning
inverse mapping functions
u = U(r,c)u = U(r,c)v = V(r,c)v = V(r,c)
copy pixel at source (u,v) copy pixel at source (u,v) to destination (r,c)to destination (r,c)
for (r,c) = polygon pixelfor (r,c) = polygon pixel
AlgorithmAlgorithm
![Page 37: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocument.in/reader035/viewer/2022062816/56815980550346895dc6be3f/html5/thumbnails/37.jpg)
ComparisonComparison
Possibility ofPossibility of aliasing aliasing
NoNoYesYesCalculation Calculation ofof
fractional area of pixel fractional area of pixel coveragecoverage
a bita bit complicated complicated as it as it involves image scanninginvolves image scanning
easyeasy if the mapping if the mapping function is knownfunction is knownEase Ease of implementationof implementation
image image texture texturetexture texture image imagePrinciplePrinciple
Inverse Inverse mappingmapping
Forward Forward mappingmapping
![Page 38: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocument.in/reader035/viewer/2022062816/56815980550346895dc6be3f/html5/thumbnails/38.jpg)
ComparisonComparison
Aliasing – filtering/resampling techniques can be applied for inverse mapping
![Page 39: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocument.in/reader035/viewer/2022062816/56815980550346895dc6be3f/html5/thumbnails/39.jpg)
ComparisonComparison
Yes, but can be avoided Yes, but can be avoided with simple filtering/resawith simple filtering/resa
mplingmplingYesYesPossibility ofPossibility of aliasing aliasing
NoNoYesYesCalculation Calculation ofof
fractional area of pixel fractional area of pixel coveragecoverage
a bita bit complicated complicated as it as it involves image scanninginvolves image scanning
easyeasy if the mapping if the mapping function is knownfunction is knownEase Ease of implementationof implementation
image image texture texturetexture texture image imagePrinciplePrinciple
Inverse Inverse mappingmapping
Forward Forward mappingmapping
We used inverse mapping
![Page 40: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocument.in/reader035/viewer/2022062816/56815980550346895dc6be3f/html5/thumbnails/40.jpg)
Mapping functionsMapping functions
Simple linear transformationsSimple linear transformations Translation, scaling, etcTranslation, scaling, etc parallelogramsparallelograms only only very fastvery fast
Linear scan-line interpolationLinear scan-line interpolation Based on Based on proportionproportion any any quadrilateralsquadrilaterals
not suitable
feasible
![Page 41: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocument.in/reader035/viewer/2022062816/56815980550346895dc6be3f/html5/thumbnails/41.jpg)
Linear scan-line interpolationLinear scan-line interpolation
Idea of Idea of interpolationinterpolation
(x1, y1)
(x2, y2)
a
1-a
(x, y) = a(x2, y2) + (1-a)(x1, y1)
![Page 42: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocument.in/reader035/viewer/2022062816/56815980550346895dc6be3f/html5/thumbnails/42.jpg)
Linear scan-line interpolationLinear scan-line interpolation
(r4, c4)
(r1, c1)
(r3, c3) (r2, c2)
(r5, c5)
(r, c)
image
(u1, v1)(u3, v3)
texture map
(u2, v2)
(u4, v4)
(u5, v5)
For a particular scanline,
(u, v)
![Page 43: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocument.in/reader035/viewer/2022062816/56815980550346895dc6be3f/html5/thumbnails/43.jpg)
After mappingAfter mapping
a bit strange !
![Page 44: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocument.in/reader035/viewer/2022062816/56815980550346895dc6be3f/html5/thumbnails/44.jpg)
Shadow mappingShadow mapping Mapping of surface Mapping of surface brightnessbrightness Retain the brightness of the Retain the brightness of the original original
surfacesurface
Method:Method: Convert the original surface Convert the original surface HSVHSV Get the Get the VV value value Replace the Replace the VV value of the mapped surface value of the mapped surface
![Page 45: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocument.in/reader035/viewer/2022062816/56815980550346895dc6be3f/html5/thumbnails/45.jpg)
After Shadow mappingAfter Shadow mapping
More natural
!
![Page 46: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocument.in/reader035/viewer/2022062816/56815980550346895dc6be3f/html5/thumbnails/46.jpg)
Future WorkFuture Work
Anti-aliasingAnti-aliasing Mapping different shapes like cansMapping different shapes like cans Speed optimizationSpeed optimization Movie manipulationMovie manipulation Use of 3D markersUse of 3D markers
![Page 47: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocument.in/reader035/viewer/2022062816/56815980550346895dc6be3f/html5/thumbnails/47.jpg)
Q & AQ & A