computer and machine vision - electrical, computer & energy...
Post on 23-May-2020
3 Views
Preview:
TRANSCRIPT
March 28, 2014 Sam Siewert
Computer and Machine
Vision
Lecture Week 11
Part-2
Segmentation, Camera
Calibration and Feature Alignment
Outline of Week 11 Exam #1 Results Overview and Solutions
Wrap up of SIFT and David Lowe’s Paper
– Primitive Operations to Accelerate SIFT
– Uses of SIFT and Comparison to Hough and
RANSAC
Introduction to Segmentation, Camera
Calibration and Feature Alignment
Sam Siewert 2
Recall Image Pyramid from
Chapter 3 in CV Page 133 – A fundamental for scale invariance (SIFT search for keypoint candidates in different scales)
Resolution Decimation – Reduction in Resolution, or downscaling
Resolution Interpolation – Increased Resolution, or upscaling (Can’t create data where there was none in original, but we can interpolate)
OpenCV Image Pyramids – Low Pass Filtering (Gaussian kernel convolution) followed by pixel decimation (removal of odd or even numbered rows and columns)
Sam Siewert 3
L=0, e.g. 9x9
L=1, 5x5
L=2, 3x3
Rows 0…8, Col 0…8
Drop rows 1,3,5,7 to
go to L=1 5x5
Drop rows 1,3 to go to
L=2 3x3
Feature Vector Concept General Concept - Summarize Data with Fewer Data points
E.g. an Intensity Histogram, Feature Vector for a Color Image – If Histogram(Image_A) = Histogram(Image_B), Image_A = Image_B more likely
– Much like a Fingerprint for Human Identification (not definitive, but narrows search)
– The Intensity Histogram is Not a Robust Feature Vector (Lighting Sensitivity)
– Increasing Dimensions, Normalization, Scale, Orientation and Quality Improves
Sam Siewert 4
R
G
B
R
G
B
Image Gradient Concept
Sam Siewert 5
Example Feature Vectors
Any N dimensional vector that represents and object
A form of compression for the reference database and faster compare
Vector and Distance Metric Definitions are Key, But Once Defined, Simple Distance Measures Can Be Applied
– Corners of Objects
– Radial Distance Profile (Centroidal)
– Various Histograms Derived from Scans with Binning (Blobs)
Sam Siewert 6
Feature Vectors
Unique Patterns That Define Objects of Interest
How to Generalize Keypoints of Feature Vectors is the Challenge
Bio-inspired (Human Observation) is A Promising Approach, but Search is Costly!
Operation in Real-Time?
PCA – Principal Component Analysis
Sam Siewert 7
http://en.wikipedia.org/wiki/File:GaussianScatterPCA.png
http://en.wikipedia.org/wiki/File:Sift_keypoints_filtering.jpg
N Dimensional Distance Harder to Visualize than 3D Vector Differences, but Same Concept as Euclidian
Quick Review of Euclidian – One Dimension: abs(p1 – p2), or Sqrt((p1 – p2)
2)
– Two Dimension: Sqrt((p1-q1)2 + (p2 – q2)
2)
– Three Dimension: Sqrt((p1-q1)2 + (p2 – q2)
2 + (p3-q3)
2)
– N Dimension: Sqrt((p1-q1)2 + (p2 – q2)
2 + … + (pn-qn)
2)
Sam Siewert 8
Other Distance Measures … Defines the Difference Between {A} and {B} in General with a Metric (Number)
Euclidian Distance is One Such Metric
Manhattan Distance is Another (Sum of Orthogonal Line Segments on a Grid Between 2 Points)
Hamming Distance is Another - # of Positions at Which 2 Equal Length Strings Have Different Symbols – E.g. 1111_0000 and 0000_1111 have maximum distance or “toned” and “roses” has distance of 3 in ASCII
Sam Siewert 9
Final Words on SIFT Re-Read David Lowe’s Paper
1. Scale Space Extrema Detection - Find Best Candidate Image
Locations to Form Keypoint feature vectors (scale invariant through search in Gaussian pyramid)
2. Keypoint Localization - Filter Out Keypoints (to remove minimum contrast and noisy edge curvature feature vectors)
3. Keypoint Orientation – So Descriptors are relative to orientation of the keypoint (to provide orientation invariance) and for magnitude to select proper level in Gaussian image
4. Keypoint Descriptor – Representation in database is a feature vector with magnitude, orientation for nxn sub-regions with vectors of m dimensions (e.g. Figure 7 2x2 with 8x8 feature vector)
Notes from the Paper on the overall Algorithm follow …
Sam Siewert 10
SIFT Interesting Aspects Feature Selection with Mathematical Significance Calculations
Scale, Orientation and Lighting Invariance Goals
Keypoints are Complex Feature Vectors to Use for Match
Bio-Inspired (see page 14, local image descriptor) – Edelman, Intrator and Poggio on Neurons for 3D recognition (gradient at orientation with spatial frequency – key trigger in visual cortex)
Sam Siewert 11
Image Correspondence and
Registration Historically Features as Simple as
Intensity and Color Vectors in an n x n ROI
Have Been Used to Register Images
Simpler Feature Vectors Work for Left-
Eye, Right-Eye Images (Simple Baseline
Separation of Known Distance)
Still Complicated by Camera Calibration
Sam Siewert 12
Hough and RANSAC Recall Hough – Idea is to Fit Threshold Points to Lines,
Circles/Elipses, Shape (RANSAC fits to Lines)
– Feature Matching as Well
SIFT Results in Many More Keypoint feature vectors compared to Hough
Advantage for Correspondence and Recognition (Like more Detailed Fingerprint)
Likewise, Sometimes a Disadvantage – Large Search for Match Detection
Sam Siewert 13
Alternatives in Machine
Vision
Shape and Pattern
Recognition Alternatives to
Hough and SIFT
Sam Siewert
14
Centroidal Profile Find Centroid Based on Threshold and Pixel COM (As Discussed Before)
Define Shape In Terms of Radius as a Function of Angle Around Centroid (Shape Signature)
Sam Siewert 15
From Computer and
Machine Vision, E.R.
Davies, page 271.
Centroidal Profile Problems Notched or Partially Occluded Objects
Computationally Complex
Must Have Accurate Centroid First
Suggested for Inspection Applications (Within Shape Tolerances?)
Sam Siewert 16
From Computer and
Machine Vision, E.R.
Davies, page 272.
Segmentation, Camera
Calibration and Feature
Alignment
Sam Siewert
17
Segmentation (CV Chapter 5) Foreground and Background – E.g. Background Elimination (Simplify to 2 regions over time)
– Simple Threshold (Simplify to 2 regions by intensity)
– Edge Detection to Define Contour Boundaries (Gradient separation)
Clustering Methods – Use Distance Measures to Assign Pixels to a Center Value (could be
random color or intensity) – E.g. OpenCV K-means Clustering
– Divide Image of N pixels into K clusters with Each Belonging to Cluster with Nearest Mean
Watershed – Gradient Lines (Like Contour Map) Define Regions, Fill Between them
Sam Siewert 18
http://en.wikipedia.org/wiki/K-means_clustering
Segmentation Use
Divides Image Logically Rather than Arbitrary Selection or ROI (Region of Interest) as a Grid Does
Localized Feature Vectors in Each ROI
Classification and Object Recognition Step
Defines Features Rather than Looking for Feature Matches (E.g. Lines in Image like Hough)
Sam Siewert 19
Cameras and Calibration
Basics for 2D and
Considerations for 3D
Mapping and Ranging
Sam Siewert
20
Camera Calibration
Intrinsics – Features of the Camera Itself – Optical Characteristics (Fish
Eye – wider in center, Hourglass – thinner in center)
– Detector Characteristics – Pixels with Value at Zero Illumination (Christmas Light) or Pixels with Low Value, Pixel Type (Bayer Filter, or Other)
Extrinsics – Camera Mount, Thermal Issues, Vibration, Orientation and Viewpoint Sam Siewert 21
http://en.wikipedia.org/wiki/Bayer_filter
CV Ch. 6 & Learning OpenCV Ch. 11
Read to Understand Basic Concepts – Coordinate Transformation Between Pixel and Real
World Coordinates E.g. Tilt, Pan, Zoom of Camera and Location in X, Y, Z of target of Interest in Scene
Why is this an issue and why is calibration required? – Camera optical distortions
– Lighting conditions
– Thermal and Mechanical Perturbations (change in camera mount over time)
– Detector Manufacturing Imperfection (Same Camera Model, Different Image from Same Scene)
– Radial Distortions
Practical Methods Provided in Learning OpenCV and Here
Sam Siewert 22
Sam Siewert 23
Camera Mount Basic Concepts Single Camera Tilt/Pan Object Tracking (Common Configuraiton)
– 2 Axes of Rotation (Others are Fixed)
– +/- 45 deg Tilt Rotation Servo (typical, could be up to 90 deg)
– +/- 45 deg Pan Rotation Servo (typical, could be up to 180 deg)
– Side or Rear Mounting of Tilt Servo
Ideally No Vibration or Rotation, Just Commanded Tilt/Pan
Sam Siewert 24
Dual Camera Tilt/Pan Tracking Baseline with 2 Fixed Cameras (for Stereo Ranging)
– Pan Servo Pans the Entire Baseline
– Tilt Servo Tilts Pan Servo
– Side Mounting Plate
Ideally Baseline Separation is Constant (no bending,
twisting of the beam)
Sam Siewert 25
Pixel Coordinates Define Image Coordinates to Track Object Centroid
Tilt/Pan With Servos to Keep Target Centroid in FOV Center
E.g. ½ Resolution 320x240 AR=4:3 Image from Analog Camera
Reference Target
Pixel Address
160 , 120
NTSC 320 Pixel Column
Format
N T
S C
2 4
0 P
i x e
l R
o w
F o
r m a t
Frame Origin
Pixel Address 0 , 0
X
Y
Sam Siewert 26
Stereo Ranging with Common Tilt/Pan Fixed
Camera Baseline
Sam Siewert 27
Pixel to Servo Calibration at Distance
Pan 1 Servo Increment and Find Centroid X Pixel Change
Tilt 1 Servo Increment and Find Centroid Y Pixel Change
Reference Target
Pixel Address
160 , 120
Tilt One Servo
Increment
Pan One Servo
Increment
NTSC 320 Pixel Column
Format
N T
S C
2 4 0
P i x
e l R
o w
F o
r m a
t
Tilt Shifted Frame
Pan Shifted
Frame
Pan Shifted
Target Pixel
Address 158 , 120
Tilt Shifted Target
Pixel Address
160 , 122
Sam Siewert 28
Characterize Camera FOV at Distances
FOV Width/Height Linear as a Function of Distance
Pixels/Inch Not Necessarily Linear as Function of Distance
Use to Calibrate Servo Step Size (Gain) for Target Centering
– For Servo Step Sizes of 1 Increment in Tilt/Pan, Camera Will Track Slowly
– For Larger Servo Step Sizes May Over-shoot
– Determine Dead-bands (Servo Limits for Pixel Change Accuracy)
Calibrate with Reference Image
at Known Distance Generalize and Expand Auto-Calibration Sequence as Described for N x M known Targets
Most Common is the Chess/Checker Board
OpenCV Support for Calibration Using Chess Board
Sam Siewert 29
Sam Siewert 30
Finding Centroid In Image Target Known by Color, Shape, or Brightness
Raster to Find Target Edges and Max Width and Height Scanline for Symmetric targets
Mark Centroid in Image for Easy Debug
Noise in Image Will Cause Centroid Error – Use Averaging
Pan Right to Move Target Left in Image
Tilt Up to Move Target Down in Image
Sam Siewert 31
Tips for Practical Real-time Calibration Consider Controlled Lighting or For Mobile Robots, On-Board Lighting with LEDs (Control your Own Lighting) Use Frame Grabber ADC Sensitivity Settings to Control Brightness Consider Automatic Calibration Sequences – Place Reference Targets to Set Pixel Step Size as a Function of Servo
Step Size – Use Stereo Range Estimation to Determine Distance to Target and Set
Tracking Gains for Current Distance
When Target is Lost, Go Into Search Mode – Start from Max Tilt/Pan and Raster to Min Tilt/Pan to Find Target – For Search Modes Use Coarser Step Size
Consider Small Servo Step Size, But High Frame Rate and Servo Command Rate Be Careful of Processor Overload with Image Processing at 30 FPS
Feature Alignment
Mechanical vs. Digital Image
Processing
Sam Siewert
32
Mosaics by Scan Mirror and
Gimbal Tilt, Pan, Rotate Gimbal to Stabilize Field of View Electro-mechanically (Works, but Expensive) – Steadicam (Automatic Gimbal to Hold FoV)
– CCTV Security Cameras
Scan Mirror in Optical Path to Stabilize Images (Spitzer Space Telescope) – Scan Mirror Moves in Counter-veiling
Direction of Telescope Rotation (Ideally Single Axis)
– Works Because Perturbations in Deep Space are Minimal
Sam Siewert 33
http://en.wikipedia.org/wiki/Steadicam
Digital Stabilization Let Camera Vibrate, Tilt/Pan/Rotate on Non-
Rigid Mounts – Digitally Stabilize
– Find Successive Corresponding Feature
Keypoints in Images Over time
– Re-align Center of Each Image
– Crop Outermost Pixels
Common Use of SIFT and Other Feature
Registration Methods (Hough, RANSAC, etc.)
Sam Siewert 34
Mosaics iOS and Android Panorama Feature – Successive Images are Taken
– Correspondence Found at Keypoints
– Stitch Images Together and Crop Overlaps
Alternative is Tilt/Pan with Gimbal and Scan Mirror – Tilt or Pan at Continuous Known Rate
– Rotate Scan Mirror in Optical Path in Opposite Direction to Stare at Target FoV
– At Known Radial Distance Snap Back to Stabilize Next Image
– Same Concept as Staring at a Fixed Object from a Moving Vehicle
Sam Siewert 35
Stereo Correspondence Use Similar Triangles To Estimate Distance – Right-Eye, Left-Eye Images with Known Intrinsics and Extrinsics
– Keypoints Found in Both are Common Target Centroid
– Planar? Out of Plane Causes Error
– Vanishing Points
Need Numerous Common Targets to Form Passive Depth Map
Sam Siewert 36
Human eye concept from
http://en.wikipedia.org/wiki/Binocular_disparity
The dl, dr, f must be known from intrinsic calibration
The b is known extrinsic, and solve for d
top related