![Page 1: Real-Time Camera Tracking in the “1st & Ten” Systemon-demand.gputechconf.com/gtc/2015/presentation/S5187-Louis-Ge… · Real-Time Camera Tracking in the “1st & Ten” System](https://reader035.vdocument.in/reader035/viewer/2022063014/5fcf2bc02a0329626d252ca9/html5/thumbnails/1.jpg)
Real-Time Camera Tracking in the “1st & Ten” System Louis Gentry and Rand Pendleton
![Page 2: Real-Time Camera Tracking in the “1st & Ten” Systemon-demand.gputechconf.com/gtc/2015/presentation/S5187-Louis-Ge… · Real-Time Camera Tracking in the “1st & Ten” System](https://reader035.vdocument.in/reader035/viewer/2022063014/5fcf2bc02a0329626d252ca9/html5/thumbnails/2.jpg)
• “1st & Ten” is a real-time visual effects system designed to insert virtual graphics in live football broadcasts with proper in-scene perspective.
• Invented by Sportvision in 1998 and debuted on ESPN Sunday Night Football that same year.
• Relies on hardware sensors for accuracy.
What is “1st & Ten”?
1st & Ten, Line of Scrimmage, and Down and Distance
1st & Ten
![Page 3: Real-Time Camera Tracking in the “1st & Ten” Systemon-demand.gputechconf.com/gtc/2015/presentation/S5187-Louis-Ge… · Real-Time Camera Tracking in the “1st & Ten” System](https://reader035.vdocument.in/reader035/viewer/2022063014/5fcf2bc02a0329626d252ca9/html5/thumbnails/3.jpg)
1st and Ten Demo
![Page 4: Real-Time Camera Tracking in the “1st & Ten” Systemon-demand.gputechconf.com/gtc/2015/presentation/S5187-Louis-Ge… · Real-Time Camera Tracking in the “1st & Ten” System](https://reader035.vdocument.in/reader035/viewer/2022063014/5fcf2bc02a0329626d252ca9/html5/thumbnails/4.jpg)
Process for enhancing life video from a locked-down camera.
Camera sensor setup
Camera calibration
Chromakey calibration
Video capture, enhancement,
and output
Sensor System Workflow
![Page 5: Real-Time Camera Tracking in the “1st & Ten” Systemon-demand.gputechconf.com/gtc/2015/presentation/S5187-Louis-Ge… · Real-Time Camera Tracking in the “1st & Ten” System](https://reader035.vdocument.in/reader035/viewer/2022063014/5fcf2bc02a0329626d252ca9/html5/thumbnails/5.jpg)
• Pros
– Once calibrated, it “just works”
– Works even on snow games
– Minimal tuning required
– Not affected by motion blur
• Cons
– Requires fixed camera position
– No reliable hardware-based image stabilization
– Cost of custom electronics
– Must be onsite with the TV production
Sensor System Pros and Cons
Pictured: Assembled, instrumented camera
Pictured: Sportvision instrumentation box
![Page 6: Real-Time Camera Tracking in the “1st & Ten” Systemon-demand.gputechconf.com/gtc/2015/presentation/S5187-Louis-Ge… · Real-Time Camera Tracking in the “1st & Ten” System](https://reader035.vdocument.in/reader035/viewer/2022063014/5fcf2bc02a0329626d252ca9/html5/thumbnails/6.jpg)
• Most movable broadcast cameras have prohibitive constraints:
– Weight
– Data connectivity
• Use an optical solution where sensors aren’t practical
• In 2004, Sportvision pioneered an optical tracking system for enabling “1st & Ten” on football replays.
“1st & Ten” on Movable Cameras?
Pictured: skycam
Pictured: Yellow Line as drawn by the skycam
![Page 7: Real-Time Camera Tracking in the “1st & Ten” Systemon-demand.gputechconf.com/gtc/2015/presentation/S5187-Louis-Ge… · Real-Time Camera Tracking in the “1st & Ten” System](https://reader035.vdocument.in/reader035/viewer/2022063014/5fcf2bc02a0329626d252ca9/html5/thumbnails/7.jpg)
• Requires no instrumentation/sensors
• Image processing on CPU
• Designed for replays
• Debuted on ESPN Sunday Night Football
Legacy Optical System
![Page 8: Real-Time Camera Tracking in the “1st & Ten” Systemon-demand.gputechconf.com/gtc/2015/presentation/S5187-Louis-Ge… · Real-Time Camera Tracking in the “1st & Ten” System](https://reader035.vdocument.in/reader035/viewer/2022063014/5fcf2bc02a0329626d252ca9/html5/thumbnails/8.jpg)
Set initial camera for first frame
Track at 2x or better speed
Apply manual corrections as
needed
Stream camera results
Reset for next shot
Optical System Workflow
3/18/2015 8
Proprietary and Confidential
![Page 9: Real-Time Camera Tracking in the “1st & Ten” Systemon-demand.gputechconf.com/gtc/2015/presentation/S5187-Louis-Ge… · Real-Time Camera Tracking in the “1st & Ten” System](https://reader035.vdocument.in/reader035/viewer/2022063014/5fcf2bc02a0329626d252ca9/html5/thumbnails/9.jpg)
• Limited CPU computing
• Limited feature search regions
• Greater than desired number of noisy measurements
• Limited to analyzing every other frame
• Often requires manual correction of tracks
Limitations of Legacy Optical System
![Page 10: Real-Time Camera Tracking in the “1st & Ten” Systemon-demand.gputechconf.com/gtc/2015/presentation/S5187-Louis-Ge… · Real-Time Camera Tracking in the “1st & Ten” System](https://reader035.vdocument.in/reader035/viewer/2022063014/5fcf2bc02a0329626d252ca9/html5/thumbnails/10.jpg)
• Reduced compute cost
• More compute power
• Better scalability
How Does Today’s GPU Compute Power Help?
![Page 11: Real-Time Camera Tracking in the “1st & Ten” Systemon-demand.gputechconf.com/gtc/2015/presentation/S5187-Louis-Ge… · Real-Time Camera Tracking in the “1st & Ten” System](https://reader035.vdocument.in/reader035/viewer/2022063014/5fcf2bc02a0329626d252ca9/html5/thumbnails/11.jpg)
• Limited to template matching
• Averaged 115 point and 60 line measurements per solve
• Required interpolation of camera solution
Legacy Implementation
![Page 12: Real-Time Camera Tracking in the “1st & Ten” Systemon-demand.gputechconf.com/gtc/2015/presentation/S5187-Louis-Ge… · Real-Time Camera Tracking in the “1st & Ten” System](https://reader035.vdocument.in/reader035/viewer/2022063014/5fcf2bc02a0329626d252ca9/html5/thumbnails/12.jpg)
NVIDIA QUADRO graphics card
Supermicro motherboard
New CUDA-Based Optical System
Intel CPU
AJA video capture card
![Page 13: Real-Time Camera Tracking in the “1st & Ten” Systemon-demand.gputechconf.com/gtc/2015/presentation/S5187-Louis-Ge… · Real-Time Camera Tracking in the “1st & Ten” System](https://reader035.vdocument.in/reader035/viewer/2022063014/5fcf2bc02a0329626d252ca9/html5/thumbnails/13.jpg)
Live Optical Tracking Test
![Page 14: Real-Time Camera Tracking in the “1st & Ten” Systemon-demand.gputechconf.com/gtc/2015/presentation/S5187-Louis-Ge… · Real-Time Camera Tracking in the “1st & Ten” System](https://reader035.vdocument.in/reader035/viewer/2022063014/5fcf2bc02a0329626d252ca9/html5/thumbnails/14.jpg)
• More sophisticated feature detection
– Strong features to track
– Less matching ambiguity
– Reduced drift and noise in calibrations
• Increased number of high-quality measurements
Improvements Using the GPU
![Page 15: Real-Time Camera Tracking in the “1st & Ten” Systemon-demand.gputechconf.com/gtc/2015/presentation/S5187-Louis-Ge… · Real-Time Camera Tracking in the “1st & Ten” System](https://reader035.vdocument.in/reader035/viewer/2022063014/5fcf2bc02a0329626d252ca9/html5/thumbnails/15.jpg)
• Disambiguate features – Tried industry standard techniques (Harris, SIFT, FAST, etc.)
– Utilized a combination of industry and proprietary techniques
• More robust features – Better differentiation between hash mark corners in close proximity
– Reduced false positives (particularly along yard lines)
Improved Detection
![Page 16: Real-Time Camera Tracking in the “1st & Ten” Systemon-demand.gputechconf.com/gtc/2015/presentation/S5187-Louis-Ge… · Real-Time Camera Tracking in the “1st & Ten” System](https://reader035.vdocument.in/reader035/viewer/2022063014/5fcf2bc02a0329626d252ca9/html5/thumbnails/16.jpg)
Metrics Comparison
0
1000
2000
3000
4000
5000
Legacy Optical Current Optical
Number of Tracked Features
![Page 17: Real-Time Camera Tracking in the “1st & Ten” Systemon-demand.gputechconf.com/gtc/2015/presentation/S5187-Louis-Ge… · Real-Time Camera Tracking in the “1st & Ten” System](https://reader035.vdocument.in/reader035/viewer/2022063014/5fcf2bc02a0329626d252ca9/html5/thumbnails/17.jpg)
• Primarily memory bound – Had to use creative methods to hide latency
• Adhering to known CUDA optimization guidelines is a must – Coalesced memory access
– Minimizing shared memory bank conflicts
– Efficient use of shared memory
– Avoid divergent warps
– Use pinned memory to reduce CPU/GPU transfer costs
Lessons Learned
![Page 18: Real-Time Camera Tracking in the “1st & Ten” Systemon-demand.gputechconf.com/gtc/2015/presentation/S5187-Louis-Ge… · Real-Time Camera Tracking in the “1st & Ten” System](https://reader035.vdocument.in/reader035/viewer/2022063014/5fcf2bc02a0329626d252ca9/html5/thumbnails/18.jpg)
Final Thoughts
• Always more to do – Optical solutions are often domain/context specific
– Often need multiple solutions to cover different camera angles
• Never enough compute – Need sophisticated algorithms
– Real-time requirement adds hard constraints
![Page 19: Real-Time Camera Tracking in the “1st & Ten” Systemon-demand.gputechconf.com/gtc/2015/presentation/S5187-Louis-Ge… · Real-Time Camera Tracking in the “1st & Ten” System](https://reader035.vdocument.in/reader035/viewer/2022063014/5fcf2bc02a0329626d252ca9/html5/thumbnails/19.jpg)
Please complete the Presenter Evaluation sent to you by email or through the GTC Mobile App.