![Page 1: Computer and Machine Visionecee.colorado.edu/~siewerts/extra/ecen5043/ecen5043_doc/... · 2014-10-01 · – Industry Standard for Machine Vision Automation – E.g. Inspection Systems](https://reader033.vdocument.in/reader033/viewer/2022050312/5f74cd10780a9f452d2b5784/html5/thumbnails/1.jpg)
January 27, 2014 Sam Siewert
Computer and Machine
Vision
Lecture Week 3
Part-1
![Page 2: Computer and Machine Visionecee.colorado.edu/~siewerts/extra/ecen5043/ecen5043_doc/... · 2014-10-01 · – Industry Standard for Machine Vision Automation – E.g. Inspection Systems](https://reader033.vdocument.in/reader033/viewer/2022050312/5f74cd10780a9f452d2b5784/html5/thumbnails/2.jpg)
Outline of Week 3
Processing Images and Moving Pictures –
High Level View and Computer
Architecture for it
Linux Platforms for Computer/Machine
Vision
I/O, Memory and Processing Challenges
Sam Siewert 2
![Page 3: Computer and Machine Visionecee.colorado.edu/~siewerts/extra/ecen5043/ecen5043_doc/... · 2014-10-01 · – Industry Standard for Machine Vision Automation – E.g. Inspection Systems](https://reader033.vdocument.in/reader033/viewer/2022050312/5f74cd10780a9f452d2b5784/html5/thumbnails/3.jpg)
Old School Moving Picture Media
and Cameras NTSC OTA (1941, 1953 color, 2009 dead)
– Analog, Interlaced, Continuous Broadcast Transmission or CCTV (Closed Circuit TV)
– Coax Cable or Tuner with Immediate CRT Display
– No Buffers, No Routing, No De-mux
– No Compression
Analog Cable
AM/FM OTA
Film Projectors
Sam Siewert 3
![Page 4: Computer and Machine Visionecee.colorado.edu/~siewerts/extra/ecen5043/ecen5043_doc/... · 2014-10-01 · – Industry Standard for Machine Vision Automation – E.g. Inspection Systems](https://reader033.vdocument.in/reader033/viewer/2022050312/5f74cd10780a9f452d2b5784/html5/thumbnails/4.jpg)
Modern Digital Cameras Camera Link – High Frame Rates
– High Data Rates and Resolutions
– Industry Standard for Machine Vision Automation
– E.g. Inspection Systems
– E.g. – Sony, IDT, National Instruments
SD-SDI and HD-SDI – Standard and High Definition Synchronous Digital Interface
– Standard for Studios, Broadcast
Digital Cinema – Red Camera
– 1080p, 2K, 4K Resolutions and Much Higher
– Automated Digital Delivery and Projection
Webcams and Mobile Phone Cameras – Very Low Cost
– Proprietary
– Performance Varies Dramatically
Sam Siewert 4
![Page 5: Computer and Machine Visionecee.colorado.edu/~siewerts/extra/ecen5043/ecen5043_doc/... · 2014-10-01 · – Industry Standard for Machine Vision Automation – E.g. Inspection Systems](https://reader033.vdocument.in/reader033/viewer/2022050312/5f74cd10780a9f452d2b5784/html5/thumbnails/5.jpg)
Differences Analog vs Digital Encoding for Transmission
– Digital Allows for Image Processing
– Adds Latency
– Requires Compression for Packet Switched Networks and Storage
Routed (Diversely), Buffered
Compressed (MPEG, JPEG) to Lower Bit-rates
Multiplexed (Shares Transmission Carrier for Audio, Video, Channels)
Transported by IP (Large Packets)
Continuous Transmission – Analog or Constant Bit-Rate / Frame-Rate
Sam Siewert 5
![Page 6: Computer and Machine Visionecee.colorado.edu/~siewerts/extra/ecen5043/ecen5043_doc/... · 2014-10-01 · – Industry Standard for Machine Vision Automation – E.g. Inspection Systems](https://reader033.vdocument.in/reader033/viewer/2022050312/5f74cd10780a9f452d2b5784/html5/thumbnails/6.jpg)
E.g. UAV Latency and Jitter
Verification of Video Frame Latency Telemetry for UAV Systems Using A Secondary Optical Method, Sam Siewert, Muhammad Ahmad, Kevin Yao
Sam Siewert 6
![Page 7: Computer and Machine Visionecee.colorado.edu/~siewerts/extra/ecen5043/ecen5043_doc/... · 2014-10-01 · – Industry Standard for Machine Vision Automation – E.g. Inspection Systems](https://reader033.vdocument.in/reader033/viewer/2022050312/5f74cd10780a9f452d2b5784/html5/thumbnails/7.jpg)
NTSC (Analog TV)
Sam Siewert 7
AM Video to CRT
FM Audio
Chroma Added Later
Odd/Even Lines (Interlaced)
29.97 FPS (30 before color)
Vertical Blanking (CRT Retrace Time, Closed Captioning)
525 Lines, 262.5 per Field, 60 Fields per Second
http://en.wikipedia.org/wiki/File:Ntsc_channel.svg
![Page 8: Computer and Machine Visionecee.colorado.edu/~siewerts/extra/ecen5043/ecen5043_doc/... · 2014-10-01 · – Industry Standard for Machine Vision Automation – E.g. Inspection Systems](https://reader033.vdocument.in/reader033/viewer/2022050312/5f74cd10780a9f452d2b5784/html5/thumbnails/8.jpg)
Linux in Computer Vision
Embedded Solutions – Texas Instruments OMAP (Beagle xM, Bone)
– Numerous ARM SoCs (NVIDIA, Qualcomm, Broadcomm, …)
Scalable Solutions – Multi-Core (Xeon Phi)
– Vector Processing – CUDA, OpenCL GPU and GP-GPU
Computer and Machine Vision is I/O, Memory and Processing Intensive
Sam Siewert 8
![Page 9: Computer and Machine Visionecee.colorado.edu/~siewerts/extra/ecen5043/ecen5043_doc/... · 2014-10-01 · – Industry Standard for Machine Vision Automation – E.g. Inspection Systems](https://reader033.vdocument.in/reader033/viewer/2022050312/5f74cd10780a9f452d2b5784/html5/thumbnails/9.jpg)
Camera Interfaces
CCD (Charge Coupled Device) or CMOS (Common Metal Oxide Substrate) Detector – Integration Time for Photo-sensitive Elements in Array (to Build
up Charge)
– Read-out Time to Sample Elements in Array
Luminance and Chroma Analog to Digital Conversion
Double Buffer for Read-out + Processing
Frame Capture – http://www.cse.uaa.alaska.edu/~ssiewert/a485_doc/Frame-
Capture-Chips/
– Host Interface over PCI Bus or USB
Sam Siewert 9
![Page 10: Computer and Machine Visionecee.colorado.edu/~siewerts/extra/ecen5043/ecen5043_doc/... · 2014-10-01 · – Industry Standard for Machine Vision Automation – E.g. Inspection Systems](https://reader033.vdocument.in/reader033/viewer/2022050312/5f74cd10780a9f452d2b5784/html5/thumbnails/10.jpg)
Digital Video Transport QoS Latency – To Tune in a Program, Turn-on
– To Deliver a Video Frame or Audio PCM Sample
– To Start, FF, REW, Start-Over, Pause
Bandwidth – Resolution, Lossy/Lossless Compression, High Motion
– Pixel Encoding for Color
– Frame Rate
– Constant Bit-rate Transport?
– Variable Bit-rate Transport and Encoding?
Jitter – Decode and Presentation Rates
– Elasticity in Decode to Presentation Buffering Necessary
Sam Siewert 10
![Page 11: Computer and Machine Visionecee.colorado.edu/~siewerts/extra/ecen5043/ecen5043_doc/... · 2014-10-01 · – Industry Standard for Machine Vision Automation – E.g. Inspection Systems](https://reader033.vdocument.in/reader033/viewer/2022050312/5f74cd10780a9f452d2b5784/html5/thumbnails/11.jpg)
January 27, 2014 Sam Siewert
Linux System Options
(Linux for Image Processing, Camera
Interfacing and Computer Vision)
![Page 12: Computer and Machine Visionecee.colorado.edu/~siewerts/extra/ecen5043/ecen5043_doc/... · 2014-10-01 · – Industry Standard for Machine Vision Automation – E.g. Inspection Systems](https://reader033.vdocument.in/reader033/viewer/2022050312/5f74cd10780a9f452d2b5784/html5/thumbnails/12.jpg)
Sam Siewert 12
Processing Outline
Many-Core Linux Host(s)
– Intel Atom
– ARM
– Xeon
GP-GPU Vector Processing PCI-E Co-Processors
NVIDIA Tesla/Fermi
AMD ATI
NPTL – Native POSIX Threads Library
NPTL Example Code Walkthrough
![Page 13: Computer and Machine Visionecee.colorado.edu/~siewerts/extra/ecen5043/ecen5043_doc/... · 2014-10-01 · – Industry Standard for Machine Vision Automation – E.g. Inspection Systems](https://reader033.vdocument.in/reader033/viewer/2022050312/5f74cd10780a9f452d2b5784/html5/thumbnails/13.jpg)
Sam Siewert 13
Conceptual View of RT Resources Three-Space View of Utilization Requirements
– CPU Margin?
– IO Latency (and Bandwidth) Margin?
– Memory Capacity (and Latency) Margin?
Upper Right Front Corner – Low-Margin
Origin – High-Margin
Mobile – Must Consider Battery Life Too (Power)
CPU-Utility
IO-Utility
Memory-Utility
![Page 14: Computer and Machine Visionecee.colorado.edu/~siewerts/extra/ecen5043/ecen5043_doc/... · 2014-10-01 · – Industry Standard for Machine Vision Automation – E.g. Inspection Systems](https://reader033.vdocument.in/reader033/viewer/2022050312/5f74cd10780a9f452d2b5784/html5/thumbnails/14.jpg)
Processing – Initial Focus Processing and Scaling Frame Transformation, Encode, Decode is Critical
Memory for Buffering (Frame Transformations, CPU Integrated or GPU Offloaded – e.g. Linux VDPAU)
I/O for Networking (Transport)
I/O for Storage (On-Demand, Post, Non-Linear Editing)
Sam Siewert 14
![Page 15: Computer and Machine Visionecee.colorado.edu/~siewerts/extra/ecen5043/ecen5043_doc/... · 2014-10-01 · – Industry Standard for Machine Vision Automation – E.g. Inspection Systems](https://reader033.vdocument.in/reader033/viewer/2022050312/5f74cd10780a9f452d2b5784/html5/thumbnails/15.jpg)
Flynn’s Computer Architecture
Taxonomy Single Instruction Multiple Instruction
Single Data SISD (Traditional Uni-
processor)
MISD (Voting schemes
and active-active
controllers)
Multiple Data SIMD (e.g. SSE 4.2, GP-
GPU, Vector Processing)
MIMD (Distributed
systems (MPMD),
Clusters with MPI/PVM
(SPMD), AMP/SMP)
Sam Siewert 15
GPC has gone MIMD with SIMD Instruction Sets and SIMD Offload
(GP-GPU) NUMA vs. UMA (Trend away from UMA to NUMA or MCH vs. IOH)
SMP with One OS (Shared Memory, CPU-balanced Interrupt Handling,
Process Load Balancing, Mutli-User, Multi-Application, CPU Affinity
Possible)
MIMD - Single Program Multi-Data vs. Multi-Program Multi-Data
![Page 16: Computer and Machine Visionecee.colorado.edu/~siewerts/extra/ecen5043/ecen5043_doc/... · 2014-10-01 · – Industry Standard for Machine Vision Automation – E.g. Inspection Systems](https://reader033.vdocument.in/reader033/viewer/2022050312/5f74cd10780a9f452d2b5784/html5/thumbnails/16.jpg)
Computer and Machine Vision
Treated as a Real-time and/or Interactive
System
– Requires Predictable Response (By Deadline)
– Rate Monotonic
– Earliest Deadline First
– Least Laxity First
Covered in Depth in OS and Digital Media
(Just be Aware for Computer Vision) Sam Siewert 16
![Page 17: Computer and Machine Visionecee.colorado.edu/~siewerts/extra/ecen5043/ecen5043_doc/... · 2014-10-01 · – Industry Standard for Machine Vision Automation – E.g. Inspection Systems](https://reader033.vdocument.in/reader033/viewer/2022050312/5f74cd10780a9f452d2b5784/html5/thumbnails/17.jpg)
Sam Siewert 17
CPU Scheduling Taxonomy Execution Scheduling
Global-MP Local-Uniprocessor
Distributed Asymmetric
(AMP )
Symmetric
(SMP OS)
Preemptive Non-Preemptive
Fixed-Priority
Hybrid
Dynamic-Priority Cooperative
Batch
FCFS SJN
Co-Routine Continuation
Function
Heuristic EDF/LLF RR Timeslice
(desktop)
Multi-Frequency
Executives
Static Dynamic
Rate
Monotonic
Deadline
Monotonic
Dataflow
(Preemptive, Non-Preemptive Subtree
Under Each Global-MP Leaf)
SMT
(Micro-Paralell)
![Page 18: Computer and Machine Visionecee.colorado.edu/~siewerts/extra/ecen5043/ecen5043_doc/... · 2014-10-01 · – Industry Standard for Machine Vision Automation – E.g. Inspection Systems](https://reader033.vdocument.in/reader033/viewer/2022050312/5f74cd10780a9f452d2b5784/html5/thumbnails/18.jpg)
Sam Siewert 18
Response Latency Ci WCET
Input/Output Latency
Interference Time
Event
Sensed Interrupt Dispatch Preemption Dispatch
Interference
Completion
(IO Queued)
Actuation
(IO Completion)
Input-Latency
Dispatch-Latency
Execution Execution
Output-Latency
Time
Response Time = TimeActuation – TimeSensed
(From Release to Response)
![Page 19: Computer and Machine Visionecee.colorado.edu/~siewerts/extra/ecen5043/ecen5043_doc/... · 2014-10-01 · – Industry Standard for Machine Vision Automation – E.g. Inspection Systems](https://reader033.vdocument.in/reader033/viewer/2022050312/5f74cd10780a9f452d2b5784/html5/thumbnails/19.jpg)
Resource Scaling
Processing – Co-Processors – GPU – CUDA, OpenCL
– Many-Core – E.g. Intel Xeon Phi MICA
– FPGA – E.g. Altera Stratix
– Ideally Camera Interface
I/O – High Rate Transport – HD-SDI, Camera Link, GigE/10GE
Memory – SSD, PCIe Nand, NVM – FusionIO, Micron, Intel
– Memristor (Future)
Sam Siewert 19