![Page 1: FPGA in Advanced Vision Applications · FPGA in Advanced Vision Applications Rostislav Halaš . Regional Product Engineer . Embedded Systems . National Instruments . ni.com . 2](https://reader036.vdocument.in/reader036/viewer/2022062505/5e81a052df39ae772c0a30f7/html5/thumbnails/1.jpg)
ni.com
FPGA in Advanced Vision Applications Rostislav Halaš Regional Product Engineer Embedded Systems National Instruments
![Page 2: FPGA in Advanced Vision Applications · FPGA in Advanced Vision Applications Rostislav Halaš . Regional Product Engineer . Embedded Systems . National Instruments . ni.com . 2](https://reader036.vdocument.in/reader036/viewer/2022062505/5e81a052df39ae772c0a30f7/html5/thumbnails/2.jpg)
2 ni.com
Platform – hardware
Tools for Measurement, Control and Automated Test
Advanced Manufacturing Aerospace and Defense Consumer Electronics Energy Transportation Wireless
Platform - Software
![Page 3: FPGA in Advanced Vision Applications · FPGA in Advanced Vision Applications Rostislav Halaš . Regional Product Engineer . Embedded Systems . National Instruments . ni.com . 2](https://reader036.vdocument.in/reader036/viewer/2022062505/5e81a052df39ae772c0a30f7/html5/thumbnails/3.jpg)
3 ni.com
Multicore CPU systems • Ride the CPU frequency wave • Automatic hardware acceleration (SSE, Hyperthreading) • Making software multithreaded
• OpenMP (multiple cores) • MPI (multiple separate machines) • Vision Development Module
• Some problems divide well • Others don’t
![Page 4: FPGA in Advanced Vision Applications · FPGA in Advanced Vision Applications Rostislav Halaš . Regional Product Engineer . Embedded Systems . National Instruments . ni.com . 2](https://reader036.vdocument.in/reader036/viewer/2022062505/5e81a052df39ae772c0a30f7/html5/thumbnails/4.jpg)
4 ni.com
FPGAs • Latency • Jitter • Compute power • Pipelining • Security • Weight / Power / Heat • Complexity • Raw Clock Rates • Limited Floating Point support
![Page 5: FPGA in Advanced Vision Applications · FPGA in Advanced Vision Applications Rostislav Halaš . Regional Product Engineer . Embedded Systems . National Instruments . ni.com . 2](https://reader036.vdocument.in/reader036/viewer/2022062505/5e81a052df39ae772c0a30f7/html5/thumbnails/5.jpg)
5 ni.com
Latency - Preprocessing
Exp
Readout
Loop Period
Processing
Exp
Loop Period
![Page 6: FPGA in Advanced Vision Applications · FPGA in Advanced Vision Applications Rostislav Halaš . Regional Product Engineer . Embedded Systems . National Instruments . ni.com . 2](https://reader036.vdocument.in/reader036/viewer/2022062505/5e81a052df39ae772c0a30f7/html5/thumbnails/6.jpg)
6 ni.com
Latency – Co-processing
Exp
Readout
Loop Period
Processing
Exp
![Page 7: FPGA in Advanced Vision Applications · FPGA in Advanced Vision Applications Rostislav Halaš . Regional Product Engineer . Embedded Systems . National Instruments . ni.com . 2](https://reader036.vdocument.in/reader036/viewer/2022062505/5e81a052df39ae772c0a30f7/html5/thumbnails/7.jpg)
7 ni.com
Latency – Co-processing
Exp
Readout
Loop Period
Exp
![Page 8: FPGA in Advanced Vision Applications · FPGA in Advanced Vision Applications Rostislav Halaš . Regional Product Engineer . Embedded Systems . National Instruments . ni.com . 2](https://reader036.vdocument.in/reader036/viewer/2022062505/5e81a052df39ae772c0a30f7/html5/thumbnails/8.jpg)
8 ni.com
Complexity
Counter Analog I/O I/O with DMA
LabVIEW FPGA VHDL
~4000 lines
![Page 9: FPGA in Advanced Vision Applications · FPGA in Advanced Vision Applications Rostislav Halaš . Regional Product Engineer . Embedded Systems . National Instruments . ni.com . 2](https://reader036.vdocument.in/reader036/viewer/2022062505/5e81a052df39ae772c0a30f7/html5/thumbnails/9.jpg)
9 ni.com
FPGA Use Modes Visualization
High-speed Control
Image Preprocessing Co-Processing
Image Acquisition Device
FPGA
FPGA Inline
Processing
Acquisition Logic
Memory
Host Memory
CPU Camera
Video Generation Monitor
Image Acquisition Device
FPGA
FPGA Inline
Processing
Acquisition Logic
Memory
Host
Memory
CPU Camera
Digital or Analog I/O Actuator
Image Acquisition Device
FPGA
FPGA Inline
Processing Acquisition
Logic
Memory
Host
Memory
CPU
Camera
Image Acquisition Device
FPGA
FPGA
Processing
Acquisition Logic
Memory
Host
Memory
CPU
Camera
![Page 10: FPGA in Advanced Vision Applications · FPGA in Advanced Vision Applications Rostislav Halaš . Regional Product Engineer . Embedded Systems . National Instruments . ni.com . 2](https://reader036.vdocument.in/reader036/viewer/2022062505/5e81a052df39ae772c0a30f7/html5/thumbnails/10.jpg)
10 ni.com
Image Processing and Visualization • FPGA is directly in the path of the image data • Processes pixels as they arrive • May require some buffering—2D kernel operations • Generates and outputs images directly or send result to
host CPU
![Page 11: FPGA in Advanced Vision Applications · FPGA in Advanced Vision Applications Rostislav Halaš . Regional Product Engineer . Embedded Systems . National Instruments . ni.com . 2](https://reader036.vdocument.in/reader036/viewer/2022062505/5e81a052df39ae772c0a30f7/html5/thumbnails/11.jpg)
11 ni.com
NI FPGA Hardware
• NI FlexRIO + NI 1483 adapter module
• PCIe-1473R • Base, medium or full
configuration cameras • General purpose digital I/O
• LabVIEW FPGA example programs
• Area scan and linescan image acquisition
• Threshold • Centroid • Bayer decoding
![Page 12: FPGA in Advanced Vision Applications · FPGA in Advanced Vision Applications Rostislav Halaš . Regional Product Engineer . Embedded Systems . National Instruments . ni.com . 2](https://reader036.vdocument.in/reader036/viewer/2022062505/5e81a052df39ae772c0a30f7/html5/thumbnails/12.jpg)
12 ni.com
FlexRIO for PXI System Architecture
I/O FPGA Processor
Embedded Controllers Synchronization Data streaming Power/cooling
PXI Platform FlexRIO FPGA Module Kintex-7 FPGA Up to 2 GB of DRAM PCIe Gen 2 x 8
FlexRIO Adapter Module Camera Link Module
132 DIO PCIe
![Page 13: FPGA in Advanced Vision Applications · FPGA in Advanced Vision Applications Rostislav Halaš . Regional Product Engineer . Embedded Systems . National Instruments . ni.com . 2](https://reader036.vdocument.in/reader036/viewer/2022062505/5e81a052df39ae772c0a30f7/html5/thumbnails/13.jpg)
13 ni.com
Visualization • Image transformation
• Image warping, rotation and flip • Image compression, encryption,
and authentication
• Feature highlighting • Filtering • Shading correction
• Noise reduction • Image averaging • Retinex algorithm
![Page 14: FPGA in Advanced Vision Applications · FPGA in Advanced Vision Applications Rostislav Halaš . Regional Product Engineer . Embedded Systems . National Instruments . ni.com . 2](https://reader036.vdocument.in/reader036/viewer/2022062505/5e81a052df39ae772c0a30f7/html5/thumbnails/14.jpg)
14 ni.com
Image Processing Functions FPGAs suitable to improve images and extract basic features
• Feature Extraction – Edge, lines corners – Binary objects – Color
• Measurements – Centroid – Area measurements
• Preprocessing – Image transforms – Image operators – Shading correction – Bayer decoding – Color space conversion – 1D & 2D FFT – Filtering (smooth/sharpen) – Binary morphology
![Page 15: FPGA in Advanced Vision Applications · FPGA in Advanced Vision Applications Rostislav Halaš . Regional Product Engineer . Embedded Systems . National Instruments . ni.com . 2](https://reader036.vdocument.in/reader036/viewer/2022062505/5e81a052df39ae772c0a30f7/html5/thumbnails/15.jpg)
15 ni.com
Image Processing Functions
FPGAs not suitable for certain high-level algorithms
• Object-level vision functions – Pattern matching – OCR/OCV – Barcode reading – Some geometric measurements – Classification
![Page 16: FPGA in Advanced Vision Applications · FPGA in Advanced Vision Applications Rostislav Halaš . Regional Product Engineer . Embedded Systems . National Instruments . ni.com . 2](https://reader036.vdocument.in/reader036/viewer/2022062505/5e81a052df39ae772c0a30f7/html5/thumbnails/16.jpg)
16 ni.com
High-Speed Control
• FPGA is directly in the path of the image data • FPGA generates and outputs control commands directly
Image Acquisition Device
FPGA Inline
Processing
Acquisition Logic
Memory
Host
Memory
CPU Camera
Digital I/O Actuator
FPGA
![Page 17: FPGA in Advanced Vision Applications · FPGA in Advanced Vision Applications Rostislav Halaš . Regional Product Engineer . Embedded Systems . National Instruments . ni.com . 2](https://reader036.vdocument.in/reader036/viewer/2022062505/5e81a052df39ae772c0a30f7/html5/thumbnails/17.jpg)
17 ni.com
• Laser alignment/steering – Beam profile/position measurements – Low latency control output
• High-speed sorting – Segmentation – Measure parameters of contaminant – Trigger rejection valves
• In Air Sorting – Image and inspect falling product – Low jitter requirement for decision
making and IO
High-Speed Control
![Page 18: FPGA in Advanced Vision Applications · FPGA in Advanced Vision Applications Rostislav Halaš . Regional Product Engineer . Embedded Systems . National Instruments . ni.com . 2](https://reader036.vdocument.in/reader036/viewer/2022062505/5e81a052df39ae772c0a30f7/html5/thumbnails/18.jpg)
18 ni.com
Example: Medical Imaging
Challenge Develop the signal processing backend for an Optical Coherence Tomography machine. High Level Requirements Sample at 800 MS/s Control fast steering mirrors to perform raster scan Imaging in real-time Stream image data over the network
![Page 19: FPGA in Advanced Vision Applications · FPGA in Advanced Vision Applications Rostislav Halaš . Regional Product Engineer . Embedded Systems . National Instruments . ni.com . 2](https://reader036.vdocument.in/reader036/viewer/2022062505/5e81a052df39ae772c0a30f7/html5/thumbnails/19.jpg)
19 ni.com
FlexRIO Optimized for Deployment
I/O FPGA Processor
FlexRIO Adapter Module Interchangeable I/O Analog, Digital, RF Custom I/O with MDK
132 DIO
Controller for FlexRIO Kintex-7 FPGA Dual-Core ARM Processor High Speed Serial NI Linux Real-Time OS Optimized for Size, Weight, Power
![Page 20: FPGA in Advanced Vision Applications · FPGA in Advanced Vision Applications Rostislav Halaš . Regional Product Engineer . Embedded Systems . National Instruments . ni.com . 2](https://reader036.vdocument.in/reader036/viewer/2022062505/5e81a052df39ae772c0a30f7/html5/thumbnails/20.jpg)
20 ni.com
OCT Solution with Controller for FlexRIO
2 Ch 800 MS/s 12-bit
Processor
Galvo Control
1 GbE
Remote Client
NI 5772 Write to Client
Acquisition Engine
Form Image
DRAM
Configuration Read from Client
![Page 21: FPGA in Advanced Vision Applications · FPGA in Advanced Vision Applications Rostislav Halaš . Regional Product Engineer . Embedded Systems . National Instruments . ni.com . 2](https://reader036.vdocument.in/reader036/viewer/2022062505/5e81a052df39ae772c0a30f7/html5/thumbnails/21.jpg)
21 ni.com
Image Co-processing
Image Acquisition Device
FPGA
FPGA Processing
Acquisition Logic
Memory
Host
Memory
CPU
Camera
![Page 22: FPGA in Advanced Vision Applications · FPGA in Advanced Vision Applications Rostislav Halaš . Regional Product Engineer . Embedded Systems . National Instruments . ni.com . 2](https://reader036.vdocument.in/reader036/viewer/2022062505/5e81a052df39ae772c0a30f7/html5/thumbnails/22.jpg)
22 ni.com
MOTION
VISION
Image Capture
CAPTURED IMAGES Move
Complete
Image Feedback
Visual Servo Control
![Page 23: FPGA in Advanced Vision Applications · FPGA in Advanced Vision Applications Rostislav Halaš . Regional Product Engineer . Embedded Systems . National Instruments . ni.com . 2](https://reader036.vdocument.in/reader036/viewer/2022062505/5e81a052df39ae772c0a30f7/html5/thumbnails/23.jpg)
23 ni.com
Control Loop
Position Setpoint
Actuator
Position Feedback
Image Processing
Coordinate Transform
Camera
Image Processing
Coordinate Transform
Camera
5ms
Visual Servo Control: Direct Servo
![Page 24: FPGA in Advanced Vision Applications · FPGA in Advanced Vision Applications Rostislav Halaš . Regional Product Engineer . Embedded Systems . National Instruments . ni.com . 2](https://reader036.vdocument.in/reader036/viewer/2022062505/5e81a052df39ae772c0a30f7/html5/thumbnails/24.jpg)
24 ni.com
Target Performance Comparison Example
![Page 25: FPGA in Advanced Vision Applications · FPGA in Advanced Vision Applications Rostislav Halaš . Regional Product Engineer . Embedded Systems . National Instruments . ni.com . 2](https://reader036.vdocument.in/reader036/viewer/2022062505/5e81a052df39ae772c0a30f7/html5/thumbnails/25.jpg)
25 ni.com
IC-3173 EtherCAT Master: Machine Controller
(EtherCAT)
AKD EtherCAT Servo Drive
GigE (PoE)
(EtherCAT)
Touchscreen HMI
To the network or PC
To industrial I/O
![Page 26: FPGA in Advanced Vision Applications · FPGA in Advanced Vision Applications Rostislav Halaš . Regional Product Engineer . Embedded Systems . National Instruments . ni.com . 2](https://reader036.vdocument.in/reader036/viewer/2022062505/5e81a052df39ae772c0a30f7/html5/thumbnails/26.jpg)
26 ni.com
Future Looking Projects Using FPGA Co-processing
System On Module
![Page 27: FPGA in Advanced Vision Applications · FPGA in Advanced Vision Applications Rostislav Halaš . Regional Product Engineer . Embedded Systems . National Instruments . ni.com . 2](https://reader036.vdocument.in/reader036/viewer/2022062505/5e81a052df39ae772c0a30f7/html5/thumbnails/27.jpg)
27 ni.com
You Might Want to Use an FPGA for Vision…
• If latency or jitter is critical • If power consumption is critical • If you have to speed up throughput • If you can pipeline your algorithms • If you have to reduce the amount of data or aggregate multiple high-speed
channels • If you are using algorithms that can take advantage of the FPGA
architecture • If the FPGA is already in the image path • If any of the above give you a competitive advantage