challenges of using v4l2 to capture and process video ...2 eugen hristev embedded linux engineer at...
TRANSCRIPT
![Page 1: Challenges of using V4L2 to capture and process video ...2 Eugen Hristev Embedded Linux Engineer at Microchip Part of the MPU32 Linux team Developing second stage and third stage bootloaders](https://reader036.vdocument.in/reader036/viewer/2022081619/60fd1a6d307b3a538627df54/html5/thumbnails/1.jpg)
A Leading Provider of Smart, Connected and Secure Embedded Control Solutions
Challenges of Using V4L2 to Capture and Process Video Sensor Images
October 27, 2020
Eugen Hristev© Copyright 2020, Microchip Technology, Inc.
Creative Commons BY-SA 3.0 license.
Corrections, suggestions, contributions and translations are welcome
![Page 2: Challenges of using V4L2 to capture and process video ...2 Eugen Hristev Embedded Linux Engineer at Microchip Part of the MPU32 Linux team Developing second stage and third stage bootloaders](https://reader036.vdocument.in/reader036/viewer/2022081619/60fd1a6d307b3a538627df54/html5/thumbnails/2.jpg)
2
Eugen Hristev
Embedded Linux Engineer at Microchip
Part of the MPU32 Linux team
Developing second stage and third stage bootloaders
Developing Linux kernel device drivers
Maintaining and developing Microchip V4L2 drivers:Atmel sensor controller, atmel sensor interface
![Page 3: Challenges of using V4L2 to capture and process video ...2 Eugen Hristev Embedded Linux Engineer at Microchip Part of the MPU32 Linux team Developing second stage and third stage bootloaders](https://reader036.vdocument.in/reader036/viewer/2022081619/60fd1a6d307b3a538627df54/html5/thumbnails/3.jpg)
3
Agenda
How digital sensors work
What happens with digital data until it is turned into a real photo
Issues that can occur during this process
How can V4L2 help tuning dedicated hardware and software in getting better picture quality
![Page 4: Challenges of using V4L2 to capture and process video ...2 Eugen Hristev Embedded Linux Engineer at Microchip Part of the MPU32 Linux team Developing second stage and third stage bootloaders](https://reader036.vdocument.in/reader036/viewer/2022081619/60fd1a6d307b3a538627df54/html5/thumbnails/4.jpg)
4
System diagram
Image sensor
Bayer interpolation
(edge detection)
WB correction
Gains, offsetsexposure correction
Contrast settings,
Brightness settings
Resulting image
User (v4l2-ctl)
User
space
Kernel
space
V4L2 subsystem interface
Hardware pipeline
/dev/videoX
Sensor controller driver
Pixel stream
Pixel stream
Pixel stream Pixel stream
![Page 5: Challenges of using V4L2 to capture and process video ...2 Eugen Hristev Embedded Linux Engineer at Microchip Part of the MPU32 Linux team Developing second stage and third stage bootloaders](https://reader036.vdocument.in/reader036/viewer/2022081619/60fd1a6d307b3a538627df54/html5/thumbnails/5.jpg)
5
Digital video sensor
How does a sensor work ?
![Page 6: Challenges of using V4L2 to capture and process video ...2 Eugen Hristev Embedded Linux Engineer at Microchip Part of the MPU32 Linux team Developing second stage and third stage bootloaders](https://reader036.vdocument.in/reader036/viewer/2022081619/60fd1a6d307b3a538627df54/html5/thumbnails/6.jpg)
6
Capture the light
Sensors operate by capturing ambient lightSensors have an array of light sensitive pixels
BAYER array
![Page 7: Challenges of using V4L2 to capture and process video ...2 Eugen Hristev Embedded Linux Engineer at Microchip Part of the MPU32 Linux team Developing second stage and third stage bootloaders](https://reader036.vdocument.in/reader036/viewer/2022081619/60fd1a6d307b3a538627df54/html5/thumbnails/7.jpg)
7
BAYER array
Why do we have more green pixels than red/blue pixels ?
Why was this pattern chosen ?
Why each pixel has only one color ?
Do we lose color information ?
How do we convert pixel color into bits ?
How do we get a real photo out of this pattern ?
![Page 8: Challenges of using V4L2 to capture and process video ...2 Eugen Hristev Embedded Linux Engineer at Microchip Part of the MPU32 Linux team Developing second stage and third stage bootloaders](https://reader036.vdocument.in/reader036/viewer/2022081619/60fd1a6d307b3a538627df54/html5/thumbnails/8.jpg)
8
Pixel data inside controller
We have pixel data. What is next ?
![Page 9: Challenges of using V4L2 to capture and process video ...2 Eugen Hristev Embedded Linux Engineer at Microchip Part of the MPU32 Linux team Developing second stage and third stage bootloaders](https://reader036.vdocument.in/reader036/viewer/2022081619/60fd1a6d307b3a538627df54/html5/thumbnails/9.jpg)
9
BAYER interpolation
Each pixel will obtain information from its neighbors
![Page 10: Challenges of using V4L2 to capture and process video ...2 Eugen Hristev Embedded Linux Engineer at Microchip Part of the MPU32 Linux team Developing second stage and third stage bootloaders](https://reader036.vdocument.in/reader036/viewer/2022081619/60fd1a6d307b3a538627df54/html5/thumbnails/10.jpg)
10
What can go wrong ?
We interpolate. But is this flawless ?
![Page 11: Challenges of using V4L2 to capture and process video ...2 Eugen Hristev Embedded Linux Engineer at Microchip Part of the MPU32 Linux team Developing second stage and third stage bootloaders](https://reader036.vdocument.in/reader036/viewer/2022081619/60fd1a6d307b3a538627df54/html5/thumbnails/11.jpg)
11
The edge problem
Pitfall of the demosaicing
![Page 12: Challenges of using V4L2 to capture and process video ...2 Eugen Hristev Embedded Linux Engineer at Microchip Part of the MPU32 Linux team Developing second stage and third stage bootloaders](https://reader036.vdocument.in/reader036/viewer/2022081619/60fd1a6d307b3a538627df54/html5/thumbnails/12.jpg)
12
The edge problem
Pitfall of the demosaicing
![Page 13: Challenges of using V4L2 to capture and process video ...2 Eugen Hristev Embedded Linux Engineer at Microchip Part of the MPU32 Linux team Developing second stage and third stage bootloaders](https://reader036.vdocument.in/reader036/viewer/2022081619/60fd1a6d307b3a538627df54/html5/thumbnails/13.jpg)
13
The edge problem
What can we do ?
Edge interpolation : sometimes, but not always !
![Page 14: Challenges of using V4L2 to capture and process video ...2 Eugen Hristev Embedded Linux Engineer at Microchip Part of the MPU32 Linux team Developing second stage and third stage bootloaders](https://reader036.vdocument.in/reader036/viewer/2022081619/60fd1a6d307b3a538627df54/html5/thumbnails/14.jpg)
14
Inside the system
Image sensor
Bayer interpolation
(edge detection)
Resulting image
User (v4l2-ctl)
User
space
Kernel
space
V4L2 subsystem interface
Hardware pipeline
/dev/videoX
Sensor controller driver
Pixel stream
Pixel stream
![Page 15: Challenges of using V4L2 to capture and process video ...2 Eugen Hristev Embedded Linux Engineer at Microchip Part of the MPU32 Linux team Developing second stage and third stage bootloaders](https://reader036.vdocument.in/reader036/viewer/2022081619/60fd1a6d307b3a538627df54/html5/thumbnails/15.jpg)
15
The color problem
How we see the light
![Page 16: Challenges of using V4L2 to capture and process video ...2 Eugen Hristev Embedded Linux Engineer at Microchip Part of the MPU32 Linux team Developing second stage and third stage bootloaders](https://reader036.vdocument.in/reader036/viewer/2022081619/60fd1a6d307b3a538627df54/html5/thumbnails/16.jpg)
16
Light has a temperature
![Page 17: Challenges of using V4L2 to capture and process video ...2 Eugen Hristev Embedded Linux Engineer at Microchip Part of the MPU32 Linux team Developing second stage and third stage bootloaders](https://reader036.vdocument.in/reader036/viewer/2022081619/60fd1a6d307b3a538627df54/html5/thumbnails/17.jpg)
17
Temperature and color
![Page 18: Challenges of using V4L2 to capture and process video ...2 Eugen Hristev Embedded Linux Engineer at Microchip Part of the MPU32 Linux team Developing second stage and third stage bootloaders](https://reader036.vdocument.in/reader036/viewer/2022081619/60fd1a6d307b3a538627df54/html5/thumbnails/18.jpg)
18
Temperature and colorWhat color is the water in this photo ? What about the rock ?
![Page 19: Challenges of using V4L2 to capture and process video ...2 Eugen Hristev Embedded Linux Engineer at Microchip Part of the MPU32 Linux team Developing second stage and third stage bootloaders](https://reader036.vdocument.in/reader036/viewer/2022081619/60fd1a6d307b3a538627df54/html5/thumbnails/19.jpg)
19
Temperature and colorWhat about now ?
![Page 20: Challenges of using V4L2 to capture and process video ...2 Eugen Hristev Embedded Linux Engineer at Microchip Part of the MPU32 Linux team Developing second stage and third stage bootloaders](https://reader036.vdocument.in/reader036/viewer/2022081619/60fd1a6d307b3a538627df54/html5/thumbnails/20.jpg)
20
What the human brain can see
Remember old black and white photos ?
Can you actually see colors in black and white ?
![Page 21: Challenges of using V4L2 to capture and process video ...2 Eugen Hristev Embedded Linux Engineer at Microchip Part of the MPU32 Linux team Developing second stage and third stage bootloaders](https://reader036.vdocument.in/reader036/viewer/2022081619/60fd1a6d307b3a538627df54/html5/thumbnails/21.jpg)
21
What the human brain can see
What about Sepia ?
The sensor does not have a brain !
We will need to adjust the color such that they look more natural in the specific light of the scenery
![Page 22: Challenges of using V4L2 to capture and process video ...2 Eugen Hristev Embedded Linux Engineer at Microchip Part of the MPU32 Linux team Developing second stage and third stage bootloaders](https://reader036.vdocument.in/reader036/viewer/2022081619/60fd1a6d307b3a538627df54/html5/thumbnails/22.jpg)
22
What the human eye sees
Before white
balancing
After white
balancing
![Page 23: Challenges of using V4L2 to capture and process video ...2 Eugen Hristev Embedded Linux Engineer at Microchip Part of the MPU32 Linux team Developing second stage and third stage bootloaders](https://reader036.vdocument.in/reader036/viewer/2022081619/60fd1a6d307b3a538627df54/html5/thumbnails/23.jpg)
23
What the human eye sees
Before white
balancingAfter white
balancing
![Page 24: Challenges of using V4L2 to capture and process video ...2 Eugen Hristev Embedded Linux Engineer at Microchip Part of the MPU32 Linux team Developing second stage and third stage bootloaders](https://reader036.vdocument.in/reader036/viewer/2022081619/60fd1a6d307b3a538627df54/html5/thumbnails/24.jpg)
24
What the human eye sees
After white
balancing
Before white
balancing
![Page 25: Challenges of using V4L2 to capture and process video ...2 Eugen Hristev Embedded Linux Engineer at Microchip Part of the MPU32 Linux team Developing second stage and third stage bootloaders](https://reader036.vdocument.in/reader036/viewer/2022081619/60fd1a6d307b3a538627df54/html5/thumbnails/25.jpg)
25
What the human eye sees
After white
balancing
Before white
balancing
![Page 26: Challenges of using V4L2 to capture and process video ...2 Eugen Hristev Embedded Linux Engineer at Microchip Part of the MPU32 Linux team Developing second stage and third stage bootloaders](https://reader036.vdocument.in/reader036/viewer/2022081619/60fd1a6d307b3a538627df54/html5/thumbnails/26.jpg)
26
What can we do to adapt ?
We need to teach the sensor to adapt
![Page 27: Challenges of using V4L2 to capture and process video ...2 Eugen Hristev Embedded Linux Engineer at Microchip Part of the MPU32 Linux team Developing second stage and third stage bootloaders](https://reader036.vdocument.in/reader036/viewer/2022081619/60fd1a6d307b3a538627df54/html5/thumbnails/27.jpg)
27
Question 1:
What is the average color of this photo ?
![Page 28: Challenges of using V4L2 to capture and process video ...2 Eugen Hristev Embedded Linux Engineer at Microchip Part of the MPU32 Linux team Developing second stage and third stage bootloaders](https://reader036.vdocument.in/reader036/viewer/2022081619/60fd1a6d307b3a538627df54/html5/thumbnails/28.jpg)
28
Question 2:
What is the average color of this photo ?
![Page 29: Challenges of using V4L2 to capture and process video ...2 Eugen Hristev Embedded Linux Engineer at Microchip Part of the MPU32 Linux team Developing second stage and third stage bootloaders](https://reader036.vdocument.in/reader036/viewer/2022081619/60fd1a6d307b3a538627df54/html5/thumbnails/29.jpg)
29
Grey world
We need to make sure that Grey is grey for us, and grey for the sensor. In the ambient light, of course...
![Page 30: Challenges of using V4L2 to capture and process video ...2 Eugen Hristev Embedded Linux Engineer at Microchip Part of the MPU32 Linux team Developing second stage and third stage bootloaders](https://reader036.vdocument.in/reader036/viewer/2022081619/60fd1a6d307b3a538627df54/html5/thumbnails/30.jpg)
30
Algorithm inside the driver
We need four frames captured.
The hardware will compute a color histogram of each BAYERcomponent
This will tell us how much of each color we have in the frame.Will count the number of pixels versus the pixel value.
By histogram, this is greenish, with low blue
![Page 31: Challenges of using V4L2 to capture and process video ...2 Eugen Hristev Embedded Linux Engineer at Microchip Part of the MPU32 Linux team Developing second stage and third stage bootloaders](https://reader036.vdocument.in/reader036/viewer/2022081619/60fd1a6d307b3a538627df54/html5/thumbnails/31.jpg)
31
Algorithm inside the driver
We need to apply Grey World: everything is Grey !
Adjust gains and offsets such that the histogram will be nearly identic for each component.
Compute the average of R, B, RG, BG. This is
Keep Green as a constant and adjust gains for red and blue, by dividing with the average,
for each channel,
Ideally, we should see that the blue component will be aligned with the others.
Ʋ𝐾𝑖,𝑗 =𝐾𝑖,𝑗
LJ𝐾
LJ𝐾
Ʋ𝐾𝑖,𝑗
![Page 32: Challenges of using V4L2 to capture and process video ...2 Eugen Hristev Embedded Linux Engineer at Microchip Part of the MPU32 Linux team Developing second stage and third stage bootloaders](https://reader036.vdocument.in/reader036/viewer/2022081619/60fd1a6d307b3a538627df54/html5/thumbnails/32.jpg)
32
Algorithm inside the driver
We need four frames captured.
The hardware will compute a color histogram of each BAYERcomponent
This will tell us how much of each color we have in the frame.Will count the number of pixels versus the pixel value.
Histogram is now aligned and
stretched
![Page 33: Challenges of using V4L2 to capture and process video ...2 Eugen Hristev Embedded Linux Engineer at Microchip Part of the MPU32 Linux team Developing second stage and third stage bootloaders](https://reader036.vdocument.in/reader036/viewer/2022081619/60fd1a6d307b3a538627df54/html5/thumbnails/33.jpg)
33
Gains and offsets
User Controls
brightness 0x00980900 (int) : min=-1024 max=1023 step=1 default=0 value=0 flags=slider
contrast 0x00980901 (int) : min=-2048 max=2047 step=1 default=16 value=16 flags=slider
white_balance_automatic 0x0098090c (bool) : default=1 value=1 flags=update
do_white_balance 0x0098090d (button) : flags=inactive, write-only, volatile, execute-on-write
gamma 0x00980910 (int) : min=0 max=3 step=1 default=3 value=3 flags=slider
red_component_gain 0x009819c0 (int) : min=0 max=8191 step=1 default=512 value=512 flags=inactive, slider, volatile
blue_component_gain 0x009819c1 (int) : min=0 max=8191 step=1 default=512 value=512 flags=inactive, slider, volatile
green_red_component_gain 0x009819c2 (int) : min=0 max=8191 step=1 default=512 value=512 flags=inactive, slider, volatile
green_blue_component_gain 0x009819c3 (int) : min=0 max=8191 step=1 default=512 value=512 flags=inactive, slider, volatile
red_component_offset 0x009819c4 (int) : min=-4095 max=4095 step=1 default=0 value=0 flags=inactive, slider, volatile
blue_component_offset 0x009819c5 (int) : min=-4095 max=4095 step=1 default=0 value=0 flags=inactive, slider, volatile
green_red_component_offset 0x009819c6 (int) : min=-4095 max=4095 step=1 default=0 value=0 flags=inactive, slider, volatile
green_blue_component_offset 0x009819c7 (int) : min=-4095 max=4095 step=1 default=0 value=0 flags=inactive, slider, volatile
V4L2 to the rescue:
![Page 34: Challenges of using V4L2 to capture and process video ...2 Eugen Hristev Embedded Linux Engineer at Microchip Part of the MPU32 Linux team Developing second stage and third stage bootloaders](https://reader036.vdocument.in/reader036/viewer/2022081619/60fd1a6d307b3a538627df54/html5/thumbnails/34.jpg)
34
Do white balance button
User Controls
brightness 0x00980900 (int) : min=-1024 max=1023 step=1 default=0 value=0 flags=slider
contrast 0x00980901 (int) : min=-2048 max=2047 step=1 default=16 value=16 flags=slider
white_balance_automatic 0x0098090c (bool) : default=1 value=0 flags=update
do_white_balance 0x0098090d (button) : flags=write-only, execute-on-write
gamma 0x00980910 (int) : min=0 max=3 step=1 default=3 value=3 flags=slider
red_component_gain 0x009819c0 (int) : min=0 max=8191 step=1 default=512 value=930 flags=slider
blue_component_gain 0x009819c1 (int) : min=0 max=8191 step=1 default=512 value=3198 flags=slider
green_red_component_gain 0x009819c2 (int) : min=0 max=8191 step=1 default=512 value=549 flags=slider
green_blue_component_gain 0x009819c3 (int) : min=0 max=8191 step=1 default=512 value=714 flags=slider
red_component_offset 0x009819c4 (int) : min=-4095 max=4095 step=1 default=0 value=-128 flags=slider
blue_component_offset 0x009819c5 (int) : min=-4095 max=4095 step=1 default=0 value=-48 flags=slider
green_red_component_offset 0x009819c6 (int) : min=-4095 max=4095 step=1 default=0 value=-152 flags=slider
green_blue_component_offset 0x009819c7 (int) : min=-4095 max=4095 step=1 default=0 value=-136 flags=slider
v4l2-ctl –set-ctrl=do_white_balance=1
# [ 2528.410573] atmel-sama7g5-isc e1408000.xisc: Completed one time white-balance adjustment.
![Page 35: Challenges of using V4L2 to capture and process video ...2 Eugen Hristev Embedded Linux Engineer at Microchip Part of the MPU32 Linux team Developing second stage and third stage bootloaders](https://reader036.vdocument.in/reader036/viewer/2022081619/60fd1a6d307b3a538627df54/html5/thumbnails/35.jpg)
35
Do white balance button
![Page 36: Challenges of using V4L2 to capture and process video ...2 Eugen Hristev Embedded Linux Engineer at Microchip Part of the MPU32 Linux team Developing second stage and third stage bootloaders](https://reader036.vdocument.in/reader036/viewer/2022081619/60fd1a6d307b3a538627df54/html5/thumbnails/36.jpg)
36
Auto white balance (AWB)
Cameras will do this for you.
But how good is this ?
Is the scenery always grey ?
Experiment with your smartphone !
Other algorithms for white balance ?
Sensor correlation: find which of the elementary possible lights are there
Grey world variations: find the grey object in the scenery
Photoshop WB: Black is black, White is white.
Manual tuning ! Towards Embedded Linux Camera...
![Page 37: Challenges of using V4L2 to capture and process video ...2 Eugen Hristev Embedded Linux Engineer at Microchip Part of the MPU32 Linux team Developing second stage and third stage bootloaders](https://reader036.vdocument.in/reader036/viewer/2022081619/60fd1a6d307b3a538627df54/html5/thumbnails/37.jpg)
37
Inside the system
Image sensor
WB correction
Gains, offsetsResulting image
User (v4l2-ctl)
User
space
Kernel
space
V4L2 subsystem interface
Hardware pipeline
/dev/videoX
Pixel streamPixel stream
Sensor controller driver
![Page 38: Challenges of using V4L2 to capture and process video ...2 Eugen Hristev Embedded Linux Engineer at Microchip Part of the MPU32 Linux team Developing second stage and third stage bootloaders](https://reader036.vdocument.in/reader036/viewer/2022081619/60fd1a6d307b3a538627df54/html5/thumbnails/38.jpg)
38
Light quantityDoes it matter how much light we capture ?
![Page 39: Challenges of using V4L2 to capture and process video ...2 Eugen Hristev Embedded Linux Engineer at Microchip Part of the MPU32 Linux team Developing second stage and third stage bootloaders](https://reader036.vdocument.in/reader036/viewer/2022081619/60fd1a6d307b3a538627df54/html5/thumbnails/39.jpg)
39
Exposure
How much light do we need to capture ?
Too much light
![Page 40: Challenges of using V4L2 to capture and process video ...2 Eugen Hristev Embedded Linux Engineer at Microchip Part of the MPU32 Linux team Developing second stage and third stage bootloaders](https://reader036.vdocument.in/reader036/viewer/2022081619/60fd1a6d307b3a538627df54/html5/thumbnails/40.jpg)
40
Again, trade-off
Too little light
![Page 41: Challenges of using V4L2 to capture and process video ...2 Eugen Hristev Embedded Linux Engineer at Microchip Part of the MPU32 Linux team Developing second stage and third stage bootloaders](https://reader036.vdocument.in/reader036/viewer/2022081619/60fd1a6d307b3a538627df54/html5/thumbnails/41.jpg)
41
ExposureAgain, histogram comes to our aid
![Page 42: Challenges of using V4L2 to capture and process video ...2 Eugen Hristev Embedded Linux Engineer at Microchip Part of the MPU32 Linux team Developing second stage and third stage bootloaders](https://reader036.vdocument.in/reader036/viewer/2022081619/60fd1a6d307b3a538627df54/html5/thumbnails/42.jpg)
42
Exposure
Autoexposure: adapt exposure to fit the histogram
![Page 43: Challenges of using V4L2 to capture and process video ...2 Eugen Hristev Embedded Linux Engineer at Microchip Part of the MPU32 Linux team Developing second stage and third stage bootloaders](https://reader036.vdocument.in/reader036/viewer/2022081619/60fd1a6d307b3a538627df54/html5/thumbnails/43.jpg)
43
Exposure
V4L2 controls directly to the subdevice
# v4l2-ctl -L -d /dev/v4l-subdev1
User Controls
exposure 0x00980911 (int) : min=14 max=8333 step=1 default=8333 value=8260
gain 0x00980913 (int) : min=256 max=46088 step=1 default=5120 value=5120
vertical_flip 0x00980915 (bool) : default=0 value=0
![Page 44: Challenges of using V4L2 to capture and process video ...2 Eugen Hristev Embedded Linux Engineer at Microchip Part of the MPU32 Linux team Developing second stage and third stage bootloaders](https://reader036.vdocument.in/reader036/viewer/2022081619/60fd1a6d307b3a538627df54/html5/thumbnails/44.jpg)
44
Brightness
Positive brightness applied
# v4l2-ctl -L
User Controls
brightness 0x00980900 (int) : min=-1024 max=1023 step=1 default=0 value=256
flags=slider
![Page 45: Challenges of using V4L2 to capture and process video ...2 Eugen Hristev Embedded Linux Engineer at Microchip Part of the MPU32 Linux team Developing second stage and third stage bootloaders](https://reader036.vdocument.in/reader036/viewer/2022081619/60fd1a6d307b3a538627df54/html5/thumbnails/45.jpg)
45
Brightness
Negative brightness applied
# v4l2-ctl -L
User Controls
brightness 0x00980900 (int) : min=-1024 max=1023 step=1 default=0 value=-256
flags=slider
![Page 46: Challenges of using V4L2 to capture and process video ...2 Eugen Hristev Embedded Linux Engineer at Microchip Part of the MPU32 Linux team Developing second stage and third stage bootloaders](https://reader036.vdocument.in/reader036/viewer/2022081619/60fd1a6d307b3a538627df54/html5/thumbnails/46.jpg)
46
Brightness vs exposure
Why do we need brightness if we have exposure ?
Is the other way around true ?
What happens with pixel data ?
![Page 47: Challenges of using V4L2 to capture and process video ...2 Eugen Hristev Embedded Linux Engineer at Microchip Part of the MPU32 Linux team Developing second stage and third stage bootloaders](https://reader036.vdocument.in/reader036/viewer/2022081619/60fd1a6d307b3a538627df54/html5/thumbnails/47.jpg)
47
How can we adjust colors
Can we adjust the ratio between the chroma channels ?
To make it easier, use the YUV representation : adjust blue Chroma and red Chroma
V4L2 controls can alter this hardware block
![Page 48: Challenges of using V4L2 to capture and process video ...2 Eugen Hristev Embedded Linux Engineer at Microchip Part of the MPU32 Linux team Developing second stage and third stage bootloaders](https://reader036.vdocument.in/reader036/viewer/2022081619/60fd1a6d307b3a538627df54/html5/thumbnails/48.jpg)
48
Contrast
Low contrast applied
# v4l2-ctl -L
User Controls
contrast 0x00980901 (int) : min=-2048 max=2047 step=1 default=16 value=8
flags=slider
![Page 49: Challenges of using V4L2 to capture and process video ...2 Eugen Hristev Embedded Linux Engineer at Microchip Part of the MPU32 Linux team Developing second stage and third stage bootloaders](https://reader036.vdocument.in/reader036/viewer/2022081619/60fd1a6d307b3a538627df54/html5/thumbnails/49.jpg)
49
Contrast
High contrast applied
# v4l2-ctl -L
User Controls
contrast 0x00980901 (int) : min=-2048 max=2047 step=1 default=16 value=8
flags=slider
![Page 50: Challenges of using V4L2 to capture and process video ...2 Eugen Hristev Embedded Linux Engineer at Microchip Part of the MPU32 Linux team Developing second stage and third stage bootloaders](https://reader036.vdocument.in/reader036/viewer/2022081619/60fd1a6d307b3a538627df54/html5/thumbnails/50.jpg)
50
Brightness and Contrast
![Page 51: Challenges of using V4L2 to capture and process video ...2 Eugen Hristev Embedded Linux Engineer at Microchip Part of the MPU32 Linux team Developing second stage and third stage bootloaders](https://reader036.vdocument.in/reader036/viewer/2022081619/60fd1a6d307b3a538627df54/html5/thumbnails/51.jpg)
51
Inside the system
Image sensor
exposure correction
Contrast settings,
Brightness settingsResulting image
User (v4l2-ctl)
User
space
Kernel
space
V4L2 subsystem interface
Hardware pipeline
/dev/videoX
Sensor controller driver
Pixel stream Pixel stream
![Page 52: Challenges of using V4L2 to capture and process video ...2 Eugen Hristev Embedded Linux Engineer at Microchip Part of the MPU32 Linux team Developing second stage and third stage bootloaders](https://reader036.vdocument.in/reader036/viewer/2022081619/60fd1a6d307b3a538627df54/html5/thumbnails/52.jpg)
52
Summary
Digital sensors need tuning
We can use a dedicated pipeline to achieve this
We can control the pipeline using Linux and V4L2
We can have an embedded Linux Camera , with sliders, buttons as interface
![Page 53: Challenges of using V4L2 to capture and process video ...2 Eugen Hristev Embedded Linux Engineer at Microchip Part of the MPU32 Linux team Developing second stage and third stage bootloaders](https://reader036.vdocument.in/reader036/viewer/2022081619/60fd1a6d307b3a538627df54/html5/thumbnails/53.jpg)
53
Summary
Image sensor
Bayer interpolation
(edge detection)
WB correction
Gains, offsets
exposure correction
Contrast settings,
Brightness settings
Resulting image
User (v4l2-ctl)
User
space
Kernel
space
V4L2 subsystem interface
Hardware pipeline
/dev/videoX
Sensor controller driver
Pixel stream
Pixel stream
Pixel stream Pixel stream
![Page 54: Challenges of using V4L2 to capture and process video ...2 Eugen Hristev Embedded Linux Engineer at Microchip Part of the MPU32 Linux team Developing second stage and third stage bootloaders](https://reader036.vdocument.in/reader036/viewer/2022081619/60fd1a6d307b3a538627df54/html5/thumbnails/54.jpg)
54
Demo time
![Page 55: Challenges of using V4L2 to capture and process video ...2 Eugen Hristev Embedded Linux Engineer at Microchip Part of the MPU32 Linux team Developing second stage and third stage bootloaders](https://reader036.vdocument.in/reader036/viewer/2022081619/60fd1a6d307b3a538627df54/html5/thumbnails/55.jpg)
55
Questions?
![Page 56: Challenges of using V4L2 to capture and process video ...2 Eugen Hristev Embedded Linux Engineer at Microchip Part of the MPU32 Linux team Developing second stage and third stage bootloaders](https://reader036.vdocument.in/reader036/viewer/2022081619/60fd1a6d307b3a538627df54/html5/thumbnails/56.jpg)
56
Resources and Availability
⚫ Atmel ISC driver which served as a base for this presentationhttps://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/media/platform/atmel
⚫ Images from this presentation are licensed under CC-BA-SY license
⚫ Where license was not specified, some images source link was provided ⚫ Otherwise, all images are personal property of the author, and were taken with SAMA7G5-EK board with SAMA7G5 SoC,using image sensor Sony imx219 and Sony imx274
https://arindamdhar.com/wp-content/uploads/2017/07/wb-button.jpghttps://www.pathpartnertech.de/wp-content/uploads/2020/03/Figure-1.jpghttps://starecat.com/content/wp-content/uploads/light-temperature-comparison-visual-1000k-to-10000k.jpghttps://www.xrite.com/categories/calibration-profiling/colorchecker-classichttps://a.fsdn.com/con/app/proj/v4l2ucp/screenshots/228307.jpg/max/max/1https://i1.wp.com/digital-photography-school.com/wp-content/uploads/2016/01/PlusMinusButton.jpg?fit=750%2C430&ssl=1https://www.researchgate.net/publication/235350557_Combining_Gray-World_assumption_White-Point_correction_and_power_transformation_for_automatic_white_balancehttps://i.ytimg.com/vi/ZK0KX4uKhLM/maxresdefault.jpg