NEDERLANDSFORENSISCHINSTITUUT
Nederlands Forensisch Instituut Laan van Ypenburg 6, 2497 GB Den Haag
Examining Photo Response Non-
Uniformity for the Comparison of
Cameras
Zeno Geradts PhD / Maarten van der Mark BS / Wiger van Houten MS
Partially funded by the European Commission within the project FIDIS
www.fidis.net
NEDERLANDSFORENSISCHINSTITUUT
Overview• Introduction• Noise and PRNU sources• Comparison of denoising
methods• Application to YouTube videos• Performance• Statistics (Bayesian)• Conclusions
NEDERLANDSFORENSISCHINSTITUUT
FIDIS
• Network of Excellence within the European Union, with 20 partners ranging from Universities, Privacy protection agencies and companies such as IBM and Microsoft
• www.fidis.net
NEDERLANDSFORENSISCHINSTITUUT
Introduction• Each image sensor has a unique
`fingerprint’, the PRNU pattern, that is detectable in all images the sensor produces
• This `fingerprint’ is used to establish the image origin
NEDERLANDSFORENSISCHINSTITUUT
Noise and PRNU sources• Different types of noise:
• 2 Categories:I. Random noise
I. Temporally variableII. Statistical distributionsIII. Can be reduced by averaging multiple frames
II. Pattern noise (PRNU and FPN)I. Not temporally variable but systematic (spatial
variable)
• How do image sensors work?• CCD image sensors• CMOS Active Pixel Sensors
NEDERLANDSFORENSISCHINSTITUUT
CCD image sensors• Basic building block (gate)• Silicon substrate • Apply positive voltage to the gate• Electron-hole pairs generated in depletion region are confined• Read out• Charge converted to voltage at the sense node
NEDERLANDSFORENSISCHINSTITUUT
CMOS APS image sensors
• Cpd: photodiode capacitor
• M1: reset transistor• M2: source follower• M3: row select
transistor• M4: bias transistor
NEDERLANDSFORENSISCHINSTITUUT
Noise and PRNU sources (2)
• The characteristic pattern extracted from the images contains 2 systematic contributions:• Multiplicative PRNU, depends on illumination• Fixed Pattern Noise (FPN), does not depend
on illumination
• Also: common components in the extracted pattern (e.g. CFA interpolation)
NEDERLANDSFORENSISCHINSTITUUT
Noise and PRNU sources (3)
• FPN (general):• Crystal defects in the silicon
lattice introduced during growth• Impurities• The size of the detector/potential well• Contamination during fabrication• Non-uniform oxide/gate thickness• In CMOS: additional sources (each
transistor)
NEDERLANDSFORENSISCHINSTITUUT
Noise and PRNU sources (4)• PRNU
The depth of the detector/potential well• Larger active area: more incident
photons• Non-uniform oxide layer: results in non-
uniform potential wells• Deeper potential well: more photons
absorbed (wavelength dependent)• In CMOS: additional sources
NEDERLANDSFORENSISCHINSTITUUT
Image denoising – pattern extraction• Denoising algorithms do not discriminate
between noise and image details. There is always a tradeoff.
• PRNU is a nonperiodic discontinuous signal
• Pattern = Image – F(Image) , F is the denoising filter
• Correlate the patterns from `questioned’ videos with reference patterns
NEDERLANDSFORENSISCHINSTITUUT
Old method• Gaussian smoothing filter• Advantages:
• Simple• Very fast
• Disadvantages:• Distorts edge integrity• Image residue left behind in
the pattern
NEDERLANDSFORENSISCHINSTITUUT
New method• Wavelet based denoising
filter [Lukas et al]*• Disadvantages
• Slower• Diadic images only
• Advantages• Preserves edges (edge
detection)• Spatial adaptive • Works really well
*Digital Camera Identification from Sensor Pattern Noise (2005) – Lukas, Fridrich, Goljan
NEDERLANDSFORENSISCHINSTITUUT
YouTube (1)• Accepts large amount of input
formats:Xvid, DivX, WMV, 3GP, …
• Downloadable as H.264/MPEG-4 AVC with e.g. keepvid.com
• Maximum resolution (H.264): 480x360
• Aspect ratio generally does not change
NEDERLANDSFORENSISCHINSTITUUT
YouTube (2)• When the resolution of the video
uploaded to YouTube is smaller than 480x360, generally no resolution change occurs (exceptions)
• When the resolution of the video exceeds 480x360, the resolution is changed to 480x360 or lower (depending on the aspect ratio)e.g. 640x480 480x360
640x360 480x270
NEDERLANDSFORENSISCHINSTITUUT
In practice – natural video• Record 30 seconds of natural
video, recorded with XVID/WMV• Upload this video to YouTube• Download the video with
keepvid.com• Extract individual frames• Estimate noise pattern
NEDERLANDSFORENSISCHINSTITUUT
In practice – flatfield video• If natural video downloaded has a
resolution lower than 480x360, record the flatfield video (RAW) with the same resolution, use these frames
• Otherwise (e.g. 640x480), record flatfield video in native resolution, and upload (download) them to (from) YouTube• Alternative: resize flatfield video to match
the dimension of the YouTube video (e.g. 640x480 480x360)
NEDERLANDSFORENSISCHINSTITUUT
Performance (1)• Works really well when natural video is
recorded in native resolution, both with XVID and WMV9 (Messenger) with large range of codec settings, even for shorter samples
• Works reasonably well when aspect ratio has been changed
• Problematic with very low resolution (Vodafone: 176x144)
• Does not always work when the video was binned during recording (e.g. native resolution of 640x480, recorded in 320x240x)
NEDERLANDSFORENSISCHINSTITUUT
Performance (2) – Creative Live! IM Webcam
• Problem: need to set two parameters for extracting the patterns
• These parameters depend on a large amount of variables: • Content• Compression• Codec• Resolution• etc.
• Only from empirical data; impossible in casework
NEDERLANDSFORENSISCHINSTITUUT
Comparison – Creative Live! IM, 640x480 native, recorded in 352x288 • Old method
(σ=0.6, threshold 5, 4x4 averaging):Camera Correlatio
n
11.5 0.2048
11.6 0.1893
11.4 0.1669
11.1 0.1273
11.2 0.1191
11.7 0.1061
New Method (σ_n=4.5, σ_f=4.5):
Camera Correlation
11.4 0.0531
11.1 0.0271
11.2 0.003
11.7 -0.0106
11.5 -0.0237
11.6 -0.0263
NEDERLANDSFORENSISCHINSTITUUT
Logitech Communicate STX (1)• Logitech :
640x480 (native), XVID q4
• Logitech : 640x480 (native), variable XVID quality
NEDERLANDSFORENSISCHINSTITUUT
Logitech Communicate STX (2)• Logitech :
320x240, XVID q4
• Logitech : 320x240, variable XVID quality
NEDERLANDSFORENSISCHINSTITUUT
Test yourself sourceforge.net search PRNU
NEDERLANDSFORENSISCHINSTITUUT
Likelihood ratioLikelihood ratio
•Comparing patterns•Value of the evidence:
NEDERLANDSFORENSISCHINSTITUUT
Likelihood ratioLikelihood ratio• In practice:
• No overlapping histograms• We do not know the origin of the questioned
image
NEDERLANDSFORENSISCHINSTITUUT
Likelihood ratioLikelihood ratio•To artificially find the pdf of Hd at Hp
we use an estimator•Example: add-constant estimatorSuppose you look at the colours of passing cars:• 1 blue car, 1 red car, and 1 green car• Q: what is the chance the next car is blue?• A: 1/3? Or smaller?• A: (1+1)/7 for blue; 1/7 for a new colour
NEDERLANDSFORENSISCHINSTITUUT
Likelihood ratioLikelihood ratio
•More complicated estimators available, but the essence is the same: reserve a probability for unseen species, based upon the data we encountered so far
•Simple Good-Turing approach
NEDERLANDSFORENSISCHINSTITUUT
Likelihood ratioLikelihood ratio
•SGT: problems:• Car colours (green, blue, red, etc) is
discrete, but some colours exist that can be seen as both (blue or green?)
• We are considering correlation values, continuous data. We have to divide the values into `species’ ourselves
NEDERLANDSFORENSISCHINSTITUUT
Likelihood ratio, recapLikelihood ratio, recap
NEDERLANDSFORENSISCHINSTITUUT
Conclusion (1)Conclusion (1)
•We understand the origins of the pattern noise that is used to perform device identification
•We understand which random noise sources are present
•A likelihood ratio can be found, but there are some unanswered questions before this approach can be used
NEDERLANDSFORENSISCHINSTITUUT
Conclusions (2)• It is possible to identify the source
video camera based on the PRNU pattern, even after the video has been uploaded to YouTube
• The new method, although computationally more intensive, performs much better
NEDERLANDSFORENSISCHINSTITUUT
Conclusions (3)• Limitations: (very) low resolution
videos, changed aspect ratios, ‘subsampled’ recordings
• No way to find the ideal parameter directly from the video or frames: this is a problem when the abovementioned limitations are met
NEDERLANDSFORENSISCHINSTITUUT
Future research
• Using the methods on databases of child pornography to link cameras
• Improvement of algorithms• Validation in casework