*

** * ** ** **

2015.6.26

Emergence in Vision: Nonlinear Models for Motion Illusion and Motion Sharpening

H. Miike, K. Otaka, A. Osa, and K. Miura Graduate School of Science and Engineering, Yamaguchi University, Tokiwadai, Ube, Japan

Emerging Phenomena in Nature

• Chemical Reaction

• Biological Self-Organization

• Convection, Tornado, Typhoon

Contents 1. Background : Emergence in Chemical Reaction 2. Emergence in Information Technology (Self-Organized Image Processing ) 3. Emergence in Vision Modeling Motion Illusions

Reproduction of Motion Sharpening 4. Conclusion

2015.6.26

1. Background: Emergence in Chemical Reaction

2015.6.26 1. Background: Emergence in Chemical Reaction

(A) Spatial-temporal Pattern Formation

Chemical Spiral Waves by Oragonator Model (A. Nomura)

Chemical Rotors in “When Time Breaks Down” by Arthur T. Winfree

Reaction-Diffusion Model for BZ-reaction • Oregonator Model (Tyson Version)

})1({12

ququfwuuuD

tu

u

)(2 wuwDtw

w

u : Activator w : Inhibitor

1entCoefficiDiffusion:wu DD

2015.6.26

1. Background: Emergence in Chemical Reaction (B) Turing Patterns (Static Pattern Formation)

Stationary Pattern Formation by Fitz-Hugh Nagumo Model with Turing Instability (A. Nomura)

S.Kondo 1995

V. Casters, P. De Kepper, et al. 1990

Reaction-Diffusion Model for Static Pattern Formation

• Fitz-Hugh Nagumo Model with Turing Condition

}))(1({12 vauuuuDtu

u

)(2 bvuvDtv

v

u : Activator v : Inhibitor

1ConditionTuring:vu DD

2015.6.26

1. Background: Emergence in Chemical Reaction (C) Reaction-Diffusion-Convection Pattern

H.Miike et al., 1987.12.24Self-Organized Oscillation of Convection in BZ-SolutionExciting Spiral Waves.

Emergence in Chemical Reaction with Hierarchical Structures: Spiral Flow Waves in Chemical Spiral Waves of BZ-Reaction, T. Sakurai et al., 2000

Spiral Flow Waves Self-Organized in the Chemical Spiral Pattern of BZ-reaction

• Rotating Direction of Hydrodynamic Flow

• Rotating Flow Waves in Spiral Shape

• Flow Waves = Synchronization of Local Nonlinear Flow Oscillator ?

. Emergence in Information Technology

2015.6.26 . Emergence in Information Technology

(A) Image Processing by BZ-Reaction ?

The pattern formation is not ordinary image processing. 1) The processed image does not settle down to a stationary pattern. 2) The pattern dynamics is rather oscillatory.

. Emergence in Information Technology (B) Self-Organized Image Processing

• A. Nomura et al., DAAAM Intern. Conf., Vienna, 1999 Solving random-dot stereogram with a reaction-diffusion

model under the Turing instability, pp.385-386

• An extended FitzHugh-Nagumo (FHN) model with the Turing condition

bvuvDtv

vauuuuDtu

v

u

2

2 11 },))(({

1,uv DD

(B) Self-Organized Image Processing (Low Level Vision)

a b c (b) Result of the proposed method a=0.1, b=1.0,

10-3, Dv/Du=8

(a) Initial u (b) Proposed (c) Marr’s method

a b c

(a) Initial u (b) Proposed (c) Ordinary method

(b) Result of the proposed method a=0.1, b=5.92,

10-3, Dv/Du=8

A. Nomura et al., 1999

(C)How to realize edge detection in a self-organized fashion? Two possible mechanisms are considered by

introducing a “Discrete FitzHugh-Nagumo Model”. (1) Importance of Discreetness in Space: dv=Dv/(��)2

(2) Necessity of Turing Condition: dv>>du?

� �

����

���

�

����

������

bvuvVddtdv

vuauuuUddtdu

v

u

4

)1)((14�

i, jTarget Pixel ui, j

Neighbour Pixel U

i

j Variables u=ui, j(t), v=vi, j(t)

(C) How to realize edge detection by the proposed model in a self-organized fashion?

Two possible mechanisms are considered: (1) Importance of Discreetness in Space: dv=Dv/(��)2

Discrete Continuous dv=2 dv=200

(a) Edge Pattern (b) Turing Pattern

(C) How to realize edge detection by the proposed model in a self-organized fashion?

Two possible mechanisms are considered: (2) Necessity of Turing Condition: dv>>du? Turing Non-Turing

dv/du=4.0 dv/du=0.67

(a) Edge Pattern 1 (b) Edge Pattern 2

Phase Diagram in d v-���pace: 2-D

Edge Point[ ]

Disappear [ ]

Edge Line [ ]

Edge Line [ 6]

Disappear [ ]

Stationary [ ]

2-D Behavior in dv -� Space

0.1 dv 500

10-1

�

10

-4

� �

����

���

�

����

������

bvuvVddtdv

vuauuuUddtdu

v

u

4

)1)((14�

Discrete Continuous

Phase Diagram in d v-���pace: 3-D

Disappear [ ]

Edge point [ ]

Edge Line [ ]

Edge Plane [ ]

Stationary [ ]

0.1 dv 500

10-1

�

10

-4

� �

����

���

�

����

������

bvuvVddtdv

vuauuuUddtdu

v

u

4

)1)((14�

3-D Behavior in dv - � Space Discrete Continuous

ST-Chaos [ ]

Summary 1: Curious Pattern Formation in a Discrete FitzHugh-Nagumo Model

• Conditions Realizing Self-Organized Edge Detection

1. Moderate Discreteness in Spatial Connection: 2<dv(du)<10, ��u<0.01 2. Turing Condition is not Essential: dv: Control Width of the Edge ��v: Control Propagation of the Wave • Spatial Temporal Chaos is Observed in 3-D Model (Difference Between 2-D and 3-D)

. Emergence in Vision

2015.6.26 . Emergence in Vision

(A) Optical illusions: http://www.ritsumei.ac.jp/~akitaoka/

Muller-Lyer, 1889

Ponzo, 1913 Herman, 1870

Mach Band

Kanizsa, 1980

Foot steps, 2001

Shepard, 1981 Chevreul illusion

Fraser and Wilcox, 1979

Understanding The Characteristics of Visual Signal Processing by Modeling Optical Illusion

Positive Aspects of Optical Illusion

• Visual Function to Recognize 3-D Real World

• New Entrance to Understand the Characteristics of Visual Signal Processing

Negative Aspects of Optical Illusion

• Visual Hallucination Caused by Mental Disease or Drug (Abnormal Perception)

• Misunderstanding or Error in Perception (Normal Perception)

a. Geometric Optical Illusions

Size Magnification

Angle Illusion

3.25

Angle Illusion of Load Lines A. Osa et al., Gland Plix in the Optical Illusion Contest , 2010

Angle Illusion of Load Lines Cues for 3-D World Recognition Enhance the Illusion

b. Peripheral Drift Illusions: Rotating Snakes

c. Motion Illusions: Footsteps Illusion

S. Anstis, Footsteps and inchworms: Illusions show that contrast affects apparent speed, Perception,Vol.30 (2001), No.7, pp.785-794

. Emergence in Vision: Motion Illusion Modeling Footsteps Illusion • An Extended FitzHugh-Nagumo (FHN) Model with

Continuous Input.

– Variables u = ui,j(t), v = vi,j(t) – Input I = Ii,j(t) – Output ui,j(t) – Initial Values ui,j(t) = vi,j(t) = 0

bvuvVddtdv

IvuauuuUddtdu

v

u

4

)1)((14

I=0.2I=0.1I=0.0

K. Miura et al., IEEJ Trans. EIS, Vol.129 (2009), pp.1156-1161

• Parameters for Input I(t) – Stripe Width10pix. Brightness 0.1 & 0.9 – Moving Object Width 23pix. Brightness 0.2 & 0.8

Results Image Sequence of Output ui,j(t)

We have to visualize intermittency of the motion.

Input I(t) Output u(t)

FHN Model

Visualization Space-Time Plot

• Visualizing Time Evolution of Motion Frame Difference Space-Time Plot

x’ x

x x’ Space

Time Spatial-Temporal Plot

Aperture Stop

Full Line Constant

Broken Line Intermittency

Time

Time Meaning of Space-Time Plot

Space

(1) Dependence on the Width of Stripe • Visual Illusion: Depends on the Width.

5 pixels 10 pixels 20 pixels

(2) Dependence on the Brightness of the Moving Object • Visual Illusion: Depends on the Contrast Change

Brightness 0.2 Brightness 0.5 Brightness 0.8

Higher Contrast Brings Clear Intermittency.

Discussion Cause of Motion Illusion • Interpretation by Sunaga et al.(2008) – Dislocation Illusion Induced by the Stimulus Image

is the Main Cause of the Motion Illusion.

• Characteristics of FitzHugh-Nagumo Model – Smoothing Effect controlled by �du – Contrast Enhancement Threshold Firing

Characteristics of FitzHugh-Nagumo Model Enhance the Dislocation Illusion.

� �

����

���

�

����

�������

bvuvVddtdv

IvuauuuUddtdu

v

u

4

)1)((14�

Summary 2 Footsteps Illusion

• Characteristics of Footsteps Illusion are Reproduced by the Extended FitzHugh-Nagumo Model.

• Reproduced Characteristics: – The Less Stripe Width Brings the Less Illusion. – The More Brightness of Moving Object Brings the More

Intermittency of the Motion. – Brightness Change Appears in the Moving Object.

Emergence in Vision: B Motion Sharpening (Higher Level Vision) • Best Example of Motion Sharpening:

Faint Pattern of Spiral Flow Waves is Clearly Imaged in Our Vision System. How to visualize the spiral flow waves ?

Motion Sharpening

Moving Object Looks Sharp than Still Image. Ramachandran et al.(1974), Bex et al. (1995), Hammett & Bex (1996)

Ordinary USM in Space

Proposed USM in Time

Image Sharpening Algorithm:

• Unsharp Masking in Space (USM)

• Unsharp Mask= Smoothing + Reversal Processing (negative to positive) • Convolution Carnell for USM:

(a) Original Image (b) Unsharp Mask (C) Sharpening Image

-1 0

-1 5 -1

0 -1 0

The reason why USM brings sharp perception

• Chevreul Brightness Illusion (Mach Band Effect) • USM Enhances Lateral

Inhibition Function of Retina (Low Level Vision).

Chevreul illusion (Mach Band Effect)

Lateral Inhibition

USM is an Image Enhance Method Utilizing Lateral Inhibition Function Observed in Chevreul Illusion.

Visual Perception

Physical Brightness Change

Realizing Motion Sharpening:

• There is no image enhancement method utilizing motion sharpening.

• We proposed a temporal unsharp masking method (t-USM, A. Osa et al, 2009) .

• Explanation of Motion Sharpening: 1) Pääkkönen & Morgan Model: Bipolar Time Impulse Response

Function of Visual System (2001) 2) Hammett et al. Model: Magnocellular cell route in LGN* is

sensitive to motion but saturates easily to signal intensity (2004).

* Retina � LGN (Lateral Geniculate Nucleus) ���Visual Cortex ���Brain

Pääkkönen & Morgan : Bipolar Impulse Response Model

Impulse Response of Visual System h(t)

h(t)=Positive Gaussian Negative Gaussian

Test Stimulus

Test Stimulus

Perceived Stimulus

Perceived Stimulus

Pääkkönen & Morgan 2001

Experiment Data Fitting Function

�1 Small �2 Large Edge Sharpening

Edge Blurring

Proposed Method: Temporal USM based on the Visual Model by Pääkkönen & Morgan

f(x, y, t)f(ff x(( , y, t) �T : Time Window

� ����

��

�2

2,,

11,,

T

Tjav jtyxf

Ttyxf

�

�� � � � �tyxftyxftyxf avt ,,,,1,,1 �� ���

A Simplified Impulse Response Function i(t)

i(t)=Positive Gaussian Negative Gaussian

tFrame

Time

t�T+1: Time Window

(1+�)f(x,y,t) A Temporal Filter Equivalent to the Proposed Method: t-USM

�tyxfav ,,��

i(t)

�1 Small �2 Large

�1

�2

(1+((1

Result 1: Spiral Flow Waves in BZ Reaction

(a) Original Image Sequence (b) Filtered Image Sequence by t-USM

Result 2: Fly-Through Image Sequence

• �T=4 Frames: Time Window���������� ��������

• Mountain and landform become clear. • Natural Preservation of 3-D Feeling.

USM Original

T-USM

Edge Sharpness is increasing. Contrast is Enhanced.

Summary 3 Motion Sharpening

• Image Sequence Sharpening and Enhancement is Realized Based on Motion Sharpening.

• Characteristics of the Proposed Method: – There is no enhancement to still object in the sequence. – Depending on the shape of edge and moving velocity,

edge sharpening and contrast enhancement are realized. – Moving objects hard to detect in the respective still

image can be enhanced .

Next Subject = Finding Nonlinear Model to Realize Motion Sharpening in a Self-Organized Fashion

Conclusion

2015.6.26

. Conclusion • We have proposed discrete FitzHugh-Nagumo models.

The models show curious characteristics as follow. 1) Edge detection and figure-ground separation are realized

in a self-organized fashion. 2) Footsteps illusion and its psychological features are

reproduced by extending the discrete FHN-model. • Considering physiological mechanism of motion sharpening

we proposed an algorithm to reproduce the phenomena. • Thus, we believe that complex phenomena observed in

our visual system are explained by nonlinear dynamics. This can be “Emergence in Vision”.

Emergence in Vision: Nonlinear Models for Motion Illusion and Motion Sharpening

� �

����

���

�

����

�������

bvuvVddtdv

IvuauuuUddtdu

v

u

4

)1)((14�

Related Studies on Visionary Emergence: Understanding Visual Processing and Visual Illusions Based on Nonlinear Sciences. • L. Kuhnert et al., Image processing using light-sensitive chemical waves,

Nature, 337(1989), pp.244-247 • E. Ueyama et al., Figure-ground separation from motion with reaction-difuson

equation, IEICE, J81-D- (1998), pp.2767-2778 • A. Nomura et al., Solving random-dot stereograms with a reaction- diffusion

model , in Proc. 10th International DAAAM 1999, p.385 • N.G. Rambidi et al., Image processing using light-sensitive chemical waves,

Physics Letters A, 298(2002), pp.375-382 • A. Nomura et al., Realizing visual functions with the reaction-diffusion

mechanism, J. Phys. Soc. Japan, 72(2003), pp.2383-2393 • M. Ebihara et al., Image processing by a discrete reaction-diffusion system, in

Proc. Third IASTED Int. Conf., 1(2003), pp.378-385 • K. Miura et al., Self-organized feature extraction in a three-dimensional

discrete reaction-diffusion system, Forma, 23(2008), pp.19-23 • A. Nomura et al., Reaction-diffusion algorithm for stereo disparity detection,

Machine Vision and Applications, 20(2009), pp.175-187 • K. Miura et al., A simulation of the footsteps illusion using a reaction diffusion

model, IEEJ, 129(2009), pp. 1156-1161