Dynamical systems,
information and time series
Stefano Marmi
Scuola Normale Superiore
http://homepage.sns.it/marmi/
Lecture 1- European University Institute
September 18, 2009
• Lecture 1: An introduction to dynamical systems and to time series. Periodic and
quasiperiodic motions. (Tue Jan 13, 2 pm - 4 pm Aula Bianchi)
• Lecture 2: A priori probability vs. statistics: ergodicity, uniform distribution of
orbits. The analysis of return times. Kac inequality. Mixing (Sep 25)
• Lecture 3: Shannon and Kolmogorov-Sinai entropy. Randomness and deterministic
chaos. Relative entropy and Kelly's betting. (Oct 9)
• Lecture 4: Time series analysis and embedology: can we distinguish deterministic
chaos in a noisy environment? (Oct 30)
• Lecture 5: Fractals and multifractals. (Nov 6)
Sept 18, 2009Dynamical systems, information and time
series - S. Marmi2
References:
• Fasano Marmi "Analytical Mechanics" Oxford University Press Chapter 13
• Benjamin Weiss: “Single Orbit Dynamics”, AMS 2000.
• Daniel Kaplan and Leon Glass: "UnderstandingNonlinear Dynamics" Springer (1995)
• Sauer, Yorke, Casdagli: Embedology. J. Stat. Phys. 65 (1991) 579-616
• Michael Small "Applied Nonlinear Time SeriesAnalysis" World Scientific
• K. Falconer: "Fractal Geometry - MathematicalFoundations and Applications" John Wiley,
The slides of all lectures will be available at my personal webpage: http://homepage.sns.it/marmi/
Sept 18, 2009Dynamical systems, information and time
series - S. Marmi3
An overview of today’s lecture
• Dynamical systems
• What is ergodic theory? From statistics to probability
• Determinism and randomness
• Statistical induction, backtesting and black swans
• Information, uncertainty and entropy
• Risk management and information theory
Sept 18, 2009Dynamical systems, information and time
series - S. Marmi4
Dynamical systems
• A dynamical system is a couple (phase space, time evolution law)
• The time variable can be discrete (evolution law = iteration of a map) or continuous (evolution law = flow solving a differential equation)
• The phase space is the set of all possible states (i.e. initial conditions) of our system
• Each initial condition uniquely determines the time evolution (determinism)
• The system evolves in time according to a fixed law (iteration of a map, differential equation, etc.)
• Often (but not necessarily) the evolution law is not linear
Sept 18, 2009Dynamical systems, information and time
series - S. Marmi5
Period 20
02 →04 →08 →16 →32 →64 →28 →56 →24 →48 →96 →92 →84 →68 →36 →72 →44 →88 →76 →52 →04 →08 →16 →…
Sept 18, 2009Dynamical systems, information and time
series - S. Marmi8
Not quite periodic…
2 →4 →8 →1 →3 →6 →1 →2 →5 →1 → 2 →4 →8 →1 →3 →6 →1 →2 →5 →1 → 2 →4 →8 →1 →3 →6 →1 →2 →5 →1 → 2 →4 →8 →1 →3 →6 →1 →2 →5 →1
→ 2 →4 →8 →1 →3 →6 →1 →2 →5 →1 → 2 →4 →8 →1 →3 →7→1 →2 →5 →1 →
Sept 18, 2009Dynamical systems, information and time
series - S. Marmi9
Quasiperiodicity
The sequence
2 →4 →8 →1 →3 →6 →1 →2 →5 →1 → 2 →4 →8 →1 →3 →6 →1 →2 →5 →1 → 2 →4 →8 →1 →3 →6 →1 →2 →5 →1 → 2 →4 →8 →1 →3 →6 →1 →2 →5 →1 → 2 →4 →8 →1 →3 →6 →1 →2 →5 →1 → 2 →4 →8 →1 →3 →7→1 →2 →5 →1 →
is the result of a symbolic analysis of the iteration of the map
log xn = log xn−1 + log 2 (mod 1) ,
Since log10 2 = 0.3010…. ≈ 3/10 for a long time the sequenceseems to be periodicSept 18, 2009
Dynamical systems, information and time series - S. Marmi
10
Quasiperiodic dynamics
id f knFor some sequence kn
Sept 18, 2009 11Dynamical systems, information and time
series - S. Marmi
• Quasiperiodic = periodic if precision is finite, but the period→∞ if the precision of measurements is improved
• More formally a dynamics f is quasiperiodic if
return times
ff1nk
Renormalization
approach
1nk
The simplest dynamical systems
• The phase space is the circle: S=R/Z
• Case 1: quasiperiodic dynamics
θ(n+1)=θ(n)+ω (mod 1)
(ω irrational, for example
θ=(√5-1)/2=0.618033989…)
• Case 2: chaotic dynamicsθ(n+1)=2θ(n)(mod 1)
Sept 18, 2009Dynamical systems, information and time
series - S. Marmi12
ω
ω
T(θ) = θ +ω (mod 1).
phase space X = [0, 1)
T : θ → 2 θ (mod 1).
13Dynamical systems, information and time
series - S. MarmiSept 18, 2009
Sensitivity to initial conditions
• « ... For, in respect to the latter branch of the
supposition, it should be considered that the most
trifling variation in the facts of the two cases might
give rise to the most important miscalculations, by
diverting thoroughly the two courses of events, very
much as, in arithmetic, an error which, in its own
individuality, may be inappreciable, produces, at
length, by dint of multiplication at all points of the
process, a result enormously at variance with truth.
... »
Sept 18, 2009 15Dynamical systems, information and time
series - S. Marmi
The mystery of Marie Roget, 1843
Edgar Allan Poe (1809-1849)Sept 18, 2009 16
Dynamical systems, information and time series - S. Marmi
Sensitivity to initial conditions
For the doubling map on the circle (case 2) one has
θ(N)- θ’(N)=2N (θ(0)- θ’(0)) even if the initialdatum is known with a 10 digit accuracy, after 40 iterations one cannot even say if the iterates are largerthan ½ or not
In quasiperiodic dynamics this does not happen: for the rotations on the circle one has θ(N)- θ’(N)= θ(0)- θ’(0)
and long term prediction is possible
The dynamics of the doubling maps is heterogeneous and unpredictable, quasiperiodic dynamics is homogeneousand predictable
Sept 18, 2009Dynamical systems, information and time
series - S. Marmi17
Heterogeneous dynamics of the doubling map
If θ(0) = 1/3 = 0.333.., θ(n) = θ(0) for all n: stationarity
If θ(0) = 3/11 = 0.272727.. θ(2n) = θ(0)
θ(2n+1) = θ(1) = 0.72727..
Period 2
If θ(0) = 1/7 = 0.142857142857.. Period 6
If θ(0) = p - 3 = 0.1415926.. , aperiodic, random…. (?)
Sept 18, 2009 18Dynamical systems, information and time
series - S. Marmi
Chaotic dynamics
Sensitive dependence on initial conditions
Density of periodic orbits
Some form of irreducibility (topological transitivity,
ergodicity, etc.)
Information is produced at a positive rate: positive entropy (Lyapunov exponents)
Sept 18, 2009 19Dynamical systems, information and time
series - S. Marmi
Topological transitivity
A dynamical system is topologically transitive if it hasa dense orbit.
In the case of the doubling map: x(n+1)=2x(n) (mod 1)
One observes that it corresponds to a shift on the binarydigits of : if x(n)=Σa(n,k)2-k then
x(n+1)= Σa(n,k)2-(k-1)
The orbit ofx=.01000110110000010100111001011101110000000100100011010001010110011110001001101010111100110111101111…is dense
Sept 18, 2009 20Dynamical systems, information and time
series - S. Marmi
Ergodic theory
The focus of the analysis is mainly on the asymptotic ditribution of the orbits, and not on transient phenomena.
Ergodic theory is an attempt to study the statistical behaviour of orbits of dynamical systems restricting the attention to their asymptotic distribution.
One waits until all transients have been wiped off and looks for an invariant probability measure describing the distribution of typical orbits.
Sept 18, 2009Dynamical systems, information and time
series - S. Marmi21
Ergodic theory: the setup (measure
preserving transformations, stationary
stochastic process)
X phase space, μ probability measure on X
Φ:X → R observable, μ(Φ) = ∫X Φ dμ expectation valueof Φ
A measurable subset of X (event). A dynamics T:X→X induces a time evolution:
on observables Φ → Φ T
on events A →T-1(A)
T is measure-preserving if μ(Φ)= μ(Φ T) for all Φ, equivalently μ(A)=μ(T-1(A)) for all A
Sept 18, 2009 22Dynamical systems, information and time
series - S. Marmi
Measure theory vs. probability
theory
Sept 18, 2009Dynamical systems, information and time
series - S. Marmi23
Birkhoff theorem and ergodicity
Birkhoff theorem: if T preserves the measure μ then
almost surely the time averages of the observables
exist (statistical expectations). The system is ergodic
if these time averages do not depend on the orbit
(statistics and a-priori probability agree)
Sept 18, 2009Dynamical systems, information and time
series - S. Marmi24
Law of large numbers:
Statistics of orbits =
a-priori probability
“Historia magistra vitae” or the
mathematical foundation of backtesting
Even without assuming ergodicity, Birkhoff
theorem shows that:
• Time averages exist and they give rise to an
experimental statistics to compare with theory
• Past and future time averages agree almost
everywhere
Sept 18, 2009Dynamical systems, information and time
series - S. Marmi25
Ergodicity and time averages appliedto the powers of 2
Ergodicity of irrational rotations implies that in the sequenceof the most significant digit of the powers of 2
Sept 18, 2009Dynamical systems, information and time
series - S. Marmi26
7 appears more frequently than 8 even if looking at the first iterates one would guess otherwise
The doubling map and coin tosses
Two measure preserving dynamical systems
which are conjugate by a measurable
isomorphism have the same orbits statistics
…The doubling map x→2x (mod 1) and the
Bernoulli scheme B(1/2,1/2) have the same
probabilistic properties
…But the first is clearly deterministic
Sept 18, 2009Dynamical systems, information and time
series - S. Marmi27
Determinism, randomness and the
physical laws (T. Tao, http://terrytao.wordpress.com/2007/04/05
/simons-lecture-i-structure-and-randomness-in-fourier-analysis-and-number-theory/)
It may well be that the universe itself is completely deterministic (though this
depends on what the “true” laws of physics are, and also to some extent on
certain ontological assumptions about reality), in which case randomness is
simply a mathematical concept, modeled using such abstract mathematical
objects as probability spaces. Nevertheless, the concept of pseudorandomness
- objects which “behave” randomly in various statistical senses - still makes
sense in a purely deterministic setting. A typical example are the digits
of π=3.1415926535897932385…this is a deterministic sequence of digits, but
is widely believed to behave pseudorandomly in various precise senses (e.g.
each digit should asymptotically appear 10% of the time). If a deterministic
system exhibits a sufficient amount of pseudorandomness, then random
mathematical models (e.g. statistical mechanics) can yield accurate predictions
of reality, even if the underlying physics of that reality has no randomness in it.28
Dynamical systems, information and time series - S. Marmi
Sept 18, 2009
Truely random?
1 0 1 1 0 0 1 1 1 0 1 0 1 0 0 0 0 0 1 1 1 1 0 0 1 1 1 0 1 1 0 0 0 0 1 0 1 1 0 0 1 0 0 1 1 1 1 0 1 1 0 1 0 0 1 0 0 0 1 1 1 0 1 0 0 0 0 1 0 1 0 0 1 0 1 0 0 00 0 0 0 1 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 1 0 0 0 1 0 0 1 1 1 0 0 0 0 1 1 0 0 1 0 0 1 1 1 1 1 0 0 0 1 0 0 0 1 0 1 1 1 1 1 1 1 1 0 0 0 0 0 1 1 0 1 1 0 0 0 1 1 1 1 0 0 1 0 1 0 1 1 1 0 1 0 1 0 1 1 0 1 0 0 1 0 0 0 0 1 0 0 0 1 0 0 1 0 1 0 0 1 0 0 0 0 0 0 1 1 1 1 1 1 0 1 0 0 0 0 0 0 1 1 0 1 0 1 0 0 0 0 0 1 0 0 0 1 1 0 0 0 0 0 1 0 0 1 1 0 1 1 1 1 1 0 0 0 1 0 1 0 1 0 0 1 1 0 0 1 0 1 1 0 1 0 1 0 1 0 1 1 0 1 1 1 1 0 0 1 1 0 0 0 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 0 0 01 0 0 1 0 1 1 1 0 1 1 1 0 1 1 0 0 0 1 1 1 0 0 1 1 1 1 0 1 1 0 0 1 1 0 1 0 1 1 1 1 1 0 1 1 1 1 1 1 1 0 0 1 0 0 1 0 1 1 0 0 0 0 1 0 1 0 1 1 0 1 1 1 1 1 1 1 10 1 1 0 0 1 0 1 1 1 1 0 1 1 0 1 1 1 1 0 1 0 0 0 1 0 0 1 1 1 0 0 1 1 1 0 1 0 1 0 1 1 1 0 1 0 1 0 0 1 1 1 1 0 0 1 0 0 1 0 0 0 0 1 1 1 1 0 1 0 0 0 0 0 1 1 1 11 0 1 0 0 0 0 0 0 0 1 1 0 1 0 1 0 1 1 0 1 1 1 0 1 0 0 1 0 1 0 0 1 1 1 0 1 1 1 1 1 1 1 0 0 0 1 0 0 0 0 1 0 0 1 1 1 0 1 1 1 0 0 0 1 0 1 0 0 1 1 0 1 1 0 0 0 01 1 0 1 1 0 1 0 0 0 1 1 0 1 0 1 1 0 1 1 1 1 0 0 0 1 1 1 0 1 1 1 0 0 0 0 0 0 0 0 1 1 1 1 0 1 1 1 1 1 1 0 1 0 1 1 1 0 1 0 1 0 1 0 1 1 1 0 0 0 0 1 0 1 0 1 1 11 0 1 1 0 1 0 0 1 0 0 0 1 1 0 0 1 1 0 0 0 1 1 0 1 0 1 0 1 1 1 1 1 0 0 1 0 0 0 0 1 1 1 0 0 1 0 1 1 1 0 1 1 1 0 1 0 1 0 0 0 1 1 0 1 0 1 0 1 1 0 0 0 1 0 0 1 0 1 1 0 1 0 0 0 0 1 1 0 0 1 1 0 0 0 1 1 1 1 1 1 1 1 0 0 1 0 0 0 1 0 1 0 0 1 1 1 1 0 1 1 1 0 1 1 1 0 1 1 1 1 0 1 0 0 0 0 0 1 1 0 1 0 0 0 0 1 1 0 0 1 0 1 1 1 11 0 0 1 1 1 0 0 0 0 0 0 1 0 0 1 1 0 0 0 0 0 1 0 0 0 1 1 0 1 0 0 0 0 1 1 1 0 0 1 0 1 1 1 1 1 0 0 1 0 1 0 0 1 0 0 1 1 0 1 1 0 0 1 0 0 1 0 1 0 0 0 1 0 0 1 1 11 0 0 1 0 0 0 1 1 0 0 0 1 1 0 1 0 0 0 1 1 1 0 1 0 0 1 1 0 1 0 1 1 0 1 1 1 0 1 0 1 1 1 0 0 1 0 0 0 1 0 0 1 0 1 0 0 0 0 1 0 0 1 0 0 0 0 0 0 0 0 1 1 1 1 0 0 01 0 1 1 1 0 0 0 1 0 1 0 1 1 0 0 1 0 1 0 0 0 0 0 0 0 1 1 1 1 1 1 0 0 0 1 0 0 0 1 0 0 1 0 0 0 1 1 1 0 0 0 0 0 0 0 1 0 0 1 1 0 0 0 0 0 0 0 0 1 0 0 1 0 1 0 0 00 1 0 1 0 1 0 0 0 1 0 1 0 1 1 1 0 0 1 0 0 1 0 1 1 1 1 1 0 1 1 1 1 0 0 1 0 0 0 0 1 0 0 0 1 1 0 0 0 0 0 0 0 1 0 0 1 1 1 0 0 0 0 0 1 0 1 1 0 0 1 0 1 1 1 1 0 00 0 0 1 0 1 0 1 0 0 1 0 0 1 1 0 0 1 0 0 0 1 1 1 0 0 0 0 1 0 1 1 1 1 1 1 1 0 0 1 1 1 1 1 1 1 1 1 1 1 0 1 1 1 1 1 0 1 0 0 1 1 0 0 0 0 1 0 1 1 0 0 0 0 0 0 0 1 0 0 0 0 1 1 0 0 0 1 0 0 1 1 1 0 0 0 0 1 1 0 0 0 0 0 0 1 0 0 0 0 1 1 1 1 1 1 0 0 0 0 1 1 0 1 1 0 1 1 1 0 0 0 1 1 0 1 1 1 0 0 0 1 0 1 1 0 0 1 1 1 1 1 1 0 1 10 0 1 0 0 0 1 0 0 1 0 0 0 0 1 1 0 0 1 0 0 1 1 1 0 0 0 1 1 1 0 1 0 0 1 1 0 1 0 0 0 0 1 0 0 0 0 0 0 1 0 0 1 1 0 1 1 0 1 0 1 0 1 0 1 1 0 0 1 1 1 0 0 1 0 0 0 1 0 1 0 1 0 1 1 1 0 1 0 0 1 0 1 1 0 1 0 0 0 1 0 0 0 1 1 1 1 0 0 0 0 0 1 1 1 0 0 1 1 0 0 1 0 1 0 1 1 1 0 1 1 0 1 1 0 1 0 0 0 1 0 0 0 1 0 0 0 0 0 1 0 0 1 0 1 0 0 0 1 1 1 0 1 0 0 0 0 0 1 1 0 0 0 1 0 0 0 0 1 1 1 1 1 1 0 1 1 0 0 1 0 0 1 1 0 0 0 0 1 1 0 1 0 0 1 1 1 1 1 1 1 0 1 1 1 0 1 0 1 0 0 1 1 0 1 1 1 1 1 0 1 1 1 10 0 0 0 1 1 0 0 0 0 1 1 1 0 1 0 1 1 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 0 0 1 0 1 0 1 0 0 1 1 0 0 1 0 0 1 0 1 1 0 1 0 0 0 1 0 0 0 0 1 0 0 0 0 0 1 1 0 0 1 0 0 1 11 0 0 0 1 1 0 1 1 0 0 0 0 0 0 1 0 1 0 0 0 0 1 0 1 1 1 1 1 0 0 1 0 1 0 1 1 1 1 1 0 1 1 0 0 1 1 0 1 0 1 0 1 0 0 1 1 1 0 0 1 0 1 1 1 1 1 1 1 0 1 1 0 0 1 1 1 10 0 1 0 1 0 1 1 1 1 0 1 0 0 1 0 1 1 1 0 0 1 0 0 1 1 0 0 1 1 0 0 0 1 1 0 0 0 1 0 0 0 0 1 0 0 0 1 1 0 1 0 1 0 0 0 0 1 1 0 1 1 0 0 0 0 0 0 0 1 1 0 1 0 0 0 0 1 0 0 0 0 0 1 0 1 0 0 1 0 1 0 0 1 0 1 1 0 0 0 0 1 0 1 0 0 1 1 0 0 0 0 1 0 1 1 1 0 0 0 1 0 1 0 0 1 1 1 0 0 0 0 0 1 1 0 0 1 1 0 0 1 0 1 0 0 0 1 1 0 1 0 1 0 1 0 0 0 0 1 1 0 1 0 0 0 1 0 1 1 1 1 0 0 1 1 0 0 1 0 0 0 0 0 0 1 0 0 0 1 1 0 0 0 0 1 0 0 0 1 0 0 1 1 1 0 0 0 0 0 0 1 1 0 1 0 0 1 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 00 1 1 0 1 1 1 0 1 0 0 0 1 0 0 0 0 0 1 0 1 0 0 1 0 0 0 1 0 1 1 1 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 1 0 1 1 1 0 1 1 0 1 1 1 0 1 1 1 0 0 0 1 0 1 0 0 0 1 1 0 0 01 1 1 1 1 0 1 0 0 0 0 1 1 1 1 0 1 1 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1 1 0 1 0 0 1 0 0 1 0 1 0 0 0 0 1 0
29Dynamical systems, information and time
series - S. MarmiSept 18, 2009
www.random.org
•
True Random Number ServiceIntroduction to Randomness and Random NumbersRANDOM.ORG is a true random number service that generates randomness via atmospheric noise. This page explains why it's hard (and interesting) to get a computer to generate proper random numbers.
Random numbers are useful for a variety of purposes, such as generating data encryption keys, simulating and modeling complex phenomena and for selecting random samples from larger data sets. They have also been used aesthetically, for example in literature and music, and are of course ever popular for games and gambling. When discussing single numbers, a random number is one that is drawn from a set of possible values, each of which is equally probable, i.e., a uniform distribution. When discussing a sequence of random numbers, each number drawn must be statistically independent of the others.
30Dynamical systems, information and time
series - S. MarmiSept 18, 2009
Examples of time-series in natural and
social sciences
• Weather measurements (temperature, pressure, rain, wind speed, …) .
If the series is very long …climate
• Earthquakes
• Lightcurves of variable stars
• Sunspots
• Macroeconomic historical time series (inflation, GDP,
employment,…)
• Financial time series (stocks, futures, commodities, bonds, …)
• Populations census (humans or animals)
• Physiological signals (ECG, EEG, …)
Sept 18, 2009Dynamical systems, information and time
series - S. Marmi31
Stochastic or chaotic?
• An important goal of time-series analysis is to
determine, given a times series (e.g. HRV) if the
underlying dynamics (the heart) is:
– Intrinsically random
– Generated by a deterministic nonlinear chaotic
system which generates a random output
– A mix of the two (stochastic perturbations of
deterministic dynamics)
Sept 18, 2009Dynamical systems, information and time
series - S. Marmi32
Deterministic or truly random?
Sept 18, 2009 33Dynamical systems, information and time
series - S. Marmi
Logit and logistic
The logistic map x→L(x)=4x(1-x) preserves the probability
measure dμ(x)=dx/(π√x(1-x))
The transformation h:[0,1] →R, h(x)=lnx-ln(1-x) conjugates L
with a new map G
h L=G h
definined on R. The new invariant probability measure is
dμ(x)=dx/[π(ex/2 + e-x/2 )]
Clearly G and L have the same dynamics (the differ only by a
coordinates change)
Sept 18, 2009Dynamical systems, information and time
series - S. Marmi35
Embedding method
Sept 18, 2009Dynamical systems, information and time
series - S. Marmi36
• Plot x(t) vs. x(t- ), x(t-2 ), x(t-3 ), …
• x(t) can be any observable
• The embedding dimension is the # of delays
• The choice of and of the dimension are critical
• For a typical deterministic system, the orbit will be
diffeomorphic to the attractor of the system (Takens
theorem)
Sept 18, 2009Dynamical systems, information and time
series - S. Marmi37
Dynamics, probability, statistics and
the problem of induction
• The probability of an event (when it exists) is almost always
impossible to be known a-priori
• The only possibility is to replace it with the frequencies measured by
observing how often the event occurs
• The problem of backtesting
• The problem of ergodicity and of typical points: from a single series
of observations I would like to be able to deduce the invariant
probability
• Bertrand Russell’s chicken (turkey in the USA)
http://www.edge.org/3rd_culture/taleb08/taleb08_index.html
Bertrand Russel(The Problems of Philosophy, Home University Library, 1912. Chapter VI On Induction) Available at the page http://www.ditext.com/russell/rus6.html
Domestic animals expect food when they see the person who feeds them. We know that all these rather crude expectations of uniformity are liable to be misleading. The man who has fed the chicken every day throughout its life at last wrings its neck instead, showing that more refined views as to the uniformity of nature would have been useful to the chicken.
Sept 18, 2009Dynamical systems, information and time
series - S. Marmi38
http://www.edge.org/3rd_culture/taleb08/taleb08_index.html
Sept 18, 2009Dynamical systems, information and time
series - S. Marmi39