1

Introduction to Stochastic Processes and Markov Chain

Prof. Dr. Md. Asaduzzaman Shah Department of Statistics

University of Rajshahi

2

Variable

Values varies from individual to individual

Stochastic Variable

Families of random variable

3

One dimensional process

Classified into four types of processes

(i) Discrete time, discrete state space (ii) Discrete time, continuous state space (iii) Continuous time, discrete state space (iv) Continuous time, continuous state space

4

Discrete time Discrete state space

5

Examples

A fair coin is tossed n times and the outcomes represented by Xn. Here, Toss No. treated as ‘time’ and the result/outcome considered as ‘space’ of the experiment. Both time and space are discrete.

Toss No. 1 2 3 … n-1 n

Outcomes T H H … H T

6

Experiment and Its Outcomes

Suppose a fair coin is tossed twice. Sample space and four simple events are

4321 ,,,,,, TTTHHTHH

TTandTHHTHH 4321 ,,

7

Counting Heads

0)()(

1)()(

1)()(

2)()(

4

3

2

1

TTXX

THXX

HTXX

HHXX

8

Probability

Now the values 0,1 and 2 have probabilities ¼, 2/4=1/2 and ¼. So, X(t) = X(ω) = 0,1,2 are stochastic variable with t=0,1,2.

9

Discrete time Continuous state space (Velocity of a car in a time interval (0,t)

10

DC

In a stochastic process {X(t), t€T} in which time parameter, t is discrete, however, the state space, velocity X(t), which is increasing or decreasing is continuous.

Time(Road) 10:00:00 12:30:00 18:45:00 24:00:00

Velocity 60m/h 45m/h 26m/h 70m/h

11

Continuous time, discrete state space

12

CD

The number of passengers waiting at Dhaka Bus Terminal, Rajshahi on Friday, August 09, 2011 from (10-11 AM).Time and tide wait for none(Continuous).

Time 10:00 10:15 10:20 11:00

Passengers

waiting460 345 220 370

13

More examples

Long Route Buses: Hanif, National , Green Line, etc. that’s are started in a particular time.

Patients are admitted at Rajshahi Medical College Hospital (RMCH) in a particular day.

Customer’s are waiting in a super-market for getting Eid Fashion

14

Continuous time Continuous state space

15

CC

If we measure the temperature of a certain place in a time interval (0,t) for each day is an example of CTCSS, because time interval is continuous as well as temperature which is not fixed, it is rising/falling following a day follows morning to noon, noon to evening, evening to night, so state space is also continuous.

16

Loss variables character

** Making Laugh to see the teeth (^^^^^^) instead of monitoring the defective teeth

** Slapping to see the number of eye drops (weeping),

17

Notations

Discrete random variable represented by Xn or X(n).

Continuous random variable indicated by Xt or X(t).

Stochastic processes are defined by the notations { Xn} and {X(t)}

18

Differences

Discrete Continuous

Family represented

by {Xn, n=0,1,2,…,} Family represented by {Xt, t€T} or

{X(t), t€T}, T is finite/infinite.

Parameter ‘n’ Parameter ‘t’ interpreted as time, some times it represents characters like as, (i) distance, (ii) length, (iii) thickness and so on.

Stochastic sequence Stochastic process

19

At a Glance

Xn

X(t)

Xn

Time

X(t)

Thrown/Toss No.

Distance/Length/

Thickness.

Outcomes of ExperimentOr Space

20

Relationship

In some of the cases, the r.v Xn, i.e. members of the family {Xn, n>=0} are mutually independent, but more often they are not independent. We generally come across Processes whose members are mutually dependent.

Throwing a fair coin 20 times by 20 peoples, outcomes will be different instead of, a person will throw the coin 20 times. The relationship among them is often of great importance.

The nature of dependence could be infinitely varied. Dependence of such special types, which occurs quite often and is

of great importance. According to the nature of dependence relationship existing among

the members of the family, we may describe or mathematically formulate them by some Stochastic Processes.

21

Stationary Processesstochastic process { Xn, n>=1 } is time independent (not necessary hours, months, years) i.e. covariance stationary.

Let Xn, n>=1 be uncorrelated random variables with mean 0 and variance 1.

nmifnmif

mn

mnmnmn

XXE

XEXEXXEXXmnC

01

),(

)(.)(),(,cov,

22

1. Poisson Process

Consider the process {X(t), t€T} with

;...;3,2,1,0,0,!

)()(Pr

nn

tentX

nt

23

(i) the mean function and (ii) the variance

ttXEtm )()(

ttX )(var

24

Non-Stationary

Both are functions of t. Therefore, the stochastic process {Xn, n>=1} is non-stationary/evolutionary. The distribution (Poisson) of the stochastic process {Xn, n>=1} is functionally dependent on t.

25

2. Mean and Co-variance

tddtDDEtXEtm 2121)()(

222

212

11

22

2221

21

21

2221

21

2121

))(()()(

)())(()(

)()(

)()()()(

))(()()(

DEDVartimesofproduct

ddtimesofsumDEDVar

dstddtsd

DEstDDEtsDE

sDDtDDEsXtXE

26

Variance

)(2

))(()(var)(

))(()()(var

22

22

221

21

21

221

22

221

22

22

dtdtddtddt

tXEtXtXE

tXEtXEtX

27

The Process is non-stationary, i.e. evolutionary

22

21)(),(cov),( tssXtXtsC

28

3. Expection

2210)( tDtDDtX

0.00.0)()()()]([ 22

210 ttDEtDEtDEtXE

29

Covariance

22

2210

2210

2210

2210

1

)()()})({(

))(()).(()}()({)(),(

stts

sDsDDEtDtDDEsDsDDtDtDDE

sXEtXEsXtXEsXtXCov

30

Variance

422

41

20

2210 1)()()()var{)}({ ttDvtDvtDvtDtDDtXVar

31

Mean stationary but Process non-stationary

Therefore, the stochastic process, X(t)=D0 + D1 t + D2 t2, where Di, i=0,1,2 are

uncorrelated random variables with mean 0 and variance 1, is dependent on the time parameter t and s, so it is non-stationary. But the mean of the process is stationary.

32

Briefly: Time, also treated as distance, length, thickness etc.

StationaryTime- Independent

Non-Stationary

Time- Dependent

33

Problem

Manufacturers A and B are competing with each other in a restricted market. Over the year, A’s customers have exhibited a high degree of loyalty as measured by the fact that customers using A’s product 80 percent of the time. Also former customers purchasing the product from B have switched back to A’s 60 percent of the time.

(a) Construct and interpret the state transition matrix terms of (i) retention and loss, (ii) retention and gain.

(b) Calculate the probability of a customer purchasing A’s product at the end of the second period, Draw the transition probability diagrams and the transition trees.

34

State Transition P

Current Purchase

(n=0)

Next Purchase (n=1)

A B

A 0.8 0.2

B 0.6 0.4

35

Interpretation of Conditional Probability

The probability of a customer’s purchase at the next step (n=1) depends upon the product that he bought previously (n=0), i.e., current purchase.

36

Retention and Loss (Rows of P),

P11= 0.80

means that a customer now using A’s product will again purchase A’s product at the next purchase in 8 of 10 times. This implies retention to A’s product.

P12= 0.20

means that the customer now using A’s product will switch over to B’s product at the next purchase in 2 pot of 10 times. This implies loss of A’s product.

37

Retention and Gain (Columns of P),

P11= 0.80

means that the customer now using A’s product will again purchase A’s product next time in 8 out of 10 times. This implies retention of A’s product.

P12= 0.60

shows that the customer now using B’s product will purchase A’s product next time in 6 out of 10 times. This implies gain to A’s product.

38

Second Row and Column

A similar interpretation holds for the second row of P

The second column of P can be interpreted similarly.

39

Transition Probability Diagram

0.80 A B 0.40

0. 200.

60

40

Transition Trees

A

A

B

A

A

B

B

0.80

0.20

0.80

0.20

0.40

0.60

41

QUESTIONNAIRE ON PROFESSIONS (Research Project & Field Studies)

•AgricultureBusinessServiceOthers

42

Generations and their professions

(Please put tick () mark on the Boxes)

Agriculture

Service

Others

Business

Great Grand FatherGrand Father

Father

43

Raw data table

ProfessionsGreat Grand Father Grand Father Father

A B S O A B S O A B S O

X X X X X X X X X

X X X X X X X X X

X X X X X X X X X X X X X X X X X X

X X X X X X X X X

44

NUMERICAL FIGURES IN DIFFERENT STATES OF CLASSES

20062421149Total

165434O

60501S

1810116B

1600157138A

OSBA

TotalG G Father

G Father

Pro

fess

ions

Professions

45

ESTIMATED TRANSITION PROBABILITY

A B S O

A 0.8625 0.0437 0.0937 0.0000

B 0.3333 0.6111 0.0000 0.0555

S 0.1666 0.0000 0.8333 0.0000

O 0.2500 0.1875 0.2500 0.3125

G G Father

G Father

Pro

fess

ions

Professions

46

EXPECTED STAY IN EACH SOCIAL CLASS AND PRAISE MEASURE OF SOCIAL MOBILITY

1.4904 1.0064 1.4999 Others

1.82552.73905.0000 Service

2.2702 1.21152.7503 Business

2.2854 1.2253 2.8003 Agriculture

Class )1(

1)(

ijj p

E

)1(

1)(

jj p

E

)(

)(

j

jj E

E

47

COMMENT

Last column of the above Table states that the generation of Great Grand Father of Grand Father, the agriculture community has most tendency to adjust their children to agriculture. Then comes the Business community and third after Agriculture and Business community comes the Service community. Finally comes the others community.

48

Bartholomew Model

k

j

k

iiji jipf

1 1

49

This measure depends on which generation we choose as our base. Replace fi by pj we get

measure of social mobility given by

10,1 1

DjippDk

j

k

iijj

50

Conditions

D = 0,

the society is perfectly stable i.e, no mobility takes place

D = 1,

the society is perfectly mobile.

51

Application Bartholomew Method

D = p1 [ p12+ 2p13 + 3p14 ] + p2 [ p21+ p23

+ 2p24 ] +

p3 [ 2p31+ p32 + p34 ] + p4 [ 3p41+ 2p42

+ p43 ]

= 0.2242

From the Bartholomew co-efficient of mobility we see that the society of that time was quite stable one.Indicates that the society in the generation of Great Father to Grand Father had a very good deg of mobility which is an indicator of the economic and social progress the society is going through in this generation.

52

Chi-Square Test

Simplest and most commonly used non-parametric test in statistical work. The quantity describes the magnitude of discrepancy theory and observation.

53

Hypothesis

The transition probability matrix (t.p.m) in internal is completely independent of the lithology at the immediate underlying point. Under this hypothesis the expected t.p.m would consist of rows that are all identical to the fixed probability vector.

54

Theory

ectedEobservedO

E

EO

j j

jj

exp,

,)(4

1

22

55

Values

)(48.9

)(39.142

05.0,4

2

talculated

calculated

56

Decision

So the null hypothesis is rejected as the tabulated value is lower than the calculated value.

57

NUMERICAL FIGURES IN DIFFERENT STATES OF CLASSES

TotalA B S O

A 75 31 40 3 149

B 0 15 6 0 21

S 5 3 15 1 24

O 3 0 1 2 6

Total 149 21 24 6 200

Professions

Pro

fess

ions

Father

G Father

58

EXPECTED STAY IN EACH SOCIAL CLASS AND PRAISE MEASURE OF SOCIAL MOBILITY

Class

Agriculture 1.8923

Business 1.2845

Service 2.3233

Others 1.3529

)(

)(

j

jj E

E

59

COMMENT

From the above Table, the generation from Grand Father to Father, the service community has most tendency to adjust their son to service. Second comes to agriculture community and third comes the business.

60

D-index & Chi-square

Using the measure of mobility as given by Bartholomew, D = 0.6031, A good degree of mobility, indicates fast social and economic progress.

Null hypothesis accepted, generations have independence in profession selections

61

Occupational Mobility : Markov Approach

Human Societies Stratified

Income Occupation Social Status

Residence Place

62

societies move on class

• In our society children do not always follow their fathers’ footsteps.

• In a free society a person has some degree of choice about changing his job or moving house.

• The inherent uncertainty of individual behavior in these situations means that the future development of the mobility process cannot be predicted with certainty but only in terms of probability.

63

MODELS FOR SOCIAL MOBILITY

• A very simple model for the development of a single family line and then, investigate the consequences of unrealistic features. The fundamental requirement in a model is that it must specify the way in which changes in social occur.

64

Assumption of model• Chance of moving depends only on the

• present class • but not on the

• past class/remote past• If the movement can be regarded as taking place at

discrete points in time the appropriate model becomes a simple

• Markov chain• Changes are governed by transition probabilities

which are independent of time. 65

Markov

• Andrei Andreivich Markov

• Alive 67 years

• From1856 to 1922

• Russian Mathematician

• Idea first occurred to him when he was watching an opera of the famous Russian writer Pushkin’s.

66

Notations

• Pij, the probability that the son of a father in class i is in class j (since the system is closed)

• 1

1

k

jijP

k is the number of classes and P, the matrix of transition probabilities.

TPpTP )0()(

67

Day-to-Day Inventory

Problem : The number of units of an item that are withdrawn from inventory on a day-to-day basis is a Markov chain in which requirements for tomorrow depend on today’s requirements. A one day transition matrix is given below :

68

Number Units Withdrawn from Inventory

6.03.01.012

4.03.03.010

0.04.06.05

12105

Today

Tomorrow

69

Target

• (a) Tree diagram showing inventory requirements on two consecutive days.

• (b) A two-day transition matrix

• (c) How a two-day transition matrix help manager for inventory management.

70

Transition Tree

5 10

5

12

5

10

12

5

10

12

10

12

5

0.6

0.4

0.0

0.6

0.3

0.1

0.4

0.3

0.3

0.0

0.4

0.6

71

Transition Probabilities

48.0)1.0)(0.0()3.0)(4.0()6.0)(6.0(211 p

36.0)3.0)(0.0()3.0)(4.0()4.0)(6.0(212 p

16.0)6.0)(0.0()4.0)(4.0()0.0)(6.0(213 p

48.0)1.0)(0.0()3.0)(4.0()6.0)(6.0(211 p

72

Transition Probabilities

31.0)1.0)(4.0()3.0)(3.0()6.0)(3.0(221 p

33.0)3.0)(4.0()3.0)(3.0()4.0)(3.0(222 p

36.0)6.0)(4.0()4.0)(3.0()0.0)(3.0(223 p

73

Transition Probabilities

21.0)1.0)(6.0()3.0)(3.0()6.0)(1.0(231 p

31.0)3.0)(6.0()3.0)(3.0()4.0)(1.0(232 p

48.0)6.0)(6.0()4.0)(3.0()0.0)(3.0(233 p

74