## 1. Definition

Let be a stochastic process, taking on a finite or countable number of values.

is a DTMC if it has the Markov property: Given the present, the future is independent of the past

We define , since has the stationary transition probabilities, this probability is not depend on n.

Transition probabilities satisfy

## 2. n Step transition Probabilities

__Proof:__

## 3. Example: Coin Flips

## 4. Limiting Probabilities

__Theorem: __For an irreducible, ergodic Markov Chain, exists and is independent of the starting state i. Then is the unique solution of and .

Two interpretation for

- The probability of being in state i a long time into the future (large n)
- The long-run fraction of time in state i

Note:

- If Markov Chain is irreducible and ergodic, then interpretation 1 and 2 are equivalent
- Otherwise, is still the solution to , but only interpretation 2 is valid.