probabilty nad random processes auto saved)
TRANSCRIPT
8/8/2019 Probabilty Nad Random Processes Auto Saved)
http://slidepdf.com/reader/full/probabilty-nad-random-processes-auto-saved 1/48
Randomness arises because of
o random nature of the generation mechanism
o l imited understanding of the signal dynamics inherent imprecision in
measurement, observation, etc.
For example, thermal noise appearing in an electronic device is generated due to random motionof electrons. We have deterministic model for weather prediction; it takes into account of the
factors affecting weather. We can locally predict the temperature or the rainfall of a place on the basis of previous data. Probabilistic models are established from observation of a random
phenomenon. While probability is concerned with analysis of a random phenomenon, statisticshelp in building such models from data.
Deterministic versus probabilistic models
A d eterministic mod el can be used for a physical quantity and the process generating it provided
sufficient information is available about the initial state and the dynamics of the processgenerating the physical quantity. For example,
y We can determine the position of a particle moving under a constant force if we know theinitial position of the particle and the magnitude and the direction of the force.
y We can determine the current in a circuit consisting of resistance, inductance and
capacitance for a known voltage source applying Kirchoff's laws.
Many of the physical quantities are rand om in the sense that these quantities cannot be predicted
with certainty and can be described in terms of probabilistic mod els only. For example,
y The outcome of the tossing of a coin cannot be predicted with certainty. Thus theoutcome of tossing a coin is random.
y The number of ones and zeros in a packet of binary data arriving through a
communication channel cannot be precisely predicted is random.
y The ubiquitous noise corrupting the signal during acquisition, storage and transmissioncan be modelled only through statistical analysis.
y Probability in Electrical Engineeringy A signal is a physical quantity that varyies with time. The physical quantity is
converted into the electrical form by means of some transducers . For example, the time-varying electrical voltage that is generated when one speaks through a telephone is a
signal. More generally, a signal is a stream of information representing anything fromstock prices to the weather data from a remote-sensing satellite.
8/8/2019 Probabilty Nad Random Processes Auto Saved)
http://slidepdf.com/reader/full/probabilty-nad-random-processes-auto-saved 2/48
y
Figure 1 A sample of a speech signal
Figure 2 An ECG signal
An analog signal is defined for a continuum of values of domain parameter and it can take a continuous range of values.
A d igital signal is defined at discrete points and also takes a discrete set of
values.
As an example, consider the case of an analog-to-digital (AD) converter. The input to the AD
converter is an analog signal while the output is a digital signal obtained by taking the samples of the analog signal at periodic intervals of time and approximating the sampled values by a
discrete set of values.
8/8/2019 Probabilty Nad Random Processes Auto Saved)
http://slidepdf.com/reader/full/probabilty-nad-random-processes-auto-saved 3/48
R andom Signal
Many of the signals encountered in practice behave randomly in part or as a whole in thesense that they cannot be explicitly described by deterministic mathematical functions such as a
sinusoid or an exponential function. Randomness arises because of the random nature of thegeneration mechanism. Sometimes, limited understanding of the signal dynamics also
necessitates the randomness assumption. In electrical engineering we encounter many signalsthat are random in nature. Some examples of random signals are:
i. R adar signal: Signals are sent out and get reflected by targets. The reflected signals are
received and used to locate the target and target distance from the receiver. The received
signals are highly noisy and demand statistical techniques for processing.
ii. Sonar signal: Sound signals are sent out and then the echoes generated by some targets
are received back. The goal of processing the signal is to estimate the location of thetarget.
iii. Speech signal: A time-varying voltage waveform is produced by the speaker speaking
over a microphone of a telephone. This signal can be modelled as a randomsignal. A sample of the speech signal is shown in Figure 1.
iv. Biomedical signals: Signals produced by biomedical measuring devices like ECG,
EEG, etc., can display specific behaviour of vital organs like heart and brain. Statisticalsignal processing can predict changes in the waveform patterns of these signals to detect
abnormality. A sample of ECG signal is shown in Figure 2.v. Communication signals: The signal received by a communication receiver is generally
corrupted by noise. The signal transmitted may the digital data like video or speech andthe channel may be electric conductors, optical fiber or the space itself. The signal is
modified by the channel and corrupted by unwanted disturbances in different stages,collectively referred to as noise.
These signals can be described with the help of probability and other concepts in statistics.Particularly the signal under observation is considered as a realization of a rand om process or a
stochastic process.T
he terms rand om processes, stochastic processes and ran
d om signals areused synonymously.
A deterministic signal is analysed in the frequency-domain through Fourier series and
Fourier transforms. We have to know how random signals can be analysed in the frequencydomain.
8/8/2019 Probabilty Nad Random Processes Auto Saved)
http://slidepdf.com/reader/full/probabilty-nad-random-processes-auto-saved 4/48
R andom Signal Processing
P rocessing refers to performing any operations on the signal. The signal can be amplified,integrated, differentiated and rectified. Any noise that corrupts the signal can also be reduced by
performing some operations. Signal processing thus involves
o Amplification
o F iltering
o I ntegration and d ifferentiation
o Nonlinear operations like rectification, squaring, mod ulation, d emod ulation etc.
These operations are performed by passing the input signal to a system that performs the
processing. For example, filtering involves selectively emphasising certain frequency
components and attenuating others. In low-pass filtering illustrated in Fig.4, high-frequencycomponents are attenuated
.
Figure 4 Low-pass filtering
Figure 5 Low-pass filtering of Filtering of signals from noise
8/8/2019 Probabilty Nad Random Processes Auto Saved)
http://slidepdf.com/reader/full/probabilty-nad-random-processes-auto-saved 5/48
Special processing techniques are to be developed to deal with random signals. In the following we
outline two illustrative applications.
Signal estimation and detection
A problem frequently come across in signal processing is the estimation of the true valueof the signal from the received noisy data. Consider the received noisy signal given by
where is the desired transmitted signal buried in the noise .
Simple frequency selective filters cannot be applied here, because random noise cannot be
localized to any spectral band and does not have a specific spectral pattern. We have to do this by dissociating the noise from the signal in the probabilistic sense. O ptimal filters like the
Wiener filter, ad aptive filters and Kalman filter deals with this problem.
In estimation, we try to find a value that is close enough to the transmitted signal. The
process is explained in Figure 6. Detection is a related process that decides the best choice out of a finite number of possible values of the transmitted signal with minimum error probability. In
binary communication, for example, the receiver has to decide about 'zero' and 'one' on the basisof the received waveform. Signal detection theory, also known as d ecision theory, is based on
hypothesis testing and other related techniques and widely applied in pattern classification, targetdetection etc.
Figure 6 Signal estimation problem
Source and Channel Coding
One of the major areas of application of probability theory is I nformation theory and
cod ing. In 1948 Claude Shannon published the paper " A mathematical theory of communication" which lays the foundation of modern digital communication. Following are
two remarkable results stated in simple languages :
y Digital data is efficiently represented with number of bits for a symbol decided by its probability of occurrence.
y The data at a rate smaller than the channel capacity can be transmitted over a noisychannel with arbitrarily small probability of error. The channel capacity again is
determined from the probabilistic descriptions of the signal and the noise.
8/8/2019 Probabilty Nad Random Processes Auto Saved)
http://slidepdf.com/reader/full/probabilty-nad-random-processes-auto-saved 6/48
Basic Concepts of Set Theory
The modern approach to probability based on axiomatically defining probability asfunction of a set. A background on the set theory is essential for understanding probability.
Some of the basic concepts of set theory are introduced here.
Set
A set is a well defined collection of objects. These objects are called elements or members of the set. Usually uppercase letters are used to denote sets.
Example 1
is a set and are its elements.
y The elements of a set are enumerated within a pair of curly brackets as shown in thisexample.
y Instead of listing all the elements, we can represent a set in the set-builder notation
specifying some common properties satisfied by the elements. Thus the set
represents the set . We read ' ' as 'all
such that'. Particularly, if a set is infinite having infinite number of elements or listingall the elements of the set is cumbersome, such a representation is useful.
y If an element x is a member of the set A, we write . If x is not a member of A, wewrite . In the above example, and .
y The null set or empty set is the set that does not contain any element. A null set is
denoted by .
Some commonly used sets and their standard notations
o Set of natural numbers --- N.
o Set of integers --- .
o Set of rational numbers --- Q.
o Set of real numbers --- .
o Set of complex numbers --- C.
8/8/2019 Probabilty Nad Random Processes Auto Saved)
http://slidepdf.com/reader/full/probabilty-nad-random-processes-auto-saved 7/48
o Open interval on the real line
o Closed interval on the real line
o Semi open intervals
Subset and Superset
A set A is called a subset of B (or B is called the superset of A), denoted by
, if all the elements of A are also elements of B. Thus
.
If A is a subset of B and there is at least one element in B which is not an element of A,
then A is called a proper subset of B. We write
Example 2 Let .
Then, .
o A set A is a subset of itself .
o implies that .
o The null set is a subset of every set.
o If the set A is finite with n number of elements, then A has subsets.
For example, the set of binary digits has subsets.
These are: .
Universal set
We always consider all sets for the problem under consideration to be a subset of a (large)
set called the universal set. For the binary digital communication problem, the set may
be considered as the universal set. We shall denote the universal set by the symbol S.
Example 3
In discussion involving English letters, the alphabet of the English language may be
considered as the universal set.
Equality of two sets
8/8/2019 Probabilty Nad Random Processes Auto Saved)
http://slidepdf.com/reader/full/probabilty-nad-random-processes-auto-saved 8/48
Two sets A and B are equal if and only if they have the same elements. Thus,
o A = B if and only if and . In other words,
We take above definition of the equality of two sets to establish identities involving set theoretic
operation.
Set operations
We can combine events by set operation to get other events. Following set operations are useful:
UNION : The union of two sets A and B, denoted by is defined as the set of
elements that are either in A or in B or both. In set builder notation, we write
INTER SECTION : The intersection of two sets A and B, denoted by , is defined as
the set of elements that are common to both A and B. We can write,
DIFFERENCE : The d ifference of two sets A and B, denoted by is the set of thoseelements of A which do not belong to B. Thus,
Similarly,
COMPLEMENT : The complement of a set A, denoted by , is defined as the set of allelements which are not in A.
Clearly,
Example 4
8/8/2019 Probabilty Nad Random Processes Auto Saved)
http://slidepdf.com/reader/full/probabilty-nad-random-processes-auto-saved 9/48
DISJOINT : Two sets A and B are called d isjoint if
In the above example, B and C are disjoint sets.
Venn Diagram
The sets and set operations can be illustrated by means of the V enn d iagrams. A rectangle
is used to represent the universal set and a circle is used to represent any set in it.
Following are a few illustrations of V enn d iagram:
Properties of set operations
1. Identity properties
2. Commutative properties
8/8/2019 Probabilty Nad Random Processes Auto Saved)
http://slidepdf.com/reader/full/probabilty-nad-random-processes-auto-saved 10/48
3. Associative properties
4. Distributive properties
5. Complementary properties
6. De Morgans laws
2. These properties can be proved easily and verified using the Venn diagram. They can be
used to derive useful results.
3. For example, Putting in
4.
we get,
5. 6.
Cartesian Product of two Sets
Consider two sets A and B, not necessarily from the same universal set. Then the set of all
ordered pairs , where and called the Cartesian product of A and B and is
denoted by . Thus
Example 5 Suppose
8/8/2019 Probabilty Nad Random Processes Auto Saved)
http://slidepdf.com/reader/full/probabilty-nad-random-processes-auto-saved 11/48
Remark
y is denoted by . y Some important Cartesian products
o
y We can extend the set theoretic operations to arbitrary number of sets. Particularly, if we
have a sequence of sets
y we can define ,
y Thus we can define the union and the intersection of countably infinite number of sets.
o
Power set
The set of all the subsets of a set S is called the power set of S and denoted by .
Thus
Note the following
y A collection of sets is called a class of sets. Each element of a class is a set. Thus is a
class.
y If S contains n elements, contains elements.
y If S is finite, then is closed under the set theoretic operation of unions, intersection
and complementation,
Thus for all ,
y If S is an infinite set, may not be closed under countably infinite unions or countably
infinite intersections. Such situations lead to serious mathematical problems.
8/8/2019 Probabilty Nad Random Processes Auto Saved)
http://slidepdf.com/reader/full/probabilty-nad-random-processes-auto-saved 12/48
y Sigma Algebra
y This necessitates defining a subset of which is closed under these countably infinite
set theoretic operations. Such a family of sub sets of is called sigma-algebra and
defined below.
Let S be a set. A sigma field F is a subset of the power set with the following
properties:
Thus F is closed under complementation and countable union operations.
Remark
y From (1) and (2) it follows that .
y is the smallest sigma-algebra formed from the subsets of .
y If is a finite set, then is a sigma-algebra.
y is closed under countable intersection operations also.
Suppose . Then and therefore,
.
Example 6 Suppose
Then the power set is a sigma algebra.
Example 7 The collection is the smallest sigma algebra containing .
For a finite set ,
the power set is the largest sigma algebra containing .
Example 8 Consider the class of all open intervals of the form on the real line.
Here and .
8/8/2019 Probabilty Nad Random Processes Auto Saved)
http://slidepdf.com/reader/full/probabilty-nad-random-processes-auto-saved 13/48
Consider the sequence of subsets in given by
It can be shown that
Hence the collection of open intervals of the real line is not a sigma-algebra.
Remark
The minimum sigma field containing is called the Borel field and denoted by B. The
same Borel field is obtained if is considered as the class of all closed intervals of the form
or the semi-open intervals of the form or . It can be shown that B contains
all the open intervals, closed intervals and semi-open intervals.
Problem Set 1
Problem 01: Consider the universal set Define two subset
and Find
a.
b.
c. d.
e.
Problem 02: Use Venn diagram to show that
a.
b.
c.
Problem 03: Consider the universal set
a. Find the power set
b. Find the smallest sigma algebra containing the subset
8/8/2019 Probabilty Nad Random Processes Auto Saved)
http://slidepdf.com/reader/full/probabilty-nad-random-processes-auto-saved 14/48
Problem 01: State whether the following statements about the subsets , , of a universal
set are true ( ) or false ( ).
a. f.
b. g.
c. h.
d. i.
e. j.
Probability Concepts
Before we give a definition of probability, let us examine the following concepts:
1. R
and om Experiment : An experiment is a random experiment if its outcome cannot be predicted precisely. One out of a number of outcomes is possible in a random
experiment. A single performance of the random experiment is called a trial.
2. S ample S pace : The sample space is the collection of all possible outcomes of a
random experiment. The elements of are called sample points.
y A sample space may be finite , countably infinite or uncountable.
y A finite or countably infinite sample space is called a d iscrete sample space.
y An uncountable sample space is called a continuous sample space
3. Event: An event A is a subset of the sample space such that probability can be assigned to
it. Thus
y
y For a discrete sample space, all subsets are events.
8/8/2019 Probabilty Nad Random Processes Auto Saved)
http://slidepdf.com/reader/full/probabilty-nad-random-processes-auto-saved 15/48
y is the certain event (sure to occur) and is the impossible event.
Figure 1
Consider the following examples.
Example 1 Tossing a fair coin
The possible outcomes are H (head) and T (tai l ). The associated sample space is It
is a finite sample space. The events associated with the sample space are: and .
Example 2 Throwing a fair die:
The possible 6 outcomes are:
. . .
The associated finite sample space is .Some events are
And so on.
Example 3 Tossing a fair coin until a head is obtained
We may have to toss the coin any number of times before a head is obtained. Thus the possibleoutcomes are :
H, TH,TTH,TTTH, ..
How many outcomes are there? The outcomes are countable but infinite in number. The
countably infinite sample space is .
8/8/2019 Probabilty Nad Random Processes Auto Saved)
http://slidepdf.com/reader/full/probabilty-nad-random-processes-auto-saved 16/48
Example 4 Picking a real number at random between -1 and +1
The associated sample space is
Clearly is a continuous sample space.
Example 5 Output of a radio receiver at any time
Suppose the output voltage of a radio receiver at any time t is a value lying between -5 V and5V.
The associated sample space is continuous and given by
Clearly is a continuous sample space.
The probability of an event is a number assigned to the event. Let us see how we can
define probability.
Classical definition of probability ( Laplace 1812)
Consider a random experiment with a finite number of outcomes If all the outcomes of theexperiment are equally likely , the probability of an event is defined by
where
Example 6 A fair die is rolled once. What is the probability of getting a 6 ?
Here and
Example 7 A fair coin is tossed twice. What is the probability of getting two heads'?
Here and .
Total number of outcomes is 4 and all four outcomes are equally likely.
Only outcome favourable to is {HH}
8/8/2019 Probabilty Nad Random Processes Auto Saved)
http://slidepdf.com/reader/full/probabilty-nad-random-processes-auto-saved 17/48
Discussion
y The classical definition is limited to a random experiment which has only a finite number
of outcomes. In many experiments like that in the above examples, the sample space isfinite and each outcome may be assumed equally likely.' In such cases, the counting
method can be used to compute probabilities of events.
y Consider the experiment of tossing a fair coin until a head' appears.As we have
discussed earlier, there are countably infinite outcomes. Can you believe that all theseoutcomes are equally likely?
y The notion of equally likely is important here. Equally likely means equally probable.
Thus this definition presupposes that all events occur with equal probability . Thus thedefinition includes a concept to be defined.
R elative-frequency based definition of probability( von Mises, 1919)
If an experiment is repeated times under similar conditions and the event occurs in times,
then
Example 8 Suppose a die is rolled 500 times. The following table shows the frequency eachface.
We see that the relative frequencies are close to . How do we ascertain that these relative
frequencies will approach to as we repeat the experiments infinite no of times?
Discussion This definition is also inadequate from the theoretical point of view.
We cannot repeat an experiment infinite number of times. How do we ascertain that the above ratio will converge for all possible sequences
of outcomes of the experiment?
3. Axiometic definition of probability ( Kolmogorov, 1933)
4. We have earlier defined an event as a subset of the sample space. Does each subset of the sample space forms an event?
8/8/2019 Probabilty Nad Random Processes Auto Saved)
http://slidepdf.com/reader/full/probabilty-nad-random-processes-auto-saved 18/48
The answer is yes for a finite sample space. However, we may not be able to assign probability meaningfully to all the subsets of a continuous sample space. We have to
eliminate those subsets. The concept of the sigma algebra is meaningful now.
Definition Let be a sample space and a sigma field defined over it. Let
be a mapping from the sigma-algebra into the real line such that for each , thereexists a unique . Clearly is a set function and is called probability, if itsatisfies the following three axioms.
5. 6. Figure 2
Discussion
y The triplet is called the probability space.
y Any assignment of probability assignment must satisfy the above three axioms
y If ,
8/8/2019 Probabilty Nad Random Processes Auto Saved)
http://slidepdf.com/reader/full/probabilty-nad-random-processes-auto-saved 19/48
This is a special case of axiom 3 and for a d iscrete sample space , this simpler versionmay be considered as the axiom 3. We shall give a proof of this result below.
y The events A and B are called mutually exclusive .
Basic results of probability
From the above axioms we established the following basic results:
1.
Suppose,
Then
Therefore
Thus which is possible only if
2. If We have ,
3. y where where
We have,
4. y If
8/8/2019 Probabilty Nad Random Processes Auto Saved)
http://slidepdf.com/reader/full/probabilty-nad-random-processes-auto-saved 20/48
We have,
We can similarly show that ,
5. If
We have ,
6. We can apply the properties of sets to establish the following result for
,
T
he following generalization is known as the principle inclusion-exclusion.7. y Principle of Inclusion-exclusion
Suppose .
Then ,
8.
Discussion
We require some rules to assign probabilities to some basic events in . For other events
we can compute the probabilities in terms of the probabilities of these basic events.9. Probability assignment in a discrete sample space
Consider a finite sample space . Then the sigma algebra is defined by the power set
of S. For any elementary event , we can assign a probability P( si ) such that,
8/8/2019 Probabilty Nad Random Processes Auto Saved)
http://slidepdf.com/reader/full/probabilty-nad-random-processes-auto-saved 21/48
10.
For any event , we can define the probability
In a special case, when the outcomes are equi-probable, we can assign equal probability p
to each elementary event.
Example 9 Consider the experiment of rolling a fair die considered in example 2.
Suppose represent the elementary events. Thus is the event of getting
1', is the event of getting '2' and so on.
Since all six disjoint events are equiprobable and we get ,
Suppose is the event of getting an odd face. Then
8/8/2019 Probabilty Nad Random Processes Auto Saved)
http://slidepdf.com/reader/full/probabilty-nad-random-processes-auto-saved 22/48
Example 9 Consider the experiment of rolling a fair die considered in example 2.
Suppose represent the elementary events. Thus is the event of getting
1', is the event of getting '2' and so on.
Since all six disjoint events are equiprobable and we get ,
Suppose is the event of getting an odd face. Then
Example 9 Consider the experiment of rolling a fair die considered in example 2.
Suppose represent the elementary events. Thus is the event of getting1', is
the event of getting '2' and so on.
Since all six disjoint events are equiprobable and we get ,
Suppose is the event of getting an odd face. Then
Probability assignment in a continuous space
Suppose the sample space S is continuous and un-countable. Such a sample space arises when the
outcomes of an experiment are numbers. For example, such sample space occurs when the experiment
consists in measuring the voltage, the current or the resistance. In such a case, the sigma algebra
8/8/2019 Probabilty Nad Random Processes Auto Saved)
http://slidepdf.com/reader/full/probabilty-nad-random-processes-auto-saved 23/48
consists of the Borel sets on the real line.
Suppose and is a non-negative integrable function such that,
For any Borel set ,
defines the probability on the Borel sigma-algebra B .
We can similarly define probability on the continuous space of etc.
Example 11 Suppose
Then for
Example 11 Suppose
Then for
8/8/2019 Probabilty Nad Random Processes Auto Saved)
http://slidepdf.com/reader/full/probabilty-nad-random-processes-auto-saved 24/48
Problem 01: Let be a probability space and A, B, C be three events. Find expression for the following events:
i. only A occurs
ii. both A and B but not C occur
iii. at least one event occurs
iv. at least two events occur
v. at most one event occurs
vi. at most two events occur
vii. exactly one of events A,B,C occurs
viii. exactly two of events A,B,C occurs
ix. all three events occur
x. none of the three events occurs
Problem 02: Verify that the following are valid probability laws using three axioms of
probability.
a. The binomial law:
b. Poisson's laws:
c. eometric probability law:
b. Problem 03: Let be a probability space and A, B, C be three events. Prove thefollowing:
c.
8/8/2019 Probabilty Nad Random Processes Auto Saved)
http://slidepdf.com/reader/full/probabilty-nad-random-processes-auto-saved 25/48
d.
e.
Problem 04: Suppose is a probability space and be two mutually exclusive
events. If and , find the probabilities that
a. either A or B occurs
b. only A occursc. both A and B occurs
d. neither A nor B occurs
Problem 05: Suppose is a probability space and A and B any two events. Given
and Find
a. maximum value of
b. minimum value of
c. maximum value of
d. minimum value of
Problem 06: Suppose in the problem 5, Find .
Probability Using CountingMethod
In many applications we have to deal with a finite sample space and the elementaryevents formed by single elements of the set may be assumed equiprobable. In this case, we candefine the probability of the event A according to the classical definition discussed earlier:
where = number of elements favorable to A and n is the total number of elements in the
sample space .
Thus calculation of probability involves finding the number of elements in the sample
space and the event A . Combinatorial rules give us quick algebraic formulae to find the
elements in .We briefly outline some of these rules:
8/8/2019 Probabilty Nad Random Processes Auto Saved)
http://slidepdf.com/reader/full/probabilty-nad-random-processes-auto-saved 26/48
1. Product rule Suppose we have a set A with m distinct elements and the set B with n
distinct elements and . Then contains mn ordered
pair of elements. This is illustrated in Fig for m=5 and n=4 n other words if we can
choose element a in m possible ways and the element b in n possible ways then the
ordered pair ( a, b ) can be chosen in mn possible ways.
Figure 1 Illustration of the product rule
The above result can be generalized as follows:
The number of distinct k -tupples in
is where
represents the number of distinct elements in .
Example 1 A fair die is thrown twice. What is the probability that a 3 will appear at least once.
Solution: The sample space corresponding to two throws of the die is illustrated in the following
table. Clearly, the sample space has elements by the product rule. The event
corresponding to getting at least one 3 is highlighted and contains 11 elements. Therefore, the
8/8/2019 Probabilty Nad Random Processes Auto Saved)
http://slidepdf.com/reader/full/probabilty-nad-random-processes-auto-saved 27/48
required probability is .
2. Sampling with replacement and ordering
Suppose we have to choose k objects from a set of n objects. Since sampling is withordering, each ordered arrangements of k objects is to be considered. Further, after everychoosing, the object is placed back in the set. In this case, the number of distinct ordered
k - tupples = . If a random experiment has n outcomes, if the experiment is repeated k times, then
the total number of outcomes = .
3. S ampling without replacement and with or d ering
Suppose we have to choose k objects from a set of n objects by picking one object after
another at random.
In this case the first object can be chosen from n objects, the second object can be chosen
from n-1 objects, and so. Therefore, by applying the product rule, the number of distinctordered k -tupples in this case is
The number is called the permutation of n objects taking k at a time and denoted by .Thus
Clearly ,
8/8/2019 Probabilty Nad Random Processes Auto Saved)
http://slidepdf.com/reader/full/probabilty-nad-random-processes-auto-saved 28/48
Example 2 Birthday problem - Given a class of students, what is the probability of two students in the
class having the same birthday? Plot this probability vs. number of students and be surprised!.
Let be the number of students in the class.
The plot of probability vs number of students is shown in above table. Observe the steep rise in
the probability in the beginning. In fact this probability for a group of 25 students is greater than 0.5 and
that for 60 students onward is closed to 1. This probability for 366 or more number of students is exactly
one.
8/8/2019 Probabilty Nad Random Processes Auto Saved)
http://slidepdf.com/reader/full/probabilty-nad-random-processes-auto-saved 29/48
8/8/2019 Probabilty Nad Random Processes Auto Saved)
http://slidepdf.com/reader/full/probabilty-nad-random-processes-auto-saved 30/48
Solution :
9 balls can be picked from a population of 15 balls in .
Therefore the required probability is
5. Arranging n objects into k specific groups Suppose we want to partition a set of n distinct
elements into k distinct subsets of sizes respectively so that
. Then the total number of distinct partitions is
This can be proved by noting that the resulting number of partitions is
Example 4 What is the probability that in a throw of 12 dice each face occurs twice.
Solution: The total number of elements in the sample space of the outcomes of a single throw of
12 dice is
The number of favourable outcomes is the number of ways in which 12 dice can be arranged in six
groups of size 2 each group 1 consisting of two dice each showing 1, group 2 consisting of two dice
each showing 2 and so on.
Therefore, the total number distinct groups
Hence the required probability is
8/8/2019 Probabilty Nad Random Processes Auto Saved)
http://slidepdf.com/reader/full/probabilty-nad-random-processes-auto-saved 31/48
Problem 01:
In a communication system, data are transmitted randomly in groups of 8 bits called
byte. If each bit is either 1' or 0' and there is no loss in the transmitting medium, what isthe probability that a received byte contains at least two 1' - s.
Problem 02:
A box contains 80 good capacitors and 10 defective capacitors. A student picks up 2capacitors at random. What is the probability that both of them are defective?
Problem 03:
5 letters are to be placed in 5 addressed envelop. Find the probability that
i. Each letter is placed correctlyii. More than two letters are placed correctly
iii. None of the letters are placed correctly
Problem 04: 6 balls are to be randomly placed in six cells. Find the probability that
i. Only one cell is occupied.
ii. Each cell is occupied.
Problem 05:
4-letters words are formed from the English alphabet. If the repetition of the letters is
not allowed, what is the probability that a randomly selected word starts with the letter 'A'?
Problem 06: You have to place n balls in m ( m > n ) cells. Find the probability that a given cellcontains exactly k -balls in the following cases:
i. The balls are undistinguishable
( Bose-Einstein statistics )ii. The balls are distinct
( Boltzmann Maxwell distribution )
iii. The balls cannot be distinguished and at most one ball is allowed in a box.2. Conditional probability
3. Consider the probability space . Let A and B two events in . We ask the following question
Given that A has occurred , what is the probability of B?
The answer is the cond itional probability of B given A denoted by . We
8/8/2019 Probabilty Nad Random Processes Auto Saved)
http://slidepdf.com/reader/full/probabilty-nad-random-processes-auto-saved 32/48
shall develop the concept of the conditional probability and explain under what condition
this conditional probability is same as .
Let us consider the case of equiprobable events discussed earlier. Let sample
points be favourable for the joint event .4.
5. Figure 16.
Clearly ,
This concept suggests us to define conditional probability. The probability of an event B under the
condition that another event A has occurred is called the conditional probability of B given A and defined
by
8/8/2019 Probabilty Nad Random Processes Auto Saved)
http://slidepdf.com/reader/full/probabilty-nad-random-processes-auto-saved 33/48
We can similarly define the conditional probability of A given B , denoted by .
From the definition of conditional probability, we have the joint probability of two
events A and B as follows
Example 1 Consider the example tossing the fair die. Suppose
Example 2 A family has two children. It is known that at least one of the children is a girl. What is the
probability that both the children are girls?
A = event of at least one girl
B = event of two girls
Clearly ,
Conditional probability and the axioms of probability
In the following we show that the conditional probability satisfies the axioms of
probability.
8/8/2019 Probabilty Nad Random Processes Auto Saved)
http://slidepdf.com/reader/full/probabilty-nad-random-processes-auto-saved 34/48
By definition
Axiom 1 :
Axiom 2 :
We have ,
Axiom 3 :
Consider a sequence of disjoint events .
We have ,
Figure 2
8/8/2019 Probabilty Nad Random Processes Auto Saved)
http://slidepdf.com/reader/full/probabilty-nad-random-processes-auto-saved 35/48
Note that the sequence is also sequence of disjoint events.
1. Properties of Conditional Probabilities
If , then
We have ,
2. Chain Rule of Probability
8/8/2019 Probabilty Nad Random Processes Auto Saved)
http://slidepdf.com/reader/full/probabilty-nad-random-processes-auto-saved 36/48
2. Chain Rule of Probability
3. T
heorem of T
otal ProbabilityL
et be n events such thatThen for any event B,
P roof : We have and the sequence is disjoint.
8/8/2019 Probabilty Nad Random Processes Auto Saved)
http://slidepdf.com/reader/full/probabilty-nad-random-processes-auto-saved 37/48
Figure 3
R emark
(1) A decomposition of a set S into 2 or more disjoint nonempty subsets is called a partition of
S .The subsets form a partition of S if
(2) The theorem of total probability can be used to determine the probability of a complex eventin terms of related simpler events. This result will be used in Bays' theorem to be discussed to the
end of the lecture.
Example 3 Suppose a box contains 2 white and 3 black balls. Two balls are picked at random without
replacement.
Let = event that the first ball is white and
Let = event that the first ball is black.
Clearly and form a partition of the sample space corresponding to picking two balls from
the box.
Let B = the event that the second ball is white. Then .
8/8/2019 Probabilty Nad Random Processes Auto Saved)
http://slidepdf.com/reader/full/probabilty-nad-random-processes-auto-saved 38/48
Independent events
Two events are called independent if the probability of occurrence of one event does not affect
the probability of occurrence of the other. Thus the events A and B are independent if
and
where and are assumed to be non-zero.
Equivalently if A and B are independent, we have
or --------------------
wo events A and B are called statistically dependent if they are not independent. Similarly, we can define
the independence of n events. The events are called independent if and only if
Example 4 Consider the example of tossing a fair coin twice. The resulting sample space is given by
and all the outcomes are equiprobable.
Let be the event of getting tail' in the first toss and be the event
of getting head' in the second toss. Then
8/8/2019 Probabilty Nad Random Processes Auto Saved)
http://slidepdf.com/reader/full/probabilty-nad-random-processes-auto-saved 39/48
and
Again, so that
Hence the events A and B are independent.
Example 5 Consider the experiment of picking two balls at random discussed in example 3.
In this case, and .
Therefore, and and B are dependent.
Bayes' Theorem
This result is known as the Baye's theorem. The probability is called the a priori probability and
is called the a posteriori probability. Thus the Bays' theorem enables us to determine the a
8/8/2019 Probabilty Nad Random Processes Auto Saved)
http://slidepdf.com/reader/full/probabilty-nad-random-processes-auto-saved 40/48
posteriori probability from the observation that B has occurred. This result is of practical
importance and is the heart of Baysean classification, Baysean estimation etc.
Example 6
In a binary communication system a zero and a one is transmitted with probability 0.6 and 0.4
respectively. Due to error in the communication system a zero becomes a one with a probability 0.1 and
a one becomes a zero with a probability 0.08. Determine the probability (i) of receiving a one and (ii)
that a one was transmitted when the received message is one.
Let S be the sample space corresponding to binary communication. Suppose be event of
transmitting 0 and be the event of transmitting 1 and and be corresponding events of
receiving 0 and 1 respectively.
Given and
Example 7 In an electronics laboratory, there are identically looking capacitors of three makes
in the ratio 2:3:4. It is known that 1% of ,1.5% of are defective.
What percentage of capacitors in the laboratory are defective? If a capacitor picked at defective is found
to be defective, what is the probability it is of make ?
Let D be the event that the item is defective. Here we have to find .
8/8/2019 Probabilty Nad Random Processes Auto Saved)
http://slidepdf.com/reader/full/probabilty-nad-random-processes-auto-saved 41/48
8/8/2019 Probabilty Nad Random Processes Auto Saved)
http://slidepdf.com/reader/full/probabilty-nad-random-processes-auto-saved 42/48
We can easily show that
where is the probability defined on the events of This is because, the
event in S occurs whenever in occurs, irrespective of the event in .
Also note that
Figure 1
Independent Experiments
In many experiments, the events and are independent for every selection of
and . Such experiments are called independent experiments.
In this case can write
8/8/2019 Probabilty Nad Random Processes Auto Saved)
http://slidepdf.com/reader/full/probabilty-nad-random-processes-auto-saved 43/48
Example 1
Consider the experiments of rolling a fair die and tossing a fair coin sequentially. What is the
probability that a '2' and a 'head' will occur?
Solution: Suppose is the sample space of the experiment of rolling of a six-faced fair die
and is the sample space of the experiment of tossing of a fair die.
Example 2
In a digital communication system transmitting 1 and 0, 1 is transmitted twice as often as0. If two bits are transmitted in a sequence, what is the probability that both the bits will be 1?
Solution:
Generalization
We can similarly define the sample space corresponding to n
experiments and the Cartesian product of events
8/8/2019 Probabilty Nad Random Processes Auto Saved)
http://slidepdf.com/reader/full/probabilty-nad-random-processes-auto-saved 44/48
.
If the experiments are independent, we can write
where is probability defined on the event of .
Bernoulli trial
Suppose in an experiment, we are only concerned whether a particular event A has
occurred or not. We call this event as the success' with probability and the
complementary event as the failure' with probability . Such a random
experiment is called Bernoulli trial.
Probability of
Success :
Failure :
Binomial Law
We are interested in finding the probability of k successes' in n independent Bernoullitrials.
This probability is given by
Consider n independent repetitions of the Bernoulli trial. Let S be the sample space
associated with each trial and we are interested in a particular event and its complement
such that and . If A occurs in a trial, then we have a
success' otherwise a 'failure'.
Thus the sample space corresponding to the n repeated trials is .
Any event in S is of the form where some s are A and remaining
s are .
8/8/2019 Probabilty Nad Random Processes Auto Saved)
http://slidepdf.com/reader/full/probabilty-nad-random-processes-auto-saved 45/48
Using the property of independent experiment we have,
.
If k 's are A and the remaining n - k 's are , then
But there are number of events in S with k number of A' s and n - k number of s.
For example, if , the possible events are
We also note that all the events are mutually exclusiv
Hence the probability of k successes in n independent repetitions of the Bernoulli trial is given by
The above law is known as the binomial probability law.
A typical plot of vs k for n=20 and a particluar value of p is shown in the figure.
8/8/2019 Probabilty Nad Random Processes Auto Saved)
http://slidepdf.com/reader/full/probabilty-nad-random-processes-auto-saved 46/48
8/8/2019 Probabilty Nad Random Processes Auto Saved)
http://slidepdf.com/reader/full/probabilty-nad-random-processes-auto-saved 47/48
Example 5 In a binary communication system, bit error occurs with a probability of . What is the
probability of getting at least one error bit in a message of 8 bits?
Solution:
Here we can consider the sample space
Approximations of the Binomial probabilities
Two interesting approximations of the binomial probabilities are very important .
Case 1: Suppose n is very large and p is very small and a constant.