![Page 1: Probability with Engineering Applications ECE 313 ...publish.illinois.edu/varshney/files/2017/08/Lecture-12.pdf · Probability with Engineering Applications ECE 313 – Section C](https://reader030.vdocument.in/reader030/viewer/2022021711/5b5858a27f8b9a88698c1315/html5/thumbnails/1.jpg)
Probability with Engineering Applications ECE 313 – Section C – Lecture 12
Lav R. Varshney
25 September 2017
1
![Page 2: Probability with Engineering Applications ECE 313 ...publish.illinois.edu/varshney/files/2017/08/Lecture-12.pdf · Probability with Engineering Applications ECE 313 – Section C](https://reader030.vdocument.in/reader030/viewer/2022021711/5b5858a27f8b9a88698c1315/html5/thumbnails/2.jpg)
Exams
• Oct 11, Wed, 8:45 - 10pm
– Upto first five checkpoints
• Nov 15, Wed, 8:45 - 10pm
• Dec 18, Mon, 1:30 - 4:30pm
![Page 3: Probability with Engineering Applications ECE 313 ...publish.illinois.edu/varshney/files/2017/08/Lecture-12.pdf · Probability with Engineering Applications ECE 313 – Section C](https://reader030.vdocument.in/reader030/viewer/2022021711/5b5858a27f8b9a88698c1315/html5/thumbnails/3.jpg)
The approximate relative frequencies of the various positions for a single pig, using a standardized surface, a trap-door rolling device, and a sample size of 11,954, are:
[J. C. Kern, “Pig Data and Bayesian Inference on Multinomial Probabilities,” Journal of Statistics Education, vol. 14, no. 3, 2006.
Non-parametric representation
3
Position Percentage
Side (no dot) 34.9%
Side (dot) 30.2%
Razorback 22.4%
Trotter 8.8%
Snouter 3.0%
Leaning Jowler 0.61%
![Page 4: Probability with Engineering Applications ECE 313 ...publish.illinois.edu/varshney/files/2017/08/Lecture-12.pdf · Probability with Engineering Applications ECE 313 – Section C](https://reader030.vdocument.in/reader030/viewer/2022021711/5b5858a27f8b9a88698c1315/html5/thumbnails/4.jpg)
Common probability distributions
[https://blog.cloudera.com/blog/2015/12/common-probability-distributions-the-data-scientists-crib-sheet/]
![Page 5: Probability with Engineering Applications ECE 313 ...publish.illinois.edu/varshney/files/2017/08/Lecture-12.pdf · Probability with Engineering Applications ECE 313 – Section C](https://reader030.vdocument.in/reader030/viewer/2022021711/5b5858a27f8b9a88698c1315/html5/thumbnails/5.jpg)
Parametric representation
Probability mass function Parameter
Bernoulli 𝑝
Binomial 𝑛, 𝑝
Geometric 𝑝
Poisson 𝜆
How do we find the values of the parameters from data?
![Page 6: Probability with Engineering Applications ECE 313 ...publish.illinois.edu/varshney/files/2017/08/Lecture-12.pdf · Probability with Engineering Applications ECE 313 – Section C](https://reader030.vdocument.in/reader030/viewer/2022021711/5b5858a27f8b9a88698c1315/html5/thumbnails/6.jpg)
Non-random parameter estimation
• Collect data and then estimate parameter using the observed data
• Suppose we decide an experiment is accurately modeled by a random variable 𝑋 with pmf 𝑝𝜃, where 𝜃 is an unknown (but not random) parameter
• When experiment is performed, suppose we observe a particular value 𝑘 for 𝑋
![Page 7: Probability with Engineering Applications ECE 313 ...publish.illinois.edu/varshney/files/2017/08/Lecture-12.pdf · Probability with Engineering Applications ECE 313 – Section C](https://reader030.vdocument.in/reader030/viewer/2022021711/5b5858a27f8b9a88698c1315/html5/thumbnails/7.jpg)
Non-random parameter estimation
• According to probability model, the probability of 𝑘 being the observed value for 𝑋 (before experiment was performed), would have been 𝑝𝜃 𝑘
• It is said that the likelihood that 𝑋 = 𝑘 is 𝑝𝜃 𝑘
![Page 8: Probability with Engineering Applications ECE 313 ...publish.illinois.edu/varshney/files/2017/08/Lecture-12.pdf · Probability with Engineering Applications ECE 313 – Section C](https://reader030.vdocument.in/reader030/viewer/2022021711/5b5858a27f8b9a88698c1315/html5/thumbnails/8.jpg)
Maximum likelihood estimation
• One standard approach for estimating non-random parameters is to maximize the likelihood of the observed data, by choosing the best value of the non-random parameter
• The MLE of 𝜃 for observation 𝑘 is denoted
𝜃 𝑀𝐿 𝑘 is the value of 𝜃 that maximizes the likelihood 𝑝𝜃 𝑘 with respect to 𝜃
![Page 9: Probability with Engineering Applications ECE 313 ...publish.illinois.edu/varshney/files/2017/08/Lecture-12.pdf · Probability with Engineering Applications ECE 313 – Section C](https://reader030.vdocument.in/reader030/viewer/2022021711/5b5858a27f8b9a88698c1315/html5/thumbnails/9.jpg)
Maximizing expressions
• The best way we know to analytically maximize functions is to take the derivative with respect to the parameter we are maximizing, setting equal to zero, and solving.
• Often it may be easier to consider monotonic functions of the thing we care about, since the extremal value is the same.
– Optimize log-likelihood, rather than likelihood directly, since log is a monotonic function
![Page 10: Probability with Engineering Applications ECE 313 ...publish.illinois.edu/varshney/files/2017/08/Lecture-12.pdf · Probability with Engineering Applications ECE 313 – Section C](https://reader030.vdocument.in/reader030/viewer/2022021711/5b5858a27f8b9a88698c1315/html5/thumbnails/10.jpg)
Example 1
• Suppose 𝑋 has a Poisson distribution with some parameter 𝜆 > 0 which is unknown and a particular value 𝑘 for 𝑋 is observed.
• Find the maximum likelihood estimate, 𝜆 𝑀𝐿
• (Here 𝜆 plays the role of 𝜃 in the definition of ML estimation)
![Page 11: Probability with Engineering Applications ECE 313 ...publish.illinois.edu/varshney/files/2017/08/Lecture-12.pdf · Probability with Engineering Applications ECE 313 – Section C](https://reader030.vdocument.in/reader030/viewer/2022021711/5b5858a27f8b9a88698c1315/html5/thumbnails/11.jpg)
Example 2
• Suppose 𝑋 has the geometric distribution with some parameter 𝑝 which is unknown and a particular value 𝑘 for 𝑋 is observed.
• Find the maximum likelihood estimate, 𝑝 𝑀𝐿
![Page 12: Probability with Engineering Applications ECE 313 ...publish.illinois.edu/varshney/files/2017/08/Lecture-12.pdf · Probability with Engineering Applications ECE 313 – Section C](https://reader030.vdocument.in/reader030/viewer/2022021711/5b5858a27f8b9a88698c1315/html5/thumbnails/12.jpg)
Example 3
• Suppose we have a weighted coin, that shows heads with some unknown probability 𝑝 each time it is flipped
• We flip it some large and known number 𝑛 times and heads shows on 𝑘 of the flips
• Find the ML estimate of 𝑝
![Page 13: Probability with Engineering Applications ECE 313 ...publish.illinois.edu/varshney/files/2017/08/Lecture-12.pdf · Probability with Engineering Applications ECE 313 – Section C](https://reader030.vdocument.in/reader030/viewer/2022021711/5b5858a27f8b9a88698c1315/html5/thumbnails/13.jpg)
Unseen elements problem
![Page 14: Probability with Engineering Applications ECE 313 ...publish.illinois.edu/varshney/files/2017/08/Lecture-12.pdf · Probability with Engineering Applications ECE 313 – Section C](https://reader030.vdocument.in/reader030/viewer/2022021711/5b5858a27f8b9a88698c1315/html5/thumbnails/14.jpg)
[https://vtspecialcollections.wordpress.com/2015/01/15/i-j-jack-good-virginia-techs-own-bletchley-park-connection/]
Unseen elements problem
![Page 15: Probability with Engineering Applications ECE 313 ...publish.illinois.edu/varshney/files/2017/08/Lecture-12.pdf · Probability with Engineering Applications ECE 313 – Section C](https://reader030.vdocument.in/reader030/viewer/2022021711/5b5858a27f8b9a88698c1315/html5/thumbnails/15.jpg)
[http://aperiodical.com/2014/07/an-alan-turing-expert-watches-the-the-imitation-game-trailer/]
![Page 16: Probability with Engineering Applications ECE 313 ...publish.illinois.edu/varshney/files/2017/08/Lecture-12.pdf · Probability with Engineering Applications ECE 313 – Section C](https://reader030.vdocument.in/reader030/viewer/2022021711/5b5858a27f8b9a88698c1315/html5/thumbnails/16.jpg)
[http://ethw.org/2004_IEEE_Conference_on_the_History_of_Electronics]
![Page 17: Probability with Engineering Applications ECE 313 ...publish.illinois.edu/varshney/files/2017/08/Lecture-12.pdf · Probability with Engineering Applications ECE 313 – Section C](https://reader030.vdocument.in/reader030/viewer/2022021711/5b5858a27f8b9a88698c1315/html5/thumbnails/17.jpg)
Example 4
• Suppose 𝑋 is drawn at random from the numbers 1 through 𝑛, with each possibility being equally likely but 𝑛 is unknown
• We observe 𝑋 = 𝑘
• Find 𝑛 𝑀𝐿 as a function of 𝑘