Maximum Likelihood Estimation in Statistics
In the field of statistics, Maximum Likelihood Estimation (MLE) is a crucial method for estimating the parameters of a statistical model. The process involves finding the values of parameters that maximize the likelihood function based on observed data. This summary covers the concept of MLE, how to find estimators for normal distributions, two-parameter estimation for normals, the significance of unbiased MLEs, and the distinction between biased and unbiased estimators. Understanding MLE is essential for statistical analysis and model fitting.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
MLE continued CSE 312 Summer 21 Lecture 22
Finding ? for Normals 1 2? 1 ? 2?? ?2 ln = ?=1 ln ? ??ln = ?=1 Setting ? = 0 and solving: ? ?? ? ? ?=1 ?? ? ? ?=1 Check using the second derivative test: ?? ? = 0 ?=1 ??= ? ? ? = ? ?2 ??2ln( ) = ? Second derivative is negative everywhere, so log-likelihood is concave down and average of the ??is a maximizer.
Two Parameter Estimation for Normals If you get independent samples ?1,?2, ,??from a ?(?,?2) where ? and ?2are unknown, the maximum likelihood estimates of the normal is: ? 2 ?=1 ?? 1 ? ?=1 and ??= ?? ? ??2 = ?? ? The maximum likelihood estimator of the mean is the sample mean is the estimate of ? is the average value of all the data points. The MLE for the variance is: the variance of the experiment choose one of the ??at random sample mean that
Summary Given: an event ? (usually ? i.i.d. samples from a distribution with unknown parameter ?). 1. Find likelihood (?;?) Usually (??;?) for discrete and ?(??;?) for continuous 2. Maximize the likelihood. Usually: A. Take the log (if it will make the math easier) B. Take the derivative C. Set the derivative to 0 and solve 3. Use the second derivative test to confirm you have a maximizer
Biased One property we might want from an estimator is for it to be unbiased unbiased. An estimator ? is unbiased if ? ? = ? The expectation is taken over the randomness in the samples we drew. (those samples are random variables). So, we re not consistently overestimating or underestimating. If an estimator isn t unbiased then it s biased biased.
Are our MLEs unbiased? ? ?=1 ?? ??= ? 1 ?? ?? = 1 ? ? ?? = 1 ? ? ? = ? ? ?? = Unbiased!
Are our MLEs biased? Our estimate for the probability coin-flips being heads (if we generalized a bit) would be num heads total flips Fill out the poll everywhere so Kushal knows how long to explain Go to pollev.com/cse312su21
Are our MLEs biased? Our estimate for the probability coin-flips being heads (if we generalized a bit) would be num heads total flips What is ? total flips = ? ? Unbiased! num heads ?? =? ? ?= ? Fill out the poll everywhere so Kushal knows how long to explain Go to pollev.com/cse312su21
Is MLE for Variance of Normal unbiased? 2] ? ??2 = ?[1 ?? ? ? ?=1 ?? 2 2) 1 ?? ?? 1 ?? (?? 2 2?? ??+ = ?? = ?? 2 2 2 1 ?? ?? 1 ?? ?? 1 ?? ?? 2 2 ?? ??+ ? 2 2 ?? ? ??+ ? 2 ? = ?? = ?? = ?? 2 2 2 1 ?? ?? 2 1 1 ? ? ?? ?? ? 2 2 ? 2 ? = ?? = ?? = ? ?? ?? 2) = ??? ?? ??? 2 ? 2 (??? ?? + ? = ? ?? ?? = ???(??) + ? ?? ?? ?? ?? ? 1 1 = ??? ?? ??? = ??? ?? 1 = ??? ?? ?2??? ?? = ??? ?? ?2 ? ??? ?? ???? ?? =? 1 ???? ?? =? 1 ? ?2
Not Unbiased 2] ? ??2 = ?[1 =? 1 ?? ? ? ?=1 ?? ??2 Which is not what we wanted. This is a biased estimator. But it s not too biased An estimator ? is consistent if lim ? ? ? = ?
Correction The MLE slightly underestimates the true variance. You could correct for this! Just multiply by This would give you a formula of: ? ? 1. 2 ? 1 1 ? ?? ? ? ?=1 ? ?? 2 where 1 ?? is the sample mean. ?? ? 1 ?=1 = Called the sample variance because it s the variance you estimate if you want an (unbiased) estimate of the variance given only a sample. If you took a statistics course, you probably learned this as the definition of variance. ??
Whats with the ? 1? Soooooooooo, why is the MLE off? Intuition 1: when we re comparing to the real mean, ?1doesn t affect the real mean (the mean is what the mean is regardless of what you draw). But when you compare to the sample mean, ?1 pulls the sample mean toward it, decreasing the variance a tiny bit. Intuition 2: We only have ? 1 degrees of freedom with the mean and ? 1 of the data points, you know the final data point. Only ? 1 of the data points have information the last is fixed by the sample mean.
Why does it matter? When statisticians are estimating a variance from a sample, they usually divide by ? 1 instead of ?. They also (with unknown variance) generally don t use the CLT to estimate probabilities. They aren t using the ()table, they re using a different table based on the altered variance estimates.
Why use MLEs? Are there other estimators? If you have a prior distribution over what values of ? are likely, combining the idea of Bayes rule with the idea of an MLE will give you Maximum a posteriori probability estimation (MAP) You pick the maximum value of (?|?) starting from a known prior over possible values of ?. (?|?) (?) (?) argmax = argmax (?|?) (?) (?) is a constant, so the argmax is unchanged if you ignore it. Note when prior is constant, you get MLE!
MLE of Continuous Uniform Distribution Let ?1,?2, ,?? be independent samples from the continuous uniform distribution, ?~????(0,?). What is the MLE ? for ?? Fill out the poll everywhere so Kushal knows how long to explain Go to pollev.com/cse312su21
MLE of Continuous Uniform Distribution Let ?1,?2, ,?? be independent samples from the continuous uniform distribution, ?~????(0,?). What is the MLE ? for ?? 1 ? ? 0= ? ? ?1,?2, ,??;? = ?=1 ? ?? ;? = The derivative is 0 only when ? = When ? = , ;? = 0. This is a minimum and not a maximum. Additionally, we know that all ?? 0,? . Hence, if ? < max ?1,?2, ,?? , ;? = 0 ? ??? ?= ?? ? 1
MLE of Continuous Uniform Distribution When ? = , ;? = 0. This is a minimum and not a maximum. Additionally, we know that all ?? 0,? . Hence, if ? < max ?1,?2, ,?? , ;? = 0 This point of discontinuity, max ?1,?2, ,?? is where the likelihood is maximized. Hence the MLE is ? = max ?1,?2, ,??
