Understanding the Gaussian Distribution and Its Properties

This insightful content dives into the Gaussian Distribution, including its formulation for multidimensional vectors, properties, conditional laws, and examples. Explore topics like Mahalanobis distance, covariance matrix, elliptical surfaces, and the Gaussian distribution as a Gaussian function. Discover how to derive conditional distributions within the Gaussian framework.

• Gaussian Distribution
• Properties
• Multivariate
• Conditional Law
• Probability

keyaan Follow

Uploaded on Aug 03, 2024 | 0 Views

Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

Presentation Transcript

1. The Gaussian Distribution CSC 411/2515 Adopted from PRML 2.3

2. The Gaussian Distribution For a ?-dimensional vector ?, the multivariate Gaussian distribution takes the form: 1 ? 2 exp 1 2? ? 1? ? ? ?|?, = 1 2 2? Motivations: Maximum of the entropy Central limit theorem

3. The Gaussian Distribution: Properties The law is a function of the Mahalanobis distance from ? to ?: 2= ? ? 1? ? The expectation of ? under the Gaussian distribution is: ? ? = ? The covariance matrix of ? is: cov ? =

4. The Gaussian Distribution: Properties The law (quadratic function) is constant on elliptical surfaces: ?? are the eigenvalues of ?? are the associated eigenvectors

5. The Gaussian Distribution: more examples Contours of constant probability density: in which the covariance matrix is: a) of general form b) diagonal c) proportional to the identity matrix

6. Conditional Law Given a Gaussian distribution ? ?|?, with , ? = ??,?? ? = ??,?? 1 ?? ?? ?? ?? ?? ?? ?? ?? = = What s the conditional distribution ?(??|??)?

7. Conditional Law Given a Gaussian distribution ? ?|?, with , ? = ??,?? ? = ??,?? 1 ?? ?? ?? ?? ?? ?? ?? ?? = = What s the conditional distribution ?(??|??)? 1 2? ? 1? ? = 1 2?? ?? 1 2?? ?? ???? ?? 1 ???? ?? 2?? ?? ???? ?? 1 ???? ?? 2?? ??

8. Conditional Law What s the conditional distribution ?(??|??)? 1 2? ? 1? ? = 1 2? 1? + ? 1? + const 1= ?? 1??|?= ???? ???? ?? ?|? ?|? Using the definition: 1 ?? ?? ?? ?? ?? ?? ?? ?? = = 1 1 ?? 1 ?? ??= ?? ?? ?? ??= ?? ?? ?? 1 ?? ?? 1 Inverse partition identity: 1 ??? 1 ? ? ? ? ? ? = ? ?? 1? 1 = ? 1?? ? 1+ ? 1???? 1

9. Conditional Law The conditional distribution ? ???? is a Gaussian with: 1?? ?? ??|?= ??+ ?? ?? 1 ?? ?|?= ?? ?? ?? The form using precision matrix: 1?? ?? ??|?= ??+ ?? ?? ?|?= ??

10. Marginal Law The marginal distribution is given by: ? ?? = ? ??,????? Picking out those terms that involve ??, we have 1 ? = 1 1? +1 ???? ?? ????+ ?? 1? 1? 2? ?? 2?? ?? ?? 2 ? = ???? ???? ?? Integrate over ??(unnormalized Gaussian) exp 1 ???? ?? 1? 1? ?? ?? ??? 2 The integral is equal to the normalization term

11. Marginal Law After integrating over ??, we pick out the remaining terms: 1 ????+ ???? +1 1? + const ????+ ?? 2? ?? 2?? ? = ???? ???? ?? The marginal distribution is a Gaussian with ? ?? = ?? cov ?? = ??

12. Short Summary 1 ?? ?? ?? ?? ?? ?? ?? ?? = = Conditional distribution: 1 ? ???? = ? ??|??|?, ?? 1 ??(?? ??) ??|?= ?? ?? Marginal distribution: ? ?? = ? ??|??, ??

13. Bayes theorem for Gaussian variables Setup: ? ? = ? ?|?, 1 ? ?|? = ? ?|A? + ?,L 1 What s the marginal distribution ? ? and conditional distribution ? ?|? ? How about first compute ? ? , where ? = ?,? ? ? is a Gaussian distribution, consider the log of the joint distribution ln? ? =ln? ? + ln? ? ? = 1 2? ? ? ? 1 2? A? + ? L ? A? + ? + const

14. Bayes theorem for Gaussian variables The same trick (consider the second order terms), we get ? ? ? = A? + ? 1 A 1 1A cov ? = ? 1+ A 1A We can then get ? ? and ? ?|? by marginal and conditional laws!

15. Maximum likelihood for the Gaussian in which the observation Assume we have X = ?1, ,?? ?? are assumed to be drawn independently from a multivariate Gaussian, the log likelihood function is given by ? ln? X ?, = ?? ln2? ? 2ln 1 ?? ? 1?? ? 2 ?=1 2 Setting the derivative to zero, we obtain the solution for the maximum likelihood estimator: ? ? ??ln? X ?, = 1?? ? = 0 ?=1 ? ? ?ML=1 ML=1 ? ?=1 ?? ? ?=1 ?? ?ML ?? ?ML

16. Maximum likelihood for the Gaussian The empirical mean is unbiased in the sense ? ?ML = ? However, the maximum likelihood estimate for the covariance has an expectation that is less that the true value: ? ML =? 1 ? ? We can correct it by multiplying ML by the factor ? 1

17. Conjugate prior for the Gaussian The maximum likelihood framework only gives point estimates for the parameters, we would like to have uncertainty estimation (confidence interval) for the estimation Introducing prior distributions over the parameters of the Gaussian We would like the posterior ? ? ? ? ? ?(?|?) has the same form as the prior (Conjugate prior!) The conjugate prior for ? is a Gaussian The conjugate prior for precision is a Gamma distribution

18. The Gaussian Distribution: limitations A lot of parameters to estimate ? +?2+? approximation (e.g., diagonal variance matrix) Maximum likelihood estimators are not robust to outliers: Student s t-distribution (bottom left) : structured 2 Not able to describe periodic data: von Mises distribution Unimodel distribution: Mixture of Gaussian (bottom right)

19. The Gaussian Distribution: frontiers Gaussian Process Bayesian Neural Networks Generative modeling (Variational Autoencoder)