The Gaussian Distribution and Its Properties

 
The 
Gaussian
 Distribution
 
CSC 411/2515
 
Adopted from PRML 2.3
 
The Gaussian Distribution
 
Motivations:
 
Maximum of the entropy
Central limit theorem
 
The Gaussian Distribution: Properties
 
The Gaussian Distribution: Properties
 
The law
 
(quadratic function)
 is constant on elliptical surfaces
:
 
The Gaussian Distribution: more examples
 
Contours of constant probability density
:
 
in which the covariance matrix is:
a)
of general form
b)
diagonal
c)
proportional to the identity matrix
 
Conditional Law
 
Conditional Law
 
Conditional Law
 
Using the definition:
 
Inverse partition identity:
 
Conditional Law
 
The form using 
precision matrix
:
 
Marginal Law
 
The marginal distribution is given by:
 
The integral is equal to the normalization term
 
Marginal Law
 
The marginal distribution is a Gaussian with
 
Short Summary
 
Conditional distribution:
 
Marginal distribution:
 
Bayes’ theorem for Gaussian variables
 
Setup:
 
Bayes’ theorem for Gaussian variables
 
The same trick (consider the second order terms), we get
 
Maximum likelihood for the Gaussian
 
Setting the derivative to zero, we obtain the solution for the
maximum likelihood estimator:
 
Maximum likelihood for the Gaussian
 
The empirical mean is unbiased in the sense
 
However, the maximum likelihood estimate for the covariance
has an expectation that is less that the true value:
 
Conjugate prior for the Gaussian
 
Introducing prior distributions over the parameters of the
Gaussian
 
The maximum likelihood framework only gives point estimates
for the parameters, we would like to have uncertainty
estimation (confidence interval) for the estimation
 
The Gaussian Distribution: limitations
 
Maximum likelihood estimators are not robust to outliers:
Student’s t-distribution (bottom left)
 
Not able to describe periodic data: 
von Mises distribution
 
Unimodel distribution: 
Mixture of Gaussian (bottom right)
 
The Gaussian Distribution: frontiers
 
Gaussian Process
 
Bayesian Neural Networks
 
Generative modeling (Variational Autoencoder)
 
……
Slide Note
Embed
Share

This insightful content dives into the Gaussian Distribution, including its formulation for multidimensional vectors, properties, conditional laws, and examples. Explore topics like Mahalanobis distance, covariance matrix, elliptical surfaces, and the Gaussian distribution as a Gaussian function. Discover how to derive conditional distributions within the Gaussian framework.

  • Gaussian Distribution
  • Properties
  • Multivariate
  • Conditional Law
  • Probability

Uploaded on Aug 03, 2024 | 0 Views


Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

E N D

Presentation Transcript


  1. The Gaussian Distribution CSC 411/2515 Adopted from PRML 2.3

  2. The Gaussian Distribution For a ?-dimensional vector ?, the multivariate Gaussian distribution takes the form: 1 ? 2 exp 1 2? ? 1? ? ? ?|?, = 1 2 2? Motivations: Maximum of the entropy Central limit theorem

  3. The Gaussian Distribution: Properties The law is a function of the Mahalanobis distance from ? to ?: 2= ? ? 1? ? The expectation of ? under the Gaussian distribution is: ? ? = ? The covariance matrix of ? is: cov ? =

  4. The Gaussian Distribution: Properties The law (quadratic function) is constant on elliptical surfaces: ?? are the eigenvalues of ?? are the associated eigenvectors

  5. The Gaussian Distribution: more examples Contours of constant probability density: in which the covariance matrix is: a) of general form b) diagonal c) proportional to the identity matrix

  6. Conditional Law Given a Gaussian distribution ? ?|?, with , ? = ??,?? ? = ??,?? 1 ?? ?? ?? ?? ?? ?? ?? ?? = = What s the conditional distribution ?(??|??)?

  7. Conditional Law Given a Gaussian distribution ? ?|?, with , ? = ??,?? ? = ??,?? 1 ?? ?? ?? ?? ?? ?? ?? ?? = = What s the conditional distribution ?(??|??)? 1 2? ? 1? ? = 1 2?? ?? 1 2?? ?? ???? ?? 1 ???? ?? 2?? ?? ???? ?? 1 ???? ?? 2?? ??

  8. Conditional Law What s the conditional distribution ?(??|??)? 1 2? ? 1? ? = 1 2? 1? + ? 1? + const 1= ?? 1??|?= ???? ???? ?? ?|? ?|? Using the definition: 1 ?? ?? ?? ?? ?? ?? ?? ?? = = 1 1 ?? 1 ?? ??= ?? ?? ?? ??= ?? ?? ?? 1 ?? ?? 1 Inverse partition identity: 1 ??? 1 ? ? ? ? ? ? = ? ?? 1? 1 = ? 1?? ? 1+ ? 1???? 1

  9. Conditional Law The conditional distribution ? ???? is a Gaussian with: 1?? ?? ??|?= ??+ ?? ?? 1 ?? ?|?= ?? ?? ?? The form using precision matrix: 1?? ?? ??|?= ??+ ?? ?? ?|?= ??

  10. Marginal Law The marginal distribution is given by: ? ?? = ? ??,????? Picking out those terms that involve ??, we have 1 ? = 1 1? +1 ???? ?? ????+ ?? 1? 1? 2? ?? 2?? ?? ?? 2 ? = ???? ???? ?? Integrate over ??(unnormalized Gaussian) exp 1 ???? ?? 1? 1? ?? ?? ??? 2 The integral is equal to the normalization term

  11. Marginal Law After integrating over ??, we pick out the remaining terms: 1 ????+ ???? +1 1? + const ????+ ?? 2? ?? 2?? ? = ???? ???? ?? The marginal distribution is a Gaussian with ? ?? = ?? cov ?? = ??

  12. Short Summary 1 ?? ?? ?? ?? ?? ?? ?? ?? = = Conditional distribution: 1 ? ???? = ? ??|??|?, ?? 1 ??(?? ??) ??|?= ?? ?? Marginal distribution: ? ?? = ? ??|??, ??

  13. Bayes theorem for Gaussian variables Setup: ? ? = ? ?|?, 1 ? ?|? = ? ?|A? + ?,L 1 What s the marginal distribution ? ? and conditional distribution ? ?|? ? How about first compute ? ? , where ? = ?,? ? ? is a Gaussian distribution, consider the log of the joint distribution ln? ? =ln? ? + ln? ? ? = 1 2? ? ? ? 1 2? A? + ? L ? A? + ? + const

  14. Bayes theorem for Gaussian variables The same trick (consider the second order terms), we get ? ? ? = A? + ? 1 A 1 1A cov ? = ? 1+ A 1A We can then get ? ? and ? ?|? by marginal and conditional laws!

  15. Maximum likelihood for the Gaussian in which the observation Assume we have X = ?1, ,?? ?? are assumed to be drawn independently from a multivariate Gaussian, the log likelihood function is given by ? ln? X ?, = ?? ln2? ? 2ln 1 ?? ? 1?? ? 2 ?=1 2 Setting the derivative to zero, we obtain the solution for the maximum likelihood estimator: ? ? ??ln? X ?, = 1?? ? = 0 ?=1 ? ? ?ML=1 ML=1 ? ?=1 ?? ? ?=1 ?? ?ML ?? ?ML

  16. Maximum likelihood for the Gaussian The empirical mean is unbiased in the sense ? ?ML = ? However, the maximum likelihood estimate for the covariance has an expectation that is less that the true value: ? ML =? 1 ? ? We can correct it by multiplying ML by the factor ? 1

  17. Conjugate prior for the Gaussian The maximum likelihood framework only gives point estimates for the parameters, we would like to have uncertainty estimation (confidence interval) for the estimation Introducing prior distributions over the parameters of the Gaussian We would like the posterior ? ? ? ? ? ?(?|?) has the same form as the prior (Conjugate prior!) The conjugate prior for ? is a Gaussian The conjugate prior for precision is a Gamma distribution

  18. The Gaussian Distribution: limitations A lot of parameters to estimate ? +?2+? approximation (e.g., diagonal variance matrix) Maximum likelihood estimators are not robust to outliers: Student s t-distribution (bottom left) : structured 2 Not able to describe periodic data: von Mises distribution Unimodel distribution: Mixture of Gaussian (bottom right)

  19. The Gaussian Distribution: frontiers Gaussian Process Bayesian Neural Networks Generative modeling (Variational Autoencoder)

More Related Content

giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#