Understanding Principal Component Analysis (PCA) in Data Analysis
Introduction to Principal Component Analysis (PCA) by J.-S. Roger Jang from MIR Lab, CSIE Dept., National Taiwan University. PCA is a method for reducing dataset dimensionality while preserving spatial characteristics. It has applications in line/plane fitting, face recognition, and machine learning. This summary covers the characteristics, comparison with K-Means clustering, examples of PCA projections, problem definition, projection angles, eigenvalues, eigenvectors, and mathematical formulations related to PCA. Explore the key concepts and applications of PCA in data analysis.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Principal Component Analysis (PCA) J.-S. Roger Jang ( ) MIR Lab, CSIE Dept. National Taiwan University jang@mirlab.org, http://mirlab.org/jang 2024/9/21
Intro. to PCA PCA (Principal Component Analysis) Characteristics: For unlabeled data An effective method for reducing a dataset s dimensionality while keeping spatial characteristics as much as possible A linear transform with solid mathematical foundation Applications Line/plane fitting Face recognition Machine learning ... 2/23
Comparison: PCA & K-Means Clustering Common goal: Reduction of unlabeled data PCA: dimensionality reduction Objective function: Variance Quiz! K-means clustering: data count reduction Objective function: Distortion | | | ?1 | ?2 | ?? | ? = 3/23
Examples of PCA Projections 2D 1D 3D 2D 4/23
Problem Definition Quiz! Input Output A dataset X of n d-dim points which are zero justified: A unity vector u such that the square sum of the dataset s projection onto u is maximized. Dataset:? = ?1,?2,...,?? ? Zero justified: ??= ? ?=1 5/23
Projection Angle between vectors Projection of x onto u ? cos? =??? ??? ? = ??? if ? = 1 cos? = ? ? Extension: What is the projection of x onto the subspace spanned by u1, u2, , um? u x = = T x x if u u cos 1 6/23
Eigenvalue & Eigenvector Quiz! Definition of eigenvector x and eigenvalue of a square matrix A: ?? = ?? or ? ?? ? = 0 x is non-zero ? ?? is singular ? ?? = 0 7/23
Demo of Eigenvectors and Eigenvalue Try eigshow in MATLAB to plot trajectories of a linear transform in 2D Cleve s comments Quiz: When x sweeps from 0 to 360 degrees, how many times will x and Ax have the same (or exactly the opposite) direction? 8/23
Mathematical Formulation Dataset representation: X is d by n, with n>d Square sum: ? ? = ?2= ??? = ??????? = ?????? Objective function with a constraint on u: ( ) ( ) u u u , | | | ?1 | ?2 | ?? | ? = = = T T T u u XX u u u max s.t. J , , 1 1 u ( 1 ) ~ = + T T T XX u u u max , J Projection of each column of X onto u: Lagrange multiplier T 1 x u Lagrange Multipliers | Geometric Meaning & Full Example T 2 x u = = T p X u T n x u 9/23
Optimization of the Obj. Function Set the gradient to zero: Arrange the eigenvalues: ? ? ?,? = ??????? + ? 1 ??? 2???? 2?? = 0 ???? = ?? = 0 ?1 ?2 ?? Max of J(u) is 1, which occurs at u=u1 Min of J(u) is d, which occurs at u=ud XXT = Covariance matrix multiplied by n-1. u is the eigenvector while is the eigenvalue When u is the eigenvector: ? u = p2= u?XX?u = u??u = ? 10/23
Facts about Symmetric Matrices A symmetric matrix have orthogonal eigenvectors corresponding to different eigenvalues. Quiz! Proof: ??1= ?1?1 ??2= ?2?2 ?2?1 ???2= ?1 ??2?2= ?2?1 ??2= ?1?1 ??2= 0 ?1 ??2 ??2 ?1 ????2= ??1 ??2 ?2 ?1?1 ?1 ??2= ?1?1 ??2= 0. Quiz: Does a symmetric matrix always have orthogonal eigenvectors? Answer: No! (Can you give an example?) 11/23
Basis Conversion Conversion between orthonormal bases = 1, if i otherwise j = T i u u =u u i j j 0, | | | = = = 1 T T U u u u U U U U I 1 2 d | | | y y 1 | | | 2 = + + + = = x 1 1 y u u u u u u U y y y 2 2 1 2 d d d | | | y d = 1 T y U x=U x Projection of x onto u1, u2, 12/23
Steps for PCA 1 ? ?=1 ? Find the sample mean: = x? 1. 1 ? 1???= Compute the covariance matrix: C = 1 ? 1 ?=1 Find the eigenvalues of (n-1)C and arrange them into descending order, ?1 ?2 ?? with the corresponding eigenvectors ?1,?2, ,??. The transformation is ? = ???, with 2. ?(x? )(x? )? 3. 4. | | | ?1 | ?2 | ?? | ? = 13/23
LS vs. TLS Quiz! Line fitting: Given a set of n points (?1,?1),(?2,?2),...,(??,??) in 2D, find a line to minimize fitting error defined next: LS (least squares) Not robust if the line is vertical! Line: ? = ?? + ? n 2 Fitting error: ? ?,? = i=1 TLS (total least squares) ???+ ? ?? Quiz: Prove that both LS and TLS lines go through the average of these n points. Line: ?? + ?? + ? = 0 ???+???+?2 ?2+?2 n Fitting error: ? ?,?,? = i=1 Hint The : shortest , y + distance between by ax + ( ) + = point a ax and c a line 0 x c 0 0 + by + 0 0 b is . 14/23 2 2 a Source
Three Steps of PCA for TLS Quiz! 2D 3D Set data average to zero. Set data average to zero. Find u1& u2via PCA. Use u2 as the normal vector of the fitting line. Find u1, u2, & u3via PCA. Use u3as the normal vector of the fitting plane. Use the normal vector and the data average to find the fitting line. Use the normal vector and the data average to find the fitting plane. 15/23
More about PCA Comparison of methods for dim. reduction PCA: For unlabeled data Unsupervised leaning LDA (linear discriminant analysis): For classifying labeled data Supervised learning If d>>n, then we need to have a workaround for computing the eigenvectors 16/23
Example of PCA IRIS dataset projection 17/23
PCA Not for Classification PCA is not designed for classification directly. Ideal situation Adversary situation 18/23
LDA for Classification LDA projection onto directions that can best separate data of different classes. Adversary situation for PCA Ideal situation for LDA 19/23
Exercise 1 Given two vectors x=[3, 4], y=[5, 5], find the projection of y onto x. 20/23
Exercise 2 Given a point [1 2 3] and a plane x+2y+2z=14, find the distance between the point and the plane. 21/23
Exercise 3 Find the eigenvalues and eigenvectors of A=[0 3; 1 2]. 22/23
Exercise 4 Given a data matrix X=[2 0 3 -1; 0 -2 -3 1], compute the variance after projecting the dataset onto its first principal component. 23/23
