Principles and Statistics of Exploratory Factor Analysis

Exploratory Factor Analysis Principles and Statistics HSE Psychometric School August 2019 Prof. dr. Gavin T. L. Brown University of Auckland Ume University

Logic of Factor Analysis If an opinion is real and stable the mathematical pattern of responses to items related to the opinion should be highly correlated there should be little Noise or Error or Residual unexplained after the pattern is found Correlations between items of same factor should be higher than their correlations with items they are not related to Note the similarity to reliability theory in CTT

Effect of a Factor The factor simplifies the data Data reduction, dimension reduction Instead of reporting multiple items, we report one factor score and improves confidence in interpretations The weight of multiple measures ensures better estimation of opinion or attitude or ability The factor reduces chance effects The factor reduces error BUT it is not Directly observed or easily estimated

Getting to a Factor Direct observation/measurement Answers to test questions Response to rating statements Measures of volume, mass, area Indirect inference about latent trait or ability Factors that are derived from the statistical interaction of directly observable measures Is it real, if it cannot be directly observed? Is evolution not real just because it happens on a time scale too difficult for us to observe?

Interpretability You should be guided by theory and common sense as well as the factor loadings from the output You must be able to understand and interpret a factor if you are going to extract it You should try different factor models to ensure you have identified the best fitting and most interpretable solution, 1-factor, 2-factor model, 3-factor model, etc. 5

Factors Vectors through n-dimensional space that minimize distance of items to line (like regression & correlation) NB NB: Correlation of factors r = 0.00 when orthogonal

Very Simple Structure Thurstone, L. L. (1931). Multiple factor analysis. Psychological Review, 38(5), 406-427. doi:10.1037/h0069792 Every variable loads strongly on 1 factor All loadings on a factor are either large & positive or near zero Every row in matrix has at least one zero Every column contains at least k zeros Thus, all items fall into mutually exclusive groups with high loadings on 1 factor and weak to zero loadings on all other factors

Factor 1 2 3 4 5 6 7 8 ce5 ce3 ce2 ce1 ce6 ce4 sq2 bd5 bd1 bd4 bd2 ig1 bd3 ig2 ig3 si3 si4 si2 si5 si1 ti6 ti2 ti1 ti3 sf1 sq1 sf4 sf2 sf3 pe2 pe1 ti5 ti4 0.767 0.735 0.731 0.714 0.559 0.455 0.342 0.140 0.142 0.121 SCoA 8 factors -0.221 0.146 0.163 0.187 0.122 0.279 0.138 0.116 0.693 0.641 0.619 0.585 0.569 0.564 0.530 0.501 0.136 -0.147 0.152 0.171 -0.184 -0.190 0.101 -0.115 Values <.10 suppressed 0.111 -0.151 -0.200 -0.118 0.255 -0.154 Normally <.30 ignored -0.129 -0.249 -0.101 0.779 0.663 0.617 0.573 0.470 0.281 0.128 Generally this has Simple Structure with 1 or 2 exceptions 0.106 0.214 -0.114 0.120 0.153 0.159 0.105 0.157 -0.451 0.187 0.211 0.748 0.631 0.559 0.479 0.377 0.350 0.186 0.124 0.112 0.112 0.297 -0.112 0.246 0.105 -0.122 0.308 0.651 0.532 0.129 -0.115 -0.120 0.188 0.101 0.135 0.151 0.138 0.183 0.218 0.158 0.134 0.603 0.528 -0.124 0.106 0.220 0.199 0.486 0.475

Factor analysis estimation Find , , lambda(the estimated factor loading matrix) and , psi (the diagonal matrix containing the estimated specific variances), So that they reproduce as accurately as possible the sample covariance matrix Communality: the variance shared with the other variables via the common factors. specific or unique: variance relates to the variability in Xi not shared with other variables factor analysis is essentially unaffected by the rescaling of the variables. So can be applied to either the covariance matrix or the correlation matrix because the results are essentially equivalent. (Note that this is not the same as when using principal components analysis

Factors: Regression matrices X1 = 1f + u1, Subject Subject Classics Classics French French English English Classics 1.00 A 3-item factor regression set X2 = 2f + u2, French 0.83 1.00 X3 = 3f + u3, --------------------------------------------------------------------------- General formula General formula X = X = f + u f + u 11 1? q1 ?? ?? English 0.78 0.67 1.00 ?1 ?? ?1 = , f f = , u u = French Overlapping covariance Classics NB. Random disturbances u u are ALL uncorrelated with each other and factors f. THUS: Inter-correlations of items come from f NOT from correlated residuals English FA does not try to account for all the observed variance, only that shared through the common factors (on the diagonal). Goal of accounting for the covariances or correlations between manifest variables

Communality the proportion of variance of a variable that is accounted for by the common factors; 2 major methods for estimation the square of the multiple correlation coefficient of Xi with the other observed variables. OR the largest of the absolute values of the correlation coefficients between Xi and one of the other variables. ranges from 0-1; is considered to be low if the communality for a variable is below 0.5. 11

Steps in EFA Are you sure you should do EFA? Don t you have some theory or model or expectations as to how items group? If so, start with CFA. 1. Data preparation and inspection 2. Factor extraction 3. Factor rotation 4. Labeling factors - have you got a theory that explains why the items group the way they do? 5. Testing factors for coherence 12

Sample size To prepare the input data and determine whether factor analysis is appropriate Sample Sample Items per Items per factor factor Cases per Cases per variable variable NB. Inadmissible solutions can occur about 2% when N=400 50 12 4 100 8 12 200 6 10 500 3 5 Costello, A. B., & Osborne, J. W. (2005). Best practices in exploratory factor analysis: Four recommendations for getting the most from your analysis. Practical Assessment Research & Evaluation, 10(7), http://www.pareonline.net/pdf/v10n17.pdf. Marsh, H. W., & Hau, K. T. (1999). Confirmatory factor analysis: Strategies for small sample sizes. In R. H. Hoyle (Ed.), Statistical strategies for small sample research (pp. 251-284). London: SAGE Publications. Marsh, H. W., Hau, K.-T., Balla, J. R., & Grayson, D. (1998). Is more ever too much? The number of indicators per factor in confirmatory factor analysis. Multivariate Behavioral Research, 33(2), 181-220. doi:10.1207/s15327906mbr3302_1 13

Sample size Overall sample size interpretive guidelines Comrey, A. L., & Lee, H. B. (1992). A first course in factor analysis (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates, Inc. 50 = very poor 100 = poor 200 = fair 300 = good 500 = very good 1000+ = excellent 14

Factorability Factorability of the correlation matrix Intercorrelations among variables > 0.3 Anti-image matrix diagonals > 0.5 Kaiser-Meyer-Olkin (KMO) Measures of sampling adequacy > 0.6 Significant Bartlett s Test of Sphericity 15

Factor extraction Extraction methods Principal Components (PC): NOT Factor Analysis Principal Axis Factoring (PAF) Maximum Likelihood (ML) 16

Extraction by Variable types Ordinal Binary category: use Asymptotic distribution free estimator Up to 4 categories: use WLSMV or PAF estimator 5 + categories: use ML estimator or ML robust estimator Continuous Use ML estimator NB. PCA is not factor analysis (no errors); so don t think it s the same or do it

Principal Factor Analysis (PAF) To get values for communalities without knowing the factor loadings. If using the correlation matrix of manifest variables, 2 frequently used methods: the square of the multiple correlation coefficient of Xi with the other observed variables. the largest of the absolute values of the correlation coefficients between Xi and one of the other variables. Each of these will lead to higher values for the initial communality when Xi is highly correlated with at least some of t he other manifest variables, which is essentially what is required. The PCA of communalities is run iteratively, leading to eigenvectors that estimate factors in model; re-estimate communalities; repeat until convergence is achieved

Maximum Likelihood Estimation method of estimating the parameters by maximizing a likelihood function, so that under the assumed statistical model the observed data is most probable. assumes that the data being analysed have a multivariate normal distribution If the sample resembles the population through proper sampling then the sample values are likely to represent the population values. The function F=0, if T + is equal to S; otherwise F>0. Likelihood estimates of the loadings and the specific variances are found by minimising F with respect to these parameters. ln=likelihood function of particular values for the data; = item correlation matrix; T = transposed inversion of matrix; = communalities matrix on diagonal; S = data covariance matrix; q=# of variables in the data matrix 19

MLE or PAF MLE: if data are relatively normally distributed because it allows for the computation of a wide range of indexes of the goodness of fit of the model [and] permits statistical significance testing of factor loadings and correlations among factors and the computation of confidence intervals. (p. 277). PAF: If multivariate normality is severely violated ML or PAF give best results, depending on whether your data are generally normally- distributed or significantly non-normal, respectively. Fabrigar, L. R., Wegener, D. T., MacCallum, R. C., & Strahan, E. J. (1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4(3), 272-299. doi:10.1037/1082- 989X.4.3.272

EFA extraction CE factor PCA PCA MLE MLE ITEM INITIAL COMMUNALITY INITIAL COMMUNALITY CE1 1.000 .589 .474 .500 CE2 1.000 .657 .540 .592 CE3 1.000 .707 .580 .667 CE4 1.000 .510 .379 .403 CE5 1.000 .712 .591 .671 CE6 1.000 .610 .485 .525 21 Done in SPSS

Data type Level of measurement All variables must be suitable for correlational analysis and they should be in: Interval or ratio scale Ordinal scale/Likert scale Jamovi Ordinal in step icon Ratio in ruler icon 22

Different estimators in Jamovi PCA MLE PAF

Factor Loadings 2 kinds of relations between the variable and factor, with the influence of other factors partialled out the correlation coefficient (Structure matrix) regression coefficient (Pattern matrix) the squared factor loading is the percentage of variance in the variable that is explained by a factor 1-SMC = error or residual; we want error to be small 24

Rotation - To transform the initial pattern matrix into simple structure for easier interpretation Initial solution Two basic types of factor rotation Why rotate a factor loading matrix? Interpretability 25

Why rotate a factor loading matrix? After rotation, the lines of best fit are re-arranged to optimally go through clusters of shared variance among the variables Then the factor loadings and the factor can be more readily interpreted The rotated factor structure is simpler and more easily interpretable 26

Two basic types of factor rotation Orthogonal Rotation Oblique Rotation (Oblimin) (Varimax) 27

Orthogonal vs. Oblique Rotations Consider purpose of factor analysis If in doubt, try both rotations Consider interpretability Look at correlations between factors in oblique solution, if it is more than 0.1 then go with oblique rotation 28

Orthogonal Rotation (OR) Orthogonal rotation shifts the vectors in order to maintain 90o between each vector. This produces the simplest structure Angles that meet at the ceiling of a square room are vectors perfectly orthogonal (90o to each other) F1 R1 = angle of rotation F2 29 R2

Orthogonal Rotation (OR) Varimax rotation: is the most popular orthogonal rotation technique enables each factor have high loadings on a small number of variables and low loadings on the others 30

Oblique Rotation (OBR) Oblique rotation shifts the factors in the factor space greater or less than 90o of the factors to one another to achieve the best simple structure phenomena are likely to be correlated with each other 31

Jamovi: 2 rotations what s different? 'Maximum likelihood' extraction with 'oblimin' rotation Factor 'Maximum likelihood' extraction with 'varimax' rotation Factor 1 2 3 Uniqueness 1 2 3 Uniqueness ce1 ce2 ce3 ce5 pe1 pe2 ce6 ce4 0.811 0.791 0.603 0.449 ce2 ce1 ce3 ce5 ce6 ce4 pe1 pe2 0.693 0.673 0.616 0.545 0.320 0.338 0.321 0.43255 0.32940 0.34540 0.34795 0.00500 0.50379 0.29201 0.56474 0.32940 0.43255 0.34540 0.34795 0.29201 0.56474 0.00500 0.50379 0.454 0.542 0.751 0.489 0.399 1.008 0.456 0.831 0.442 0.944 0.519 0.307 0.365

Factor names The name should : encapsulate the substantive nature of the factor enable others to grasp its meaning TEST the label with others 33

Best Factor Analysis Maximum Likelihood Estimation (MLE) Best estimate of population values Not principal components which includes error Oblique Rotation Not orthogonal At least 3 items per factor Loading >.30, no off-loadings >.30 At least 5 cases per item Preferably 10 Bandalos, D. L., & Finney, S. J. (2010). Factor analysis: Exploratory and confirmatory. In G. R. Hancock & R. O. Mueller (Eds.), The reviewer's guide to quantitative methods in the social sciences (pp. 93-114). New York: Routledge. Costello, A. B., & Osborne, J. W. (2005). Best practices in exploratory factor analysis: Four recommendations for getting the most from your analysis. Practical Assessment Research & Evaluation, 10(7).

SCoA factors [affect/social + external attributions] (MLE, Oblimin rotation, regressions) Factor 1 2 Factor Correlation Matrix F1 is the CE and PE as expected F2 is the SF and SQ items as expected Item SQ2 is problematic The correlation suggests 2 factors are tenable The fit says that the difference between data and this model is more than chance (p<.001), so the discrepancy is not due to chance. Is the model wrong? ce5 .909 -.153 Factor 1 2 ce3 .799 .017 1 1.000 .598 ce6 .751 -.045 2 .598 1.000 ce2 .742 .057 ce1 .691 .004 Goodness-of-fit Test ce4 .635 .019 Chi-Square df Sig. pe1 .537 .178 332.445 64 .000 pe2 .498 .231 sq2 .320 .308 sf4 -.053 .668 sf3 .021 .629 sf1 -.061 .596 sf2 .183 .554 sq1 .111 .465