Which Factor Retention Method Performs Best? A Monte Carlo Simulation for Psychological and Educational Measurement

Bhilan Rhiyu Antama; Edi Istiyono; Moh. Khairudin; Syukrul Hamdi; Heni Setiyaningsih

doi:10.55681/armada.v4i9.3847

Authors

Bhilan Rhiyu Antama Educational Research and Evaluation, Graduate School, Yogyakarta State University, Indonesia
Edi Istiyono Educational Research and Evaluation, Graduate School, Yogyakarta State University, Indonesia
Moh. Khairudin Educational Research and Evaluation, Graduate School, Yogyakarta State University, Indonesia
Syukrul Hamdi Educational Research and Evaluation, Graduate School, Yogyakarta State University, Indonesia
Heni Setiyaningsih Educational Research and Evaluation, Graduate School, Yogyakarta State University, Indonesia

DOI:

https://doi.org/10.55681/armada.v4i9.3847

Keywords:

Dimensionality Assessment, Exploratory Graph Analysis, Factor Retention, Monte Carlo Simulation, Parallel Analysis

Abstract

Determining the appropriate number of factors is the most critical decision in exploratory factor analysis (EFA), but evidence-based guidance for the context of skewed ordinal data remains limited. This study compares the performance of three factor retention methods: optimal coordinates (OC), parallel analysis (PA), and exploratory graph analysis (EGA) through Monte Carlo simulations with 1000 replications using negatively skewed 5-category Likert scale data. Controlled conditions included the number of factors (2, 3, 5), sample size (300, 500, 900, 1500), factor loadings (0.4; 0.7), and inter-factor correlations (0.0; 0.3). The results show that PA achieved the highest accuracy (94.7%), followed by EGA (92.8%), while OC lagged far behind (36.8%). PA achieved perfect accuracy on large samples with high loadings, whereas OC exhibited a tendency toward under-extraction that increased drastically as the number of factors increased. EGA outperformed PA in minimizing over-extraction but was more prone to under-extraction in complex structures. These findings recommend PA as the primary method and EGA as a complementary method in Likert-scale-based research and confirm that algorithmic scree plots are not suitable for use as the sole criterion for factor retention.

Downloads

Download data is not yet available.

References

Auerswald, M., & Moshagen, M. (2019). How to determine the number of factors to retain in exploratory factor analysis: A comparison of extraction methods under realistic conditions. Psychological Methods, 24(4), 468–491. https://doi.org/10.1037/MET0000200

Brandenburg, N., & Papenberg, M. (2022). Reassessment of Innovative Methods to Determine the Number of Factors: A Simulation-Based Comparison of Exploratory Graph Analysis and Next Eigenvalue Sufficiency Test. Psychological Methods, 29(1), 21–47. https://doi.org/10.1037/MET0000527

Cattell, R. B. (1966). The Scree Test For The Number Of Factors. Multivariate Behavioral Research, 1(2), 245–276. https://doi.org/10.1207/S15327906MBR0102_10

Christensen, A. P., Garrido, L. E., Guerra-Peña, K., & Golino, H. (2023). Comparing community detection algorithms in psychometric networks: A Monte Carlo simulation. Behavior Research Methods 2023 56:3, 56(3), 1485–1505. https://doi.org/10.3758/S13428-023-02106-4

Cosemans, T., Rosseel, Y., & Gelper, S. (2022). Exploratory Graph Analysis for Factor Retention: Simulation Results for Continuous and Binary Data. Educational and Psychological Measurement, 82(5), 880–910. https://doi.org/10.1177/00131644211059089

Fabrigar, L. R., MacCallum, R. C., Wegener, D. T., & Strahan, E. J. (1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4(3), 272–299. https://doi.org/10.1037/1082-989X.4.3.272

Finch, W. H. (2023). A Comparison of Methods for Determining the Number of Factors to Retain with Exploratory Factor Analysis of Dichotomous Data. Psych 2023, Vol. 5, Pages 1004-1018, 5(3), 1004–1018. https://doi.org/10.3390/PSYCH5030067

Garrido, L. E., Abad, F. J., & Ponsoda, V. (2016). Are fit indices really fit to estimate the number of factors with categorical variables? Some cautionary findings via monte carlo simulation. Psychological Methods, 21(1), 93–111. https://doi.org/10.1037/MET0000064

Golino, H. F., & Epskamp, S. (2017). Exploratory graph analysis: A new approach for estimating the number of dimensions in psychological research. PLOS ONE, 12(6), e0174035. https://doi.org/10.1371/JOURNAL.PONE.0174035

Golino, H., Shi, D., Christensen, A. P., Garrido, L. E., Nieto, M. D., Sadana, R., Thiyagarajan, J. A., & Martínez-Molina, A. (2019). Investigating the performance of Exploratory Graph Analysis and traditional techniques to identify the number of latent factors: A simulation and tutorial. Psychological Methods, 25(3), 292–320. https://doi.org/10.1037/MET0000255

Goretzko, D. (2022). Factor Retention in Exploratory Factor Analysis With Missing Data. Educational and Psychological Measurement, 82(3), 444–464. https://doi.org/10.1177/00131644211022031

Goretzko, D., & Bühner, M. (2020). One model to rule them all? Using machine learning algorithms to determine the number of factors in exploratory factor analysis. Psychological Methods, 25(6), 776–786. https://doi.org/10.1037/MET0000262

Goretzko, D., & Bühner, M. (2022). Factor Retention Using Machine Learning With Ordinal Data. Applied Psychological Measurement, 46(5), 406–421. https://doi.org/10.1177/01466216221089345

Goretzko, D., Siemund, K., & Sterner, P. (2024). Evaluating Model Fit of Measurement Models in Confirmatory Factor Analysis. Educational and Psychological Measurement, 84(1), 123–144. https://doi.org/10.1177/00131644231163813

Horn, J. L. (1965). A Rationale and Test for the Number of Factors in Factor Analysis. Psychometrika, 30(2), 179–185. https://doi.org/10.1007/BF02289447

Howard, M. C. (2016). A Review of Exploratory Factor Analysis Decisions and Overview of Current Practices: What We Are Doing and How Can We Improve? International Journal of Human-Computer Interaction, 32(1), 51–62. https://doi.org/10.1080/10447318.2015.1087664

Howard, M. C., & Henderson, J. (2023). A review of exploratory factor analysis in tourism and hospitality research: Identifying current practices and avenues for improvement. Journal of Business Research, 154, 113328. https://doi.org/10.1016/J.JBUSRES.2022.113328

Lee, H., & Cham, H. (2024). Comparing Accuracy of Parallel Analysis and Fit Statistics for Estimating the Number of Factors With Ordered Categorical Data in Exploratory Factor Analysis. Educational and Psychological Measurement, 84(6), 1173–1202. https://doi.org/10.1177/00131644241240435

Lim, S., & Jahng, S. (2019). Determining the number of factors using parallel analysis and its recent variants. Psychological Methods, 24(4), 452–467. https://doi.org/10.1037/MET0000230

Lorenzo-Seva, U., & Ferrando, P. J. (2024). Determining Sample Size Requirements in EFA Solutions: A Simple Empirical Proposal. Multivariate Behavioral Research, 59(5), 899–912. https://doi.org/10.1080/00273171.2024.2342324

Markos, A., & Tsigilis, N. (2024). Dimensionality assessment in ordinal data: a comparison between parallel analysis and exploratory graph analysis. Frontiers in Psychology, 15, 1359111. https://doi.org/10.3389/FPSYG.2024.1359111/TEXT