Impact of Consequences and Contexts on Effort in International Large-Scale Assessments Presentation
The study explores how different consequences and contexts influence effort in International Large-Scale Assessments (ILSAs), focusing on the presentation to the AEA-Europe Conference in Dublin, Ireland. By examining test-taking motivation across jurisdictions like Shanghai and New Zealand, the research sheds light on the varying impact of stakes and motivations on student performance. The proposed model and experimental conditions aim to uncover the intricate relationships between student conceptions of assessment, test-taking motivation, and performance outcomes.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Different Consequences and Contexts Produce different effort on ILSAs Presentation to the AEA-Europe Conference, Dublin, Ireland (November, 2022) Anran Zhao, Gavin T. L. Brown*, and Kane Meissel Faculty of Education and Social Work, The University of Auckland *=presenter; gt.brown@auckland.ac.nz Download from: https://doi.org/10.17608/k6.auckland.21317484.v1
Sources Zhao, A. (2021). Sources of Sample Bias in PISA: Selectivity and Effort Differences between Shanghai and New Zealand [unpublished Ph.D. thesis, The University of Auckland]. Auckland, NZ. Zhao, A., Brown, G. T. L., & Meissel, K. (2020). Manipulating the consequences of tests: how Shanghai teens react to different consequences. Educational Research and Evaluation, 26(5-6), 221-251. https://doi.org/10.1080/13803611.2021.1963938 Zhao, A., Brown, G. T. L., & Meissel, K. (2022). New Zealand students test-taking motivation: an experimental study examining the effects of stakes. Assessment in Education: Principles, Policy & Practice, 29(4), 397-421. https://doi.org/10.1080/0969594X.2022.2101043
ILSA PISA, TIMSS, etc. reports assume that greater performance is explained by greater ability rather than motivation Between-country comparisons are made assuming score is a pure and accurate measure But the importance of ILSAs at the student level differs across jurisdictions (Ekl f, Pave i , & Gr nmo, 2014) or contexts (Knekta, 2017)
Test-taking motivation matters TTM predicts performance it is possible that country reputation consequences may elicit different effort in different societies, rather than ability Students in Shanghai scored world best for PISA in 2009 and 2012, but, given the importance of political-ideological education in China (Chen & Brown, 2018), that score MAY depend upon TTM effort for national reputation.
This study between-subjects experiment: 3 conditions The consequence of a hypothetical test (i.e., none, my country, or me personally) in Shanghai (n=1003) and New Zealand (n=479) jurisdictions that range from high to middle on PISA. Instruments four factor Student Conceptions of Assessment (Brown, 2008) inventory before being randomly assigned to experimental condition TTM estimated for effort, the importance of the test, and their anxiety for that test; adapted from Knekta and Ekl f (2015) and Thelk et al. (2009). Analysis Confirmatory factor analysis for measurement models, invariance testing between jurisdictions structural equation modeling to examine relations between SCoA and TTM.
Measurement Model Results SCoA measurement model bi-factor: four orthogonal specific factors and one general factor, dropping two items [i.e., sq1 and pe1]); correspondence to both NZ and Shanghai data TTM measurement model invariant within jurisdictions across consequence conditions, but only configural equivalence across jurisdictions
Structural Models Shanghai New Zealand
Structural Equation Model Results invariant within Shanghai across the three experimental conditions, only configural invariance in New Zealand, supports ecological rationality (Rieskamp & Reimer, 2007) in how students respond; that is, student TTM changes in New Zealand according to consequence, but doesn t in Shanghai.
Mean Score differences by Jurisdiction & Condition Only TTM Compared to no or personal stakes NZ big differences SH small to no differences
Test Importance by Condition & Jurisdiction
Impact of Condition by Jurisdiction on Effort, Anxiety 18/22
SCoA Validation The bi-factor structure of the SCoA replicated previous studies (Matos et al., 2019; Weekers et al., 2009). SCoA had a statistically significant, largely through the general factor, but small impact on test-taking motivation in both the Shanghai and New Zealand samples. in the Shanghai sample, the stronger the general conception of test as improving learning, being related to external attributions, class climate and personal emotions, the more students thought tests were important, stimulated greater anxiety, and exerted more effort. in the New Zealand sample, students general conception of test was not a significant predictor of their reported anxiety.
TTM Regardless of condition, greater anxiety resulted in greater effort in both jurisdictions. consistent with control-value theory in that negative emotion anxiety is an activating force for greater effort and performance (Pekrun et al., 2002). Non-invariance in the structural model between Shanghai and New Zealand predominantly because of the country at stakes condition, indicating students from the two contexts perceived this consequence quite differently.
Conclusion students in high-performing societies might have tried harder on PISA than students elsewhere, because of the importance of national reputation within that jurisdiction. Motivational factors (i.e., Effort, Importance, and Anxiety) were all greater as consequences shifted from none to country at stake and were highest in individual consequences. margin of increase on motivational factors by stakes was trivial or small for Shanghai and moderate to large for New Zealand. Corroborates Gneezy et al. (2019) and Chen and Brown (2018), in that there may be no such thing as a low-consequence test in Shanghai insofar as test-taking motivation is concerned. Thus, PISA scores from Shanghai may reflect the continuing importance of being tested in that society, as much as greater competence.
References Chen, J., & Brown, G. T. L. (2018). Chinese secondary school students' conceptions of assessment and achievement emotions: endorsed purposes lead to positive and negative feelings. Asia Pacific Journal of Education, 38(1), 91-109. https://doi.org/10.1080/02188791.2018.1423951 Ekl f, H., Pave i , B. J., & Gr nmo, L. S. (2014). A Cross-National Comparison of Reported Effort and Mathematics Performance in TIMSS Advanced. Applied Measurement in Education, 27(1), 31-45. https://doi.org/10.1080/08957347.2013.853070 Gneezy, U., List, J. A., Livingston, J. A., Qin, X., Sadoff, S., & Xu, Y. (2019). Measuring success in education: The role of effort on the test itself. American Economic Review: Insights, 1(3), 291 308. https://doi.org/10.1257/aeri.20180633 Knekta, E. (2017). Are all Pupils Equally Motivated to do Their Best on all Tests? Differences in Reported Test-Taking Motivation within and between Tests with Different Stakes. Scandinavian Journal of Educational Research, 61(1), 95- 111. https://doi.org/10.1080/00313831.2015.1119723 Matos, D. A. S., Brown, G. T. L., & Gomes, C. (2019). Bifactor invariance analysis of Student Conceptions of Assessment Inventory. Psico-USF, 24(4), 737-750. https://doi.org/http://dx.doi.org/10.1590/1413-82712019240411 Pekrun, R., Goetz, T., Titz, W., & Perry, R. P. (2002). Academic emotions in students self-regulated learning and achievement: A program of qualitative and quantitative research. Educational Psychologist, 37(2), 91-105. https://doi.org/10.1207/S15326985EP3702_4 Rieskamp, J., & Reimer, T. (2007). Ecological rationality. In R. F. Baumeister & K. D. Vohs (Eds.), Encyclopedia of Social Psychology (pp. 273-275). Sage. https://doi.org/10.4135/9781412956253.n166 Weekers, A. M., Brown, G. T. L., & Veldkamp, B. P. (2009). Analyzing the dimensionality of the Students' Conceptions of Assessment (SCoA) inventory. In D. M. McInerney, G. T. L. Brown, & G. A. D. Liem (Eds.), Student perspectives on assessment: What students can tell us about assessment for learning. (pp. 133-157). Information Age Publishing.
