Analysis of National Student Survey Results: Insights on Student Engagement by Dr. Alex Buckley
The presentation by Dr. Alex Buckley from the University of Edinburgh on December 4, 2018 emphasized that there are few significant differences in university and course-level ratings across institutions, questioning their suitability for constructing league tables. The discussion highlighted the importance of statistical significance, confidence intervals, and comparisons with sector averages in interpreting NSS data. Various subject areas and their evaluation were also examined.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Analysing NSS results and implications for student engagement Dr Alex Buckley University of Edinburgh, 4thDecember 2018
[A]t the university level, there are relatively few universities that differ significantly from the mean across all universities and, at the course level, there is even a smaller portion of differences that are statistically significant. This suggests the inappropriateness of these ratings for the construction of league tables. (Cheng and Marsh 2010)
Statistical significance and confidence intervals 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Example 1 Example 2 Institution A Institution B
1. How different are the scores from other institutions? 2. How different are they from last year? 3. How different are they from the sector average?
100% 90% 80% 70% 60% 50% 86% 85% 40% 72% 69% 62% 30% 20% 10% 0% Year 2012 Year 2013 Year 2014 Year 2015 Year 2016
Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q12 Q13 Q14 Q15 Q16 Q17 Q18 Q19 Q20 Q21 Q22 Q23 Q24 Q25 Q26 Q27 ( 004) ? Pharm acol ogy, ? Toxi col ogy? - - - - - ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND and?Pharm acy + + + + ( 010) ? Medi cal ?Technol ogy ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ( 011) ? Others? i n?Subj ects? al l i ed? t o? + ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND Medi ci ne - ( 012) ? Bi ol ogy ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND - - ( 015) ? Mi crobi ol ogy ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ( 016) ? Mol ecul ar?Bi ol ogy, ? - ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND Bi ophysi cs?and? Bi ochemi stry - - - - - - - ( 017) ? Others? i n?Bi ol ogi cal ? Sci ences ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND - ( 019) ? Psychol ogy ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND - - - - - ( 025) ? Chem i stry ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND + + + + ( 026) ? Physi cs? and?Astronom y ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND - ( 032) ? Mathem ati cs? and? Stati st i cs ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ( 033) ? Operati onal ?Research ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND - + + ( 035) ? Com puter? Sci ence ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ( 037) ? Mechani cal , ?Producti on?and? - - ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND Manufacturi ng? Engi neeri ng ( 039) ? Naval ?Archi tecture ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ( 040) ? El ectroni c? and?El ectri cal ? + + + + ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND Engi neeri ng ( 041) ? Ci vi l ? Engi neeri ng ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ( 042) ? Chem i cal , ?Process? and? - - - - - - - - - - - - - - - - - ND ND ND ND ND ND ND ND ND ND Energy? Engi neeri ng + - - + ( 047) ? Archi tecture ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ( 052) ? Econom i cs ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND - ( 053) ? Pol i ti cs ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND + - ( 058) ? Soci al ?Work ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND + - - - - - + + - + - - ( 060) ? Law ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND + + - + + + + ( 061) ? Busi ness? studi es ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND + + ( 062) ? Marketi ng ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ( 063) ? Managem ent ? studi es ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ( 064) ? Hum an? Resource? - ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND Managem ent ( 065) ? Fi nance ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ( 066) ? Accounti ng ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ( 067) ? Touri sm , ? Transport? and? ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND Travel ( 068) ? Others? i n?Busi ness? and? ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND Adm i ni strati ve? studi es - - - ( 073) ? Journal i sm ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND - - ( 075) ? Engl i sh?studi es ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ( 079) ? French? studi es ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND - ( 082) ? I beri an? studi es ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND - - ( 090) ? Hi story ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ( 105) ? Academ i c? studi es? i n? - ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND Educati on - - - + - - ( 106) ? Others? i n?Educati on ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND - - - - - - - - - - - - ( 108) ? I ni ti al ?Teacher? Trai ni ng ND ND ND ND ND ND ND ND ND ND ND ND ND ND ND
27 - Total questions - Questions that differ from at least one other - (6%) 17 uni - 24% - Questions that differ from sector average 6 - Questions that differ from previous year - 2%
[T]here is an assumption that when teachers transmit feedback information to students these messages are easily decoded and translated into action. Yet, there is strong evidence that feedback messages are invariably complex and difficult to decipher, and that students require opportunities to construct actively an understanding of them (e.g. through discussion) before they can be used to regulate performance (Nicol and Macfarlane-Dick 2006)
[T]here is an assumption that when students transmit feedback information to teachers these messages are easily decoded and translated into action. Yet, there is strong evidence that feedback messages are invariably complex and difficult to decipher, and that teachers require opportunities to construct actively an understanding of them (e.g. through discussion) before they can be used to regulate performance (Nicol and Macfarlane-Dick 2006)
Working with students to explore NSS data Incorporating critique of NSS questionnaire and analysis of NSS data into statistics module Helping SU to use NSS to develop yearly priorities and objectives Funding a student researcher to work with the SU to explore the text comments Commissioning NSS analysis from student reps, supported by staff Staff & student conference to reflect on NSS scores and shape institutional policy and practice (examples from Buckley 2012)
A number of institutions have in effect handed over the administration of local questionnaires and interpretation of the NSS to students, and there is some evidence that NSS scores improve when students are engaged in this way. (Gibbs 2012)
There is a critical difference between going through the empty ritual of participation and having the real power needed to affect the outcome of the process. (Arnstein 1969)
... universities should treat learning as not yet wholly solved problems and hence always in research mode. (von Humboldt 1810, cited by Elton 2005) [T]he teaching-research nexus should be what distinguishes higher education from high school or vocational education. An important aim of higher education is to support students to realize not only the contested uncertain nature of knowledge, but also the importance and the fascination of pursuing that knowledge. (Healey and Jenkins 2006)
Questions for discussion 1. If you are involved in student voice activities, where on the ladder of participation do they sit? 2. Is the interpretation of your NSS results treated as a not yet wholly solved problem ?
References Arnstein, S. (1969). A ladder of citizen participation. Journal of the American Planning Association 35(4): 216-224 Buckley, A. (2012) Making it Count: Reflecting on the National Student Survey in the process of enhancement. York: Higher Education Academy. Cheng, J. and Marsh, H. (2010) National Student Survey: Are differences between universities and courses reliable and meaningful? Oxford Review of Education 36(6): 693- 712 Elton, L. (2005) Scholarship and the research and teaching nexus. In: Barnett, R. (ed.) Reshaping the university: new relationships between research, scholarship and teaching. Maidenhead: McGraw-Hill/Open University Press, 108 18. Gibbs, G. (2012). Implications of Dimensions of Quality in a Market Environment. York: HEA. Healey, M. and Jenkins, A. (2006), Strengthening the Teaching-Research Linkage in Undergraduate Courses and Programs , New Directions for Teaching and Learning 107: 45-55 Nicol, D. and Macfarlane-Dick, D. (2006) Formative assessment and self regulated learning: a model and seven principles of good feedback practice , Studies in Higher Education 31(2): 199-218
