Pilot Study on Reliability and Usability in Computational Thinking Challenge
"Explore a pilot study on reliability and usability of a computerized competency-based assessment, Computational Thinking Challenge (CTC), focusing on the development and evaluation of psychometric properties. Learn about the background research, method stages, and findings of this study conducted by Rina Lai at the University of Cambridge."
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Computational Thinking Challenge: A Pilot Study on Reliability and Usability Rina Lai Faculty of Education, University of Cambridge Cambridge Computing Education Research Symposium 1st April, 2020
Background Research Motivation 1. There is a paucity of CT assessment tools for teachers; this is a central factor that impedes the promotion of computational thinking (CT) in the classroom (Rich & Hodges, 2017). 2. Psychometrically sound, reliable and valid CT assessments are scarce, if not absent, in the current literature. 2
Background Research Focus 1. Develop a computerised competency-based assessment: Computational Thinking Challenge (CTC) 2. Evaluate the reliability, validity, and psychometric properties of CTC 3
Background Computational Thinking Challenge (CTC) Computerised Competency-based assessment that measures context/language-neutral programming skills and CT problem-solving skills Hybrid approach to testing with a focus on quality formative feedback 4
Method Stage 1 examines the internal consistency reliability 19 children ages ranged from 14.9 to 16.2 (Mage=15.51; SDage = 0.35) 17 (89%) males and 2 (11%) females Stage 2 compares reliability between the computer-based and paper-based version of CTC 24 children ages ranged from 16.9 to 18.4 (Mage= 17.5; SDage = 0.37) 21 (87.5%) males and 3 (12.5%) females Stage 3 investigates user s experience in terms of difficulty/ clarity of items and motivation/ enjoyment Same sample as stage 2 5
Findings Stage 1 CTC demonstrates good reliability, Cronbach s = .79 (Mscale =9.21; SDscale = 3.73; 2=13.95). Stage 2 Paper-based version of CTC has relatively poor reliability, Cronbach s =.59 compared to the computer-based version that demonstrates high reliability, Cronbach s = .78. Paper-based Computer-based N Cronbach s Mean Variance Standard Deviation 12 .59 9.83 9.06 3.01 12 .78 8.75 14.20 3.77 6
Findings Stage 3 How did you find the problems in the CTC? Moderately easy Slightly easy Slightly difficult Moderately difficult Neither easy nor difficult N=1 (4.3% ) N =2 (8.7%) N =7 (30.4%) N =1 (4.3%) N =12 (52.2%) How stimulating, interesting, or fun were the problems? Extremely Very Slightly Not at all Moderately N =2 (8.7%) N =3 (13%) N =6 (26.1%) N =1 (4.3%) N =11 (47.8% ) How well did you understand the problems and the questions asked? Moderately well Slightly well Not well at all Very well N =5 ( 21.7%) N =6 (26.1%) N =1 (4.3%) N = 11 (47.8%) How much did you enjoy CTC? A great deal A lot Not at all A moderate amount A little N =3 (13%) N =5 (21.7%) N =1 (4.3%) N =7 (30.4%) N =7 (30.4%) 7
Whats next? Currently Main study is in the process 20 + UK schools have participated (thank you!) Next: Item Response Theory How difficult/ discriminative is each item? How much information does CTC provide over the range of CT competency? How does each item contribute to the test information? Teacher survey Educational Data Mining Explore emerging performance patterns CTC participation 8
