Enhancing Novice Programming Environments with Data-Driven Hints

Thomas Price, Yihuan Dong and
Tiffany Barnes
Generating Data-driven Hints for
Open-ended Programming
EDM 2016
 
1
Motivation
–
Intelligent Tutoring Systems (
ITSs
) are effective learning
tools in Computer Science
–
They improve performance in and out of the tutor 
(Corbett 2001)
–
But, they are not commonly used to teach CS
Introduction
 
2
Motivation
–
Intelligent Tutoring Systems (
ITSs
) are effective learning
tools in Computer Science
–
They improve performance in and out of the tutor 
(Corbett 2001)
–
But, they are not commonly used to teach CS
–
Novice Programming Environments (
NPEs
) are popular
tools for introducing programming
–
They improve retention 
(Moskal 2004)
 and test scores 
(Dann 2012)
–
But, they lack support for struggling students
Introduction
 
3
Motivation
–
Intelligent Tutoring Systems (
ITSs
) are effective learning
tools in Computer Science
–
They improve performance in and out of the tutor 
(Corbett 2001)
–
But, they are not commonly used to teach CS
–
Novice Programming Environments (
NPEs
) are popular
tools for introducing programming
–
They improve retention 
(Moskal 2004)
 and test scores 
(Dann 2012)
–
But, they lack support for struggling students
–
Goal
: Combine these two effective technologies
–
Create ITS-like hints for NPEs
Introduction
 
4
NPE Example - Snap
!
Background
Novice Programming Environments
 
–
Visual, block-based
language
–
Programmable sprites
–
Low floor, high ceiling
–
All in the browser
–
Integrated with BJC
Curriculum 
(Garcia 2015)
 
5
Novice Programming Environments
–
Examples: Scratch, Alice, Greenfoot, Snap
!
–
Interesting
: Let students create games, stories and
simulations 
(Utting 2010)
–
Innovative
: Features like block-based programming
reduce the burden of syntax 
(Price 2015a)
–
Students use NPEs voluntarily and share knowledge 
(Maloney 2008)
–
Popular
: Used in classes, clubs camps and MOOCS
–
Scratch is the 22nd most popular programming language 
(TIOBE
Index, June 2016)
Background
Novice Programming Environments
6
Intelligent Tutoring Systems
–
Designed to play the role of a human tutor 
(VanLehn 2006)
–
Guide students to better learning outcomes
–
E.g., support students when they get stuck
–
Outer Loop: problem selection (using student model)
–
Inner Loop
: Step-by-step support on one problem
–
Next-step hints: answer "What do I do now?" questions
Background
Automated Hint Generation
7
Data-driven Hint Generation
–
Hint Factory 
(Barnes 2008)
–
Use previous students' data to model problem solving strategies
–
Successfully applied to many domains (e.g. logic proofs)
–
Overcomes challenges of expert models:
–
High cost of expert time (Murray 1999)
–
"Expert blind spot" (Alibali 2001)
–
Difficult to construct for large problem spaces
Background
Automated Hint Generation
8
Interaction Networks
Background
Automated Hint Generation
9
(Eagle 2012)
The Hint Factory
Background
Automated Hint Generation
10
The Guessing Game
GuessingGame:
  Say( 
"Welcome to the Guessing Game!"
 )
  
answer
 ← Ask( 
"What is your name?"
 )
  Say( Join( 
"Hello "
, answer ) )
  
number
 ← Random( 
1
, 
10
 )
  doUntil ( 
answer
 == 
number
 ):
    
answer
 ← Ask( 
"Guess a number"
 )
    if ( 
answer
 == 
number
 ):
      Say( 
"Correct!"
 )
    else:
      if ( 
answer
 > 
number
 ):
        Say( 
"Too high!"
 )
      if ( 
answer
 < 
number
 ):
        Say( 
"Too low!"
 )
Background
Hint Generation in Programming
 
11
Programming States
Background
Hint Generation in Programming
12
GuessingGame:
GuessingGame:
  Say( )
GuessingGame:
  Say( 
"Welcome to the Guessing Game!"
 )
GuessingGame:
  Say( 
"Welcome to the Guessing Game!"
 )
  
answer
 ←
GuessingGame:
  Say( 
"Welcome to the Guessing Game!"
 )
  
answer
 ← Ask( 
"What is your name?"
 ))
GuessingGame:
  Say( 
"Welcome to the Guessing Game!"
 )
  
answer
 ← Ask( 
"What is your name?"
 )
  Say( Join( 
"Hello "
, answer ) )
  
number
 ← Random( 
1
, 
10
 )
  doUntil ( 
answer
 == 
number
 ):
    
answer
 ← Ask( 
"Guess a number"
 )
    if ( 
answer
 == 
number
 ):
      Say( 
"Correct!"
 )
    else:
      if ( 
answer
 > 
number
 ):
        Say( 
"Too high!"
 )
      if ( 
answer
 < 
number
 ):
        Say( 
"Too low!"
 )
Abstract Syntax Trees
Background
Hint Generation in Programming
GuessingGame:
  Say( 
"Welcome to the Guessing Game!"
 )
  
answer
 ← Ask( 
"What is your name?"
 )
  Say( Join( 
"Hello "
, answer ) )
  
number
 ← Random( 
1
, 
10
 )
  doUntil ( 
answer
 == 
number
 ):
    
answer
 ← Ask( 
"Guess a number"
 )
    if ( 
answer
 == 
number
 ):
      Say( 
"Correct!"
 )
    else:
      if ( 
answer
 > 
number
 ):
        Say( 
"Too high!"
 )
      if ( 
answer
 < 
number
 ):
        Say( 
"Too low!"
 )
13
Programming States
Background
Hint Generation in Programming
14
Challenges of Using Data
–
Programming is a challenge for data-driven methods
–
Student solutions have very little overlap 
(Price 2015b)
–
Challenges overcome by:
–
Modeling the 
output 
of a program, rather than code 
(Hicks 2014)
–
Canonicalizing code so that it overlaps more 
(Rivers 2012)
–
Incorporating search to find new solutions 
(Rivers 2015)
–
Related: Automated Hint Generation
–
Start with a student program and search for a solution 
(Singh 2013)
–
Or transform into a similar expert solution 
(Zimmerman 2015)
 
Background
Hint Generation in Programming
15
Challenges of NPEs
Background
–
NPEs rely on open-ended programming assignments
–
Many variations on a correct solution
–
Students make meaningful design choices
–
Multiple, interdependent goals
–
Can we use current hint generation methods?
–
Students solutions have almost no overlap
–
Goals are difficult to automatically assess
Hint Generation in Programming
16
Hint Generation for NPEs
The CTD Algorithm
–
Contextual Tree Decomposition (CTD) Algorithm
–
Is data driven, requiring no expert model or test cases
–
Can operate with minimal overlap among students
 
17
Root Paths
The CTD Algorithm
Terminology
GuessingGame:
  Say( 
"Welcome to the Guessing Game!"
 )
  
answer
 ← Ask( 
"What is your name?"
 )
  Say( Join( 
"Hello "
, answer ) )
  
number
 ← Random( 
1
, 
10
 )
  doUntil ( 
answer
 == 
number
 ):
    
answer
 ← Ask( 
"Guess a number"
 )
    if ( 
answer
 == 
number
 ):
      Say( 
"Correct!"
 )
    else:
      if ( 
answer
 > 
number
 ):
        Say( 
"Too high!"
 )
      if ( 
answer
 < 
number
 ):
        Say( 
"Too low!"
 )
18
Root Paths
The CTD Algorithm
Terminology
GuessingGame:
  Say( 
"Welcome to the Guessing Game!"
 )
  
answer
 ← Ask( 
"What is your name?"
 )
  Say( Join( 
"Hello "
, answer ) )
  
number
 ← Random( 
1
, 
10
 )
  doUntil ( 
answer
 == 
number
 ):
    
answer
 ← Ask( 
"Guess a number"
 )
    if ( 
answer
 == 
number
 ):
      Say( 
"Correct!"
 )
    else:
      if ( 
answer
 > 
number
 ):
        Say( 
"Too high!"
 )
      if ( 
answer
 < 
number
 ):
        Say( 
"Too low!"
 )
19
Root Paths
The CTD Algorithm
Terminology
GuessingGame:
  Say( 
"Welcome to the Guessing Game!"
 )
  
answer
 ← Ask( 
"What is your name?"
 )
  Say( Join( 
"Hello "
, answer ) )
  
number
 ← Random( 
1
, 
10
 )
  doUntil ( 
answer
 == 
number
 ):
    
answer
 ← Ask( 
"Guess a number"
 )
    if ( 
answer
 == 
number
 ):
      Say( 
"Correct!"
 )
    else:
      if ( 
answer
 > 
number
 ):
        Say( 
"Too high!"
 )
      if ( 
answer
 < 
number
 ):
        Say( 
"Too low!"
 )
20
Contextual Interaction Networks (CINs)
The CTD Algorithm
Terminology
21
Root Path
Adding Students
The CTD Algorithm
Generating Hints
22
–
Input: a set of student solution paths from 
historical
data
–
Output: a 
set 
of CINs that model student problem
solving strategies
Adding Students
The CTD Algorithm
Generating Hints
GuessingGame:
  Say( 
"Welcome to the Guessing Game!"
 )
  
answer
 ← Ask( 
"What is your name?"
 )
  Say( Join( 
"Hello "
, answer ) )
23
Adding Students
The CTD Algorithm
Generating Hints
GuessingGame:
  Say( 
"Welcome to the Guessing Game!"
 )
  
answer
 ← Ask( 
"What is your name?"
 )
  Say( Join( 
"Hello "
, answer ) )
  
answer
 ← Ask( 
"Guess a number"
 )
24
Adding Students
The CTD Algorithm
Generating Hints
GuessingGame:
  Say( 
"Welcome to the Guessing Game!"
 )
  
answer
 ← Ask( 
"What is your name?"
 )
  Say( Join( 
"Hello "
, answer ) )
  
answer
 ← Ask( 
"Guess a number"
 )
  
doUntil ( 
answer
 ==
 ):
25
Adding Students
The CTD Algorithm
Generating Hints
GuessingGame:
  Say( 
"Welcome to the Guessing Game!"
 )
  
answer
 ← Ask( 
"What is your name?"
 )
  Say( Join( 
"Hello "
, answer ) )
  
answer
 ← Ask( 
"Guess a number"
 )
  
doUntil ( 
answer
 == Random(1, 10)
 ):
26
Adding Students
The CTD Algorithm
Generating Hints
GuessingGame:
  Say( 
"Welcome to the Guessing Game!"
 )
  
answer
 ← Ask( 
"What is your name?"
 )
  Say( Join( 
"Hello "
, answer ) )
  doUntil ( 
answer
 == Random(1, 10)
 ):
    
answer
 ← Ask( 
"Guess a number"
 )
27
Adding Students
The CTD Algorithm
Generating Hints
GuessingGame:
  Say( 
"Welcome to the Guessing Game!"
 )
  
answer
 ← Ask( 
"What is your name?"
 )
  Say( Join( 
"Hello "
, answer ) )
  
number
 ← Random( 
1
, 
10
 )
  doUntil ( 
answer
 == Random(1, 10)
 ):
    
answer
 ← Ask( 
"Guess a number"
 )
28
Adding Students
The CTD Algorithm
Generating Hints
GuessingGame:
  Say( 
"Welcome to the Guessing Game!"
 )
  
answer
 ← Ask( 
"What is your name?"
 )
  Say( Join( 
"Hello "
, answer ) )
  
number
 ← Random( 
1
, 
10
 )
  
doUntil ( 
answer
 ==
 ):
    
answer
 ← Ask( 
"Guess a number"
 )
29
Adding Students
The CTD Algorithm
Generating Hints
GuessingGame:
  Say( 
"Welcome to the Guessing Game!"
 )
  
answer
 ← Ask( 
"What is your name?"
 )
  Say( Join( 
"Hello "
, answer ) )
  
number
 ← Random( 
1
, 
10
 )
  
doUntil ( 
answer
 == 
number
 ):
    
answer
 ← Ask( 
"Guess a number"
 )
30
Adding Students
The CTD Algorithm
Generating Hints
GuessingGame:
  Say( 
"Welcome to the Guessing Game!"
 )
  
answer
 ← Ask( 
"What is your name?"
 )
  Say( Join( 
"Hello "
, answer ) )
  
number
 ← Random( 
1
, 
10
 )
  doUntil ( 
answer
 == 
number
 ):
    
answer
 ← Ask( 
"Guess a number"
 )
31
Providing Hints
–
Input: the set of CINs and the student's current code
–
Output: a 
set 
of hints
The CTD Algorithm
Generating Hints
32
Providing Hints
The CTD Algorithm
Generating Hints
GuessingGame:
  
number
 ← 
8
  Say( 
"Hi! Let's play."
 )
  
answer
 ← Ask( 
"Who's playing?"
 )
  Say( Join( 
"Sup "
, answer ) )
  doUntil ( 
answer
 == Random( 1, 10) ):
    
answer
 ← Ask( 
"Guess a number"
 )
33
Providing Hints
The CTD Algorithm
Generating Hints
GuessingGame:
  
number
 ← 
8
  Say( 
"Hi! Let's play."
 )
  
answer
 ← Ask( 
"Who's playing?"
 )
  Say( Join( 
"Sup "
, answer ) )
  doUntil ( 
answer
 == Random( 1, 10)
 ):
    
answer
 ← Ask( 
"Guess a number"
 )
34
Providing Hints
The CTD Algorithm
Generating Hints
GuessingGame:
  
number
 ← 
8
  Say( 
"Hi! Let's play."
 )
  
answer
 ← Ask( 
"Who's playing?"
 )
  Say( Join( 
"Sup "
, answer ) )
  doUntil ( 
answer
 == Random( 1, 10)
 ):
    
answer
 ← Ask( 
"Guess a number"
 )
35
Providing Hints
The CTD Algorithm
Generating Hints
GuessingGame:
  
number
 ← 
8
  Say( 
"Hi! Let's play."
 )
  
answer
 ← Ask( 
"Who's playing?"
 )
  Say( Join( 
"Sup "
, answer ) )
  doUntil ( 
answer
 == Random( 1, 10)
 ):
    
answer
 ← Ask( 
"Guess a number"
 )
36
Technical Evaluation
–
Used historical data:
–
51 students in an intro CS course for non-majors
–
Logs from in-lab work on the Guessing Game assignment
–
9109 program snapshots - very sparse:
–
85% of snapshots found in only one submission
–
92% of submissions had a unique solution
–
Hint generation:
–
Used CTD to generate hints for each snapshot
–
Hints for each student were generated with the other 50
students' data
Technical Evaluation
Methods
37
Hint Policies
1.
CTD All
: CTD hints using all data (n=51).
2.
CTD Exemplar
: CTD hints using data from correct
submissions  (n=32).
3.
Direct Expert*
: Direct modification towards an ideal
solution.
4.
Direct Student*
: Direct modification towards the
student's own solution.
*Generated using
 
(Zimmerman 2015)
Technical Evaluation
Methods
38
Can CTD Provide Hints Reliably?
–
Repeatedly applied hints to 50 time-slices from each
student's work
–
CTD hints could finish the assignment from all slices
–
CTD All
 policy solutions achieved 89.5-93.2% of
objectives
–
Imperfect solutions all missed the same objective
–
CTD 
Exemplar 
achieved 98.5-100%
–
Number of hints needed decreased linearly with time
Technical Evaluation
Analysis
39
Can CTD Provide Hints Reliably?
Technical Evaluation
Analysis
40
CTD All
CTD Exemplar
Do CTD Hints Satisfy Objectives?
–
Do hints complete new objectives?
–
Generated hints for all snapshots
–
Identified when a hint would complete an uncompleted
objective (before the student)
Technical Evaluation
Analysis
41
Do CTD Hints Satisfy Objectives?
Technical Evaluation
Analysis
42
–
Do hints complete new objectives?
–
Generated hints for all snapshots
–
Identified when a hint would complete an uncompleted
objective (before the student)
–
Do hints undo objectives?
–
Generated hints for each snapshot
–
For each objective a student completed:
–
Counted if a hint "undid" the complete objective
Do CTD Hints Satisfy Objectives?
Technical Evaluation
Analysis
43
Do CTD Hints Satisfy Objectives?
Technical Evaluation
Analysis
44
Are CTD Hints Reasonable?
–
Reasonable hints should get
students closer to their
submitted solution
–
Counted the percent of hints
which got students closer
–
Compared with student's own
actions
–
D.E. 
<
 C.E. 
<
 C.A. = D.S. 
<
 S.N.
–
(
blue
 = significant differences)
Technical Evaluation
Analysis
45
Percent of hints that get closer
for each policy.
Discussion
Are these results good?
–
Yes!
–
Hints are 
always 
available
 and lead to solutions
–
Despite 85% of snapshots and 92% of solutions being unique
–
They 
identify objectives
 most of the time without undoing them
–
They're 
as good as the Direct Student
 policy at pointing a
student toward their final solution
–
But... there is still much work to do
Technical Evaluation
Discussion
46
Demo!
http://go.ncsu.edu/isnap
 
 
47
Thank You!
Questions?
Thomas Price (
twprice@ncsu.edu
)
http://go.ncsu.edu/isnap
 
 
48
References
Barnes, T., & Stamper, J. (2008). Toward Automatic Hint Generation for Logic Proof Tutoring Using Historical Student Data. In
Proceedings of the 9th International Conference on Intelligent Tutoring Systems
 (pp. 373–382).
Corbett, A., & Anderson, J. (2001). Locus of Feedback Control in Computer-Based Tutoring: Impact on Learning Rate, Achievement
and Attitudes. In 
Proceedings of the SIGCHI Conference on Human Computer Interaction
 (pp. 245–252).
Dann, W., Cosgrove, D., & Slater, D. (2012). Mediated Transfer: Alice 3 to Java. In 
Proceedings of the 43rd ACM Technical
Symposium on Computer Science Education
 (pp. 141–146).
Hicks, A., Peddycord III, B., & Barnes, T. (2014). Building Games to Learn from Their Players: Generating Hints in a Serious Game. In
Proceedings of the 12th International Conference on Intelligent Tutoring Systems
 (pp. 312–317).
Moskal, B., Lurie, D., & Cooper, S. (2004). Evaluating the Effectiveness of a New Instructional Approach. 
ACM SIGCSE Bulletin
,
36
(1), 75–79.
Price, T. W., & Barnes, T. (2015a). Comparing Textual and Block Interfaces in a Novice Programming Environment. In 
Proceedings of
the 11th International Computing Education Research Conference
.
Price, T. W., & Barnes, T. (2015b). An Exploration of Data-Driven Hint Generation in an Open-Ended Programming Problem. In
Proceedings of the Workshop on Graph-Based Data Mining held at EDM’15
.
Rivers, K., & Koedinger, K. (2013). Automatic Generation of Programming Feedback: A Data-driven Approach. In 
Proceedings of the
First Workshop on AI-supported Education for Computer Science
 (pp. 50–59).
 
 
49
References, Cont.
Rivers, K., & Koedinger, K. R. (2015). Data-Driven Hint Generation in Vast Solution Spaces: a Self-Improving Python Programming
Tutor. 
International Journal of Artificial Intelligence in Education
, 
16
(1).
Singh, R., Gulwani, S., & Solar-Lezama, A. (2013). Automated Feedback Generation for Introductory Programming Assignments.
ACM SIGPLAN Notices
, 
48
(6).
Utting, I., Cooper, S., & Kölling, M. (2010). Alice, Greenfoot, and Scratch -- A Discussion. 
ACM Transactions on Computing Education
,
10
(4).
Vanlehn, K. (2006). The Behavior of Tutoring Systems. 
International Journal of Artifical Intelligence in Education
, 
16
(3), 227–265.
Zimmerman, K., & Rupakheti, C. R. (2015). An Automated Framework for Recommending Program Elements to Novices. In
Proceedings of the 30th International Conference on Automated Software Engineering
.
 
 
50
(Backup Slides… Shhh)
 
 
 
51
Hint-Chain Grades Over Time
 
 
52
Does Help Help? Yes!
–
Students with hints:
–
Complete more of the tutor, have less dropout and have higher
final course grades 
(Stamper 2013)
–
Perform better in the tutor and on assessments without hints
available 
(Corbett 2001a)
–
Data analysis of hint use 
(Beck 2008)
 suggests it:
–
Greatly increases the likelihood of getting a problem correct
–
Somewhat increases the likelihood of learning content
Related Work
Hints and Feedback in ITSs
53
Expert-model-based Tutors
–
Example: The ACT Programming Tutor (APT) 
(Anderson 1989)
–
Defines an expert model using if-then production rules
–
e.g. If goal is X, then code Y and set Z as the new goal
–
Uses model tracing to ensure students stay on an expert path
–
When the student deviates, or asks for help, offers a hint
–
Example: SQL-Tutor 
(Mitrovic 1999)
–
Defines an expert model using sets of constraints
–
e.g. When code has N left-parentheses, it must have N right
parentheses
–
When students submit work, finds violated constraints and gives
corresponding feedback
 
 
54

								

								

								

										

												
Slide Note

												

										

								


								
						

							
						


							

								

									
										
Embed
									
									
										
Share
									

									
									
									

								

								
							


							

							 
 							 
 							 
						  


							
							

								
Effective learning tools like Intelligent Tutoring Systems (ITSs) and Novice Programming Environments (NPEs) are valuable in computer science education. This study aims to combine these technologies by creating ITS-like hints for NPEs, utilizing insights from programming education research to better support students in their learning journey.


								
    
																		
  •   Programming
    • 
																	
  •    Education
    • 
																	
  •    Novice
    • 
																	
  •    Intelligent Tutoring Systems
    • 
																	
  •    Data-Driven
    • 
																

								

								

									

										kryde
										

											 kryde Follow
											

										

									

									
Uploaded on Sep 21, 2024 | 0 Views

								

								


								

									
Download Presentation

									
								

								
Please find below an Image/Link to download the presentation.

								
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
									Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.


									
									

									


										 
										 
										 
										 

									

									

									

									

										
 E N D 

									


								
Presentation Transcript

								

								
  1. Generating Data-driven Hints for Open-ended Programming Thomas Price, Yihuan Dong and  Tiffany Barnes EDM 2016 Price, Dong and Barnes (NCSU) Hints for Open-Ended Programming EDM Conference 2016 1
    2. 
            
            
            
  2. Introduction Motivation     Intelligent Tutoring Systems (ITSs) are effective learning  tools in Computer Science     They improve performance in and out of the tutor (Corbett 2001)     But, they are not commonly used to teach CS Price, Dong and Barnes (NCSU) Hints for Open-Ended Programming EDM Conference 2016 2
    3. 
            
            
            
  3. Introduction Motivation     Intelligent Tutoring Systems (ITSs) are effective learning  tools in Computer Science     They improve performance in and out of the tutor (Corbett 2001)     But, they are not commonly used to teach CS     Novice Programming Environments (NPEs) are popular  tools for introducing programming     They improve retention (Moskal 2004) and test scores (Dann 2012)     But, they lack support for struggling students Price, Dong and Barnes (NCSU) Hints for Open-Ended Programming EDM Conference 2016 3
    4. 
            
            
            
  4. Introduction Motivation     Intelligent Tutoring Systems (ITSs) are effective learning  tools in Computer Science     They improve performance in and out of the tutor (Corbett 2001)     But, they are not commonly used to teach CS     Novice Programming Environments (NPEs) are popular  tools for introducing programming     They improve retention (Moskal 2004) and test scores (Dann 2012)     But, they lack support for struggling students     Goal: Combine these two effective technologies      Create ITS-like hints for NPEs Price, Dong and Barnes (NCSU) Hints for Open-Ended Programming EDM Conference 2016 4
    5. 
            
            
            
  5. Background Novice Programming Environments NPE Example - Snap!     Visual, block-based  language     Programmable sprites     Low floor, high ceiling     All in the browser     Integrated with BJC  Curriculum (Garcia 2015) Price, Dong and Barnes (NCSU) Hints for Open-Ended Programming EDM Conference 2016 5
    6. 
            
            
            
  6. Background Novice Programming Environments Novice Programming Environments     Examples: Scratch, Alice, Greenfoot, Snap!     Interesting: Let students create games, stories and  simulations (Utting 2010)      Innovative: Features like block-based programming  reduce the burden of syntax (Price 2015a)     Students use NPEs voluntarily and share knowledge (Maloney 2008)     Popular: Used in classes, clubs camps and MOOCS     Scratch is the 22nd most popular programming language (TIOBE  Index, June 2016) Price, Dong and Barnes (NCSU) Hints for Open-Ended Programming EDM Conference 2016 6
    7. 
            
            
            
  7. Background Automated Hint Generation Intelligent Tutoring Systems     Designed to play the role of a human tutor (VanLehn 2006)     Guide students to better learning outcomes     E.g., support students when they get stuck     Outer Loop: problem selection (using student model)     Inner Loop: Step-by-step support on one problem     Next-step hints: answer "What do I do now?" questions Price, Dong and Barnes (NCSU) Hints for Open-Ended Programming EDM Conference 2016 7
    8. 
            
            
            
  8. Background Automated Hint Generation Data-driven Hint Generation     Hint Factory (Barnes 2008)     Use previous students' data to model problem solving strategies     Successfully applied to many domains (e.g. logic proofs)     Overcomes challenges of expert models:     High cost of expert time (Murray 1999)     "Expert blind spot" (Alibali 2001)     Difficult to construct for large problem spaces Price, Dong and Barnes (NCSU) Hints for Open-Ended Programming EDM Conference 2016 8
    9. 
            
            
            
  9. Background Automated Hint Generation Interaction Networks Start State Actions  (Edges) States Frequency Solution  Path Goal State (Eagle 2012) Price, Dong and Barnes (NCSU) Hints for Open-Ended Programming EDM Conference 2016 9
    10. 
            
            
            
  10. Background Automated Hint Generation The Hint Factory Hint: Go  Here You are  Here Hint: Go  Here You are  Here Price, Dong and Barnes (NCSU) Hints for Open-Ended Programming EDM Conference 2016 10
    11. 
            
            
            
  11. Background Hint Generation in Programming The Guessing Game GuessingGame: Say( "Welcome to the Guessing Game!" ) answer     Ask( "What is your name?" ) Say( Join( "Hello ", answer ) ) number     Random( 1, 10 ) doUntil ( answer == number ): answer     Ask( "Guess a number" ) if ( answer == number ): Say( "Correct!" ) else: if ( answer > number ): Say( "Too high!" ) if ( answer < number ): Say( "Too low!" ) Price, Dong and Barnes (NCSU) Hints for Open-Ended Programming EDM Conference 2016 11
    12. 
            
            
            
  12. Background Hint Generation in Programming Programming States GuessingGame: GuessingGame: Say( "Welcome to the Guessing Game!" ) answer     Ask( "What is your name?" ) Say( Join( "Hello ", answer ) ) number     Random( 1, 10 ) doUntil ( answer == number ): answer     Ask( "Guess a number" ) if ( answer == number ): Say( "Correct!" ) else: if ( answer > number ): Say( "Too high!" ) if ( answer < number ): Say( "Too low!" ) GuessingGame: Say( ) GuessingGame: Say( "Welcome to the Guessing Game!" ) GuessingGame: Say( "Welcome to the Guessing Game!" ) answer      GuessingGame: Say( "Welcome to the Guessing Game!" ) answer     Ask( "What is your name?" )) Price, Dong and Barnes (NCSU) Hints for Open-Ended Programming EDM Conference 2016 12
    13. 
            
            
            
  13. Background Hint Generation in Programming Abstract Syntax Trees GuessingGame: Say( "Welcome to the Guessing Game!" ) answer     Ask( "What is your name?" ) Say( Join( "Hello ", answer ) ) number     Random( 1, 10 ) doUntil ( answer == number ): answer     Ask( "Guess a number" ) if ( answer == number ): Say( "Correct!" ) else: if ( answer > number ): Say( "Too high!" ) if ( answer < number ): Say( "Too low!" ) Price, Dong and Barnes (NCSU) Hints for Open-Ended Programming EDM Conference 2016 13
    14. 
            
            
            
  14. Background Hint Generation in Programming Programming States Price, Dong and Barnes (NCSU) Hints for Open-Ended Programming EDM Conference 2016 14
    15. 
            
            
            
  15. Background Hint Generation in Programming Challenges of Using Data     Programming is a challenge for data-driven methods     Student solutions have very little overlap (Price 2015b)     Challenges overcome by:     Modeling the output of a program, rather than code (Hicks 2014)     Canonicalizing code so that it overlaps more (Rivers 2012)     Incorporating search to find new solutions (Rivers 2015)     Related: Automated Hint Generation     Start with a student program and search for a solution (Singh 2013)     Or transform into a similar expert solution (Zimmerman 2015) Price, Dong and Barnes (NCSU) Hints for Open-Ended Programming EDM Conference 2016 15
    16. 
            
            
            
  16. Background Hint Generation in Programming Challenges of NPEs     NPEs rely on open-ended programming assignments     Many variations on a correct solution     Students make meaningful design choices     Multiple, interdependent goals     Can we use current hint generation methods?     Students solutions have almost no overlap     Goals are difficult to automatically assess Price, Dong and Barnes (NCSU) Hints for Open-Ended Programming EDM Conference 2016 16
    17. 
            
            
            
  17. The CTD Algorithm Hint Generation for NPEs     Contextual Tree Decomposition (CTD) Algorithm     Is data driven, requiring no expert model or test cases     Can operate with minimal overlap among students Price, Dong and Barnes (NCSU) Hints for Open-Ended Programming EDM Conference 2016 17
    18. 
            
            
            
  18. The CTD Algorithm Terminology Root Paths GuessingGame: Say( "Welcome to the Guessing Game!" ) answer     Ask( "What is your name?" ) Say( Join( "Hello ", answer ) ) number     Random( 1, 10 ) doUntil ( answer == number ): answer     Ask( "Guess a number" ) if ( answer == number ): Say( "Correct!" ) else: if ( answer > number ): Say( "Too high!" ) if ( answer < number ): Say( "Too low!" ) Price, Dong and Barnes (NCSU) Hints for Open-Ended Programming EDM Conference 2016 18
    19. 
            
            
            
  19. The CTD Algorithm Terminology Root Paths GuessingGame: Say( "Welcome to the Guessing Game!" ) answer     Ask( "What is your name?" ) Say( Join( "Hello ", answer ) ) number     Random( 1, 10 ) doUntil ( answer == number ): answer     Ask( "Guess a number" ) if ( answer == number ): Say( "Correct!" ) else: if ( answer > number ): Say( "Too high!" ) if ( answer < number ): Say( "Too low!" ) Price, Dong and Barnes (NCSU) Hints for Open-Ended Programming EDM Conference 2016 19
    20. 
            
            
            
  20. The CTD Algorithm Terminology Root Paths GuessingGame: Say( "Welcome to the Guessing Game!" ) answer     Ask( "What is your name?" ) Say( Join( "Hello ", answer ) ) number     Random( 1, 10 ) doUntil ( answer == number ): answer     Ask( "Guess a number" ) if ( answer == number ): Say( "Correct!" ) else: if ( answer > number ): Say( "Too high!" ) if ( answer < number ): Say( "Too low!" ) Price, Dong and Barnes (NCSU) Hints for Open-Ended Programming EDM Conference 2016 20
    21. 
            
            
            
  21. The CTD Algorithm Terminology Contextual Interaction Networks (CINs) Root Path Price, Dong and Barnes (NCSU) Hints for Open-Ended Programming EDM Conference 2016 21
    22. 
            
            
            
  22. The CTD Algorithm Generating Hints Adding Students     Input: a set of student solution paths from historical  data     Output: a set of CINs that model student problem  solving strategies Price, Dong and Barnes (NCSU) Hints for Open-Ended Programming EDM Conference 2016 22
    23. 
            
            
            
  23. The CTD Algorithm Generating Hints Adding Students GuessingGame: Say( "Welcome to the Guessing Game!" ) answer     Ask( "What is your name?" ) Say( Join( "Hello ", answer ) ) Price, Dong and Barnes (NCSU) Hints for Open-Ended Programming EDM Conference 2016 23
    24. 
            
            
            
  24. The CTD Algorithm Generating Hints Adding Students GuessingGame: Say( "Welcome to the Guessing Game!" ) answer     Ask( "What is your name?" ) Say( Join( "Hello ", answer ) ) answer     Ask( "Guess a number" ) Price, Dong and Barnes (NCSU) Hints for Open-Ended Programming EDM Conference 2016 24
    25. 
            
            
            
  25. The CTD Algorithm Generating Hints Adding Students GuessingGame: Say( "Welcome to the Guessing Game!" ) answer     Ask( "What is your name?" ) Say( Join( "Hello ", answer ) ) answer     Ask( "Guess a number" ) doUntil ( answer == ): Price, Dong and Barnes (NCSU) Hints for Open-Ended Programming EDM Conference 2016 25
    26. 
            
            
            
  26. The CTD Algorithm Generating Hints Adding Students GuessingGame: Say( "Welcome to the Guessing Game!" ) answer     Ask( "What is your name?" ) Say( Join( "Hello ", answer ) ) answer     Ask( "Guess a number" ) doUntil ( answer == Random(1, 10) ): Price, Dong and Barnes (NCSU) Hints for Open-Ended Programming EDM Conference 2016 26
    27. 
            
            
            
  27. The CTD Algorithm Generating Hints Adding Students GuessingGame: Say( "Welcome to the Guessing Game!" ) answer     Ask( "What is your name?" ) Say( Join( "Hello ", answer ) ) doUntil ( answer == Random(1, 10) ): answer     Ask( "Guess a number" ) Price, Dong and Barnes (NCSU) Hints for Open-Ended Programming EDM Conference 2016 27
    28. 
            
            
            
  28. The CTD Algorithm Generating Hints Adding Students GuessingGame: Say( "Welcome to the Guessing Game!" ) answer     Ask( "What is your name?" ) Say( Join( "Hello ", answer ) ) number     Random( 1, 10 ) doUntil ( answer == Random(1, 10) ): answer     Ask( "Guess a number" ) Price, Dong and Barnes (NCSU) Hints for Open-Ended Programming EDM Conference 2016 28
    29. 
            
            
            
  29. The CTD Algorithm Generating Hints Adding Students GuessingGame: Say( "Welcome to the Guessing Game!" ) answer     Ask( "What is your name?" ) Say( Join( "Hello ", answer ) ) number     Random( 1, 10 ) doUntil ( answer == ): answer     Ask( "Guess a number" ) Price, Dong and Barnes (NCSU) Hints for Open-Ended Programming EDM Conference 2016 29
    30. 
            
            
            
  30. The CTD Algorithm Generating Hints Adding Students GuessingGame: Say( "Welcome to the Guessing Game!" ) answer     Ask( "What is your name?" ) Say( Join( "Hello ", answer ) ) number     Random( 1, 10 ) doUntil ( answer == number ): answer     Ask( "Guess a number" ) Price, Dong and Barnes (NCSU) Hints for Open-Ended Programming EDM Conference 2016 30
    31. 
            
            
            
  31. The CTD Algorithm Generating Hints Adding Students GuessingGame: Say( "Welcome to the Guessing Game!" ) answer     Ask( "What is your name?" ) Say( Join( "Hello ", answer ) ) number     Random( 1, 10 ) doUntil ( answer == number ): answer     Ask( "Guess a number" ) Price, Dong and Barnes (NCSU) Hints for Open-Ended Programming EDM Conference 2016 31
    32. 
            
            
            
  32. The CTD Algorithm Generating Hints Providing Hints     Input: the set of CINs and the student's current code     Output: a set of hints Price, Dong and Barnes (NCSU) Hints for Open-Ended Programming EDM Conference 2016 32
    33. 
            
            
            
  33. The CTD Algorithm Generating Hints Providing Hints GuessingGame: number     8 Say( "Hi! Let's play." ) answer     Ask( "Who's playing?" ) Say( Join( "Sup ", answer ) ) doUntil ( answer == Random( 1, 10) ): answer     Ask( "Guess a number" ) Price, Dong and Barnes (NCSU) Hints for Open-Ended Programming EDM Conference 2016 33
    34. 
            
            
            
  34. The CTD Algorithm Generating Hints Providing Hints GuessingGame: number     8 Say( "Hi! Let's play." ) answer     Ask( "Who's playing?" ) Say( Join( "Sup ", answer ) ) doUntil ( answer == Random( 1, 10) ): answer     Ask( "Guess a number" ) Price, Dong and Barnes (NCSU) Hints for Open-Ended Programming EDM Conference 2016 34
    35. 
            
            
            
  35. The CTD Algorithm Generating Hints Providing Hints GuessingGame: number     8 Say( "Hi! Let's play." ) answer     Ask( "Who's playing?" ) Say( Join( "Sup ", answer ) ) doUntil ( answer == Random( 1, 10) ): answer     Ask( "Guess a number" ) Price, Dong and Barnes (NCSU) Hints for Open-Ended Programming EDM Conference 2016 35
    36. 
            
            
            
  36. The CTD Algorithm Generating Hints Providing Hints GuessingGame: number     8 Say( "Hi! Let's play." ) answer     Ask( "Who's playing?" ) Say( Join( "Sup ", answer ) ) doUntil ( answer == Random( 1, 10) ): answer     Ask( "Guess a number" ) Price, Dong and Barnes (NCSU) Hints for Open-Ended Programming EDM Conference 2016 36
    37. 
            
            
            
  37. Technical Evaluation Methods Technical Evaluation     Used historical data:     51 students in an intro CS course for non-majors     Logs from in-lab work on the Guessing Game assignment     9109 program snapshots - very sparse:     85% of snapshots found in only one submission     92% of submissions had a unique solution     Hint generation:     Used CTD to generate hints for each snapshot     Hints for each student were generated with the other 50  students' data Price, Dong and Barnes (NCSU) Hints for Open-Ended Programming EDM Conference 2016 37
    38. 
            
            
            
  38. Technical Evaluation Methods Hint Policies 1.CTD All: CTD hints using all data (n=51). 2.CTD Exemplar: CTD hints using data from correct  submissions  (n=32). 3.Direct Expert*: Direct modification towards an ideal  solution. 4.Direct Student*: Direct modification towards the  student's own solution. *Generated using (Zimmerman 2015) Price, Dong and Barnes (NCSU) Hints for Open-Ended Programming EDM Conference 2016 38
    39. 
            
            
            
  39. Technical Evaluation Analysis Can CTD Provide Hints Reliably?     Repeatedly applied hints to 50 time-slices from each  student's work     CTD hints could finish the assignment from all slices     CTD All policy solutions achieved 89.5-93.2% of  objectives     Imperfect solutions all missed the same objective     CTD Exemplar achieved 98.5-100%     Number of hints needed decreased linearly with time Price, Dong and Barnes (NCSU) Hints for Open-Ended Programming EDM Conference 2016 39
    40. 
            
            
            
  40. Technical Evaluation Analysis Can CTD Provide Hints Reliably? CTD All CTD Exemplar Price, Dong and Barnes (NCSU) Hints for Open-Ended Programming EDM Conference 2016 40
    41. 
            
            
            
  41. Technical Evaluation Analysis Do CTD Hints Satisfy Objectives?     Do hints complete new objectives?     Generated hints for all snapshots     Identified when a hint would complete an uncompleted  objective (before the student) Price, Dong and Barnes (NCSU) Hints for Open-Ended Programming EDM Conference 2016 41
    42. 
            
            
            
  42. Technical Evaluation Analysis Do CTD Hints Satisfy Objectives? CTD All CTD Exemplar Direct Expert Direct Student Price, Dong and Barnes (NCSU) Hints for Open-Ended Programming EDM Conference 2016 42
    43. 
            
            
            
  43. Technical Evaluation Analysis Do CTD Hints Satisfy Objectives?     Do hints complete new objectives?     Generated hints for all snapshots     Identified when a hint would complete an uncompleted  objective (before the student)     Do hints undo objectives?     Generated hints for each snapshot     For each objective a student completed:     Counted if a hint "undid" the complete objective Price, Dong and Barnes (NCSU) Hints for Open-Ended Programming EDM Conference 2016 43
    44. 
            
            
            
  44. Technical Evaluation Analysis Do CTD Hints Satisfy Objectives? CTD All CTD Exemplar Direct Expert Direct Student Price, Dong and Barnes (NCSU) Hints for Open-Ended Programming EDM Conference 2016 44
    45. 
            
            
            
  45. Technical Evaluation Analysis Are CTD Hints Reasonable?     Reasonable hints should get  students closer to their  submitted solution     Counted the percent of hints  which got students closer     Compared with student's own  actions     D.E. < C.E. < C.A. = D.S. < S.N.     (blue = significant differences) Percent of hints that get closer  for each policy. Price, Dong and Barnes (NCSU) Hints for Open-Ended Programming EDM Conference 2016 45
    46. 
            
            
            
  46. Technical Evaluation Discussion Discussion Are these results good?     Yes!      Hints are always available and lead to solutions     Despite 85% of snapshots and 92% of solutions being unique     They identify objectives most of the time without undoing them     They're as good as the Direct Student policy at pointing a  student toward their final solution     But... there is still much work to do Price, Dong and Barnes (NCSU) Hints for Open-Ended Programming EDM Conference 2016 46
    47. 
            
            
            
  47. Demo! http://go.ncsu.edu/isnap Price, Dong and Barnes (NCSU) Hints for Open-Ended Programming EDM Conference 2016 47
    48. 
            
            
            
  48. Thank You! Questions? Thomas Price (twprice@ncsu.edu) http://go.ncsu.edu/isnap Price, Dong and Barnes (NCSU) Hints for Open-Ended Programming EDM Conference 2016 48
    49. 
            
            
            
  49. References Barnes, T., & Stamper, J. (2008). Toward Automatic Hint Generation for Logic Proof Tutoring Using Historical Student Data. In Proceedings of the 9th International Conference on Intelligent Tutoring Systems (pp. 373   382).  Corbett, A., & Anderson, J. (2001). Locus of Feedback Control in Computer-Based Tutoring: Impact on Learning Rate, Achievement  and Attitudes. In Proceedings of the SIGCHI Conference on Human Computer Interaction (pp. 245   252). Dann, W., Cosgrove, D., & Slater, D. (2012). Mediated Transfer: Alice 3 to Java. In Proceedings of the 43rd ACM Technical  Symposium on Computer Science Education (pp. 141   146).  Hicks, A., Peddycord III, B., & Barnes, T. (2014). Building Games to Learn from Their Players: Generating Hints in a Serious Game. In  Proceedings of the 12th International Conference on Intelligent Tutoring Systems (pp. 312   317).  Moskal, B., Lurie, D., & Cooper, S. (2004). Evaluating the Effectiveness of a New Instructional Approach. ACM SIGCSE Bulletin,  36(1), 75   79. Price, T. W., & Barnes, T. (2015a). Comparing Textual and Block Interfaces in a Novice Programming Environment. In Proceedings of  the 11th International Computing Education Research Conference. Price, T. W., & Barnes, T. (2015b). An Exploration of Data-Driven Hint Generation in an Open-Ended Programming Problem. In  Proceedings of the Workshop on Graph-Based Data Mining held at EDM   15. Rivers, K., & Koedinger, K. (2013). Automatic Generation of Programming Feedback: A Data-driven Approach. In Proceedings of the  First Workshop on AI-supported Education for Computer Science (pp. 50   59). Price, Dong and Barnes (NCSU) Hints for Open-Ended Programming EDM Conference 2016 49
    50. 
            
            
            
  50. References, Cont. Rivers, K., & Koedinger, K. R. (2015). Data-Driven Hint Generation in Vast Solution Spaces: a Self-Improving Python Programming  Tutor. International Journal of Artificial Intelligence in Education, 16(1). Singh, R., Gulwani, S., & Solar-Lezama, A. (2013). Automated Feedback Generation for Introductory Programming Assignments.  ACM SIGPLAN Notices, 48(6). Utting, I., Cooper, S., & K  lling, M. (2010). Alice, Greenfoot, and Scratch -- A Discussion. ACM Transactions on Computing Education,  10(4). Vanlehn, K. (2006). The Behavior of Tutoring Systems. International Journal of Artifical Intelligence in Education, 16(3), 227   265. Zimmerman, K., & Rupakheti, C. R. (2015). An Automated Framework for Recommending Program Elements to Novices. In  Proceedings of the 30th International Conference on Automated Software Engineering. Price, Dong and Barnes (NCSU) Hints for Open-Ended Programming EDM Conference 2016 50
    51. 
            
            
            
							

							

					

					
					
					

						
Related

						

						


							
							
							
							

						

							

									

      

        
          

            

              The Development of Support System for Novice Teachers
            

            

              
The Development of Support System for Novice Teachers

              
sadia

            

          

        
      


     

      

        
          

            

              Strategic Support for Novice Teachers in Latvia 2021-2027
            

            

              
Strategic Support for Novice Teachers in Latvia 2021-2027

              
marlon

            

          

        
      


     

      

        
          

            

              TESS 2.0: A Guide for Arkansas Novice Teachers
            

            

              
TESS 2.0: A Guide for Arkansas Novice Teachers

              
caiden

            

          

        
      


     

      

        
          

            

              Decision Analysis and Operations Research in Management
            

            

              
Decision Analysis and Operations Research in Management

              
senara

            

          

        
      


     

      

        
          

            

              Module 2: PSEA and Safe Programming Training of Trainers (ToT) by CRS HRD
            

            

              
Module 2: PSEA and Safe Programming Training of Trainers (ToT) by CRS HRD

              
barnabas

            

          

        
      


     

      

        
          

            

              Comprehensive Guide to NBDHE Format and Test-Taking Hints
            

            

              
Comprehensive Guide to NBDHE Format and Test-Taking Hints

              
jadiel

            

          

        
      


     

      

        
          

            

              Web Application Development and Programming CTE Program Overview
            

            

              
Web Application Development and Programming CTE Program Overview

              
harm_21

            

          

        
      


     

      

        
          

            

              Introduction to Programming and Computer Instructions
            

            

              
Introduction to Programming and Computer Instructions

              
buechler_c

            

          

        
      


     

      

        
          

            

              Transition to Half-Ice Hockey for Novice Players
            

            

              
Transition to Half-Ice Hockey for Novice Players

              
keal_30

            

          

        
      


     

      

        
          

            

              Development of Attosecond Theory for Nobel Prize through Verilog Programming
            

            

              
Development of Attosecond Theory for Nobel Prize through Verilog Programming

              
inia775

            

          

        
      


     

      

        
          

            

              Active Learning for Inference and Regeneration of Computer Programs: A Data-driven Approach
            

            

              
Active Learning for Inference and Regeneration of Computer Programs: A Data-driven Approach

              
syerg

            

          

        
      


     

      

        
          

            

              CS252 Systems Programming Course Overview
            

            

              
CS252 Systems Programming Course Overview

              
siip640

            

          

        
      


     

      

        
          

            

              Introduction to R Programming for Beginners: Data and Programming Basics
            

            

              
Introduction to R Programming for Beginners: Data and Programming Basics

              
jurn149

            

          

        
      


     

      

        
          

            

              Introduction to Programming Languages and Functional Programming with OCaml
            

            

              
Introduction to Programming Languages and Functional Programming with OCaml

              
ard_re

            

          

        
      


     

      

        
          

            

              ITM352 and Its Role in MIS
            

            

              
ITM352 and Its Role in MIS

              
kelilahh

            

          

        
      


     

      

        
          

            

              Functional Programming Paradigm
            

            

              
Functional Programming Paradigm

              
vshu

            

          

        
      


     

      

        
          

            

              Essential Principles of Teaching Programming Languages
            

            

              
Essential Principles of Teaching Programming Languages

              
leun_van

            

          

        
      


     

      

        
          

            

              Leveraging MPI's One-Sided Communication Interface for Shared Memory Programming
            

            

              
Leveraging MPI's One-Sided Communication Interface for Shared Memory Programming

              
jmcgl

            

          

        
      


     

      

        
          

            

              Computer Programming Principles
            

            

              
Computer Programming Principles

              
dbard

            

          

        
      


     

      

        
          

            

              Lexical Scope and Function Closures in Programming Languages
            

            

              
Lexical Scope and Function Closures in Programming Languages

              
rroge

            

          

        
      


     

      

        
          

            

              Exploring Virtual Environments in Computing: An Overview
            

            

              
Exploring Virtual Environments in Computing: An Overview

              
der_ce

            

          

        
      


     

      

        
          

            

              Enhancing Programming Skills for Blind Students with Alan and Torino Environments
            

            

              
Enhancing Programming Skills for Blind Students with Alan and Torino Environments

              
nysh_101

            

          

        
      


     							

						

					

					

				


				

			

		
		

			

				
				

					
More Related Content

					

						

														

					

				

				
			

		












 





  
  
  



giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#