Insights on Exploring Marking and Feedback in Education
E
x
p
l
o
r
e
:
M
a
r
k
i
n
g
a
n
d
F
e
e
d
b
a
c
k
(
P
h
a
s
e
1
)
E
v
a
l
u
a
t
i
o
n
i
n
s
i
g
h
t
s
Christothea Herodotou,
Senior Lecturer
Institute of Educational Technology
Exams and Assignment (E&A) Implementation Programme
•
Replace the existing eTMA system
•
Expand exams and assignments capabilities
V
i
s
i
o
n
:
“
p
r
o
v
i
d
e
a
s
e
c
u
r
e
,
r
o
b
u
s
t
a
n
d
f
l
e
x
i
b
l
e
d
i
g
i
t
a
l
a
s
s
e
s
s
m
e
n
t
p
l
a
t
f
o
r
m
t
h
a
t
i
m
p
r
o
v
e
s
t
h
e
s
t
u
d
e
n
t
e
x
p
e
r
i
e
n
c
e
”
I
n
i
t
i
a
l
f
o
c
u
s
:
r
e
p
l
a
c
e
a
n
d
e
n
h
a
n
c
e
a
s
s
i
g
n
m
e
n
t
s
u
b
m
i
s
s
i
o
n
,
m
a
r
k
i
n
g
,
a
n
d
f
e
e
d
b
a
c
k
p
r
o
c
e
s
s
W
I
S
E
f
l
o
w
i
s
t
h
e
n
e
w
e
x
a
m
s
a
n
d
a
s
s
i
g
n
m
e
n
t
s
s
y
s
t
e
m
,
o
f
f
e
r
e
d
b
y
U
N
I
w
i
s
e
.
Maria Kantirou – Project Manager and Chair of the E & A Working group
Project website:
https://learn3.open.ac.uk/course/view.php?id=301194
Background
Aim of Explore Phase 1
Herodotou, Christothea and Gillespie, Anna (2020).
Explore: Marking and Feedback (Phase 1) (Final report).
Open
University.
•
Understand how Associate Lecturers (ALs) are using WISEflow
•
Understand whether and how it aligns with OU processes and working practices
•
Identify possible gaps
R
Q
1
:
W
h
a
t
a
r
e
t
h
e
u
s
a
g
e
p
a
t
h
w
a
y
s
t
o
c
o
m
p
l
e
t
i
n
g
m
a
r
k
i
n
g
w
i
t
h
W
I
S
E
f
l
o
w
?
•
What features do ALs use/do not use?
•
Do ALs complement their practices with other tools offline or online (e.g. hand written
notes, printing out TMAs, other applications such as word doc)?
R
Q
2
:
W
h
a
t
c
h
a
l
l
e
n
g
e
s
d
o
A
L
s
f
a
c
e
w
h
e
n
m
a
r
k
i
n
g
w
i
t
h
W
I
S
E
f
l
o
w
?
• Are these challenges related to missing system features?
• Are these challenges related to system features they are not aware of or do not
know how to use?
Sample and methods
3
5
s
e
l
f
-
s
e
l
e
c
t
e
d
A
L
s
w
i
t
h
m
i
x
e
d
b
a
c
k
g
r
o
u
n
d
s
a
n
d
e
x
p
e
r
i
e
n
c
e
s
1 hour familiarisation session with WISEflow
Marked and provided feedback to 3 TMAs
Completed a questionnaire with open and closed questions
1
8
f
o
l
l
o
w
-
u
p
i
n
t
e
r
v
i
e
w
s
a
n
d
c
o
g
n
i
t
i
v
e
w
a
l
k
t
h
r
o
u
g
h
s
w
i
t
h
a
s
u
b
-
c
o
h
o
r
t
o
f
A
L
s
Data analysis: Thematic analysis, statistical analysis
Herodotou, Christothea and Gillespie, Anna (2020).
Explore: Marking and Feedback (Phase 1) (Final report).
Open
University.
What is the ALs’ overall degree of
technology (WISEflow) acceptance?
Unified Theory of Acceptance and Use of Technology (UTAUT)
•
L
o
w
A
n
x
i
e
t
y
(
M
=
1
.
8
,
S
D
=
.
9
6
)
•
Average Expectancy (M=3.1, SD=1.0)
•
Average Attitudes (M=3.1, SD=1.1)
•
R
e
l
a
t
i
v
e
l
y
h
i
g
h
I
n
t
e
n
t
i
o
n
t
o
u
s
e
t
h
e
s
y
s
t
e
m
(
M
=
3
.
8
,
S
D
=
1
.
5
)
,
•
Relatively high Effort (M=3.6, SD=.8)
•
Relatively high Facilitating conditions (M=3.5, SD=.7)
•
Relatively high Self-efficacy (M=3.5, SD=.7)
I
n
t
e
n
t
i
o
n
t
o
u
s
e
t
h
e
s
y
s
t
e
m
w
a
s
p
o
s
i
t
i
v
e
l
y
r
e
l
a
t
e
d
(
s
i
g
n
i
f
i
c
a
n
t
l
y
)
w
i
t
h
e
x
p
e
c
t
a
n
c
y
,
a
t
t
i
t
u
d
e
s
,
e
f
f
o
r
t
,
a
n
d
s
e
l
f
-
e
f
f
i
c
a
c
y
a
n
d
n
e
g
a
t
i
v
e
l
y
r
e
l
a
t
e
d
t
o
a
n
x
i
e
t
y
.
What is the overall degree of WISEflow acceptance
amongst ALs?
Herodotou, Christothea and Gillespie, Anna (2020).
Explore: Marking and Feedback (Phase 1) (Final report).
Open
University.
Advantages
of WISEflow
•
Easy management of marking within a single
online environment
•
No need to download, zip and upload files
•
Visually stimulating comments
•
through different colour highlighting, stamps and
drawing
•
Audio-feedback
•
more personal feedback through intonation
•
Navigation is easy and intuitive
Usability-
related
challenges
•
Highlight functionality
•
not precise enough
•
Comments/Annotations
•
Need to be: movable, numbered, have more formatting
options
•
Allow paste of graphs/images
•
Need for UNDO facility
•
Rubric - need for clarity
•
Problem loading
•
Not able to copy and paste in the rubric
•
Calculation of marks
•
Need for automatic addition of marks and mark checked
functionality
•
Audio feedback
•
Loading and recording problems
•
Improve recording quality (avoid jumps, repetitions)
Missing
features
•
A space to write notes
•
generic feedback to comments and rubric (e.g., concept
explanations, common errors, examples of good practice)
•
Help guide to explain:
•
highlighting functionality
•
how to use the rubric
•
Edit TMA text
•
model effective writing (e.g., literature, history, languages)
•
cut and paste into the body of the assignment (e.g., diagram,
SPSS output)
•
Accessibility issues
•
Read text software
•
Speech to text software (dictation)
•
Offline marking
•
All marking/feedback done using the Assessor Tool, as per scope
of this activity
•
Three (n=3) ALs stated that they would like to be able to work on
the go when no access to internet connection
Interview analysis
•
Confirmed insights from questionnaires
•
Assignments in varied formats
•
word docs, pdfs, statistical software outputs, power point presentations
•
PDF submissions in WISEflow
•
‘Feedback sandwich’
•
Well accommodated by WISEflow
•
Feedback by annotating the text
•
Opportunity to teach students through modelling
•
Not supported at the moment
•
Enriched feedback strategies using WISEflow
•
Colour-coding of different types of comments
Herodotou, Christothea and Gillespie, Anna (2020).
Explore: Marking and Feedback (Phase 1) (Final report).
Open
University.
Thank you,
questions?
Christothea.Herodotou@open.ac.uk
@herodotouc
Programme team email:
CSR-Exams-Assignment@open.ac.uk
This phase 1 evaluation delves into the adoption of WISEflow among Associate Lecturers (ALs), examining usage patterns, challenges faced, and overall technology acceptance based on the Unified Theory of Acceptance and Use of Technology (UTAUT).
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Explore: Marking and Feedback (Phase 1) Evaluation insights Christothea Herodotou, Senior Lecturer Institute of Educational Technology 1
Background Exams and Assignment (E&A) Implementation Programme Replace the existing eTMA system Expand exams and assignments capabilities Vision: provide a secure, robust and flexible digital assessment platform that improves the student experience Initial focus: replace and enhance assignment submission, marking, and feedback process WISEflow is the new exams and assignments system, offered by UNIwise. Maria Kantirou Project Manager and Chair of the E & A Working group Project website: https://learn3.open.ac.uk/course/view.php?id=301194 2
Aim of Explore Phase 1 Understand how Associate Lecturers (ALs) are using WISEflow Understand whether and how it aligns with OU processes and working practices Identify possible gaps RQ1: What are the usage pathways to completing marking with WISEflow? What features do ALs use/do not use? Do ALs complement their practices with other tools offline or online (e.g. hand written notes, printing out TMAs, other applications such as word doc)? RQ2: What challenges do ALs face when marking with WISEflow? Are these challenges related to missing system features? Are these challenges related to system features they are not aware of or do not know how to use? Herodotou, Christothea and Gillespie, Anna (2020). Explore: Marking and Feedback (Phase 1) (Final report). Open University. 3
Sample and methods 35 self-selected ALs with mixed backgrounds and experiences 1 hour familiarisation session with WISEflow Marked and provided feedback to 3 TMAs Completed a questionnaire with open and closed questions 18 follow-up interviews and cognitive walkthroughs with a sub- cohort of ALs Data analysis: Thematic analysis, statistical analysis Herodotou, Christothea and Gillespie, Anna (2020). Explore: Marking and Feedback (Phase 1) (Final report). Open University. 4
What is the ALs overall degree of technology (WISEflow) acceptance? Unified Theory of Acceptance and Use of Technology (UTAUT) Direct factors explaining technology acceptance Indirect factors explaining technology acceptance Performance Expectancy: The degree to which an individual believes that using the system will help him or her to attain gains in job performance Anxiety: Evoking anxious or emotional reactions when it comes to performing a behaviour (e.g., using the system). Effort expectancy: The degree of ease associated with the use of the system (perceived ease of use, complexity, and ease of use) Self-efficacy: Judgment of one s ability to use a technology to accomplish a particular task. Facilitating conditions: The degree to which an individual believes that an organizational and technical infrastructure exists to support use of the system. Attitudes: An individual s positive or negative feelings about performing the target behaviour. 5
What is the overall degree of WISEflow acceptance amongst ALs? Low Anxiety (M=1.8, SD=.96) Average Expectancy (M=3.1, SD=1.0) Average Attitudes (M=3.1, SD=1.1) Relatively high Intention to use the system (M=3.8, SD=1.5), Relatively high Effort (M=3.6, SD=.8) Relatively high Facilitating conditions (M=3.5, SD=.7) Relatively high Self-efficacy (M=3.5, SD=.7) Intention to use the system was positively related (significantly) with expectancy, attitudes, effort, and self-efficacy and negatively related to anxiety. Herodotou, Christothea and Gillespie, Anna (2020). Explore: Marking and Feedback (Phase 1) (Final report). Open University. 6
Easy management of marking within a single online environment No need to download, zip and upload files Visually stimulating comments through different colour highlighting, stamps and drawing Audio-feedback more personal feedback through intonation Navigation is easy and intuitive Advantages of WISEflow 7
Highlight functionality not precise enough Comments/Annotations Need to be: movable, numbered, have more formatting options Allow paste of graphs/images Need for UNDO facility Rubric - need for clarity Problem loading Not able to copy and paste in the rubric Calculation of marks Need for automatic addition of marks and mark checked functionality Audio feedback Loading and recording problems Improve recording quality (avoid jumps, repetitions) Usability- related challenges 8
A space to write notes generic feedback to comments and rubric (e.g., concept explanations, common errors, examples of good practice) Help guide to explain: highlighting functionality how to use the rubric Edit TMA text model effective writing (e.g., literature, history, languages) cut and paste into the body of the assignment (e.g., diagram, SPSS output) Accessibility issues Read text software Speech to text software (dictation) Offline marking All marking/feedback done using the Assessor Tool, as per scope of this activity Three (n=3) ALs stated that they would like to be able to work on the go when no access to internet connection Missing features 9
Interview analysis Confirmed insights from questionnaires Assignments in varied formats word docs, pdfs, statistical software outputs, power point presentations PDF submissions in WISEflow Feedback sandwich Well accommodated by WISEflow Feedback by annotating the text Opportunity to teach students through modelling Not supported at the moment Enriched feedback strategies using WISEflow Colour-coding of different types of comments Herodotou, Christothea and Gillespie, Anna (2020). Explore: Marking and Feedback (Phase 1) (Final report). Open University. 10
Thank you, questions? Programme team email: CSR-Exams-Assignment@open.ac.uk Christothea.Herodotou@open.ac.uk @herodotouc 11
