Framework for Evaluating Critical Thinking Apps: Enhancing Analytical Skills
Session D2
1.
A Framework for Evaluating Critical Thinking Apps
2.
New Detective Tool to Tell What Students Are
Thinking in Class
3.
Use of Social Media to Motivate Academic Inquiry
and Discussion for Developing of Critical Thinking
4.
AR Marker-Based Assessment as Learning for an
Analytical Thinking
5.
Social Media as Digital Companionship:
Scaffolding in the E-Learning Context
A
F
r
a
m
e
w
o
r
k
f
o
r
E
v
a
l
u
a
t
i
n
g
C
r
i
t
i
c
a
l
T
h
i
n
k
i
n
g
A
p
p
s
Teresa Chen, Ph.D.
California State University, Long Beach
eLearning Forum Asia
Taipei, Taiwan
May, 2018
O
v
e
r
v
i
e
w
Significance of the research
Current practices
Research questions
Audience who may benefit from the
study
Introduction
Significance
Importance of critical thinking
Apps for cultivating critical thinking
Challenges
Misalignment between the
conceptualization of critical thinking and
the content of the so-called CT Apps
Anecdotal recommendations in the “best
Apps” lists and reviews
What can be considered a critical
thinking App?
What are the design principles that are
essential in the development of a
critical thinking App?
Research Questions
Intended Audience
Apps users, including instructors (and
students)
Apps designers and developers
Critical thinking skills, dispositions,
and the strategies that support
students’ development of CT
Software (with an emphasis on Apps)
design principles that can be applied
to the design of CT Apps
Research-based tools (e.g., rubrics
and checklists) for Apps evaluation
Literature Review
1.
Formulate a research problem
2.
Conduct a literature search
3.
Evaluate search results
4.
Analyze results
5.
Interprete findings
6.
Use the findings to create and
validate an evaluation framework
Method: Research Synthesis
Three Lines of Research
Definitions of Critical Thinking
Cognitive skills and affective dispositions from the
landmark “Delphi Report” (Facione, 1990)
“Critical thinking is a habit of mind characterized by
the comprehensive exploration of issues, ideas,
artifacts, and events before accepting or formulating
an opinion or conclusion” (AACU, 2009)
Operational definition (Liu, et al., 2014)
Five dimensions, including analytical and synthetic
dimensions
Argument analysis & development; causation &
explanation
Critical-thinking assessment test (Haynes, et al.,
2016)
“Classic” and broadened definitions
Definitions
F
o
r
m
a
l
R
e
a
s
o
n
i
n
g
S
k
i
l
l
s
&
D
i
s
p
o
s
i
t
i
o
n
s
P
r
o
b
l
e
m
-
S
o
l
v
i
n
g
&
C
r
e
a
t
i
v
i
t
y
D
e
f
i
n
i
t
i
o
n
s
o
f
C
r
i
t
i
c
a
l
T
h
i
n
k
i
n
g
Pedagogy
Mixed approach (Ennis, 1989; 1997)
teaching critical thinking as a subject itself
infusing it in the other subject matter
Strategies: e.g., Argument mapping, case studies,
computer-assisted reasoning, logic modeling
(Mathias, 2015; Niu, Behar-Horenstein, & Garvan,
2013)
Explicit instruction of reasoning principles and
ample opportunities for practice
: Essential to
students’ development of critical thinking skills
(Heijltyes, van Gog, Leppink, & Pass, 2014)
Design Principles
Instructional design models (Brown &
Green, 2015; Luterbach, 2018)
Eight golden rules of interface design
(Shneiderman, et al., 2016)
Fundamental principles of interaction
(Norman, 2013)
Apps Evaluation
Tools: Frameworks, rubrics and checklists
(e.g., Cherner, Lee, Fegely, & Santaniello,
2016; Israelson, 2015; Lee & Cherner, 2015;
Ok, Kim, Kang, & Bryant, 2016)
Evaluation criteria
Subject/ content/ educational value
Design (including instructional design/
technology design/ user experience
design)
Engagement/ motivation
Three Lines of Research
1.
Content
2.
Pedagogy
Critical thinking in particular
Instructional design in general
3.
User-centered d
esign principles
Interface design
User experience design
Framework: Categories
Enables users to
1.
Evaluate evidence and its use
2.
Analyze and evaluate arguments
3.
Understand implications and
consequences
4.
Develop sound and valid arguments
5.
Understand causation and explanation
Broadened definitions:
Problem
-solve and develop creativity
A. Content
Specifically
Explicit instruction of reasoning
principles
Ample opportunities for practice
Instructional strategies
In general
learning objectives, instructional
strategies and assessment as well as
alignment among them
Bloom’s taxonomy
B. Pedagogy
C1. Design: User Interface
1.
Strive for consistency
2.
Enable frequent users to use shortcuts
3.
Offer informative feedback
4.
Design dialog to yield closure
5.
Offer simple error handling
6.
Permit easy reversal of actions
7.
Support internal locus of control
8.
Reduce short-term memory load
Eight Golden Rules (Schneiderman, et al., 2016)
C2. Design: User Experience
1.
Affordances
2.
Signifiers
3.
Constraints
4.
Mappings
5.
Feedback
6.
Conceptual model
Fundamental Principles (Norman, 2013)
Evaluation Instrument
Validation: Pilot Test
Two evaluators’ independent review of ten
CT Apps
80% in agreement with category A, 60%
with category B, 90% with C1, and 80
with C2.
Correlation with the evaluation rubric, for
assessing instructional Apps in general,
developed by Lee and Cherner (2015)
Overall recommendation: 80% in
agreement
Implications for practice
Guidelines fo
r the development of CT
Apps
Teaching and learning with CT Apps
Implications for research
Design and quality of CT Apps
Effectiveness of CT Apps (learning
outcomes)
Conclusion
Introduction to a research study by Teresa Chen, Ph.D., focusing on the importance of critical thinking apps in academia. The study explores design principles for developing effective critical thinking apps, challenges faced, and research questions on app conceptualization.
Uploaded on Sep 14, 2024 | 0 Views
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Session D2 A Framework for Evaluating Critical Thinking Apps New Detective Tool to Tell What Students Are Thinking in Class Use of Social Media to Motivate Academic Inquiry and Discussion for Developing of Critical Thinking AR Marker-Based Assessment as Learning for an Analytical Thinking Social Media as Digital Companionship: Scaffolding in the E-Learning Context 1. 2. 3. 4. 5.
A Framework for Evaluating A Framework for Evaluating Critical Thinking Apps Critical Thinking Apps Teresa Chen, Ph.D. California State University, Long Beach eLearning Forum Asia Taipei, Taiwan May, 2018
Overview Overview Introduction Literature Review Method Results and Discussion Conclusion
Introduction Significance of the research Current practices Research questions Audience who may benefit from the study
Significance Importance of critical thinking Apps for cultivating critical thinking Challenges Misalignment between the conceptualization of critical thinking and the content of the so-called CT Apps Anecdotal recommendations in the best Apps lists and reviews
Research Questions What can be considered a critical thinking App? What are the design principles that are essential in the development of a critical thinking App?
Intended Audience Apps users, including instructors (and students) Apps designers and developers
Literature Review Critical thinking skills, dispositions, and the strategies that support students development of CT Software (with an emphasis on Apps) design principles that can be applied to the design of CT Apps Research-based tools (e.g., rubrics and checklists) for Apps evaluation
Method: Research Synthesis 1. Formulate a research problem 2. Conduct a literature search 3. Evaluate search results 4. Analyze results 5. Interprete findings 6. Use the findings to create and validate an evaluation framework
Three Lines of Research Critical Thinking Definitions Pedagogical approaches Instructional design User interface design User experience design Design Principles Apps Evaluation Research-based criteria Evaluation tools
Definitions of Critical Thinking Cognitive skills and affective dispositions from the landmark Delphi Report (Facione, 1990) Critical thinking is a habit of mind characterized by the comprehensive exploration of issues, ideas, artifacts, and events before accepting or formulating an opinion or conclusion (AACU, 2009) Operational definition (Liu, et al., 2014) Five dimensions, including analytical and synthetic dimensions Argument analysis & development; causation & explanation Critical-thinking assessment test (Haynes, et al., 2016) Classic and broadened definitions
Definitions Definitions of Critical Thinking Definitions of Critical Thinking
Pedagogy Mixed approach (Ennis, 1989; 1997) teaching critical thinking as a subject itself infusing it in the other subject matter Strategies: e.g., Argument mapping, case studies, computer-assisted reasoning, logic modeling (Mathias, 2015; Niu, Behar-Horenstein, & Garvan, 2013) Explicit instruction of reasoning principles and ample opportunities for practice: Essential to students development of critical thinking skills (Heijltyes, van Gog, Leppink, & Pass, 2014)
Design Principles Instructional design models (Brown & Green, 2015; Luterbach, 2018) Eight golden rules of interface design (Shneiderman, et al., 2016) Fundamental principles of interaction (Norman, 2013)
Apps Evaluation Tools: Frameworks, rubrics and checklists (e.g., Cherner, Lee, Fegely, & Santaniello, 2016; Israelson, 2015; Lee & Cherner, 2015; Ok, Kim, Kang, & Bryant, 2016) Evaluation criteria Subject/ content/ educational value Design (including instructional design/ technology design/ user experience design) Engagement/ motivation
Three Lines of Research Critical Thinking Apps Evaluation Design Principles Evaluation Framework Evaluation Framework
Framework: Categories 1. Content 2. Pedagogy Critical thinking in particular Instructional design in general 3. User-centered design principles Interface design User experience design
A. Content Enables users to Evaluate evidence and its use Analyze and evaluate arguments Understand implications and consequences Develop sound and valid arguments Understand causation and explanation Broadened definitions: Problem-solve and develop creativity 1. 2. 3. 4. 5.
B. Pedagogy Specifically Explicit instruction of reasoning principles Ample opportunities for practice Instructional strategies In general learning objectives, instructional strategies and assessment as well as alignment among them Bloom s taxonomy
C1. Design: User Interface Strive for consistency Enable frequent users to use shortcuts Offer informative feedback Design dialog to yield closure Offer simple error handling Permit easy reversal of actions Support internal locus of control Reduce short-term memory load 1. 2. 3. 4. 5. 6. 7. 8. Eight Golden Rules (Schneiderman, et al., 2016)
C2. Design: User Experience 1. Affordances 2. Signifiers 3. Constraints 4. Mappings 5. Feedback 6. Conceptual model Fundamental Principles (Norman, 2013)
Validation: Pilot Test Two evaluators independent review of ten CT Apps 80% in agreement with category A, 60% with category B, 90% with C1, and 80 with C2. Correlation with the evaluation rubric, for assessing instructional Apps in general, developed by Lee and Cherner (2015) Overall recommendation: 80% in agreement
Conclusion Implications for practice Guidelines for the development of CT Apps Teaching and learning with CT Apps Implications for research Design and quality of CT Apps Effectiveness of CT Apps (learning outcomes)
