Quality assurance dysfunctions
Q
u
a
l
i
t
y
a
s
s
u
r
a
n
c
e
d
y
s
f
u
n
c
t
i
o
n
s
Dr Martin Boswell
, Registrar, Whitireia Community
Polytechnic and Wellington Institute of Technology,
Wellington, New Zealand
currently seconded as:
Academic Operations Senior Implementation Lead at
Te Pūkenga - New Zealand Institute of Skills and Technology
A
c
a
d
e
m
i
c
a
u
d
i
t
–
a
c
a
s
e
s
t
u
d
y
f
r
o
m
N
e
w
Z
e
a
l
a
n
d
An external QA process conducted in NZ since 1994 …
1.
Operated by non-government agency: the Academic Quality
Agency for New Zealand Universities (
)www.aqa.ac.nz
2.
Aims to assure
and
enhance quality
3.
Focus on internal QA
processes
(not outcomes)
4.
Institution level (not programmes)
5.
Peer review
6.
Recommendations but no pass/fail judgement(s)
P
h
D
:
Q
u
a
l
i
t
a
t
i
v
e
,
l
o
n
g
i
t
u
d
i
n
a
l
c
a
s
e
-
s
t
u
d
y
•
Engagement with academic audit, 1994–2019
•
Methodology: documentary evidence, 19 interviews
•
11 recommendations
recurred
…
•
Recurrence: potential challenges with engagement and
implementation of recommendations
1.
Closing the student feedback loop
2.
Monitoring PhD student progress (six-monthly reports)
3.
‘QA infrastructure and leadership’
•
Supervisors: Prof Stephen Marshall, A/Prof Kate Thornton
S
u
m
m
a
r
y
f
i
n
d
i
n
g
s
:
Q
A
d
y
s
f
u
n
c
t
i
o
n
s
P
a
r
t
1
•
Good intentions
•
Reinterpretation
•
Deferral
•
Repackaging
Some quotes used in this presentation
have been paraphrased for clarity/brevity
Institution side:
G
o
o
d
i
n
t
e
n
t
i
o
n
s
•
Background: The AQA encourages ‘enhancement initiatives’
in self-review process
•
Cycle 3 self-review (2005)
: “The university will conduct a
major planning exercise that will re-think QA, including the
roles, responsibilities and membership of Academic Board,
Academic Committee and Quality Office…”
•
Cycle 4 audit (2009)
: Modest changes to Academic Board
structure and membership; no other changes
•
Conclusion
: Avoid overpromising (institution)
R
e
i
n
t
e
r
p
r
e
t
a
t
i
o
n
•
Cycle 5 audit (2014)
: “
Urgently incorporate feedback to
students regarding actions taken in response to course
evaluations
”
•
Cycle 5 mid-cycle report-back (2019)
:
“Recommendation has
been implemented and timeliness of evaluation of course
feedback has improved”
•
In fact: no implementation; ‘timeliness’ not relevant
•
Conclusions
: Be truthful (institution). Be attentive (agency)
D
e
f
e
r
r
a
l
•
Cycle 2 audit (2000)
: Recommendation: “Create a
compendium of QA processes”
•
Cycle 3 self-review (2005)
: “We reviewed key issues raised in
the report and will start [sic] a review…”
•
Nothing about compendium of QA processes
•
Symptoms:
Actions are “ongoing” or “working group formed”
•
Conclusion
: Be attentive (agency), lest deferral by institution
become a permanent state
R
e
p
a
c
k
a
g
i
n
g
•
Cycle 2 audit (2000)
: “Close the student feedback loop”
•
Cycle 3 audit (2005)
: “Close the student feedback loop”
•
Cycle 4 self-review (2009)
: Enhancement initiative:
“We will establish [sic] a systematic process for
responding to student feedback”
•
Recommendation not implemented, now repackaged
•
Conclusion
: Be attentive (agency) and challenge failure
to implement
S
u
m
m
a
r
y
f
i
n
d
i
n
g
s
:
Q
A
d
y
s
f
u
n
c
t
i
o
n
s
P
a
r
t
2
Quality agency side:
•
Echoing
•
Reinforcement
•
Complexity
•
Superimposition
•
Abandonment
E
c
h
o
i
n
g
•
Cycle 2 self-review (2000)
: “We were advised to strongly
encourage/require staff to provide feedback about survey results
to students and explain how their concerns will be addressed”
•
Cycle 2 audit (2000)
: “Teachers should be strongly encouraged/
required to provide feedback to students about the results of
evaluations and notify how any concerns will be addressed”
•
Symptoms
: “The panel was told that …[verbatim quote from self-
review]” or “The university intends to …[ditto]”
•
Conclusion
: Echoing is an indicator of possible superficiality
R
e
i
n
f
o
r
c
e
m
e
n
t
•
Cycle 3 audit (2005)
: “The panel was pleased to hear that a
new Learning and Teaching Committee will be established
and urges that this be done as quickly as possible”
•
A suggestion, not a formal recommendation
•
Echoing is observational / Reinforcement implies action
•
Conclusion
: If a recommendation is needed, make one.
Suggestions are less likely to be implemented if they are
buried in the document text.
C
o
m
p
l
e
x
i
t
y
:
A
n
e
x
a
m
p
l
e
o
f
1
r
e
c
o
m
m
e
n
d
a
t
i
o
n
:
“The panel recommends that the University continues to develop and
implement an appropriate institution-wide quality assurance
infrastructure and institutional leadership for quality assurance and
quality enhancement activities that:
1.
builds on and enhances those existing mechanisms and procedures
that are effective in supporting a quality culture,
2.
facilitates ownership of, and engagement with, quality and
continuous improvement activities by the whole University
community, and
3.
provides assurance of institution-wide quality in research, teaching,
learning and community engagement” (
Cy4AR, 2009, p. 31
)
S
u
p
e
r
i
m
p
o
s
i
t
i
o
n
Cycle 2 audit (2000)
: (1) “Centralize the checking of PhD
student biannual progress monitoring and …
(2) analyse progress monitoring for possible systemic issues”
•
Research found only 1 instance (in 19 years) of a systemic
issue that was not of a personal nature
•
Why collect and address
personal
issues systemically?
•
Conclusion
: Did the agency create extra bureaucracy?
•
Postscript
: The university did not collate personal
information; it ‘disobeyed’ the recommendation
A
b
a
n
d
o
n
m
e
n
t
•
Cycle 5 audit report (2014)
:
“The use of student
feedback has been reviewed but there is still a
significant agenda for change in teaching and learning”
•
Cycle 5 follow-up report (2016)
: No comment from
agency about lack of implementation
•
Cycle 5 mid-cycle report (2019)
: No comment from
agency about lack of implementation
•
Conclusion
: Why bother giving the recommendation?
O
v
e
r
a
l
l
c
o
n
c
l
u
s
i
o
n
s
•
Auditors / evaluators must be experienced
•
A longitudinal approach can detect dysfunctions
•
Recommendations must be workable, concise, clear
•
Encourage candour about implementation challenges
•
Agency should not abandon earlier recommendations
•
Targeted QA scope may help avoid superficiality
•
Hierarchical structures seem to hinder quality/QA
•
Carefully consider potential superimposition
Thank you /
рақмет
!
Questions?
Thesis:
https://doi.org/10.26686/wgtn.22337005
Findings from PhD research from recently submitted thesis
This content discusses quality assurance dysfunctions in academic institutions, focusing on an external QA process in New Zealand operated by the Academic Quality Agency for New Zealand Universities. It includes a case study on academic audits, engagement with quality assurance, and summary findings on dysfunctions such as good intentions, reinterpretation, deferral, and repackaging. Recommendations for improvement are highlighted to address potential challenges in implementing quality assurance processes effectively.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Quality assurance dysfunctions Quality assurance dysfunctions Dr Martin Boswell, Registrar, Whitireia Community Polytechnic and Wellington Institute of Technology, Wellington, New Zealand currently seconded as: Academic Operations Senior Implementation Lead at Te P kenga - New Zealand Institute of Skills and Technology https://en.wikipedia.org/wiki/New_Zealand#/media/File:NZL_orthographic_NaturalEarth.svg
Academic audit Academic audit a case study from New Zealand a case study from New Zealand An external QA process conducted in NZ since 1994 1. Operated by non-government agency: the Academic Quality Agency for New Zealand Universities (www.aqa.ac.nz) 2. Aims to assure and enhance quality 3. Focus on internal QA processes (not outcomes) 4. Institution level (not programmes) 5. Peer review 6. Recommendations but no pass/fail judgement(s)
PhD: Qualitative, longitudinal case PhD: Qualitative, longitudinal case- -study study Engagement with academic audit, 1994 2019 Methodology: documentary evidence, 19 interviews 11 recommendations recurred Recurrence: potential challenges with engagement and implementation of recommendations 1. Closing the student feedback loop 2. Monitoring PhD student progress (six-monthly reports) 3. QA infrastructure and leadership Supervisors: Prof Stephen Marshall, A/Prof Kate Thornton
Summary findings: QA dysfunctions Part 1 Summary findings: QA dysfunctions Part 1 Good intentions Reinterpretation Deferral Repackaging Institution side: Some quotes used in this presentation have been paraphrased for clarity/brevity
Good intentions Good intentions Background: The AQA encourages enhancement initiatives in self-review process Cycle 3 self-review (2005): The university will conduct a major planning exercise that will re-think QA, including the roles, responsibilities and membership of Academic Board, Academic Committee and Quality Office Cycle 4 audit (2009): Modest changes to Academic Board structure and membership; no other changes Conclusion: Avoid overpromising (institution)
Reinterpretation Reinterpretation Cycle 5 audit (2014): Urgently incorporate feedback to students regarding actions taken in response to course evaluations Cycle 5 mid-cycle report-back (2019): Recommendation has been implemented and timeliness of evaluation of course feedback has improved In fact: no implementation; timeliness not relevant Conclusions: Be truthful (institution). Be attentive (agency)
Deferral Deferral Cycle 2 audit (2000): Recommendation: Create a compendium of QA processes Cycle 3 self-review (2005): We reviewed key issues raised in the report and will start [sic] a review Nothing about compendium of QA processes Symptoms: Actions are ongoing or working group formed Conclusion: Be attentive (agency), lest deferral by institution become a permanent state
Repackaging Repackaging Cycle 2 audit (2000): Close the student feedback loop Cycle 3 audit (2005): Close the student feedback loop Cycle 4 self-review (2009): Enhancement initiative: We will establish [sic] a systematic process for responding to student feedback Recommendation not implemented, now repackaged Conclusion: Be attentive (agency) and challenge failure to implement
Summary findings: QA dysfunctions Part 2 Summary findings: QA dysfunctions Part 2 Echoing Reinforcement Complexity Superimposition Abandonment Quality agency side:
Echoing Echoing Cycle 2 self-review (2000): We were advised to strongly encourage/require staff to provide feedback about survey results to students and explain how their concerns will be addressed Cycle 2 audit (2000): Teachers should be strongly encouraged/ required to provide feedback to students about the results of evaluations and notify how any concerns will be addressed Symptoms: The panel was told that [verbatim quote from self- review] or The university intends to [ditto] Conclusion: Echoing is an indicator of possible superficiality
Reinforcement Reinforcement Cycle 3 audit (2005): The panel was pleased to hear that a new Learning and Teaching Committee will be established and urges that this be done as quickly as possible A suggestion, not a formal recommendation Echoing is observational / Reinforcement implies action Conclusion: If a recommendation is needed, make one. Suggestions are less likely to be implemented if they are buried in the document text.
Complexity: Complexity: An example of 1 recommendation: The panel recommends that the University continues to develop and implement an appropriate institution-wide quality assurance infrastructure and institutional leadership for quality assurance and quality enhancement activities that: 1. builds on and enhances those existing mechanisms and procedures that are effective in supporting a quality culture, 2. facilitates ownership of, and engagement with, quality and continuous improvement activities by the whole University community, and 3. provides assurance of institution-wide quality in research, teaching, learning and community engagement (Cy4AR, 2009, p. 31)
Superimposition Superimposition Cycle 2 audit (2000): (1) Centralize the checking of PhD student biannual progress monitoring and (2) analyse progress monitoring for possible systemic issues Research found only 1 instance (in 19 years) of a systemic issue that was not of a personal nature Why collect and address personal issues systemically? Conclusion: Did the agency create extra bureaucracy? Postscript: The university did not collate personal information; it disobeyed the recommendation
Abandonment Abandonment Cycle 5 audit report (2014): The use of student feedback has been reviewed but there is still a significant agenda for change in teaching and learning Cycle 5 follow-up report (2016): No comment from agency about lack of implementation Cycle 5 mid-cycle report (2019): No comment from agency about lack of implementation Conclusion: Why bother giving the recommendation?
Overall conclusions Overall conclusions Auditors / evaluators must be experienced A longitudinal approach can detect dysfunctions Recommendations must be workable, concise, clear Encourage candour about implementation challenges Agency should not abandon earlier recommendations Targeted QA scope may help avoid superficiality Hierarchical structures seem to hinder quality/QA Carefully consider potential superimposition
Thank you / ! Questions? Thesis: https://doi.org/10.26686/wgtn.22337005
