Quality Control Coding Process in Research
Quality Control
Coding
LEARNING LIBRARY
FEBRUARY 2022
Learning objectives
●
Learn how to perform quality control on original coding
●
Define “redundant coding”
○
Learn the steps used in redundant coding
●
Define ”naïve coding”
○
Learn the steps used in naïve coding
●
Learn how to do a final quality control check
How to perform
quality control on
original coding
●
As records are coded, the supervisor will
check the following:
○
Unanswered questions
○
Caution notes
○
Citations
○
Formatting
issues with the legal text
●
Original coding checks occur d
aily, as
researchers are coding records
Define “redundant coding”
●
“Redundant coding” consists of two researchers independently coding
identical coding records
○
The supervisor compares and reviews these records to determine where the researchers
diverge
●
Redundant coding identifies:
○
Problems with the questions
○
Problems with the response set
○
Coding errors
Steps in redundant coding
1. Assigning redundant coding
●
The supervisor assigns 100% redundant coding until the rate of divergence is
below 5%
○
When the rate of divergence goes below 5%, the supervisor assigns 20% redundant coding
■
The supervisor may assign additional redundant coding as needed
●
Divergences are recorded on a Coding Review Sheet, a document allowing
researchers to explain their coding decisions in the case of a divergence
Researcher 1
Researcher 2
Code law
R1 divergences
R2 divergences
Identical
responses
Code law
2. Researcher code records
3. Supervisor reviews redundant
coding
●
Calculate the rate of divergence
●
Record divergences and errors in a Coding Review Sheet and send notes to
researchers
Calculate rate of divergence
T
h
e
r
a
t
e
o
f
d
i
v
e
r
g
e
n
c
e
i
s
c
a
l
c
u
l
a
t
e
d
b
y
d
i
v
i
d
i
n
g
t
h
e
t
o
t
a
l
n
u
m
b
e
r
o
f
d
i
v
e
r
g
e
n
c
e
s
i
n
a
b
a
t
c
h
o
f
j
u
r
i
s
d
i
c
t
i
o
n
s
(
n
u
m
e
r
a
t
o
r
)
b
y
t
h
e
t
o
t
a
l
n
u
m
b
e
r
o
f
c
o
d
e
d
v
a
r
i
a
b
l
e
s
(
d
e
n
o
m
i
n
a
t
o
r
)
.
Record divergences and errors in a coding review sheet
Jurisdiction
Iteration
Date of
iteration
Question
Researcher
Supervisor
comments
Researcher
comments
Status
4. Determine reason for divergence
●
Two types of divergences can occur:
1.
Objective:
instances where one coder answered the question incorrectly
2.
Interpretive:
instances where the coders disagreed on a response based on a different
interpretation of the law or of the question
Resolve divergences
W
h
e
n
t
h
e
r
e
i
s
a
n
o
b
j
e
c
t
i
v
e
e
r
r
o
r
,
t
h
e
r
e
s
p
o
n
s
e
s
h
o
u
l
d
b
e
r
e
c
o
d
e
d
●
If a researcher is frequently making
objective errors, additional training may be
necessary
W
h
e
n
t
h
e
r
e
i
s
a
n
i
n
t
e
r
p
r
e
t
i
v
e
e
r
r
o
r
,
t
h
e
a
r
e
s
e
v
e
r
a
l
p
o
t
e
n
t
i
a
l
r
e
s
o
l
u
t
i
o
n
s
:
●
Modify question
●
Collect additional law
●
Edit response set
●
Researchers work on the Coding Review Sheet independently and may agree or disagree on a
response after revisiting the question
●
The team meets to discuss and resolve any outstanding divergences
●
Researchers recode all of their original jurisdictions, as needed
Define “naïve coding”
●
A researcher who is naïve to the project codes 20% of the total number of
records
○
The supervisor compares and reviews these records to determine where the researchers
diverge
●
Naïve coding:
○
Ensures that the project is replicable
○
Increases the accuracy of the project with additional quality control
Steps in naïve coding
1. Assigning naïve coding
●
As coding nears completion, the supervisor assigns a naïve coder to code
20% of the total number of records
○
These records are assigned at random
●
The naïve coder reviews the research protocol and background research prior
to coding
2. Supervisor reviews naïve coding
●
Calculate the rate of divergence
●
Record divergences and errors in a Coding Review Sheet and send notes to
researchers
3. Resolve divergences
●
O
b
j
e
c
t
i
v
e
e
r
r
o
r
s
a
r
e
t
o
b
e
e
x
p
e
c
t
e
d
a
t
a
h
i
g
h
e
r
r
a
t
e
f
o
r
n
a
ï
v
e
c
o
d
i
n
g
t
h
a
n
f
o
r
r
e
d
u
n
d
a
n
t
c
o
d
i
n
g
,
b
e
c
a
u
s
e
t
h
e
n
a
ï
v
e
r
e
s
e
a
r
c
h
e
r
i
s
u
n
f
a
m
i
l
i
a
r
w
i
t
h
t
h
e
t
o
p
i
c
●
I
n
t
e
r
p
r
e
t
i
v
e
e
r
r
o
r
s
s
h
o
u
l
d
o
c
c
u
r
a
t
a
r
e
d
u
c
e
d
r
a
t
e
.
A
n
e
x
c
e
s
s
o
f
i
n
t
e
r
p
r
e
t
i
v
e
e
r
r
o
r
s
m
i
g
h
t
i
n
d
i
c
a
t
e
t
h
a
t
q
u
e
s
t
i
o
n
s
a
n
d
r
e
s
p
o
n
s
e
s
a
r
e
u
n
c
l
e
a
r
,
t
h
a
t
l
a
w
s
a
r
e
m
i
s
s
i
n
g
,
o
r
t
h
a
t
t
h
e
r
e
s
e
a
r
c
h
p
r
o
t
o
c
o
l
n
e
e
d
s
t
o
b
e
c
l
a
r
i
f
i
e
d
●
The original and naïve coder must go
through the Coding Review Sheet
independently and may agree or disagree
on a response after revisiting the question
●
The team meets to discuss and resolve
any outstanding divergences
●
The original coder recodes jurisdictions, as
needed
●
When naïve coding results in a particularly
high rate of divergence, or reveals a
systemic problem with a project, the entire
project may have to be recorded, or
questions and responses may have to be
adjusted
Final quality control checks
●
Supervisor will review all of the questions, responses, and citations prior to
publishing the project to identify any outstanding issues, including:
○
Any questions that were not answered
○
Outlier responses
○
Missing citations
○
Inconsistent caution notes
Summary
●
T
h
e
s
u
p
e
r
v
i
s
o
r
s
h
o
u
l
d
c
h
e
c
k
o
r
i
g
i
n
a
l
c
o
d
i
n
g
a
s
i
t
i
s
b
e
i
n
g
d
o
n
e
●
R
e
d
u
n
d
a
n
t
c
o
d
i
n
g
–
t
w
o
r
e
s
e
a
r
c
h
e
r
s
c
o
d
e
i
d
e
n
t
i
c
a
l
r
e
c
o
r
d
s
o
f
t
h
e
s
a
m
e
j
u
r
i
s
d
i
c
t
i
o
n
.
S
u
p
e
r
v
i
s
o
r
c
o
m
p
a
r
e
s
r
e
c
o
r
d
s
t
o
d
e
t
e
r
m
i
n
e
w
h
i
c
h
c
o
d
i
n
g
a
n
s
w
e
r
s
s
h
o
u
l
d
b
e
s
e
l
e
c
t
e
d
●
The steps involved in redundant coding are:
1.
Researchers code identical records
2.
Supervisor reviews coding
3.
Team resolves divergences
●
N
a
ï
v
e
c
o
d
i
n
g
–
a
r
e
s
e
a
r
c
h
e
r
u
n
f
a
m
i
l
i
a
r
w
i
t
h
t
h
e
p
r
o
j
e
c
t
c
o
d
e
s
2
0
%
o
f
t
h
e
t
o
t
a
l
n
u
m
b
e
r
o
f
r
e
c
o
r
d
s
f
o
r
t
h
a
t
p
r
o
j
e
c
t
●
The steps involved in naïve coding are:
1.
Supervisor assigns naïve coding
•
Naïve researcher codes 20% of the total number of records
2.
Supervisor reviews naïve coding
3.
Team resolves divergences
●
T
h
e
s
u
p
e
r
v
i
s
o
r
w
i
l
l
d
o
a
f
i
n
a
l
c
h
e
c
k
o
f
a
l
l
d
a
t
a
b
e
f
o
r
e
p
u
b
l
i
s
h
i
n
g
Learn how to ensure quality in coding processes through redundant coding, defining coding types, and conducting final quality control checks. Supervisors review researchers' work to identify issues and divergence, maintaining accuracy in research coding practices.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Quality Control Coding LEARNING LIBRARY FEBRUARY 2022
Learning objectives Learn how to perform quality control on original coding Define redundant coding Learn the steps used in redundant coding Define na ve coding Learn the steps used in na ve coding Learn how to do a final quality control check 2
As records are coded, the supervisor will check the following: How to perform quality control on original coding Unanswered questions Caution notes Citations Formatting issues with the legal text Original coding checks occur daily, as researchers are coding records
Define redundant coding Redundant coding consists of two researchers independently coding identical coding records The supervisor compares and reviews these records to determine where the researchers diverge Redundant coding identifies: Problems with the questions Problems with the response set Coding errors 4
Steps in redundant coding 1 Supervisor assigns redundant coding 2 Researchers code records 3 Supervisor reviews redundant coding 4 Team resolves divergences
1. Assigning redundant coding The supervisor assigns 100% redundant coding until the rate of divergence is below 5% When the rate of divergence goes below 5%, the supervisor assigns 20% redundant coding The supervisor may assign additional redundant coding as needed Divergences are recorded on a Coding Review Sheet, a document allowing researchers to explain their coding decisions in the case of a divergence 6
2. Researcher code records R1 divergences Code law Researcher 1 Identical responses Code law Researcher 2 R2 divergences 7
3. Supervisor reviews redundant coding Calculate the rate of divergence Record divergences and errors in a Coding Review Sheet and send notes to researchers 8
Calculate rate of divergence The rate of divergence is calculated by dividing the total number of divergences in a batch of jurisdictions (numerator) by the total number of coded variables (denominator). 9
Record divergences and errors in a coding review sheet Question Jurisdiction Supervisor comments Researcher comments Date of iteration Status Researcher Iteration
4. Determine reason for divergence Two types of divergences can occur: 1. Objective: instances where one coder answered the question incorrectly 2. Interpretive: instances where the coders disagreed on a response based on a different interpretation of the law or of the question 11
Resolve divergences When there is an objective error, the response should be recoded If a researcher is frequently making objective errors, additional training may be necessary When there is an interpretive error, the are several potential resolutions: Modify question Collect additional law Edit response set Researchers work on the Coding Review Sheet independently and may agree or disagree on a response after revisiting the question The team meets to discuss and resolve any outstanding divergences Researchers recode all of their original jurisdictions, as needed 12
Define nave coding A researcher who is na ve to the project codes 20% of the total number of records The supervisor compares and reviews these records to determine where the researchers diverge Na ve coding: Ensures that the project is replicable Increases the accuracy of the project with additional quality control 13
Steps in nave coding Supervisor assigns na ve coding Supervisor reviews na ve coding Team resolves divergences
1. Assigning nave coding As coding nears completion, the supervisor assigns a na ve coder to code 20% of the total number of records These records are assigned at random The na ve coder reviews the research protocol and background research prior to coding 15
2. Supervisor reviews nave coding Calculate the rate of divergence Record divergences and errors in a Coding Review Sheet and send notes to researchers 16
3. Resolve divergences Objective errors are to be expected at a higher rate for na ve coding than for redundant coding, because the na ve researcher is unfamiliar with the topic Interpretive errors should occur at a reduced rate. An excess of interpretive errors might indicate that questions and responses are unclear, that laws are missing, or that the research protocol needs to be clarified The original and na ve coder must go through the Coding Review Sheet independently and may agree or disagree on a response after revisiting the question The team meets to discuss and resolve any outstanding divergences The original coder recodes jurisdictions, as needed When na ve coding results in a particularly high rate of divergence, or reveals a systemic problem with a project, the entire project may have to be recorded, or questions and responses may have to be adjusted 17
Final quality control checks Supervisor will review all of the questions, responses, and citations prior to publishing the project to identify any outstanding issues, including: Any questions that were not answered Outlier responses Missing citations Inconsistent caution notes 18
Summary The supervisor should check original coding as it is being done Redundant coding two researchers code identical records of the same jurisdiction. Supervisor compares records to determine which coding answers should be selected The steps involved in redundant coding are: 1. Researchers code identical records 2. Supervisor reviews coding 3. Team resolves divergences Na ve coding a researcher unfamiliar with the project codes 20% of the total number of records for that project The steps involved in na ve coding are: 1. Supervisor assigns na ve coding Na ve researcher codes 20% of the total number of records 2. Supervisor reviews na ve coding 3. Team resolves divergences The supervisor will do a final check of all data before publishing 19
