Challenging Convictions: Hidden Failures and Bayesian Analysis

 
P
r
e
s
e
n
t
a
t
i
o
n
 
o
f
 
A
r
t
i
c
l
e
:
T
o
o
 
g
o
o
d
 
t
o
 
b
e
 
t
r
u
e
:
W
h
e
n
 
o
v
e
r
w
h
e
l
m
i
n
g
 
e
v
i
d
e
n
c
e
f
a
i
l
s
 
t
o
 
c
o
n
v
i
n
c
e
 
PRESENTATION BY:
CHRISTIAN MICHELSEN, JENS ARNBAK AND SOFUS LAGUNA KRISTENSEN
 
Written by:
Lachlan J. Gunn, François Chapeau-Blondeau, Mark
McDonnell, Bruce Davis, Andrew Allison, Derek Abbott
 
Introduction
 
Normally:
More measurements agree with your model ->  better confidence in your
model
The article introduces a ”
hidden failure state
This causes your confidence in the model to decrease with increasing
agreement with data.
This is known as: 
Verschlimmbesserung
 or 
disimprovement
.
The article analyzes this through 
B
ayesian analysis:
Theory – Bayesian Analysis
 
Bayes’ Law
 
Without hidden failure state:
 
Including a hidden failure state defined
by the variable F :
 
Determining Origin of Roman Pot
 
Roman pot found in Britain – we wish to determine whether a specific pot was made in Roman
occupied Britain or transported from Italy to Britain.
 
Two hypotheses: H
0
: Italy, H
1
: Britain
Flat prior – both are equally likely
Test for certain trace element found in British clay: error rate p
e
 = 0.3
 
Hidden Failure State – introduction of trace element during manufacturing process
Rate of contamination: p
c
 = 0.01
50 / 50 distribution of contaminated pots between Britain and Rome
If contaminated, the trace element will be measured with 90 % probability
Determining Origin of Roman Pot
Information table:
 
Plotting the resulting PMF:
 
Identifying Suspect in Identity Parade
 
 
We want to estimate the probability of correctly identifying a suspect as the perpetrator through
the use of identity parades.
 
Again two hypotheses: H
0
: Innocent, H
1
: Guilty
Flat prior: 50 / 50
False-Negative rate: the probability of falsely accusing an innocent suspect when perpetrator is in the
parade p
fn
  = 0.48
False-Positive rate: the probability of falsely accusing an innocent suspect when perpetrator is not in the
parade p
fp
 = 0.133
 
Hidden Failure State: bias in the conduction of the parade
Small probability p
c
 that the parade is biased
If the identity parade is biased, the suspect is identified as guilty 90 % of the time regardless of guilt.
Identifying Suspect in Identity Parade
Information table:
 
Identifying Suspect in Identity Parade
 
 
 
Even with just p
c
 = 1%,
the probability of guilt is never > 95%.
 
Determining Origin of Roman Pot
 
 
What if the success rate (1-p
e
) is larger than the success rate while contaminated (90 %)
 
Determining Origin of Roman Pot
 
 
What if the success rate (1-p
e
) is larger than the success rate while contaminated (90 %)
 
Determining Origin of Roman Pot
 
 
What if the success rate (1-p
e
) is larger than the success rate while contaminated (90 %)
 
Conclusion
 
 
Including hidden failure states highly changes 
the probabilistic
nature of the problem
Even a small probability of bias (hidden failure state) can
drastically reduce the confidence 
of our test
In real life, the ratio between 
(1- p
e
) 
and p
fp
 high determines the
significance of the hidden failure state
 
Questions?
Slide Note
Embed
Share

Delve into the intriguing concept of hidden failure states impacting model confidence, as explored in the article by Lachlan J. Gunn and team. Through Bayesian analysis, the article uncovers how overwhelming evidence may fail to persuade, introducing terms like Verschlimmbesserung. Case studies involving Roman pottery origins and suspect identification in identity parades shed light on the practical implications of hidden states. Dive deep into the world of uncertainty and model disimprovement in this thought-provoking piece.


Uploaded on Aug 30, 2024 | 0 Views


Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

E N D

Presentation Transcript


  1. Presentation of Article: Presentation of Article: Too good to be true: When overwhelming evidence fails to convince Written by: Lachlan J. Gunn, Fran ois Chapeau-Blondeau, Mark McDonnell, Bruce Davis, Andrew Allison, Derek Abbott PRESENTATION BY: CHRISTIAN MICHELSEN, JENS ARNBAK AND SOFUS LAGUNA KRISTENSEN

  2. Introduction Normally: More measurements agree with your model -> better confidence in your model The article introduces a hidden failure state This causes your confidence in the model to decrease with increasing agreement with data. This is known as: Verschlimmbesserung or disimprovement. The article analyzes this through Bayesian analysis:

  3. Theory Bayesian Analysis Bayes Law Without hidden failure state: Including a hidden failure state defined by the variable F :

  4. Determining Origin of Roman Pot Roman pot found in Britain we wish to determine whether a specific pot was made in Roman occupied Britain or transported from Italy to Britain. Two hypotheses: H0: Italy, H1: Britain Flat prior both are equally likely Test for certain trace element found in British clay: error rate pe = 0.3 Hidden Failure State introduction of trace element during manufacturing process Rate of contamination: pc = 0.01 50 / 50 distribution of contaminated pots between Britain and Rome If contaminated, the trace element will be measured with 90 % probability

  5. Determining Origin of Roman Pot Information table: Plotting the resulting PMF:

  6. Identifying Suspect in Identity Parade We want to estimate the probability of correctly identifying a suspect as the perpetrator through the use of identity parades. Again two hypotheses: H0: Innocent, H1: Guilty Flat prior: 50 / 50 False-Negative rate: the probability of falsely accusing an innocent suspect when perpetrator is in the parade pfn = 0.48 False-Positive rate: the probability of falsely accusing an innocent suspect when perpetrator is not in the parade pfp = 0.133 Hidden Failure State: bias in the conduction of the parade Small probability pc that the parade is biased If the identity parade is biased, the suspect is identified as guilty 90 % of the time regardless of guilt.

  7. Identifying Suspect in Identity Parade Information table:

  8. Identifying Suspect in Identity Parade Even with just pc = 1%, the probability of guilt is never > 95%.

  9. Determining Origin of Roman Pot What if the success rate (1-pe) is larger than the success rate while contaminated (90 %)

  10. Determining Origin of Roman Pot What if the success rate (1-pe) is larger than the success rate while contaminated (90 %)

  11. Determining Origin of Roman Pot What if the success rate (1-pe) is larger than the success rate while contaminated (90 %)

  12. Conclusion Including hidden failure states highly changes the probabilistic nature of the problem Even a small probability of bias (hidden failure state) can drastically reduce the confidence of our test In real life, the ratio between (1- pe) and pfp high determines the significance of the hidden failure state

  13. Questions?

Related


More Related Content

giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#