Introduction to Probability: Key Concepts and Definitions
Introduction to Probability
Text Analytics
Giuseppe Attardi
IP notice: some slides from: Dan Jurafsky, Jim Martin, Sandiway Fong, Dan Klein
Outline
Probability
Basic probability
Conditional probability
Bayes’ Theorem
Independence
Introduction to Probability
Experiment (trial)
Experiment (trial)
Repeatable
procedure with well-defined possible outcomes
Sample Space (S)
Sample Space (S)
•
the set of
all possible outcomes
•
finite or infinite
Example
•
coin toss experiment
•
possible outcomes: S = {heads, tails}
Example
•
die toss experiment
•
possible outcomes: S = {1, 2, 3, 4, 5, 6}Slide from Sandiway Fong
Sample Space
Definition of sample space depends on what we are asking
Sample Space (S): the set of all possible outcomes
Example
•
die toss experiment for whether the number is even or odd
•
possible outcomes: {even, odd}•
not
{1, 2, 3, 4, 5, 6}More definitions
Events
Events
an
event
is any subset of outcomes from the
sample space
Example
Example
die toss experiment
let A represent the event such that the outcome of the die toss experiment is
divisible by 3
A = {3, 6}
A is a subset of the sample space S= {1, 2, 3, 4, 5, 6}
Example
Example
Draw a card from a deck
•
suppose sample space S = {heart, spade, club, diamond} (four suits
)
let A represent the event of drawing a heart
let B represent the event of drawing a red card
A = {heart}
B = {heart, diamond}Counting
Counting
the Sample Space
suppose operation o
i
can be performed in
n
i
ways, then
a sequence of
k
operations
o
1
o
2
...
o
k
can be performed in
n
1
n
2
...
n
k
ways
Example
die toss experiment, 6 possible outcomes
two dice are thrown at the same time
number of sample points in sample space = 6
6 = 36
Definition of Probability
The probability law assigns to an event
The probability law assigns to an event
E
E
a nonnegative number
a nonnegative number
Written
Written
P
P
(E)
(E)
Called the probability
Called the probability
E
E
That encodes our knowledge or belief about the overall
That encodes our knowledge or belief about the overall
likelihood
likelihood
of all
of all
the elements of
the elements of
A
A
Probability must satisfy
Probability must satisfy
the Kolmogorov
the Kolmogorov
axioms
axioms
Kolmogorov Probability Axioms
Non negativity
Non negativity
P
(
A
)
0
, for every event
A
Additivity
Additivity
If
A
and
B
are two
disjoint events
, then the probability of their
union (either one or the other occurs) satisfies:
P
(
A
B
) =
P
(
A
) +
P
(
B
)
Monotonicity
Monotonicity
P
(
A
)
P
(
B
) for any
A
B
Normalization
Normalization
The probability of the entire sample space
S
is equal to 1, i.e.
P
(
S
) = 1
A
A
B
B
=
=
An example
An experiment involving a single coin toss
An experiment involving a single coin toss
There are two possible outcomes,
There are two possible outcomes,
H
H
and
and
T
T
Sample space S is
Sample space S is
{H, T}{H, T}
If coin is fair, should assign equal probabilities to 2 outcomes
If coin is fair, should assign equal probabilities to 2 outcomes
Since they have to sum to 1
Since they have to sum to 1
P
P
({H,T}) = ({H,T}) = P
P
({H}) + ({H}) + P
P
({T}) = 1.0({T}) = 1.0P
P
({H}) = 0.5({H}) = 0.5P
P
({T}) = 0.5({T}) = 0.5Another example
Experiment involving 3 coin tosses
Outcome is a 3-long string of
H
or
T
S ={HHH, HHT, HTH, HTT, THH, THT, TTH, TTT}
Assume each outcome is equiprobable
“
Uniform distribution
”
What is probability of the event that exactly 2 heads occur?
E
= {HHT, HTH, THH}P
(
E
) =
P
({HHT}) + P
({HTH}) + P
({THH})= 1/8 + 1/8 + 1/8
= 3/8
Probability definitions
In summary:
Probability of drawing a spade from 52 well-shuffled playing cards:
Probabilities of two events
If two events
If two events
A
A
and
and
B
B
are
are
independent
independent
i.e.
i.e.
P(B)
P(B)
is the same whether
is the same whether
P(A)
P(A)
occurred or not
occurred or not
t
t
hen
hen
P
P
(
(
A
A
and
and
B
B
) =
) =
P
P
(
(
A
A
)
)
·
·
P
P
(
(
B
B
)
)
Flip a fair coin twice
Flip a fair coin twice
What is the probability that they are both heads?
What is the probability that they are both heads?
Draw a card from a deck, then
Draw a card from a deck, then
put it back
put it back
, draw a card from
, draw a card from
the deck again
the deck again
What is the probability that both drawn cards are hearts?
What is the probability that both drawn cards are hearts?
How about non-uniform probabilities?
A biased coin,
A biased coin,
twice as likely to come up tails as heads,
is tossed twice
What is the probability that
What is the probability that
at least one head
at least one head
occurs?
occurs?
Sample space = {hh, ht, th, tt} (h = heads, t = tails)Sample space = {hh, ht, th, tt} (h = heads, t = tails)
Sample points/probability for the event:
Sample points/probability for the event:
ht 1/3
x 2/3 =
2/9
hh 1/3 x 1/3=
1/9
th 2/3 x 1/3 =
2/9
tt 2/3 x 2/3 = 4/9
Answer:
Answer:
5/9 =
5/9 =
0.56 (
0.56 (
sum of weights in
sum of weights in
red
red
)
)
= 1 - 4/9 (prob. of complement)
= 1 - 4/9 (prob. of complement)
Computing Probabilities
Direct counts (when outcomes are equally probable)
Direct counts (when outcomes are equally probable)
Sum of union of disjoint events
Sum of union of disjoint events
P
(
A
or
B
) =
P
(
A
) +
P
(
B
)
Product of multiple independent events
Product of multiple independent events
P
(
A
and
B
) =
P
(
A
)
·
·
P
(
B
)
Indirect probability:
Indirect probability:
P
(
A
) = 1 –
P
(
S
–
A
)
Moving toward language
What
’
s the probability of drawing a 2 from a deck of 52 cards with four 2s?
What
’
s the probability of a random word (from a random dictionary page)
being a verb?
Probability and part of speech tags
What
’
s the probability of a random word (from a random dictionary
page) being a verb?
How to compute each of these
All words = just count all the words in the dictionary
# of ways to get a verb: number of words which are verbs!
If a dictionary has 50,000 entries, and 10,000 are verbs….
P
(V)
is
10000/50000 = 1/5 = 0.20
Conditional Probability
A way to reason about the outcome of an experiment based
A way to reason about the outcome of an experiment based
on
on
partial information
partial information
In a word guessing game the first letter for the word is a
“
t
”
. What is
the likelihood that the second letter is an
“
h
”
?
How likely is it that a person has a disease given that a medical test
was negative?
A spot shows up on a radar screen. How likely is it that it corresponds
to an aircraft? Or to a failure of the screen?
More precisely
Given an experiment, a corresponding sample space
S
, and a probability law
Suppose we know that the outcome is within some given event
B
We want to quantify the likelihood that the outcome also belongs to some
other event given event
A
We need a new probability law that gives us the conditional probability of
A
given
B
P
(
A
|
B
)
An intuition
A
is
“
it
’
s raining now
”
P
(
A
)
in Tuscany is .01
B
is
“
it was raining ten minutes ago
”
P
(
A
|
B
)
means
“
what is the probability of it raining now if it was raining
10 minutes ago
”
P
(
A
|
B
)
is probably way higher than
P
(A)
Perhaps
P
(
A
|
B
)
is 0.10
Intuition: The knowledge about
B
should change our estimate of the
probability of
A
.
Conditional probability
One of the following 30 items is chosen at random
What is
P
(X)
, the probability that it is an
X
?
What is
P
(X|red)
, the probability that it is an
X
given that it is red?
S
Conditional Probability
L
L
et
et
A
A
and
and
B
B
be events
be events
P
P
(
(
B
B
|
|
A
A
)
)
= the
= the
probability
probability
of event
of event
B
B
occurring given
occurring given
event
event
A
A
occurred
occurred
definition:
definition:
P
P
(
(
B
B
|
|
A
A
) =
) =
P
P
(
(
A
A
B
B
) /
) /
P
P
(
(
A
A
)
)
A
B
Conditional Probability
Note
:
P
(
A
,
B
) =
P
(
B|A
)
· P
(
A
)
also
:
P
(
A
,
B
) =
P
(
B
,
A
)
hence
:
P
(
B|A
)
· P
(
A
) =
P
(
A|B
)
· P
(
B
)
hence
: …
A
B
A
,
B
Bayes’ Theorem
P
(
B
)
: prior probability
P
(
B
|
A
)
: posterior probability
Independence
What is
What is
P
P
(
(
A
A
,
,
B
B
)
)
if
if
A
A
and
and
B
B
are independent?
are independent?
P
P
(
(
A
A
,
,
B
B
) =
) =
P
P
(
(
A
A
) ·
) ·
P
P
(
(
B
B
)
)
iff
iff
A
A
,
,
B
B
independent
independent
P
P
(
(
heads
heads
,
,
tails
tails
) =
) =
P
P
(
(
heads
heads
) ·
) ·
P
P
(
(
tails
tails
) = 0.5 · 0.5 = 0.25
) = 0.5 · 0.5 = 0.25
Note:
Note:
P
P
(
(
A|B
A|B
)
)
= P
= P
(
(
A
A
)
)
iff
iff
A
A
,
,
B
B
independent
independent
Also:
Also:
P
P
(
(
B|A
B|A
)
)
= P
= P
(
(
B
B
)
)
iff
iff
A
A
,
,
B
B
independent
independent
Independent Events
P
(
A
) =
P
(
A
|
B
)
25/100 = 15/60
P
(
A
B
) =
P
(
A
)
•
P
(
B
)
15/100 = 25/100
•
60/100
S = 100
A
=25
B
=60
15
Independence
Independence Revisited
These four statements are equivalent:
1.
A
and
B
are independent events
2.
P
(A and B) =
P
(A)
•
P
(B)
3.
P
(A|B) =
P
(A)
4.
P
(B|A) =
P
(B)
Monty Hall Problem (Pacchi in Affari Tuoi, Italian TV)
The contestant is shown
The contestant is shown
three doors
three doors
.
.
Two of the doors have
Two of the doors have
goats
goats
behind them and one has a
behind them and one has a
car
car
.
.
The contestant
The contestant
chooses a do
chooses a do
or.
or.
Before opening the chosen door, Monty Hall
Before opening the chosen door, Monty Hall
opens a door
opens a door
that has a
that has a
goat behind it.
goat behind it.
The contestant can then
The contestant can then
switch
switch
to the other unopened door, or
to the other unopened door, or
stay
stay
with the original choice.
with the original choice.
Which is best?
Which is best?
Solution
Consider the sample space, three doors: Car, A, B
There are three options:
1.
Contestant chooses Car. If she switches, she loses; if she stays, she wins
2.
Contestant chooses A with goat. If she switches, she wins; otherwise she
loses.
3.
Contestant chooses B with goat. If she switches, she wins; otherwise she
loses.
Switching gives 2/3 chances of winning
Other Explanation
P
(choice = Car) = 1/3
P
P
(choice = Car | Open) = 1/3
(choice = Car | Open) = 1/3
Opening door does not change
P
P
(other = Car | Open) = 2/3
(other = Car | Open) = 2/3
Summary
Probability
Conditional Probability
Independence
Additional Material
http://onlinestatbook.com/chapter5/probability.html
http://onlinestatbook.com/chapter5/probability.html
Explore the fundamental concepts of probability including basic probability, conditional probability, Bayes Theorem, independence, sample space, events, counting, and the definition of probability. Learn about the significance of sample space, event subsets, and how probability laws encode knowledge or beliefs. The material covers practical examples such as die toss experiments, drawing cards from a deck, and more, highlighting the essential principles of probability theory.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Introduction to Probability Text Analytics Giuseppe Attardi IP notice: some slides from: Dan Jurafsky, Jim Martin, Sandiway Fong, Dan Klein
Outline Probability Basic probability Conditional probability Bayes Theorem Independence
Introduction to Probability Experiment (trial) Repeatable procedure with well-defined possible outcomes Sample Space (S) the set of all possible outcomes finite or infinite Example coin toss experiment possible outcomes: S = {heads, tails} Example die toss experiment possible outcomes: S = {1, 2, 3, 4, 5, 6} Slide from Sandiway Fong
Sample Space Definition of sample space depends on what we are asking Sample Space (S): the set of all possible outcomes Example die toss experiment for whether the number is even or odd possible outcomes: {even, odd} not {1, 2, 3, 4, 5, 6}
More definitions Events an event is any subset of outcomes from the sample space Example die toss experiment let A represent the event such that the outcome of the die toss experiment is divisible by 3 A = {3, 6} A is a subset of the sample space S= {1, 2, 3, 4, 5, 6} Example Draw a card from a deck suppose sample space S = {heart, spade, club, diamond} (four suits) let A represent the event of drawing a heart let B represent the event of drawing a red card A = {heart} B = {heart, diamond}
Counting Counting the Sample Space suppose operation oi can be performed in ni ways, then a sequence of k operations o1o2...ok can be performed in n1 n2 ... nk ways Example die toss experiment, 6 possible outcomes two dice are thrown at the same time number of sample points in sample space = 6 6 = 36
Definition of Probability The probability law assigns to an event E a nonnegative number Written P(E) Called the probability E That encodes our knowledge or belief about the overall likelihood of all the elements of A Probability must satisfy the Kolmogorov axioms
Kolmogorov Probability Axioms Non negativity P(A) 0, for every event A Additivity If A and B are two disjoint events, then the probability of their union (either one or the other occurs) satisfies: P(A B) = P(A) + P(B) Monotonicity P(A) P(B) for any A B Normalization The probability of the entire sample space S is equal to 1, i.e. P(S) = 1 A B =
An example An experiment involving a single coin toss There are two possible outcomes, H and T Sample space S is {H, T} If coin is fair, should assign equal probabilities to 2 outcomes Since they have to sum to 1 P({H,T}) = P({H}) + P({T}) = 1.0 P({H}) = 0.5 P({T}) = 0.5
Another example Experiment involving 3 coin tosses Outcome is a 3-long string of H or T S ={HHH, HHT, HTH, HTT, THH, THT, TTH, TTT} Assume each outcome is equiprobable Uniform distribution What is probability of the event that exactly 2 heads occur? E = {HHT, HTH, THH} P(E) = P({HHT}) + P({HTH}) + P({THH}) = 1/8 + 1/8 + 1/8 = 3/8
Probability definitions In summary: number of outcomes correspond ing to event E = ( ) P E total number of outcomes Probability of drawing a spade from 52 well-shuffled playing cards: 13 1 = = . 0 25 52 4
Probabilities of two events If two events A and B are independent i.e. P(B) is the same whether P(A) occurred or not then P(A and B) = P(A) P(B) Flip a fair coin twice What is the probability that they are both heads? Draw a card from a deck, then put it back, draw a card from the deck again What is the probability that both drawn cards are hearts?
How about non-uniform probabilities? A biased coin, twice as likely to come up tails as heads, is tossed twice What is the probability that at least one head occurs? Sample space = {hh, ht, th, tt} (h = heads, t = tails) Sample points/probability for the event: ht 1/3 x 2/3 = 2/9 th 2/3 x 1/3 = 2/9 Answer: 5/9 = 0.56 (sum of weights in red) = 1 - 4/9 (prob. of complement) hh 1/3 x 1/3= 1/9 tt 2/3 x 2/3 = 4/9
Computing Probabilities Direct counts (when outcomes are equally probable) A A P # # = ( ) S Sum of union of disjoint events P(A or B) = P(A) + P(B) Product of multiple independent events P(A and B) = P(A) P(B) Indirect probability: P(A) = 1 P(S A)
Moving toward language What s the probability of drawing a 2 from a deck of 52 cards with four 2s? P(drawingatwo) =4 52=1 13=.077 What s the probability of a random word (from a random dictionary page) being a verb? P(drawingaverb) =#ofwaystogetaverb allwords
Probability and part of speech tags What s the probability of a random word (from a random dictionary page) being a verb? P(drawingaverb) =#ofwaystogetaverb allwords How to compute each of these All words = just count all the words in the dictionary # of ways to get a verb: number of words which are verbs! If a dictionary has 50,000 entries, and 10,000 are verbs . P(V) is 10000/50000 = 1/5 = 0.20
Conditional Probability A way to reason about the outcome of an experiment based on partial information In a word guessing game the first letter for the word is a t . What is the likelihood that the second letter is an h ? How likely is it that a person has a disease given that a medical test was negative? A spot shows up on a radar screen. How likely is it that it corresponds to an aircraft? Or to a failure of the screen?
More precisely Given an experiment, a corresponding sample space S, and a probability law Suppose we know that the outcome is within some given event B We want to quantify the likelihood that the outcome also belongs to some other event given event A We need a new probability law that gives us the conditional probability of A given B P(A|B)
An intuition A is it s raining now P(A) in Tuscany is .01 B is it was raining ten minutes ago P(A|B) means what is the probability of it raining now if it was raining 10 minutes ago P(A|B) is probably way higher than P(A) Perhaps P(A|B) is 0.10 Intuition: The knowledge about B should change our estimate of the probability of A.
Conditional probability One of the following 30 items is chosen at random What is P(X), the probability that it is an X? What is P(X|red), the probability that it is an X given that it is red? O O O O O X X O O X X X O O X X O X O X O X O X X O O X O O
Conditional Probability Let A and B be events P(B|A) = the probability of event Boccurring given event Aoccurred definition:P(B|A) = P(A B) / P(A) S A B
Conditional Probability A ? ? ? =?(? ?) =?(?,?) ?(?) B A,B ?(?) Note: P(A,B) = P(B|A) P(A) also: P(A,B) = P(B,A) hence: P(B|A) P(A) = P(A|B) P(B) hence:
Bayes Theorem ? ? ? =? ? ? ?(?) ?(?) P(B): prior probability P(B|A): posterior probability
Independence What is P(A, B) if A and B are independent? P(A, B) = P(A) P(B)iffA, B independent P(heads, tails) = P(heads) P(tails) = 0.5 0.5 = 0.25 Note: P(A|B) = P(A)iff A, B independent Also: P(B|A) = P(B)iff A, B independent
Independent Events S = 100 A=25 15 B=60 Independence P(A) = P(A|B) 25/100 = 15/60 P(A B) = P(A) P(B) 15/100 = 25/100 60/100 Independence does not mean disjoint
Independence Revisited These four statements are equivalent: A and B are independent events P(A and B) = P(A) P(B) P(A|B) = P(A) P(B|A) = P(B) 1. 2. 3. 4.
Monty Hall Problem (Pacchi in Affari Tuoi, Italian TV) The contestant is shown three doors. Two of the doors have goats behind them and one has a car. The contestant chooses a door. Before opening the chosen door, Monty Hall opens a door that has a goat behind it. The contestant can then switch to the other unopened door, or stay with the original choice. Which is best?
Solution Consider the sample space, three doors: Car, A, B There are three options: 1.Contestant chooses Car. If she switches, she loses; if she stays, she wins 2.Contestant chooses A with goat. If she switches, she wins; otherwise she loses. 3.Contestant chooses B with goat. If she switches, she wins; otherwise she loses. Switching gives 2/3 chances of winning
Other Explanation P(choice = Car | Open) = 1/3 Opening door does not change P(other = Car | Open) = 2/3 P(choice = Car) = 1/3
Summary Probability Conditional Probability Independence
Additional Material http://onlinestatbook.com/chapter5/probability.html
