Checking Mathematical Proofs

How are proofs read, checked, and

understood?

Keith Weber

Rutgers University

Plan for today

•

How do students and mathematicians check proofs for

correctness?

•

How and why do mathematicians read proofs?

•

How can we measure students’ proof understanding?

•

How can we help students understand proofs better?

Plan for today

Students’ understanding of a proof depends less on how proofs are

presented and more on the activity that they engage in.

For any positive integer 

n

, if 

n

2

 is divisible by 3, then 

n

 is divisible

by 3.

Proof

. Assume that 

n

2

is a positive integer that is divisible by 3.

That is, 

n

2

 = (3

n

+1)

2

 = 9

n

2

 + 6

n

 + 1 = 3

n

(

n

+2) + 1.

Therefore, 

n

2

 is divisible by 3.

Assume 

n

2

 is even and is a multiple of 3.

Then 

n

2

 = 

(3

n

)

2

 = 9

n

2

 = (3

n

)(3

n

).

Therefore, 

n

2

 is a multiple of 3.

If we factor 

n

2

=9

n

2

, we get (3

n

)(3

n

) which means that 

n

 is divisible

by 3.

For any positive integer 

n

, if 

n

2

 is divisible by 3, then 

n

 is divisible

by 3.

Proof

. Assume that 

n

2

is a positive integer that is divisible by 3.

That is, 

n

2

 = (3

n

+1)

2

 = 9

n

2

 + 6

n

 + 1 = 3

n

(

n

+2) + 1.

Therefore, 

n

2

 is divisible by 3.

Assume 

n

2

 is even and is a multiple of 3.

Then 

n

2

 = 

(3

n

)

2

 = 9

n

2

 = (3

n

)(3

n

).

Therefore, 

n

2

 is a multiple of 3.

If we factor 

n

2

=9

n

2

, we get (3

n

)(3

n

) which means that 

n

 is divisible

by 3.

Would undergraduate math students recognize this as incorrect?

If a student did recognize this as valid, what might they be thinking?

Claim

. For any positive integer 

n

, if 

n

2

 is divisible by 3, then 

n

 is

divisible by 3.

Proof

. Let 

n

 be an integer such that 

n

2

 is divisible by 3.

Let 

k

 be an integer such that such that 

n

2

 = 3

k

.

Since 

n

2

 = 

3

k

, 

n

 x 

n

 = 3

k

.

Thus, 3|

n

.

Therefore if 

n

2

is a multiple of 3, then 

n

 is a multiple of 3.

Reading proofs and checking proofs

for correctness

Weber, K. (2008). How mathematicians determine if an argument is

a valid proof. 

Journal for Research in Mathematics Education

, 39,

431-459.

Weber, K. (2010). Mathematics majors’ perceptions of conviction,

validity, and proof. 

Mathematical Thinking and Learning

, 12, 306-

336.

Reading proofs and checking proofs

for correctness

•

We present students with proofs, in part, so they can have a

good reason to believe that theorems are true. But if they are to

obtain legitimate conviction, they need to distinguish good

proofs from bad (Selden & Selden, 2003).

•

There have been a large number of studies in which

undergraduate mathematics students have asked students to

check proofs for correctness.

•

In each study, the students have performed very poorly, and

cannot distinguish correct proofs from invalid arguments (Inglis

& Alcock, 2012; Ko & Knuth, 2013; Selden & Selden, 2003;

Weber, 2010).

Reading proofs and checking proofs

for correctness

For any positive integer 

n

, if 

n

2

 is divisible by 3, then 

n

 is divisible

by 3.

Proof

. Assume that 

n

2

is a positive integer that is divisible by 3.

That is, 

n

2

 = (3

n

+1)

2

 = 9

n

2

 + 6

n

 + 1 = 3

n

(

n

+2) + 1.

Therefore, 

n

2

 is divisible by 3.

Assume 

n

2

 is even and is a multiple of 3.

Then 

n

2

 = 

(3

n

)

2

 = 9

n

2

 = (3

n

)(3

n

).

Therefore, 

n

2

 is a multiple of 3.

If we factor 

n

2

=9

n

2

, we get (3

n

)(3

n

) which means that 

n

 is divisible

by 3.

Across two studies, 

13 out of 26 

students found this to be invalid.

Reading proofs and checking proofs

for correctness

•

The literature often says that undergraduates are at “chance

level” when checking proofs for correctness. But that’s not quite

right.

•

Students will usually confirm correct proofs are valid. But they

will also frequently evaluate incorrect proofs as valid.

•

On a positive note, if interviewers talk with students about the

proof and ask them to explain aspects of the proof, students’

performance can improve dramatically (Selden & Selden, 2003).

What do mathematicians do when they

check proofs for correctness?

•

I conducted task-based interviews with 8 mathematicians.

•

They were handed purported proofs, one-by-one, and asked to

”think aloud”  while determining if the proofs were valid.

•

Four were student-generated proofs from the Selden and

Selden (2003) study on student proof reading, Four were more

advanced proofs in number theory taken from expository

journals such as 

American Mathematical Monthly

. (One was a

foil with a mistake that I generated).

•

Mathematicians scored eight proofs or stopped when 45

minutes elapsed (whichever came first) and then asked general

questions about how they read proofs.

An interesting case

Claim

. For any positive integer 

n

, if 

n

2

 is divisible by 3, then 

n

 is

divisible by 3.

Proof

. Let 

n

 be an integer such that 

n

2

 = 3

x

, where 

x

 is an integer.

Then 3|

n

2

.

Since 

n

2

 = 

3

x

, 

n

 x 

n

 = 3

x

.

Thus, 3|

n

.

Therefore if 

n

2

is a multiple of 3, then 

n

 is a multiple of 3.

An interesting case

Claim

. For any positive integer 

n

, if 

n

2

 is divisible by 3, then 

n

 is

divisible by 3.

Proof

. Let 

n

 be an integer such that 

n

2

 = 3

x

, where 

x

 is an integer.

Then 3|

n

2

.

Since 

n

2

 = 

3

x

, 

n

 x 

n

 = 3

x

.

Thus, 3|

n

.

Therefore if 

n

2

is a multiple of 3, then 

n

 is a multiple of 3.

5 mathematicians said invalid, 2 mathematicians said valid, 1 said

he couldn’t decide.

Inglis and Alcock (2012) found 7 mathematicians said invalid, 5

valid.

What did mathematicians do?

•

Mathematicians would first check to see if the structure of the

proof was valid (i.e., the assumptions and conclusion of the

proof were sensible), and then proceed to check if each step

followed from previous assertions.

“I will first try to understand the structure of the proof, to get a feel

for the argument that’s being used. After that, if that’s reasonable,

I’ll check the individual steps to make sure each of them are valid”.

What did mathematicians do?

Line-by-line check

There were 77 instances in which a participant encountered a step

in the proof that they were initially unsure of.

What did mathematicians do?

Line-by-line check

There were 77 instances in which a participant encountered a step

in the proof that they were initially unsure of.

•

In 15 cases, they constructed a sub-proof.

•

In 33 cases, they constructed an informal argument.

•

In 19 cases, they resolved their doubt by checking the claim with

examples.

•

10 other cases were idiosyncratic and could not be classified

(i.e., coded as “other”).

What did mathematicians do?

Line-by-line check

An instance of a sub-proof:

Previous assertion:

n

 is congruent to 3 (mod 4).

Next assertion: 

Note that 

n

 is not a perfect square.

What did mathematicians do?

Line-by-line check

An instance of a sub-proof:

Previous assertion:

n

 is congruent to 3 (mod 4).

Next assertion: 

Note that 

n

 is not a perfect square.

“

So if you take an odd number and square it, 2k + 1 , and I assume

that when you square it out . . .

[the participant writes: (2k + 1)

2 

= 4k

2

 + 4k + 1 = 4(k

2

 + k) + 1]

T

hat would be, yeah, that would be 1 mod 4. OK, note that n is not

a perfect square. OK, I think I'm OK with that”

What did mathematicians do?

Line-by-line check

An instance of an informal argument:

Previous assertion:

ab

 is congruent to 3 (mod 4).

Next assertion:

 Either 

a

 is congruent to 3 (mod 4) and 

b

 is

congruent to 1 (mod 4), or 

a

 is congruent to 1 (mod 4) and 

b

 is

congruent to 3 (mod 4).

“

I

g

u

e

s

s

t

h

a

t

’

s

a

f

a

c

t

.

I

m

e

a

n

,

h

o

w

d

o

y

o

u

t

a

k

e

t

w

o

n

u

m

b

e

r

s

i

n

Z

4

a

n

d

m

u

l

t

i

p

l

y

t

h

e

m

t

o

g

e

t

h

e

r

t

o

g

e

t

t

h

r

e

e

?

”

What did mathematicians do?

Line-by-line check

An instance of a sub-proof:

Previous assertion:

n

 is congruent to 3 (mod 4).

Next assertion: 

Note that 

n

 is not a perfect square.

I'm using examples to see what, where the proof is coming from.

So 5

2

 is 25 and that's 1 mod 4, 36 is 0 mod 4, 49 is 1 mod 4, 64 is 0

mod 4. I'm thinking that, ah! So it is ... 24 times 24, that's 0 mod 4.

So a perfect square has to be 1 mod 4, doesn't it? n

2

 equals 1 mod

4 or 0 mod 4. Alright.

What did mathematicians do?

Line-by-line check

Interviewer: I noticed that at times you used examples to help you

validate the proofs.

Mathematician H: Yes. I think with the proofs with number theory,

they [examples] are a little easier to use. You can show it's true for

some and then use induction arguments to show that it's true for all

of them. Topology [Mathematician H's area of research] you don't

quite have that.

What did mathematicians do?

Line-by-line check

An example of an informal argument:

Previous assertion:

n

 is a natural number.

Next assertion: 

There exists an odd integer 

m

 and a non-zero

integer 

l 

such that 

n

 = 2

l

m

.

Math. E: Hmm . . . can we express every integer in that way? Well,

1 is / = 0, 

m

 = 1 . 2, 4, and 8 are powers, but can we express every

integer in that way? What about 3? Um, let 

m

 = 3 and / = 0. And 5,

let 

m

 = 5 and / = 0. What about 6? 6 is 2 cubed times 3 [sic], OK, I

guess this sounds reasonable.

Differences between mathematicians

and students

Mathematicians focus on structure:

•

There was one statement that proved “If 

n

2

 is divisible by 3, then

n

 is divisible by 3” by proving the converse.

–

All 20 mathematicians in Weber (2010) and Inglis and Alcock’s

(2012) study rejected the argument.

–

24 of 54 (44%) of the undergraduates in Weber (2010), Inglis and

Alcock (2012), and Selden and Selden’s (2003) study rejected the

argument.

Differences between mathematicians

and students

Mathematicians search for warrants:

•

When a mathematician sees a statement in line 

n, 

they search

for a reason (or ‘warrant’) for why that statement follows from

previous assertions in the proof.

•

Inglis and Alcock (2012) did an eye-tracking study.

–

They found that for statements requiring a warrant, mathematicians

would follow the pattern of fixating on line 

n

, then line 

n

-1, and then

line 

n

 again, more often than when a warrant was not required.

–

They also found that most undergraduate mathematics students

rarely engaged in this behavior.

–

When students were requested to explain how new statements in a

proof followed from previous assertions, their ability to evaluate

proofs improved (Alcock & Weber, 2005; Selden & Selden, 2003).

Differences between mathematicians

and students

Students tended to focus on formula and calculations:

Students tended to focus on formula and calculations:

•

In Inglis and Alcock’s (2012) study, undergraduate mathematics

students and mathematicians spent roughly an equal amount of

time dwelling on formulas.

•

But mathematicians spent 50% longer when reading statements

that did not have formulas in them.

Students read proofs quickly

•

Students would often spend two minutes or less reading the

proofs in Weber’s (2010) study, even on proofs where

mathematicians would spend five minutes or more.

For any positive integer 

n

, if 

n

2

 is divisible by 3, then 

n

 is divisible

by 3.

Proof

. Assume that 

n

2

is a positive integer that is divisible by 3.

That is, 

n

2

 = (3

n

+1)

2

 = 9

n

2

 + 6

n

 + 1 = 3

n

(

n

+2) + 1.

Therefore, 

n

2

 is divisible by 3.

Assume 

n

2

 is even and is a multiple of 3.

Then 

n

2

 = 

(3

n

)

2

 = 9

n

2

 = (3

n

)(3

n

).

Therefore, 

n

2

 is a multiple of 3.

If we factor 

n

2

=9

n

2

, we get (3

n

)(3

n

) which means that 

n

 is divisible

by 3.

Questions

•

Would writing up arguments, or checking other arguments, with

an ITP help students engage in mathematicians’ proving

processes?

Accounting for the previous data in

terms of beliefs about proving

Weber, K. (2008). How mathematicians determine if an argument is a valid

proof. 

Journal for Research in Mathematics Education

, 39, 431-459.

Weber, K. & Mejia-Ramos, J.P.  (2011). How and why mathematicians

read proofs: An exploratory study. 

Educational Studies in Mathematics

, 76,

329-344.

Mejia-Ramos, J.P. & Weber, K. (2014). How and why mathematicians read

proofs: Further evidence from a survey study. 

Educational Studies in

Mathematics

, 85, 161-173.

Weber, K. & Mejia-Ramos, J.P.  (2014). Mathematics majors’ beliefs about

proof reading. 

International Journal of Mathematics Education in Science

and Technology, 

45, 89-103.

Accounting for the previous data in

terms of beliefs about proving

Standard view:

•

Mathematicians read proofs to gain certainty (or near certainty)

that theorems are true.

•

Empirical arguments (“proof by example”) cannot provide

certainty (or near certainty) that mathematical statements are

true.

–

That’s why we need proof!

•

Mathematicians check steps in proofs with empirical arguments

alone.

Why do mathematicians check proofs

with examples?

When I read a proof in a respected journal, it is not uncommon for

me to see how the steps in a proof apply to a specific example.

This increases my confidence that the proof is correct.

8

3

%

A

G

R

E

E

8

%

D

I

S

A

G

R

E

E

When I read a proof in a respected journal and I am 

not

immediately sure that a statement in the proof is true, it is not

uncommon for me to gain a

sufficiently high level of confidence in

the statement by checking it with one or more carefully chosen

examples to assume the claim is correct and continue reading the

proof.

5

6

%

A

G

R

E

E

3

0

%

D

I

S

A

G

R

E

E

Are mathematicians checking for

correctness?

“I tend to hope that the proof will give me some insight into the

problem it was solving. Checking for validity is subordinate really.

I’m really looking more to gain some insight”.

“One other bias, I’ll just throw this out there. To be honest, when I

read papers, I don’t read the proofs. Maybe that’s bad and maybe

that’s not, if I’m convinced that the result is true, I don’t necessarily

need to read it. I can just believe it.

Checking just for correctness, I guess what I said is it’s not

important to me. If it offers me some insight, that’s often what I’m

looking for if I take the time to read the proof”

Are mathematicians checking for

correctness?

[

Understanding a proof] “

means to understand how each step

followed from the

previous one. 

I don’t always do this, even when I

referee. 

I simply don’t always have

time to look over all the details

of every proof in every paper that I read. When I read

the theorem,

I think, is this theorem likely to be true and what does the author

need to

show to prove it’s true. And then I find the big idea of the

proof and see if it will

work.

 If the big idea works, if the key idea

makes sense, probably the rest of the

details of the proof are going

to work too

"

Are mathematicians checking for

correctness?

When I read a proof in a respected journal, it is not uncommon that

I do not check the proof for correctness. Rather, I read the proof to

gain some other type of insight.

7

4

%

A

G

R

E

E

1

4

%

D

I

S

A

G

R

E

E

When I read a proof in a respected journal, 

if I understand the main

idea of the proof and think it is correct, it is not uncommon that I do

not check

that every line of the proof is correct, but trust that the

logical details are correct.

7

7

%

A

G

R

E

E

1

4

%

D

I

S

A

G

R

E

E

Are mathematicians checking for

correctness?

When I referee a manuscript, 

if I understand the main idea of the

proof and think it is correct, it is not uncommon that I do not check

that every line of the proof is correct, but trust that the logical

details are correct.

4

3

%

A

G

R

E

E

2

8

%

D

I

S

A

G

R

E

E

When I referee a manuscript and I am 

not immediately sure that a

statement in the proof is true, it is not uncommon for me to gain a

sufficiently high level of confidence in the statement by checking it

with one or more carefully chosen

examples to assume the claim is

correct and continue reading the proof.

3

5

%

A

G

R

E

E

5

2

%

D

I

S

A

G

R

E

E

Do mathematicians trust published

results?

I: One of the things you didn’t say was you would read it to be sure

the theorem was

true. Is that because it was too obvious to say or

is that not why you would read the

proof?

M6: Well, I mean, it depends. If it’s something in the published

literature then… I’ve

certainly encountered mistakes in the

published literature, but it’s not high in my

mind. So in other words I

am open to the possibility that there’s a mistake in the

proof, but I…

it’s not… [pause]

I: But you act on the assumption that it’s probably correct?

M6: Yeah, that’s right. That’s right.

M8: Now notice what I did not say. I do not try and determine if a

proof is correct. If

it’s in a journal, I assume it is. I’m much more

interested in the ideas of the proof.

Do mathematicians trust published

results?

It is not uncommon for me to believe that a proof is correct because

it is published in an academic journal.

7

2

%

A

G

R

E

E

1

2

%

D

I

S

A

G

R

E

E

When I read a proof in a respected journal, 

it is not uncommon for

me to be very confident that the proof is correct because it was

written by an

authoritative source that I trust.

8

3

%

A

G

R

E

E

7

%

D

I

S

A

G

R

E

E

When I referee a manuscript, 

it is not uncommon for me to be very

confident that the proof is correct because it was written by an

authoritative source that I trust.

3

9

%

A

G

R

E

E

4

1

%

D

I

S

A

G

R

E

E

How do students and mathematicians

beliefs differ?

A. In a good proof, every step is spelled out for the reader. The

reader should not be left

wondering where the new step in the proof

comes from.

B. When reading a good proof, I expect I will have to do some of

the work to verify the steps

in the proof myself.

How do students and mathematicians

beliefs differ?

A. In a good proof, every step is spelled out for the reader. The

reader should not be left

wondering where the new step in the proof

comes from.

B. When reading a good proof, I expect I will have to do some of

the work to verify the steps

in the proof myself.

7

5

%

o

f

u

n

i

v

e

r

s

i

t

y

m

a

t

h

e

m

a

t

i

c

s

s

t

u

d

e

n

t

s

(

N

=

1

7

5

)

p

r

e

f

e

r

r

e

d

A

.

2

7

%

o

f

m

a

t

h

e

m

a

t

i

c

i

a

n

s

(

N

=

8

3

)

p

r

e

f

e

r

r

e

d

A

.

How do students and mathematicians

beliefs differ?

If I can see how each step in a proof follows logically from previous

ones, then I understand the proof completely.

How do students and mathematicians

beliefs differ?

If I can see how each step in a proof follows logically from previous

ones, then I understand the proof completely.

7

5

%

o

f

u

n

i

v

e

r

s

i

t

y

m

a

t

h

e

m

a

t

i

c

s

s

t

u

d

e

n

t

s

a

g

r

e

e

d

w

i

t

h

t

h

i

s

s

t

a

t

e

m

e

n

t

2

3

%

o

f

m

a

t

h

e

m

a

t

i

c

i

a

n

s

a

g

r

e

e

d

How do students and mathematicians

beliefs differ?

•

Most students believed they would never spend more than 15

minutes studying a proof outside of class.

•

Most mathematicians believed that there were some proofs

where students should spend at least 30 minutes studying

outside of class.

Questions?

•

How will ITPs affect student beliefs about their responsibility

when they check proofs?

•

Would they help mathematicians (and students!) by allowing

them to focus on the “big ideas”?

What does it mean to understand a

proof? And how can we help students

understand?

Mejia-Ramos, J.P., Fuller, E., Weber, K., Samkoff

+

, A. & Rhoads

+

, K.

(2012). A model for proof comprehension in undergraduate mathematics.

Educational Studies in Mathematics

, 79, 3-18.

Mejia-Ramos, J.P., Lew, K., de la Torre, J., & Weber, K. (2017).

Developing and validating proof comprehension tests in undergraduate

mathematics. 

Research in Mathematics Education

, 19, 130-146.

•

What would it mean for a student to understand this proof?

•

How would you know if a student did understand the proof? (Or

to what extent the student understood the proof)?

Proof comprehension tests:

Motivation

•

Both mathematicians and mathematics educators want proofs to

be used as a tool for understanding.

•

Yet a literature review in 2009 found only 3 published articles on

students’ or mathematicians’ understanding of proof 

(Mejia-Ramos

& Inglis, 2009).

•

This is bad.

–

Mathematicians do not know how well students understand what

lecturers say or what they read.

–

Students are not given guidance on their own understanding.

–

Mathematics educators do not know if their suggestions actually

work.

Assessing proof comprehension:

A model

“Local” questions are questions that can be answered by looking a

small number of statements in the proof.

(1)

Meaning of terms and statements:

“Which of the following numbers are prime?”

(2) Justification of claims:

“In the proof, why does 

q

 divide (

n

 -  

p

1

p

2

…

p

n

)?”

(3) Logical status and proof framework:

“Why do we begin by assuming that there are finitely many

primes?”

Assessing proof comprehension:

A model

“Global” questions are questions that can be answered by looking

at the proof as a whole.

(1)

Summary:

“Provide a high-level summary of the main ideas of the proof”

(2) Modularizing:

“What is the logical connection between lines 3-5 and lines 6-9?”

(3) Applying the ideas of the proof in another context:

“Assuming that there are at least two primes, would the proof work

if we defined 

n

 = 

p

1

p

2

…

p

n

 – 1?”

(4) Applying the ideas of the proof to a particular example:

“

Using the ideas of the proof, 

if 2, 3, 5, and 7 were the only primes,

why would 

n

 not be divisible by 3?”

Designing proof comprehension tests:

Motivation

•

Publishing our proof comprehension model had its desired

effect. Many researchers used the model to develop and assess

methods for improving students’ proof comprehension.

•

However, there are drawbacks:

–

We have no evidence that making tests in the way we prescribed

led to valid tests (or that the researchers applied our model in a

valid way).

–

It would be nice to compare between methods and studies, which

isn’t possible if everyone is using their own in-house tests.

–

We want this to be practically useful , but asking mathematicians to

generate and then grade open-ended items is unrealistically

demanding.

What (do we think) makes a good test?

•

Face validity

•

Construct validity

•

Reliability

•

Practicality

Our process of test generation

1.

Generate at least four open-ended items for each dimension of

our model.

2.

Conduct 12 interviews with students to get realistic foils.

3.

Have our Advisory Board and three mathematicians review the

items.

4.

Ask 12 students to think aloud while answering our multiple

choice items.

5.

Give the tests to 200 students. Reduce test to 12 items by

removing items with low biserial correlations.

6.

Conduct a final validation interviews with 12 students on the

shorter tests.

Our process of test generation

1.

Generate at least four open-ended items for each dimension of

our model.

2.

Conduct 12 interviews with students to get realistic foils.

3.

Have our Advisory Board and three mathematicians review the

items.

4.

Ask 12 students to think aloud while answering our multiple

choice items.

5.

Give the tests to 200 students. Reduce test to 12 items by

removing items with low biserial correlations.

6.

Conduct a final validation interviews with 12 students on the

shorter tests.

S

t

e

p

s

1

a

n

d

2

c

r

e

a

t

e

d

a

b

a

n

k

o

f

p

l

a

u

s

i

b

l

e

i

t

e

m

s

.

T

h

e

r

e

m

a

i

n

i

n

g

s

t

e

p

s

w

e

r

e

u

s

e

d

t

o

w

e

e

d

o

u

t

b

a

d

i

t

e

m

s

a

n

d

i

m

p

r

o

v

e

p

r

o

b

l

e

m

a

t

i

c

o

n

e

s

.

Our process of test generation

1.

Generate at least four open-ended items for each dimension of

our model.

2.

Conduct 12 interviews with students to get realistic foils.

3.

Have our Advisory Board and three mathematicians review the

items.

4.

Ask 12 students to think aloud while answering our multiple

choice items.

5.

Give the tests to 200 students. Reduce test to 12 items by

removing items with low biserial correlations.

6.

Conduct a final validation interviews with 12 students on the

shorter tests.

S

t

e

p

3

i

m

p

r

o

v

e

d

t

h

e

f

a

c

e

v

a

l

i

d

i

t

y

o

f

o

u

r

i

t

e

m

,

m

a

k

i

n

g

s

u

r

e

t

h

e

t

e

s

t

p

a

s

s

e

d

m

u

s

t

e

r

w

i

t

h

e

x

p

e

r

t

s

,

a

n

d

t

h

a

t

m

a

t

h

e

m

a

t

i

c

i

a

n

s

c

a

n

a

c

e

o

u

r

t

e

s

t

.

Our process of test generation

1.

Generate at least four open-ended items for each dimension of

our model.

2.

Conduct 12 interviews with students to get realistic foils.

3.

Have our Advisory Board and three mathematicians review the

items.

4.

Ask 12 students to think aloud while answering our multiple

choice items.

5.

Give the tests to 200 students. Reduce test to 12 items by

removing items with low biserial correlations.

6.

Conduct a final validation interviews with 12 students on the

shorter tests.

S

t

e

p

s

4

a

n

d

6

w

e

r

e

u

s

e

d

t

o

i

m

p

r

o

v

e

c

o

n

s

t

r

u

c

t

v

a

l

i

d

i

t

y

.

I

n

s

h

o

r

t

,

w

e

c

r

e

a

t

e

d

i

t

e

m

s

t

h

a

t

s

t

u

d

e

n

t

s

a

n

s

w

e

r

e

d

c

o

r

r

e

c

t

l

y

o

r

i

n

c

o

r

r

e

c

t

l

y

f

o

r

t

h

e

r

i

g

h

t

r

e

a

s

o

n

s

.

Our process of test generation

1.

Generate at least four open-ended items for each dimension of

our model.

2.

Conduct 12 interviews with students to get realistic foils.

3.

Have our Advisory Board and three mathematicians review the

items.

4.

Ask 12 students to think aloud while answering our multiple

choice items.

5.

Give the tests to 200 students. Reduce test to 12 items by

removing items with low biserial correlations.

6.

Conduct a final validation interviews with 12 students on the

shorter tests.

S

t

e

p

s

5

w

a

s

u

s

e

d

t

o

i

n

c

r

e

a

s

e

r

e

l

i

a

b

i

l

i

t

y

a

n

d

p

r

a

c

t

i

c

a

l

i

t

y

(

i

.

e

.

,

a

s

h

o

r

t

e

r

t

e

s

t

w

i

t

h

h

i

g

h

i

n

t

e

r

n

a

l

r

e

l

i

a

b

i

l

i

t

y

)

.

What can our tests teach us about

students’ proof comprehension?

Is proof reading a general skill in a transition-to-proof course? Or is

it highly dependent on the proof that was read?

What can our tests teach us about

students’ proof comprehension?

Is proof reading a general skill in a transition-to-proof course? Or is

it highly dependent on the proof that was read?

What can our tests teach us about

students’ proof comprehension?

Does proof reading ability matter? Does proof comprehension

ability influence factors we care about, like course grade?

What can our tests teach us about

students’ proof comprehension?

Does proof reading ability matter? Does proof comprehension

ability influence factors we care about, like course grade?

We conducted an initial regression with Proof Grade as the

dependent variable and Math SAT, Verbal SAT, and Calc 2 Grade

as the independent variables. This accounted for 17.8% of the

variance in Proof Grade.

The model obtained by adding Primes and Fibonacci to the

dependent variables accounted for 26.0% of the variance.

What can our tests teach us about

students’ proof comprehension?

What is the dimensionality of our test? Can our test distinguish

local and global understandings of students’ proof comprehension?

What can our tests teach us about

students’ proof comprehension?

What is the dimensionality of our test? Can our test distinguish

local and global understandings of students’ proof comprehension?

No. Factor analysis showed the test was unidimensional.

What can our tests teach us about

students’ proof comprehension?

Caveat: There is a distinction between theoretical dimensionality of

a construct and the empirical dimensionality of an assessment.

However:

•

Others who have designed tests using our model and tested

dimensionality found a unidimensional construct 

(Hodds et al.,

2014)

•

Using an eye tracking study, Panse et al. (2018) found no

difference in reading behavior when students and

mathematicians were asked to check a proof for correctness or

read for understanding.

–

This does not imply that mathematicians cannot understand a proof

in different ways. But it does suggest that checking a proof for

correctness or trying to glean understanding involve the same

cognitive activities.

Summary

•

Using mathematicians’ comments, we defined proof

comprehension as consisting of:

–

Local understandings: (What do statements mean? Why are steps

valid?)

–

Global understandings: (What is the big idea? How does the proof

as a whole apply to specific examples?)

•

There is emerging evidence that proof understanding is

(empirically) a single factor construct. There is not a distinction

between local understandings (going step-by-step) and global

understandings (seeing the big picture.

Questions

•

Can students working with ITP improve local understandings?

Global understandings?

•

Is there a difference in between how local understandings and

global understandings are sought?

How can we increase students’

understanding of proofs?

•

Alcock, L., Hodds, M., Roy, S., & Inglis, M. (2015). Investigating

and improving undergraduate proof comprehension. 

Notices of

the AMS

, 

62

(7), 742-752.

How can we increase understanding of

proofs:

Two approaches

•

Present better proofs

•

Change the way students engage in them

First approach:

e-proofs

•

Alcock and Wilkinson (2011) created “e-proofs”. These were

proofs that presented proofs on a computer, where the students

received audio-commentary and annotations on what students

should be thinking about and observing as they read the proof.

•

The proof was popular with students, mathematicians, and

mathematics educators.

•

Roy et al. (2017) did a study comparing student understanding

of those who read e-proofs with those who read a standard

paper-and-pencil proof.

–

There was no significant difference on proof comprehension after

reading the proofs.

–

T

h

e

e

-

p

r

o

o

f

s

g

r

o

u

p

d

i

d

w

o

r

s

e

o

n

a

r

e

t

e

n

t

i

o

n

t

e

s

t

g

i

v

e

n

s

e

v

e

r

a

l

w

e

e

k

s

l

a

t

e

r

.

First approach:

Generic proofs

T

h

e

o

r

e

m

:

A

p

e

r

f

e

c

t

s

q

u

a

r

e

h

a

s

a

n

o

d

d

n

u

m

b

e

r

o

f

f

a

c

t

o

r

s

.

G

e

n

e

r

i

c

‘

p

r

o

o

f

’

:

L

e

t

’

s

l

o

o

k

a

t

3

6

.

W

e

s

h

o

w

i

t

h

a

s

a

n

o

d

d

n

u

m

b

e

r

o

f

f

a

c

t

o

r

s

.

W

e

c

a

n

s

y

s

t

e

m

a

t

i

c

a

l

l

y

l

i

s

t

a

l

l

p

a

i

r

s

t

h

a

t

m

u

l

t

i

p

l

y

t

o

3

6

.

1x36

2x18

3x12

4x9

6x6

All the factors of 36 appear on this list. We see that in all cases,

except the last, we are multiplying different factors together. The

reason that we can pair exactly one number with itself follows

because 36 is a perfect square.

(Adapted from Zaslavsky & Leron, 2013).

First approach:

Generic proofs

•

A generic proof has the following properties:

–

The proven statement is of the form, “For all 

n

, 

n

 has property P”.

–

The proof applies to a specific 

n

0

 which is not too simple or not too

complex.

–

The justifications for each step of the proof are common to all

elements of the scope of the claim, not something specific to 

n

0

.

–

The proof should be constructive (Rowland, 2001).

•

Generic proofs are said to improve comprehension:

“G

eneric proofs can help understanding

by enabling students to

engage with the main ideas of

the complete proof in an intuitive and

familiar

context, temporarily suspending the formidable

issues of

full generality, formalism and symbolism”

(Zaslavsky & Leron,

2013, p. 27).

First approach:

Generic proofs

•

My colleagues and I compared students’ comprehension of

similar conventional proofs and generic proofs in a randomized

controlled experiments with 106 students.

•

The generic proof students did better on tasks asking them to

apply the ideas of a proof to a specific example (although the

result was not statistically significant).

•

The generic proof students did significantly worse on the other

items in the post-test (Lew et al., 2020).

First approach:

Structured proofs

•

Leron (1983) proposed presenting ‘structured proofs’– proofs in

an outline format where overarching goals are given a

prominent heading.

•

This was touted by mathematicians and mathematics educators,

but the story is the same: Students did slightly worse when a

post-test was given (Fuller et al., 2014).

First approach:

Improving lectures

•

Gabel and Dreyfus (2017) found a method to improve lectures

by giving greater attention or presence to the key idea of the

lecture.

•

Despite the lectures being viewed favorably by mathematicains

and students, their understanding of the lecture was still limited.

First approach:

“All three of these approaches involve instructor provision of

different or extra explanations. A structured proof involves

restructuring the proof text, a generic proof involves changing its

content, and an e-Proof involves augmenting the proof with

annotations and commentary. Changing the presentation in such

ways requires substantial instructor effort, and the underwhelming

empirical results suggest that this may not be effort well spent”

(Hodds et al., 2014, p. 67).

Second approach:

Self-explanation training

Hodds et al (2014) asked students to “self-explain” after reading

each line in a proof:

•

A self-explanation is where you ask a question about and

attempt to answer after reading a line in the text.

–

Asking why new lines are true, why an idea was used, or how a

statement links to previous knowledge are self-explanations.

Parahprasing, re-wording, or monitoring understanding are not.

•

Students who were prompted to self-explain performed better on

a proof comprehension test than those who did not, and did

better on a retention test.

Second approach:

The importance of processing

I think the chan ges in formatting where researchers saying the

inferences they made, and the output of the cognitive activities that

they engaged in, and making them explicit to students:

•

e-proofs filled in all the “hidden” details of a proof, the things a

more experienced reader would know.

•

Generic proofs exemplified arguments, a common means

mathematicians use to understand proofs.

•

Structured proofs organize and modularize proof, which is a

good way to make sense of a proof.

But what led to the understanding was 

making the inferences

 and

engaging in the activity

, not (just) the results

Using ITPs to improve understanding

Using ITPs to improve understanding

Tim,

I am teaching a methods course for the first time in 28 hours. I don't know how

to do it. Any advice?

Keith

Using ITPs to improve understanding

Plus, they're really stuck on thinking "what am I gonna go?" and

you need to push them to think, "what are the kids gonna do?"

from both the perspective of the activities the kids will do and what

their work shows.  If you can get them to start making that switch

and asking better questions, you've been wildly successful.

(Tim Fukawa-Connelly, personal communication)

Questions

•

How will ITPs change the ways that students engage in reading

the proofs of others?

•

Will this kill the need to read the proof closely (since the ITP

already tells the student that the proof is correct)?

•

Or will this lead students to engage in deeper analysis, like why

the definitions are correct, or why the proof was structured the

way it is.

•

Of course, this depends not only on the technology, but how it is

used to educate students.