Statistical Aspects of Data Mining: Professor Rajan Patel
S
t
a
t
i
s
t
i
c
s
2
0
2
:
S
t
a
t
i
s
t
i
c
a
l
A
s
p
e
c
t
s
o
f
D
a
t
a
M
i
n
i
n
g
P
r
o
f
e
s
s
o
r
R
a
j
a
n
P
a
t
e
l
L
e
c
t
u
r
e
1
=
C
o
u
r
s
e
w
e
b
p
a
g
e
a
n
d
C
h
a
p
t
e
r
s
1
+
2
A
g
e
n
d
a
:
1
)
G
o
o
v
e
r
i
n
f
o
r
m
a
t
i
o
n
o
n
c
o
u
r
s
e
w
e
b
p
a
g
e
2
)
L
e
c
t
u
r
e
o
v
e
r
C
h
a
p
t
e
r
1
3
)
D
i
s
c
u
s
s
n
e
c
e
s
s
a
r
y
s
o
f
t
w
a
r
e
4
)
S
t
a
r
t
l
e
c
t
u
r
i
n
g
o
v
e
r
C
h
a
p
t
e
r
2
(
D
a
t
a
)
S
t
a
t
i
s
t
i
c
s
2
0
2
:
S
t
a
t
i
s
t
i
c
a
l
A
s
p
e
c
t
s
o
f
D
a
t
a
M
i
n
i
n
g
P
r
o
f
e
s
s
o
r
R
a
j
a
n
P
a
t
e
l
C
o
u
r
s
e
w
e
b
p
a
g
e
:
h
t
t
p
:
/
/
s
i
t
e
s
.
g
o
o
g
l
e
.
c
o
m
/
s
i
t
e
/
s
t
a
t
s
2
0
2
(
l
i
n
k
e
d
f
r
o
m
s
t
a
t
s
2
0
2
.
c
o
m
)
C
o
u
r
s
e
e
-
m
a
i
l
a
d
d
r
e
s
s
:
s
t
a
t
s
2
0
2
@
g
m
a
i
l
.
c
o
m
G
o
o
g
l
e
g
r
o
u
p
f
o
r
g
e
n
e
r
a
l
d
i
s
c
u
s
s
i
o
n
:
s
t
a
t
s
2
0
2
I
n
t
r
o
d
u
c
t
i
o
n
t
o
D
a
t
a
M
i
n
i
n
g
b
y
T
a
n
,
S
t
e
i
n
b
a
c
h
,
K
u
m
a
r
C
h
a
p
t
e
r
1
:
I
n
t
r
o
d
u
c
t
i
o
n
W
h
a
t
i
s
D
a
t
a
M
i
n
i
n
g
?
D
a
t
a
m
i
n
i
n
g
i
s
t
h
e
p
r
o
c
e
s
s
o
f
a
u
t
o
m
a
t
i
c
a
l
l
y
d
i
s
c
o
v
e
r
i
n
g
u
s
e
f
u
l
i
n
f
o
r
m
a
t
i
o
n
i
n
l
a
r
g
e
d
a
t
a
r
e
p
o
s
i
t
o
r
i
e
s
.
(
p
a
g
e
2
)
T
h
e
r
e
a
r
e
m
a
n
y
o
t
h
e
r
d
e
f
i
n
i
t
i
o
n
s
The problem/question of interest
D
a
t
a
M
i
n
i
n
g
E
x
a
m
p
l
e
s
a
n
d
N
o
n
-
E
x
a
m
p
l
e
s
D
a
t
a
M
i
n
i
n
g
:
-
C
e
r
t
a
i
n
n
a
m
e
s
a
r
e
m
o
r
e
p
r
e
v
a
l
e
n
t
i
n
c
e
r
t
a
i
n
U
S
l
o
c
a
t
i
o
n
s
(
O
’
B
r
i
e
n
,
O
’
R
u
r
k
e
,
O
’
R
e
i
l
l
y
…
i
n
B
o
s
t
o
n
a
r
e
a
)
-
G
r
o
u
p
t
o
g
e
t
h
e
r
s
i
m
i
l
a
r
d
o
c
u
m
e
n
t
s
r
e
t
u
r
n
e
d
b
y
s
e
a
r
c
h
e
n
g
i
n
e
a
c
c
o
r
d
i
n
g
t
o
t
h
e
i
r
c
o
n
t
e
x
t
(
e
.
g
.
A
m
a
z
o
n
r
a
i
n
f
o
r
e
s
t
,
A
m
a
z
o
n
.
c
o
m
,
e
t
c
.
)
N
O
T
D
a
t
a
M
i
n
i
n
g
:
-
L
o
o
k
u
p
p
h
o
n
e
n
u
m
b
e
r
i
n
p
h
o
n
e
d
i
r
e
c
t
o
r
y
-
Q
u
e
r
y
a
W
e
b
s
e
a
r
c
h
e
n
g
i
n
e
f
o
r
i
n
f
o
r
m
a
t
i
o
n
a
b
o
u
t
“
A
m
a
z
o
n
”
W
h
y
M
i
n
e
D
a
t
a
?
S
c
i
e
n
t
i
f
i
c
V
i
e
w
p
o
i
n
t
D
a
t
a
c
o
l
l
e
c
t
e
d
a
n
d
s
t
o
r
e
d
a
t
e
n
o
r
m
o
u
s
s
p
e
e
d
s
(
G
B
/
h
o
u
r
)
●
r
e
m
o
t
e
s
e
n
s
o
r
s
o
n
a
s
a
t
e
l
l
i
t
e
●
t
e
l
e
s
c
o
p
e
s
s
c
a
n
n
i
n
g
t
h
e
s
k
i
e
s
●
m
i
c
r
o
a
r
r
a
y
s
g
e
n
e
r
a
t
i
n
g
g
e
n
e
e
x
p
r
e
s
s
i
o
n
d
a
t
a
●
s
c
i
e
n
t
i
f
i
c
s
i
m
u
l
a
t
i
o
n
s
g
e
n
e
r
a
t
i
n
g
t
e
r
a
b
y
t
e
s
o
f
d
a
t
a
T
r
a
d
i
t
i
o
n
a
l
t
e
c
h
n
i
q
u
e
s
i
n
f
e
a
s
i
b
l
e
f
o
r
l
a
r
g
e
d
a
t
a
s
e
t
s
D
a
t
a
m
i
n
i
n
g
m
a
y
h
e
l
p
s
c
i
e
n
t
i
s
t
s
●
i
n
c
l
a
s
s
i
f
y
i
n
g
a
n
d
s
e
g
m
e
n
t
i
n
g
d
a
t
a
●
i
n
h
y
p
o
t
h
e
s
i
s
f
o
r
m
a
t
i
o
n
W
h
y
M
i
n
e
D
a
t
a
?
C
o
m
m
e
r
c
i
a
l
V
i
e
w
p
o
i
n
t
L
o
t
s
o
f
d
a
t
a
i
s
b
e
i
n
g
c
o
l
l
e
c
t
e
d
a
n
d
w
a
r
e
h
o
u
s
e
d
●
W
e
b
d
a
t
a
,
e
-
c
o
m
m
e
r
c
e
●
P
u
r
c
h
a
s
e
s
a
t
d
e
p
a
r
t
m
e
n
t
/
g
r
o
c
e
r
y
s
t
o
r
e
s
●
B
a
n
k
/
c
r
e
d
i
t
c
a
r
d
t
r
a
n
s
a
c
t
i
o
n
s
●
C
o
m
p
u
t
e
r
s
h
a
v
e
b
e
c
o
m
e
m
o
r
e
p
o
w
e
r
f
u
l
●
C
o
m
p
e
t
i
t
i
v
e
p
r
e
s
s
u
r
e
i
s
s
t
r
o
n
g
●
P
r
o
v
i
d
e
b
e
t
t
e
r
,
c
u
s
t
o
m
i
z
e
d
s
e
r
v
i
c
e
s
f
o
r
a
n
e
d
g
e
I
n
c
l
a
s
s
e
x
e
r
c
i
s
e
#
1
:
G
i
v
e
a
n
e
x
a
m
p
l
e
o
f
s
o
m
e
t
h
i
n
g
y
o
u
d
i
d
y
e
s
t
e
r
d
a
y
o
r
t
o
d
a
y
w
h
i
c
h
r
e
s
u
l
t
e
d
i
n
d
a
t
a
w
h
i
c
h
c
o
u
l
d
p
o
t
e
n
t
i
a
l
l
y
b
e
m
i
n
e
d
t
o
d
i
s
c
o
v
e
r
u
s
e
f
u
l
i
n
f
o
r
m
a
t
i
o
n
.
I
n
c
l
a
s
s
e
x
e
r
c
i
s
e
#
1
:
G
i
v
e
a
n
e
x
a
m
p
l
e
o
f
s
o
m
e
t
h
i
n
g
y
o
u
d
i
d
y
e
s
t
e
r
d
a
y
o
r
t
o
d
a
y
w
h
i
c
h
r
e
s
u
l
t
e
d
i
n
d
a
t
a
w
h
i
c
h
c
o
u
l
d
p
o
t
e
n
t
i
a
l
l
y
b
e
m
i
n
e
d
t
o
d
i
s
c
o
v
e
r
u
s
e
f
u
l
i
n
f
o
r
m
a
t
i
o
n
.
O
r
i
g
i
n
s
o
f
D
a
t
a
M
i
n
i
n
g
(
p
a
g
e
6
)
D
r
a
w
s
i
d
e
a
s
f
r
o
m
m
a
c
h
i
n
e
l
e
a
r
n
i
n
g
,
A
I
,
p
a
t
t
e
r
n
r
e
c
o
g
n
i
t
i
o
n
a
n
d
s
t
a
t
i
s
t
i
c
s
T
r
a
d
i
t
i
o
n
a
l
t
e
c
h
n
i
q
u
e
s
m
a
y
b
e
u
n
s
u
i
t
a
b
l
e
d
u
e
t
o
-
e
n
o
r
m
i
t
y
o
f
d
a
t
a
-
h
i
g
h
d
i
m
e
n
s
i
o
n
a
l
i
t
y
o
f
d
a
t
a
-
h
e
t
e
r
o
g
e
n
e
o
u
s
,
d
i
s
t
r
i
b
u
t
e
d
n
a
t
u
r
e
o
f
d
a
t
a
Statistics
D
a
t
a
M
i
n
i
n
g
AI/Machine Learning
2
T
y
p
e
s
o
f
D
a
t
a
M
i
n
i
n
g
T
a
s
k
s
(
p
a
g
e
7
)
P
r
e
d
i
c
t
i
v
e
M
e
t
h
o
d
s
:
U
s
e
s
o
m
e
v
a
r
i
a
b
l
e
s
t
o
p
r
e
d
i
c
t
u
n
k
n
o
w
n
o
r
f
u
t
u
r
e
v
a
l
u
e
s
o
f
o
t
h
e
r
v
a
r
i
a
b
l
e
s
.
D
e
s
c
r
i
p
t
i
v
e
M
e
t
h
o
d
s
:
F
i
n
d
h
u
m
a
n
-
i
n
t
e
r
p
r
e
t
a
b
l
e
p
a
t
t
e
r
n
s
t
h
a
t
d
e
s
c
r
i
b
e
t
h
e
d
a
t
a
.
E
x
a
m
p
l
e
s
o
f
D
a
t
a
M
i
n
i
n
g
T
a
s
k
s
●
C
l
a
s
s
i
f
i
c
a
t
i
o
n
[
P
r
e
d
i
c
t
i
v
e
]
(
C
h
a
p
t
e
r
s
4
,
5
)
●
R
e
g
r
e
s
s
i
o
n
[
P
r
e
d
i
c
t
i
v
e
]
(
c
o
v
e
r
e
d
i
n
s
t
a
t
s
c
l
a
s
s
e
s
)
●
V
i
s
u
a
l
i
z
a
t
i
o
n
[
D
e
s
c
r
i
p
t
i
v
e
]
(
i
n
C
h
a
p
t
e
r
3
)
●
A
s
s
o
c
i
a
t
i
o
n
A
n
a
l
y
s
i
s
[
D
e
s
c
r
i
p
t
i
v
e
]
(
C
h
a
p
t
e
r
6
)
●
C
l
u
s
t
e
r
i
n
g
[
D
e
s
c
r
i
p
t
i
v
e
]
(
C
h
a
p
t
e
r
8
)
●
A
n
o
m
a
l
y
D
e
t
e
c
t
i
o
n
[
D
e
s
c
r
i
p
t
i
v
e
]
(
C
h
a
p
t
e
r
1
0
)
S
o
f
t
w
a
r
e
W
e
W
i
l
l
U
s
e
:
R
C
a
n
b
e
d
o
w
n
l
o
a
d
e
d
f
r
o
m
h
t
t
p
:
/
/
c
r
a
n
.
r
-
p
r
o
j
e
c
t
.
o
r
g
/
f
o
r
W
i
n
d
o
w
s
,
M
a
c
o
r
L
i
n
u
x
D
o
w
n
l
o
a
d
i
n
g
R
f
o
r
W
i
n
d
o
w
s
:
D
o
w
n
l
o
a
d
i
n
g
R
f
o
r
W
i
n
d
o
w
s
:
D
o
w
n
l
o
a
d
i
n
g
R
f
o
r
W
i
n
d
o
w
s
:
I
n
t
r
o
d
u
c
t
i
o
n
t
o
D
a
t
a
M
i
n
i
n
g
b
y
T
a
n
,
S
t
e
i
n
b
a
c
h
,
K
u
m
a
r
C
h
a
p
t
e
r
2
:
D
a
t
a
W
h
a
t
i
s
D
a
t
a
?
A
n
a
t
t
r
i
b
u
t
e
i
s
a
p
r
o
p
e
r
t
y
o
r
c
h
a
r
a
c
t
e
r
i
s
t
i
c
o
f
a
n
o
b
j
e
c
t
E
x
a
m
p
l
e
s
:
e
y
e
c
o
l
o
r
o
f
a
p
e
r
s
o
n
,
t
e
m
p
e
r
a
t
u
r
e
,
e
t
c
.
A
n
A
t
t
r
i
b
u
t
e
i
s
a
l
s
o
k
n
o
w
n
a
s
v
a
r
i
a
b
l
e
,
f
i
e
l
d
,
c
h
a
r
a
c
t
e
r
i
s
t
i
c
,
o
r
f
e
a
t
u
r
e
A
c
o
l
l
e
c
t
i
o
n
o
f
a
t
t
r
i
b
u
t
e
s
d
e
s
c
r
i
b
e
a
n
o
b
j
e
c
t
A
n
o
b
j
e
c
t
i
s
a
l
s
o
k
n
o
w
n
a
s
r
e
c
o
r
d
,
p
o
i
n
t
,
c
a
s
e
,
s
a
m
p
l
e
,
e
n
t
i
t
y
,
i
n
s
t
a
n
c
e
,
o
r
o
b
s
e
r
v
a
t
i
o
n
Attributes
Objects
R
e
a
d
i
n
g
D
a
t
a
i
n
t
o
R
D
o
w
n
l
o
a
d
i
t
f
r
o
m
t
h
e
w
e
b
a
t
http://sites.google.com/site/stats202/data/weblog2.txt
W
h
a
t
i
s
y
o
u
r
w
o
r
k
i
n
g
d
i
r
e
c
t
o
r
y
?
> getwd()
C
h
a
n
g
e
i
t
t
o
y
o
u
r
d
e
s
k
o
p
:
>
s
e
t
w
d
(
"
/
U
s
e
r
s
/
r
a
j
a
n
/
D
e
s
k
t
o
p
"
)
R
e
a
d
i
t
i
n
:
>
d
a
t
a
<
-
r
e
a
d
.
c
s
v
(
"
w
e
b
l
o
g
2
.
t
x
t
"
,
s
e
p
=
"
"
,
h
e
a
d
e
r
=
F
)
R
e
a
d
i
n
g
D
a
t
a
i
n
t
o
R
L
o
o
k
a
t
t
h
e
f
i
r
s
t
5
r
o
w
s
:
>
d
a
t
a
[
1
:
5
,
]
V1 V2 V3 V4 V5 V6 V7 V8
1 122.178.203.210 - - [20/Jun/2011:00:00:25 -0400] GET /favicon.ico HTTP/1.1 404 2294
2 70.105.172.121 - - [20/Jun/2011:00:01:03 -0400] GET / HTTP/1.1 200 736
3 70.105.172.121 - - [20/Jun/2011:00:01:03 -0400] GET /favicon.ico HTTP/1.1 404 2290
4 70.105.172.121 - - [20/Jun/2011:00:01:03 -0400] GET /favicon.ico HTTP/1.1 404 2290
5 70.105.172.121 - - [20/Jun/2011:00:01:32 -0400] GET /original_index.html HTTP/1.1 200 3897
V9 V10
1 www.stats202.com http://www.stats202.com/original_index.html
2 stats202.com -
3 stats202.com -
4 stats202.com -
5 www.stats202.com http://stats202.com/
V11 V12
1 Opera/9.80 (X11; Linux x86_64; U; en) Presto/2.8.131 Version/11.11 -
2 Mozilla/5.0 (Windows NT 5.1; rv:2.0.1) Gecko/20100101 Firefox/4.0.1 -
3 Mozilla/5.0 (Windows NT 5.1; rv:2.0.1) Gecko/20100101 Firefox/4.0.1 -
4 Mozilla/5.0 (Windows NT 5.1; rv:2.0.1) Gecko/20100101 Firefox/4.0.1 -
5 Mozilla/5.0 (Windows NT 5.1; rv:2.0.1) Gecko/20100101 Firefox/4.0.1 -
L
o
o
k
a
t
t
h
e
f
i
r
s
t
c
o
l
u
m
n
:
>
d
a
t
a
[
,
1
]
This content discusses the course web page and Chapters 1 and 2 of Statistics 202: Statistical Aspects of Data Mining with Professor Rajan Patel. It covers information on data mining, necessary software, and the significance of mining data from both scientific and commercial viewpoints.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Statistics 202: Statistical Aspects of Data Mining Professor Rajan Patel Lecture 1 = Course web page and Chapters 1+2 Agenda: 1) Go over information on course web page 2) Lecture over Chapter 1 3) Discuss necessary software 4) Start lecturing over Chapter 2 (Data)
Statistics 202: Statistical Aspects of Data Mining Professor Rajan Patel Course web page: http://sites.google.com/site/stats202 (linked from stats202.com) Course e-mail address: stats202@gmail.com Google group for general discussion: stats202
Introduction to Data Mining by Tan, Steinbach, Kumar Chapter 1: Introduction
What is Data Mining? Data mining is the process of automatically discovering useful information in large data repositories. (page 2) There are many other definitions The problem/question of interest
Data Mining Examples and Non-Examples Data Mining: -Certain names are more prevalent in certain US locations (O Brien, O Rurke, O Reilly in Boston area) NOT Data Mining: -Look up phone number in phone directory -Group together similar documents returned by search engine according to their context (e.g. Amazon rainforest, Amazon.com, etc.) -Query a Web search engine for information about Amazon
Why Mine Data? Scientific Viewpoint Data collected and stored at enormous speeds (GB/hour) remote sensors on a satellite telescopes scanning the skies microarrays generating gene expression data scientific simulations generating terabytes of data Traditional techniques infeasible for large data sets Data mining may help scientists in classifying and segmenting data
Why Mine Data? Commercial Viewpoint Lots of data is being collected and warehoused Web data, e-commerce Purchases at department / grocery stores Bank/credit card transactions Computers have become more powerful Competitive pressure is strong Provide better, customized services for an edge
In class exercise #1: Give an example of something you did yesterday or today which resulted in data which could potentially be mined to discover useful information.
In class exercise #1: Give an example of something you did yesterday or today which resulted in data which could potentially be mined to discover useful information.
Origins of Data Mining (page 6) Draws ideas from machine learning, AI, pattern recognition and statistics Traditional techniques may be unsuitable due to - enormity of data - high dimensionality of data - heterogeneous, distributed nature of data Statistics AI/Machine Learning Data Mining
2 Types of Data Mining Tasks (page 7) Predictive Methods: Use some variables to predict unknown or future values of other variables. Descriptive Methods: Find human-interpretable patterns that describe the data.
Examples of Data Mining Tasks Classification [Predictive] (Chapters 4,5) Regression [Predictive] (covered in stats classes) Visualization [Descriptive] (in Chapter 3) Association Analysis [Descriptive] (Chapter 6) Clustering [Descriptive] (Chapter 8) Anomaly Detection [Descriptive] (Chapter 10)
Software We Will Use: R Can be downloaded from http://cran.r-project.org/ for Windows, Mac or Linux
Introduction to Data Mining by Tan, Steinbach, Kumar Chapter 2: Data
Attributes What is Data? An attribute is a property or characteristic of an object Examples: eye color of a person, temperature, etc. Objects An Attribute is also known as variable, field, characteristic, or feature A collection of attributes describe an object An object is also known as record, point, case, sample, entity, instance, or observation
Reading Data into R Download it from the web at http://sites.google.com/site/stats202/data/weblog2.txt What is your working directory? > getwd() Change it to your deskop: > setwd("/Users/rajan/Desktop") Read it in: > data<-read.csv("weblog2.txt", sep=" ",header=F)
Reading Data into R Look at the first 5 rows: >data[1:5,] V1 V2 V3 V4 V5 V6 V7 V8 1 122.178.203.210 - - [20/Jun/2011:00:00:25 -0400] GET /favicon.ico HTTP/1.1 404 2294 2 70.105.172.121 - - [20/Jun/2011:00:01:03 -0400] GET / HTTP/1.1 200 736 3 70.105.172.121 - - [20/Jun/2011:00:01:03 -0400] GET /favicon.ico HTTP/1.1 404 2290 4 70.105.172.121 - - [20/Jun/2011:00:01:03 -0400] GET /favicon.ico HTTP/1.1 404 2290 5 70.105.172.121 - - [20/Jun/2011:00:01:32 -0400] GET /original_index.html HTTP/1.1 200 3897 V9 V10 1 www.stats202.com http://www.stats202.com/original_index.html 2 stats202.com - 3 stats202.com - 4 stats202.com - 5 www.stats202.com http://stats202.com/ V11 V12 1 Opera/9.80 (X11; Linux x86_64; U; en) Presto/2.8.131 Version/11.11 - 2 Mozilla/5.0 (Windows NT 5.1; rv:2.0.1) Gecko/20100101 Firefox/4.0.1 - 3 Mozilla/5.0 (Windows NT 5.1; rv:2.0.1) Gecko/20100101 Firefox/4.0.1 - 4 Mozilla/5.0 (Windows NT 5.1; rv:2.0.1) Gecko/20100101 Firefox/4.0.1 - 5 Mozilla/5.0 (Windows NT 5.1; rv:2.0.1) Gecko/20100101 Firefox/4.0.1 - Look at the first column: > data[,1]
