Introduction to Go Programming Language
A Short Intro to Go
CS 240
“Go is a
c
o
m
p
i
l
e
d
,
c
o
n
c
u
r
r
e
n
t
,
s
t
a
t
i
c
a
l
l
y
t
y
p
e
d
,
g
a
r
b
a
g
e
-
c
o
l
l
e
c
t
e
d
language developed at Google”
- Rob Pike, 2012
What’s this Weird Language I’ve Never Heard of?
Rob Pike is the Jeff Dean of distributed systems. Here’s the article the quote is from:
https://talks.golang.org/2012/splash.article
c
o
m
p
i
l
e
d
L
i
k
e
C
,
C
+
+
c
o
n
c
u
r
r
e
n
t
L
i
k
e
E
r
l
a
n
g
s
t
a
t
i
c
a
l
l
y
t
y
p
e
d
L
i
k
e
C
,
C
+
+
,
J
a
v
a
g
a
r
b
a
g
e
-
c
o
l
l
e
c
t
e
d
L
i
k
e
J
a
v
a
a
n
d
P
y
t
h
o
n
What’s this Weird Language I’ve Never Heard of?
Why Not Use Python, Java, C++, etc?
Built for Systems.
Go preserves efficiency but has good abstractions.
Easy multi threading and IO communication.
Develop quickly
Do many things efficiently
and at the same time
Official Self Reported List of Companies that use Go:
https://github.com/golang/go/wiki/GoUsers
Seems Google Specific. Who Else Actually Uses it?
Why did
they
Choose Go?
“We built everything in Python because it was easy,
b
u
t
n
o
w
i
t
’
s
s
l
o
w
.
S
o
w
e
s
w
i
t
c
h
e
d
t
o
G
o
.
”
- Most companies using Go
But How do I Use Go?
S
t
a
r
t
h
e
r
e
:
https://tour.golang.org/list
D
i
d
n
’
t
i
n
s
t
a
l
l
G
o
?
U
s
e
t
h
e
w
e
b
I
D
E
:
https://play.golang.org/
O
t
h
e
r
R
e
s
o
u
r
c
e
s
:
Go for Pythonists
https://talks.golang.org/2013/go4python.slide#1
Go for Distributed Systems
https://talks.golang.org/2013/distsys.slide#1
Official Go Talks
https://github.com/golang/go/wiki/GoTalks
But How do I Use Go?
D
E
M
O
:
g
o
t
o
u
r
Build Software for Any System
go build file.go
Compile an executable for your machine
env GOOS=windows GOARCH=amd64 go build file.go
Compile an executable for Windows with 64 bit processor
Format your Code
COMMAND
WHAT IT DOES
gofmt file.go
Format the file.go properly
D
E
M
O
:
g
o
f
m
t
Wait, I Have Questions!
Go’s official
“Frequently Asked Questions (FAQ)”
https://golang.org/doc/faq
MapReduce
CS 240
Map Reduce
W
i
k
i
p
e
d
i
a
:
“
M
a
p
R
e
d
u
c
e
i
s
a
p
r
o
g
r
a
m
m
i
n
g
m
o
d
e
l
a
n
d
a
n
a
s
s
o
c
i
a
t
e
d
i
m
p
l
e
m
e
n
t
a
t
i
o
n
f
o
r
p
r
o
c
e
s
s
i
n
g
a
n
d
g
e
n
e
r
a
t
i
n
g
b
i
g
d
a
t
a
s
e
t
s
w
i
t
h
a
p
a
r
a
l
l
e
l
,
d
i
s
t
r
i
b
u
t
e
d
a
l
g
o
r
i
t
h
m
o
n
a
c
l
u
s
t
e
r
.
”
In other words, a general and scalable solution to deal
with big data computation on multiple machines.
Abstract Map Reduce
map(key, value) -> list(<k’, v’>)
•
Apply function to (key, value) pair
•
Outputs set of intermediate pairs
reduce(key, list<value>) -> <k’, v’>
•
Applies aggregation function to values
•
Outputs result
Split and
distribute data
Aggregate and
compute
results
Word Count – The
Hello World
of Map Reduce
Bus Car Train
Train Plane Car
Bus Bus Plane
Bus Car Train
Train Plane Car
Bus Bus Plane
Bus 1
Car 1
Train 1
Train 1
Plane 1
Car 1
Bus 2
Plane 1
Bus 2
Bus 1
Car 1
Car 1
Train 1
Train 1
Plane 1
Plane 1
Bus 3
Car 2
Train 2
Plane 2
Bus 3
Car 2
Train 2
Plane 2
Splitting
Mapping
Intermediate
Splitting
Reducing
Combining
doMap()
doReduce()
A Motivating Problem for Map Reduce
“Find me the closest Starbucks to KAUST.
Actually, I’ll give you a place and something to look for,
and you find me the closest one.
Here’s a 1 TB text file … good luck”
GPS Coordinates
Site Name
[22.3,
39.1]
Tim Hortons
[22.2,
39.1]
KAUST Library
[35.7,
139.7]
Starbucks
...
...
In KAUST
In Tokyo, Japan
A Motivating Problem for Map Reduce
0
1
2
3
4
0
3
2
1
GPS Coordinates
Site Name
[22.3,
39.1]
Tim Hortons
[22.2,
39.1]
KAUST Library
[35.7,
139.7]
Starbucks
...
...
Map to grids
Reduce to
single files
Split the File and Map Each Chunk Independently (1/2)
GPS Coordinates
Site Name
[22.3,
39.1]
Tim Hortons
[22.2,
39.1]
KAUST Library
[35.7,
139.7]
Starbucks
...
...
[42.0,
69.0]
Chanak Train Stop
[22.2,
39.2]
Burger King
...
...
...
...
...
...
...
...
Mapper
Mapper
Mapping Nodes
Split the File and Map Each Chunk Independently (2/2)
GPS Coordinates
Site Name
[22.3,
39.1]
Tim Hortons
[22.2,
39.1]
KAUST Library
[35.7,
139.7]
Starbucks
...
...
[42.0,
69.0]
Chanak Train
[22.2,
39.2]
Burger King
...
...
...
...
...
...
...
...
Mapper
Mapper
(1,2)
: [22.3, 39.1] Tim Hortons
(1,2)
: [22.2, 39.1] KAUST Library
(1,2)
: ...
(2,4)
: [35.7, 139.7] Starbucks
(2,4)
: ...
(1,3)
: [42.0, 69.0] Chanak Train
(1,3)
: ...
(1,2)
: [22.2, 39.2] Burger King
(1,2)
: ...
KEY <grid>: VALUE <locations and name>
...
Mapping Nodes
(KEY) can appear in multiple
mappers
Collect the Mapper Results and Reduce to Single Files (1/2)
Reducer
(1,2)
Reducer
(1,3), (2,4)
(1,2)
: [22.3, 39.1] Tim Hortons
(1,2)
: [22.2, 39.1] KAUST Library
(1,2)
: ...
(2,4)
: [35.7, 139.7] Starbucks
(2,4)
: ...
(1,3)
: [42.0, 69.0] Chanak Train
(1,3)
: ...
(1,2)
: [22.2, 39.2] Burger King
(1,2)
: ...
Reducing Nodes
Collect the Mapper Results and Reduce to Single Files (2/2)
Reducer
(1,2)
Reducer
(1,3), (2,4)
(1,2)
: [22.3, 39.1] Tim Hortons
(1,2)
: [22.2, 39.1] KAUST Library
(1,2)
: ...
(2,4)
: [35.7, 139.7] Starbucks
(2,4)
: ...
(1,3)
: [42.0, 69.0] Chanak Train
(1,3)
: ...
(1,2)
: [22.2, 39.2] Burger King
(1,2)
: ...
KEY <grid>: [
VALUES <locations and names>,
...]
Reducing Nodes
(1,2)
: [
[22.3, 39.1] Tim Hortons,
[22.2, 39.1] KAUST Library,
[22.2, 39.2] Burger King
...]
]
(2,4)
: [
[35.7, 139.7] Starbucks,
...]
(1,3)
: [
[42.0, 69.0] Chanak Train Stop,
...],
Word Count – The
Hello World
of Map Reduce
Bus Car Train
Train Plane Car
Bus Bus Plane
Bus Car Train
Train Plane Car
Bus Bus Plane
Bus 1
Car 1
Train 1
Train 1
Plane 1
Car 1
Bus 2
Plane 1
Bus 2
Bus 1
Car 1
Car 1
Train 1
Train 1
Plane 1
Plane 1
Bus 3
Car 2
Train 2
Plane 2
Bus 3
Car 2
Train 2
Plane 2
Splitting
Mapping
Intermediate
Splitting
Reducing
Combining
Mapper nodes
Reducer nodes
Task is automatically
distributed across five
different nodes
Hadoop: An open-source implementation
Apache Hadoop is the most popular open-source
implementation of MapReduce
Runs on top of a distributed filesystem (HDFS)
Try their MapReduce tutorial:
https://hadoop.apache.org/docs/r1.2.1/mapred_tutorial.html
How Hadoop Does it
Mapper
Mapping Nodes
Mapper
Reducer
Reducing Nodes
Reducer
Some Advice for the Assignments
•
Write modular code
•
Use comments (even to yourself)
•
Don’t forget
go fmt
(graded)
•
The clearer your code is, the more
we can help with bugs
Go (also known as Golang) is a compiled, concurrent, statically typed, and garbage-collected language developed at Google by Rob Pike in 2012. It combines efficiency with good abstractions, making it ideal for system development. Many companies have adopted Go for its ease of use, speed, and efficient multitasking capabilities. To get started with Go programming, you can explore the official resources provided by the Go community.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
A Short Intro to Go CS 240
Whats this Weird Language Ive Never Heard of? Go is a compiled, concurrent, statically typed, garbage-collected language developed at Google - Rob Pike, 2012 Rob Pike is the Jeff Dean of distributed systems. Here s the article the quote is from: https://talks.golang.org/2012/splash.article
Whats this Weird Language Ive Never Heard of? compiled concurrent statically typed garbage-collected Like C, C++ Like Erlang Like C, C++, Java Like Java and Python
Why Not Use Python, Java, C++, etc? Built for Systems. Go preserves efficiency but has good abstractions. Easy multi threading and IO communication. Develop quickly Do many things efficiently and at the same time
Seems Google Specific. Who Else Actually Uses it? Official Self Reported List of Companies that use Go: https://github.com/golang/go/wiki/GoUsers
Why did they Choose Go? We built everything in Python because it was easy, but now it s slow. So we switched to Go. - Most companies using Go
But How do I Use Go? Start here: https://tour.golang.org/list Didn t install Go? Use the web IDE: https://play.golang.org/ Other Resources: Go for Pythonists https://talks.golang.org/2013/go4python.slide#1 Go for Distributed Systems https://talks.golang.org/2013/distsys.slide#1 Official Go Talks https://github.com/golang/go/wiki/GoTalks
But How do I Use Go? DEMO: go tour
Build Software for Any System go build file.go Compile an executable for your machine env GOOS=windows GOARCH=amd64 go build file.go Compile an executable for Windows with 64 bit processor
Format your Code COMMAND gofmt file.go WHAT IT DOES Format the file.go properly DEMO: gofmt
Wait, I Have Questions! Go s official Frequently Asked Questions (FAQ) https://golang.org/doc/faq
MapReduce CS 240
Map Reduce Wikipedia: MapReduce is a programming model and an associated implementation for processing and generating big data sets with a parallel, distributed algorithm on a cluster. In other words, a general and scalable solution to deal with big data computation on multiple machines.
Abstract Map Reduce map(key, value) -> list(<k , v >) Apply function to (key, value) pair Outputs set of intermediate pairs Split and distribute data reduce(key, list<value>) -> <k , v > Applies aggregation function to values Outputs result Aggregate and compute results
Word Count The Hello World of Map Reduce Bus 2 Bus 1 Bus 3 Bus 1 Car 1 Train 1 Bus Car Train Car 1 Car 1 Car 2 Bus 3 Car 2 Train 2 Plane 2 Bus Car Train Train Plane Car Bus Bus Plane Train 1 Plane 1 Car 1 Train Plane Car Train 1 Train 1 Train 2 Bus 2 Plane 1 Bus Bus Plane Plane 2 Plane 1 Plane 1 Intermediate Splitting Mapping Splitting Combining Reducing doReduce doReduce() () doMap doMap() ()
A Motivating Problem for Map Reduce Find me the closest Starbucks to KAUST. Actually, I ll give you a place and something to look for, and you find me the closest one. Here s a 1 TB text file good luck GPS Coordinates [22.3, [22.2, [35.7, ... Site Name Tim Hortons KAUST Library Starbucks ... 39.1] 39.1] 139.7] In KAUST In Tokyo, Japan
A Motivating Problem for Map Reduce GPS Coordinates [22.3, [22.2, [35.7, ... Site Name Tim Hortons KAUST Library Starbucks ... 39.1] 39.1] 139.7] 0 1 2 3 4 0 (0,0) (0,1) (0,2) (0,3) (0,4) 1 (1,0) (1,1) (1,2) (1,3) (1,4) Reduce to single files Map to grids 2 (2,0) (2,1) (2,2) (2,3) (2,4) (3,0) (3,1) (3,2) (3,3) (3,4) 3
Split the File and Map Each Chunk Independently (1/2) Mapping Nodes Mapper GPS CoordinatesSite Name [22.3, 39.1] [22.2, 39.1] [35.7, 139.7] Starbucks ... [42.0, 69.0] [22.2, 39.2] ... ... ... ... Tim Hortons KAUST Library ... Chanak Train Stop Burger King ... ... ... ... Mapper
Split the File and Map Each Chunk Independently (2/2) KEY <grid>: VALUE <locations and name> ... Mapping Nodes (1,2): [22.3, 39.1] Tim Hortons (1,2): [22.2, 39.1] KAUST Library (1,2): ... (2,4): [35.7, 139.7] Starbucks (2,4): ... Mapper GPS Coordinates Site Name [22.3, 39.1] [22.2, 39.1] [35.7, 139.7] Starbucks ... [42.0, 69.0] [22.2, 39.2] ... ... ... ... Tim Hortons KAUST Library ... Chanak Train Burger King ... ... ... ... (1,3): [42.0, 69.0] Chanak Train (1,3): ... (1,2): [22.2, 39.2] Burger King (1,2): ... Mapper (KEY) can appear in multiple mappers
Collect the Mapper Results and Reduce to Single Files (1/2) Reducing Nodes (1,2): [22.3, 39.1] Tim Hortons (1,2): [22.2, 39.1] KAUST Library (1,2): ... (2,4): [35.7, 139.7] Starbucks (2,4): ... Reducer (1,2) (1,3): [42.0, 69.0] Chanak Train (1,3): ... (1,2): [22.2, 39.2] Burger King (1,2): ... Reducer (1,3), (2,4)
Collect the Mapper Results and Reduce to Single Files (2/2) KEY <grid>: [ VALUES <locations and names>, ...] Reducing Nodes (1,2): [ [22.3, 39.1] Tim Hortons, [22.2, 39.1] KAUST Library, [22.2, 39.2] Burger King ...] ] (1,2): [22.3, 39.1] Tim Hortons (1,2): [22.2, 39.1] KAUST Library (1,2): ... (2,4): [35.7, 139.7] Starbucks (2,4): ... Reducer (1,2) (2,4): [ [35.7, 139.7] Starbucks, ...] (1,3): [42.0, 69.0] Chanak Train (1,3): ... (1,2): [22.2, 39.2] Burger King (1,2): ... Reducer (1,3), (2,4) (1,3): [ [42.0, 69.0] Chanak Train Stop, ...],
Word Count The Hello World of Map Reduce Reducer nodes Mapper nodes Bus 2 Bus 1 Bus 3 Bus 1 Car 1 Train 1 Bus Car Train Car 1 Car 1 Car 2 Bus 3 Car 2 Train 2 Plane 2 Bus Car Train Train Plane Car Bus Bus Plane Train 1 Plane 1 Car 1 Train Plane Car Train 1 Train 1 Train 2 Bus 2 Plane 1 Bus Bus Plane Plane 2 Plane 1 Plane 1 Task is automatically distributed across five different nodes Intermediate Splitting Mapping Splitting Combining Reducing
Hadoop: An open-source implementation Apache Hadoop is the most popular open-source implementation of MapReduce Runs on top of a distributed filesystem (HDFS) Try their MapReduce tutorial: https://hadoop.apache.org/docs/r1.2.1/mapred_tutorial.html
How Hadoop Does it Mapping Nodes Reducing Nodes Mapper Reducer Mapper Reducer
Some Advice for the Assignments Write modular code Use comments (even to yourself) Don t forget go fmt (graded) The clearer your code is, the more we can help with bugs
