5. INFRASTRUCTURE
• Same config, same logging, same directory structure
• Same protocol, including JSON to Protobuf conversion
• Build and testing inTeamCity
• QA team should not know this is a Go project
Go is not special in any way
6. INFRASTRUCTURE
• Logs go to syslog and eventually to Splunk
• Metrics are collected with home-grown system based on
RRD (you will see some examples)
• HTTP based profiling is always on
7. INFRASTRUCTURE
• As of now we do not use vendoring
• We use go get for dependencies
• Sometimes we just fork projects (as with rocksdb lib)
• We use Make for building
15. DO NOT GIVE UP
for {
generateSomeLoad()
go tool pprof -alloc_objects http://.../debug/pprof/heap
go tool pprof -inuse_objects http://.../debug/pprof/heap
think()
go build -gcflags=-m foobar.go
thinkAndFix()
}
17. MEMORY PROFILING CRASH COURSE
Allocating on stack
!
• Fast
• No GC pressure
• Stack size is not fixed in Go
• Not always possible
Allocating on heap
!
• Slower
• GC pressure
• Always possible
18. type Person struct {
Name string
Age uint
}
var People []*Person
const PeopleCount = 10000000
func allocateInitial() { ...
func allocateMore() { ...
!
!
func main() {
runtime.GOMAXPROCS(runtime.NumCPU())
allocateInitial()
go allocateMore()
http.ListenAndServe("localhost:8080", nil)
}
MEMORY PROFILING CRASH COURSE
19. func allocateMore() {
for {
for i := 0; i < PeopleCount; i++ {
People = append(People, &Person{"marko", 29})
}
People = People[0:PeopleCount]
time.Sleep(10 * time.Second)
}
}
func allocateInitial() {
for i := 0; i < PeopleCount; i++ {
People = append(People, &Person{"marko", 29})
}
}
MEMORY PROFILING CRASH COURSE
20. ESCAPE ANALYSIS
$ go build -gcflags=-m
# github.com/mkevac/test002
./test.go:20: &Person literal escapes to heap
./test.go:27: &Person literal escapes to heap
31. MEMORY PROFILING CRASH COURSE (FIXED)
type Person struct {
Name [6]byte // was string
Age uint
}
var People []Person // was []*Person
const PeopleCount = 10000000
func allocateMore() {
for {
for i := 0; i < PeopleCount; i++ {
People = append(People,
Person{[6]byte{'m', 'a', 'r', 'k', 'o', 0}, 29})
}
People = People[0:PeopleCount]
time.Sleep(10 * time.Second)
}
}
32. $ go tool pprof —inuse_objects ./test003 http:/…/debug/pprof/heap
(pprof) top10
1820 of 1824 total (99.78%)
Dropped 8 nodes (cum <= 9)
flat flat% sum% cum cum%
1820 99.78% 99.78% 1820 99.78% mcommoninit
0 0% 99.78% 1820 99.78% runtime.rt0_go
0 0% 99.78% 1820 99.78% runtime.schedinit
37. GO IS NOT C
• Often it is preferable to copy a little bit,
but avoid using pointers
• We had to thoroughly inspect our code
and remove pointers everywhere we
could
38. $ go tool pprof --alloc_objects ./heaptest /tmp/…/mem.pprof
Adjusting heap profiles for 1-in-4096 sampling rate
Welcome to pprof! For help, type 'help'.
(pprof) top
Total: 29161720 objects
22648298 77.7% 77.7% 22648298 77.7% newselect
6513152 22.3% 100.0% 6513152 22.3% main.main
256 0.0% 100.0% 256 0.0% runtime.mallocinit
14 0.0% 100.0% 14 0.0% allocg
0 0.0% 100.0% 270 0.0% _rt0_go
0 0.0% 100.0% 22648298 77.7% main.loop
0 0.0% 100.0% 14 0.0% mcommoninit
NOT AN ALLOCATION
49. $ cat test.c
!
int get_error(char **error) {
*error = "error";
return 0;
}
$ cat test.go
[…skipped…]
!
func main() {
var errStr *C.char
C.get_error(&errStr)
s := C.GoString(errStr)
fmt.Println(s)
}
$ go build -gcflags=-m
# github.com/mkevac/test001
[…skipped…]
./test.go:13: moved to heap: errStr
./test.go:14: &errStr escapes to heap
./test.go:16: main ... argument does not escape
go 1.3.0
50. $ cat test.c
!
int get_error(char **error) {
*error = "error";
return 0;
}
$ cat test.go
[…skipped…]
!
func main() {
var errStr *C.char
C.get_error(&errStr)
s := C.GoString(errStr)
fmt.Println(s)
}
$ go build -gcflags=-m
# github.com/mkevac/test001
./test.go:14: main &errStr does not escape
./test.go:16: main ... argument does not escape
go 1.4.2
52. db.Put() can cause data to escape!
func SaveCoord(c Coord) error {
key := GetKey(c)
data := make([]byte, c.Size())
n, _ := c.MarshalTo(data)
return db.Put(key, data[:n])
}
59. MAPS
var m = make(map[int]int)
!
func main() {
runtime.GOMAXPROCS(runtime.NumCPU())
!
for i := 0; i < 10000000; i++ {
m[i] = i
}
!
for {
runtime.GC()
time.Sleep(5 * time.Second)
}
}
GC Pause:
500 ms
60. commit 85e7bee19f9f26dfca414b1e9054e429c448b14f
Author: Dmitry Vyukov <dvyukov@google.com>
Date: Mon Jan 26 21:04:41 2015 +0300
!
runtime: do not scan maps when k/v do not contain pointers
!
Currently we scan maps even if k/v does not contain pointers.
This is required because overflow buckets are hanging off the main table.
This change introduces a separate array that contains pointers to all
overflow buckets and keeps them alive. Buckets themselves are marked
as containing no pointers and are not scanned by GC (if k/v does not
contain pointers).
!
This brings maps in line with slices and chans -- GC does not scan
their contents if elements do not contain pointers.
!
Currently scanning of a map[int]int with 2e8 entries (~8GB heap)
takes ~8 seconds. With this change scanning takes negligible time.
!
Update #9477.
!
Change-Id: Id8a04066a53d2f743474cad406afb9f30f00eaae
Reviewed-on: https://go-review.googlesource.com/3288
Reviewed-by: Keith Randall <khr@golang.org>
expected in 1.5
61. GOTIP
gc #11 @11.995s 0%: 0+0+0+0+3 ms clock, 0+0+0+0+25 ms
cpu, 304->304->304 MB, 8 P (forced)
62. GC SUMMARY & LINKS
• If you want to write low latency apps, you have to fight GC :-(
• Debugging performance issues: http://goo.gl/jRIyGg
• Go Escape Analysis Flaws: http://goo.gl/U1wkvy
• Go execution tracer: http://goo.gl/KHLBQN
• Interface type conversion: http://goo.gl/oJDYPa
• GC debugcharts: https://github.com/mkevac/debugcharts
• GC visualisation (davecheney): http://goo.gl/ubz5DL
