Talk at Sequoia CTO Summit on programming with Go language

1. Go @ Tokopedia Qasim Zaidi

2. About Tokopedia
• Launched in 2009
• 300K Active merchants
• 8 Million Products
• Indonesia’s biggest online marketplace
• Everyone pays shipping charges
• Shipping is a function of distance
• No discounting or cash-backs, neutral marketplace.
• Most payments are offline

3. nginx
ssl termination and
proxy pass
nginx-mod_perl
nginx-mod_perl
nginx-mod_perl
The Stack
postgres
mongo
redis
proxy_pass

4. What we already use go for
• Search & Discovery
• Image Uploads
• Analytics

8. 60% of the time, it works every time.

9. nginx
ssl termination and
proxy pass
nginx-mod_perl
nginx-mod_perl
nginx-mod_perl
The Stack
postgres
mongo
redis
proxy_pass
mod_perl blocks nginx
isolation is hard
cancellation is harder

10. Challenges of scaling with mod_perl
• Each request is
synchronously
processed in a worker
• If there is one bad
request, it ends up
queuing everything else.
• Cancellation is not easy
- so even when the user
moves away, the request
continues to run.

13. In 2012, we bet on node.js

14. scaled well too - from 50K transactions in 2012 to over a million in 2015

15. node.js challenges
• Too easy to mess up (dynamic, no type checking)
• linting, code reviews
• Unbounded Concurrency is hard.
• async, promises
• Scale issues with http client for c10k
• even dns latencies can have big impact

16. Picking a new language in 2015.

17. the no QA philosophy
QA is not a different role.
Developers are responsible for
testing their code.

18. Testing in Go
• Easy to write unit tests
func TestFoo(t *testing.T) {
...
}
• run as go test
• No separate frameworks
• Benchmarking is as easy as testing

19. the no Documentation
philosophy
Code should be self explanatory.
Documentation outside of code is
never updated.

20. Documentation with GoDoc
• Comments in code - e.g.
preceding function definition
• It extracts the comments
and presents them:
• $ godoc strings Join
func Join(a []string, sep
string) string
Join concatenates the
elements of a to create a
single string. The separator
string sep is placed between
elements in the resulting
string.
// Join concatenates the elements of a to create
a single string.
// The separator string sep is placed between
elements in the resulting string.
func Join(a []string, sep string) string {
//self explanatory blah blah blah
//more of the same
}

21. godoc - examples and testing
func ExampleJoin() {
s := []string{"foo", "bar", "baz"}
fmt.Println(strings.Join(s, ", "))
// Output: foo, bar, baz
}
• Also integrated with the testing framework to provide testable
example functions.

22. godoc - webview

23. Concurrency should
be easy
And not yet forced.
Too much concurrency spoils the
code.

24. Code should be hard
to mess with
A compiler that catches bugs
and is not too slow
statically compiled binaries - no
library mess

25. Institutional knowledge
Go brings google’s institutional
knowledge to you, for free
- Groupcache
- Expvars
- Single-flights and Cancellations

26. Object Caching
• In node.js - we built in an object caching solution in
house, that can easily make a function called cached
without changing much code.
• var wrapper = cache.wrap(original);
• Can chose whether to cache in memory or redis.
• Can avoid thundering herds - single flighting

27. Enter group cache
• In memory
• distributed
• single flight
ELB
app
groupcache
app app
groupcachegroupcache
http http

28. expvars - server metrics over http
• Export variables over http
• http://localhost:8123/debug/vars
{
"cmdline": ["/var/folders/bc/27hv15_d2zvcc3n3s9dxmfg00000gn/T/go-build421732654/command-
line-arguments/_obj/exe/main","-l","debug","-s","i.sock","-c","realtest"],
"counters": {"a": 10, "b": 10},
"memstats": {"Alloc":1076016,"TotalAlloc":1801544,"Sys":5966072,"Lookups":209,"Mallocs":
7986,"Frees":4528,"HeapAlloc":1076016,"HeapSys":2097152,"HeapIdle":327680,"HeapInuse":
1769472,"HeapReleased":0,"HeapObjects":3458,"StackInuse":212992,"StackSys":
917504,"MSpanInuse":21504,"MSpanSys":32768,"MCacheInuse":8800,"MCacheSys":
16384,"BuckHashSys":1441160,"GCSys":1183744,"OtherSys":277360,"NextGC":
1436032,"LastGC":1418102095002592201,"PauseTotalNs":2744531,"PauseNs":
[480149,171430,603839,288381,494934,522995,182803,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"NumGC":7,"EnableGC":true,"DebugGC":false}
}

29. Cancellations
• User moves away, nginx sends 499, but queries keep
running.
• Not Abandoning work when user has moved away is
often wasteful
• Go’s has advisory cancellations - cancel across go-
routines
• facilitated by contexts

30. moving from ‘dynamic’ to ‘compiled’
• Deployment changes
• from git clone to compile, build & deploy
• have a way of knowing binary version that is automatic
BUILDHASH=$(shell cd src/$(SRCDIR) && git rev-parse --verify HEAD | cut -c 1-7)
go build —ldflags -X $BUILDHASH
appname —version
• config and code is separate
• make sure everything you may ever need to configure is in config/flags, can’t change
binary
appname —port
appname —configtest
appname —debug
• have good debugging (we have debug() statements that are turned on by —debug flag)

31. The go workflow
jenkins
dpkg-
buildpackagetriggers build
build dpkg using go get
ansible
srv1 srv1 srv1
elb
nginx
binary, run via upstart
deploy dpkg
restart
2
1
triggers deploy
3

32. No templates, no exceptions, weird and sometimes inconsistent.
Go is an opinionated language
“Instructions, registers, and assembler
directives are always in UPPER CASE to
remind you that assembly programming is
a fraught endeavor.”
- Rob Pike

33. -Dick Gabriel, from the golang FAQ
Who would have guessed sophistication
bought such noise?

34. Golang - missing a package manager?
• go get - but its not complete
• No dependency management, unlike npm, no easy
hermetic builds
• gopkg.in is a solution - allows versioning
• if you have to change versions, edit every single import

37. Deployment Challenges
• No Fork() - forking with subroutines ain’t easy
• Facebook grace (https://github.com/facebookgo/grace)
• PID changes - no relationship
• problems with upstart / job monitor
• log rotation

39. Moving from a script to binary
• Deployment changes (from git clone to build, package and deploy)
• Need to know which version of code is running
• appname —version
• Need to debug easily
• appname —debug - enables debug() output
• Keep configuration flexible, because in case of issues, you can’t just edit and
deploy
• appname —configtest
• appname —port (over ride config)

40. Go @ Tokopedia
• Go is a modern language, very suitable for web.
• Go is Performant, unless you are doing C.
• Go is mostly google still, and not 100% complete.
• Go is opinionated, like Plan9, and there will be a learning
curve.
• Go is slower for prototyping, e.g. when compared to
node.js

41. Questions?