Many problems in scientific computing can be effectively addressed by content-addressed storage. This paper presents the open source Keep content-addressed storage system, briefly describes the architecture and core design goals, and reviews the engineering team's experience porting a large Perl application to Go in a short time frame.

1. Keep: Open Source
Content-Addressed
Storage
How We Turned a Big Hot Mess of Perl Into
a Sweet Go Ride

2. Overview
● The problem: large-scale data management is
hard.
● The solution: content-addressed storage and
federation.
● History of Warehouse and Keep
● Keep: motivations and design goals
● What we learned from Go

3. The problem:
Data management is hard.

4. The problem
Managing data for scientific research is hard.
It's too easy to lose data:
rm result<tab> ---OH WAIT CRAP NO
./generate_results.py -o results1.csv ---OH WAIT CRAP NO
Or lose track of how we got it.
$ ls -l results/
-rw-r----- 1 twp twp 859458786 Sep 2 15:39 results1.csv
-rw-r----- 1 twp twp 758489475 Sep 3 15:51 results2.csv
-rw-r----- 1 twp twp 958747348 Sep 4 11:46 results3.csv
-rw-r----- 1 twp twp 795984373 Sep 5 17:12 results4.csv
-rw-r----- 1 twp twp 833857373 Sep 6 9:38 results5.csv
-rw-r----- 1 twp twp 894847636 Sep 7 12:46 results6.csv
-rw-r----- 1 twp twp 847476854 Oct 2 12:17 results_umm.csv
-rw-r----- 1 twp twp 766845784 Sep 12 19:08 results_wednesday_i_think.csv
-rw-r----- 1 twp twp 932875738 Sep 8 18:32 results_whatever.csv

5. The problem
Why is federation important?
Because the alternative is snail-mailing hard drives
of data all over the world.

6. The solution: Keep

7. Keep: open source content-addressed
storage.
Design goals:
● Gracefully handle data sets measured in the
terabytes and petabytes.
● Multi-tenant architecture
● Lightweight permissions system
● Minimize external dependencies
● Data federation

8. What is Content-Addressed Storage?
A very large key/value store, in which the address of a data object is the hash of its
content. Example:
$ head -c 10000000 /dev/urandom > /tmp/stuff
$ md5sum /tmp/stuff
c54a33209b03905476bf971e722c683d /tmp/stuff
This file can only be stored under the address c54a33209b03905476bf971e722c683d.
PUT /c54a33209b03905476bf971e722c683d
-> HTTP/1.1 200 OK
c54a33209b03905476bf971e722c683d+10000000
Attempting to store it under a different name yields an HTTP 4xx error.
PUT /ffffffffffffffffffffffffffffffff
-> HTTP/1.1 422 Hash mismatch in request
With CAS, if you store the same data blob twice, you always get the same key back.
Best known example: Git refs.

9. Why is CAS useful?
Permanent storage.
● Blobs cannot be overwritten.
● Even on purpose!
Determine quickly whether a large data blob is present in the store.
These characteristics make content-addressed storage extremely
well suited to any system which demands accountability on very
large data sets, such as:
● Scientific computing
● Accounting data management
● Photo retouching

10. Existing alternatives
CAS systems typically have names like: EMC,
NetApp, IBM.
Cost tends to be on the order of $2,000-$10,000
per terabyte.
Not open source, therefore, poopy.
A few open source alternatives exist, notably
Camlistore. But some missing or immature features:
● Multi-tenant support
● Blob permissions

11. Keep architecture
Data is written in blocks up to 64MB.
Smart client, dumb server. Client is responsible for:
● Structured data (directories, folders, names, etc)
● Replication
Default implementation on top of POSIX filesystem. Cheap, easy to deploy.
-rw------- 1 keep keep 14551778 Oct 10 17:06 /keep/87b/87b0f2f2eb0c1f90c6da46309a799cc0
-rw------- 1 keep keep 8154404 Oct 10 17:06 /keep/cc5/cc57ebe000aed447e1e481569e1a8abd
-rw------- 1 keep keep 7239086 Oct 10 17:06 /keep/ae8/ae8a8b29d9fb6325ee93d951cdae896f
-rw------- 1 keep keep 14455989 Oct 10 17:06 /keep/e92/e928a4d4b5c3ea903914d178bbfdb035
Keep Volumes can be implemented on top of any backend service:
● RAID
● Amazon S3
● Google Cloud Storage

12. Keep permissions
Goals:
● Require permission to read blocks
● No hard dependencies on external authentication services
Solution: permission hints.
Block requests are accompanied by a timestamped signature, e.g.:
87b0f2f2eb0c1f90c6da46309a799cc0+14551778 + Abcf33732294c3e1fe16e39cea3114c9461274645 @ 5438550a
--------- block locator string --------/ ------------ SHA-1 signature ----------/ timestamp
Signatures are derived from the block hash, the user's OAuth token, the expiration
timestamp, and a server-side signing secret.
Permission hints can be generated by the authentication server.
The Keep server can verify a valid permission signature instantly, without even having to
contact any other service.

13. From Perl to Go
Original Keep implementation, "Warehouse", written in Perl for the
Harvard Personal Genome Project.
Many drawbacks:
● Perl
● Eats ALL the memory
● Not multithreaded (see also: Perl)
● Slooooooooooooow. Slow slow slow.
● Slow.
● Did I mention Perl?

14. What did Go give us?
So we rewrote Warehouse in Go.
● 3 weeks to working prototype (supports GET and
PUT)
● 6 weeks to production (including permissions)
Advantages:
● Easier dependency management
● Clean concurrency architecture
● Rapid develop/build/test/deploy cycle

15. Dependency management: Perl
Welcome to dependency hell.
Package: libwarehouse-perl
Architecture: all
Depends: ${perl:Depends}, ${misc:Depends}, libdbi-perl, libwww-perl,
libio-stringy-perl, libtimedate-perl, libgnupg-interface-perl,
libunix-syslog-perl, libbsd-resource-perl, libio-compress-zlib-perl,
libdigest-sha-perl, bioperl, gcc, g++, libstdc++6, bison, perlmagick,
imagemagick, gnuplot, bzip2, libbz2-dev, libfftw3-3, libfftw3-dev,
libxml-simple-perl, ghostscript, xsltproc, libyaml-perl,
libjson-perl, realpath, psmisc
Recommends: libhttp-ghttp-perl, libwhisker2-perl, libfuse-perl
Description: Warehouse -- Client and Server library for the storage warehouse.
Warehouse -- Client and Server library for the Free Factory storage warehouse.
Deployment dependencies include MogileFS, MySQL, PGP, others.
The stuff of nightmares.

16. Dependency Management: Go
import (
"bufio"
"bytes"
"container/list"
"crypto/md5"
"encoding/json"
"fmt"
"github.com/gorilla/mux"
"io"
"log"
"net/http"
"os"
"regexp"
"runtime"
"strconv"
"strings"
"syscall"
"time"
)
Exactly one third-party
dependency:
github.com/gorilla/mux
(HTTP request routing).e

17. Concurrency: Perl

18. Concurrency: Keep
Go implements concurrency
in "goroutines" -- small
tasks that run
independently of each other
and may be run on other
threads.
The Go runtime is
responsible for managing
threads and scheduling
tasks.
Writing concurrent code is as
simple as:
func master() {
go start_slave()
// The slave may continue to run after master()
returns.
}
func start_slave() {
while (work_to_do()) {
...
}
}

19. Concurrency: Keep
The standard Go libraries
provide a rich set of tools for
writing concurrent
applications.
This is a complete
implementation of a working
multithreaded HTTP server in
Go.
The HTTP library's http.
Server type handles each
request in its own goroutine.
package main
import (
"log"
"net/http"
)
func main() {
http.HandleFunc("/hello", helloHandler)
log.Fatal(http.ListenAndServe(":8080", nil))
}
func helloHandler(w http.ResponseWriter, r *http.Request)
{
w.Write([]byte("Hello!n"))
}

20. Rapid development cycles
Very fast build cycles:
hitchcock:/home/twp/arvados/services/keepstore% wc handlers.go keepstore.go perms.go volume.go work_queue.go
...
1515 6183 45084 total
hitchcock:/home/twp/arvados/services/keepstore% time go build
go build 1.01s user 0.15s system 99% cpu 1.154 total
Testing:
hitchcock:/home/twp/arvados/services/keepstore% wc *_test.go
...
1830 6233 50798 total
hitchcock:/home/twp/arvados/services/keepstore% go test
...
PASS
ok _/home/twp/arvados/services/keepstore 1.018s

21. Rapid development cycles
PASS
ok _/home/twp/arvados/services/keepstore 1.018s
Why does the test take that long?
Oh right.
// Sleep for 1s, then put the block again. The volume
// should report a more recent mtime.
//
// TODO(twp): this would be better handled with a mock Time object.
// Alternatively, set the mtime manually to some moment in the past
// (maybe a v.SetMtime method?)
//
time.Sleep(time.Second)

22. Lessons from Go
You don't have to choose between performance
and rapid prototyping.
Extremely fast design and test cycle.
Where Perl celebrates "laziness, impatience, and
hubris," Go makes it easy to do the right thing.
Go makes laziness awkward.

23. Where do we go from here?
Keep is open source (AGPLv3)
Keep source code is at github.
com/curoverse/arvados/tree/master/services/kee
pstore
The full source code repository: github.
com/curoverse/arvados
We are eager for contributors and new ideas for
how to use Keep!

24. Thank you!
Tim Pierce
Senior Software Engineer, Curoverse
twp@curoverse.com
http://curoverse.com/
@qwrrty