A presentation at Twitter's official developer conference, Chirp, about why we use the Scala programming language and how we build services in it. Provides a tour of a number of libraries and tools, both developed at Twitter and otherwise.
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
The Why and How of Scala at Twitter
1.
2. The Why and How
of Scala at Twitter
Alex Payne (@al3x)
Twitter Infrastructure
3. Who’s this guy?
•Working at Twitter since
2007, before it was even a
company.
•Spent a couple years on the
API.
•Co-authored Programming
Scala (O’Reilly, 2009).
•Into programming
languages, scaling stuff.
5. So, like, Ruby or Scala?
•Both!
•Ruby mostly on the
frontend.
•Scala mostly on the
backend.
•Thrift and data stores to
connect the two.
6. Elevator Pitch
• It’s fun.
• It’s fast.
• It’s runs on a great virtual machine (JVM).
• It’s got an awesome community.
• It can borrow from Java.
7. No, seriously, why?
• A rich static type system that gets out of your
way when you don’t need it, is awesome when
you do.
• Flexible syntax makes it easy to write
readable, maintainable code.
• Traits for cross-cutting modularity.
• Lots of OOP goodness.
8. I’m not sold yet.
• How about the choice between immutable and
mutable variables?
• Powerful functional programming: mapping,
filtering, folding, currying, so much more.
• Lazy values.
• Pattern matching.
• XML literals, and clean syntax for working
with XML documents built right in!
9. What about concurrency?
• You can do Actor concurrency (message
passing), kinda like Erlang.
• Or, use threads.
• Or, use all the java.util.concurrent tools.
• Or, use other JVM concurrency frameworks
(ex: Netty, Apache Mina).
13. We build services.
• Isolated components, can swap them out.
• Talk to the rest of the system via Thrift (for
now, maybe Avro in the future).
• Independently tested for correctness, load.
• Can have custom operational properties.
14. Scala Services at Twitter
• Kestrel: queuing.
• Flock (and Gizzard): social graph store.
• Hawkwind: people search.
• Hosebird: streaming API.
• more in the works all the time...
16. We build libraries.
• Reusable chunks of code that we can share
between projects.
• We open source them.
• We try to keep the “NIH” to a minimum, but
we also have very particular requirements.
17. Ostrich
• Gather stats from inside your running process.
• Counters, gauges, and timings.
• Share stats many ways: JMX, JSON-over-
HTTP, plain text Telnet-style socket, log files.
22. xrayspecs
• A set of extensions to the fantastic Specs BDD
test framework.
• Handles testing concurrency, time, creating
temporary folders.
• The concurrency features have been rolled into
the main distribution of Specs!
23. xrayspecs Examples
“response arrives from server” in {
get(“/example”) must eventually(notBe(null))
}
“time should stop” in {
Time.freeze()
val nowSnapshot = Time.now
Thread.sleep(30)
Time.now mustEqual nowSnapshot
}
24. scala-json
• A cleaned-up version of the official Scala JSON
codec.
• Makes clever use of parser combinators – a
good way to learn about them.
• Heavily tested, fixes a bunch of edge cases.
26. Other Twitter Libraries
• Naggati: build protocols for Apache Mina.
• Smile: a memcached client that uses Actors.
• Querulous: a nice database client.
• Jackhammer: a load testing framework
(coming soon).
• probably stuff I'm leaving out...
27. Stuff We Use
• There's great open source code in the Scala
community.
• We try to use and contribute back to third-
party projects.
• Follow @implicit_ly for new Scala releases.
• Also subscribe to Repopular Scala.
28. sbt – the simple build tool
• Scala’s answer to Ant and Maven.
• Sets up new projects.
• Maintains project configuration, build tasks,
and dependencies in pure Scala. Totally open-
ended.
• Interactive console.
• Will run tasks as soon as files in your project
change – automatically compile and run tests!
29. specs
• As aforementioned: great way to do BDD
testing.
• Set up pre- and post-conditions for
individual tests, suites, etc.
• Extremely flexible and extensible.
• Great, responsive maintainer.
• Supports mocking (we like Mockito).
30. The IDE Question
• We've had the best luck with IntelliJ IDEA,
but it's still pretty rough.
• Most of us use a plain text editor like Emacs,
VIM, or TextMate.
• Combined with sbt, it's actually a really nice
workflow!