Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
sbt,
history of JSON libraries,
microservices, and
schema evolution
Eugene Yokota (@eed3si9n)

February, 2017
• Scala hobbyist since 2010
• scalaxb (XML data binding)
• treehugger.scala
• sbt-assembly, sbt-buildinfo, etc
• “learning...
Lightbend Production Suite
big picture: development at scale
• How do you scale a technological organization?
• Goal: Sustainable development
development at scale
• Bezos mandate (written circa 2002 before AWS) https://plus.google.com/
+RipRowan/posts/eVeouesvaVX
microservice is a soc...
Brief history of
JSON libraries
Eugene Yokota (@eed3si9n)

2016
Brief history of JSON libraries
Dispatch JSON
literaljson
Lift JSON
sjson
Spray JSON
Play JSON
Argonaut
json4s
Circe
Jawn
...
2008
Two books
Programming in Scala
import scala.util.parsing.combinator._
class JSON extends JavaTokenParsers {
def value : Parser[Any] ...
• Chapter 5. Writing a library: working with JSON data
Real World Haskell
data JValue = JString String
| JNumber Double
| ...
2009
• https://github.com/dispatch/dispatch/commit/
41edb939baa5c6edb4378c1bd8e1d2f10f3350f2
• Contributed by Jorge Ortiz
• Par...
• https://github.com/jonifreeman/literaljson
• Authored by Joni Freeman
• Custom parser
• Values stored in AST, JValue
• O...
2010
• https://github.com/debasishg/sjson
• Authored by Debasish Ghosh
• sjson: Now offers Type Class based JSON
Serialization ...
Typeclass
• Allows adding capability to a class after the fact in a
typesafe manner.
Typeclass
trait Eq[A] {
  def equals(x: A, y: A...
JsonFormat
trait CanRead[A] {
  def reads(json: JsValue): A
}
trait CanWrite[A] {
  def writes(a: A): JsValue
}
trait Json...
JsonFormat
• JsonFormat typeclass allows conversion to and
from an arbitrary type to a JSON AST.
2011
• https://github.com/spray/spray-json
• Authored by Mathias Dönitz
• Original parser
• Ported AST from Dispatch JSON and t...
• https://github.com/playframework/playframework/commit/
63448578b15dcc7bf4806878c7b3aa4c74193af6
• Started out as port of...
2012
• http://argonaut.io/
• Purely functional JSON library
• Authored by Mark Hibberd, Tony Morris, Sean Parsons
• Uses Scalaz...
2013
• https://github.com/json4s/json4s
• Fork of Lift-json. JSON library that is not strongly tied to a web
framework.
json4s
2014
• https://github.com/non/jawn
• Authored by Erik Osheim (@d6)
• Backend-independent JSON parser
Jawn
• http://rapture.io/mod/json
• Authored by Jon Pretty
• Backend-independent JSON library
Rapture JSON
2015
• https://github.com/travisbrown/circe
• Authored by Travis Brown
• Port of Argonaut
Circe
2016
• https://github.com/mdedetrich/scala-json-ast
• Authored by Matthew de Detrich
• Aiming to be the common JSON AST
Scala J...
• https://github.com/eed3si9n/sjson-new
• Backend-independent typeclass based JSON codec
• No macros
sjson-new
Brief history of JSON libraries
Dispatch JSON
literaljson
Lift JSON
sjson
Spray JSON
Play JSON
Argonaut
json4s
Circe
Jawn
...
Contract-first approach
Eugene Yokota (@eed3si9n)

2016
• “Serialization” tends to start from a programming language construct, and it
generates String or byte array.
• Data bind...
• sealed traits
• case classes
Representing data in Scala
• sealed traits
• case classes
Representing data in Scala
Cannot evolve in a binary compatible way.
Representing data in Scala
class Greeting(name: String) {
  def copy(name: String = name): Greeting = ???
  def unapply(v:...
• sealed traits
• case classes
• Cannot evolve in a binary compatible way.
• But generating equals, hash, and toString is ...
Contraband
• http://www.scala-sbt.org/contraband/
• Contraband is a description language for your datatypes and APIs, currently
targe...
Record types
package com.example
@target(Scala)
## Character represents the characters in Star Wars.
type Character {
name...
@since annotation
package com.example
@target(Scala)
type Greeting {
value: String!
x: Int @since("0.2.0")
}
Enumeration types
package com.example
@target(Scala)
## Star Wars trilogy.
enum Episode {
NewHope
Empire
Jedi
}
Interfaces
package com.example
@target(Scala)
## Character represents the characters in Star Wars.
interface Character {
n...
Record type example
package com.example
@target(Scala)
type Person {
name: String!
age: Int
}
Record type example
// DO NOT EDIT MANUALLY
package com.example
final class Person private (
  val name: String,
  val age...
Record type example
  override def toString: String = {
    "Person(" + name + ", " + age + ")"
  }
  protected[this] def ...
Record type example
object Person {
  def apply(name: String, age: Option[Int]): Person =
new Person(name, age)
  def appl...
Record type example
> val x = Person("Alice", 20)
> x.withAge(21)
Contraband can derive sjson-new codecs
JSON codec generation
package com.example
@target(Scala)
type Person {
name: String!
age: Int
}
JSON codec generation
package generated
import _root_.sjsonnew.{ deserializationError,
serializationError, Builder, JsonFo...
JSON codec generation
scala> import sjsonnew.support.scalajson.unsafe.{ Converter,
CompactPrinter, Parser }
scala> import ...
JSON codec generation
scala> val s = CompactPrinter(j)
s: String = {"name":"Bob","age":20}
scala> val x = Parser.parseUnsa...
• http://www.scala-sbt.org/contraband/
• Contraband is a description language for your datatypes and APIs, currently
targe...
• An opportunity for datatype-generic programming.
• For example, one backend is Int
Contraband
• An opportunity for datatype-generic programming.
• For example, one backend is Int (Murmurhash support)
• Builder API is...
sbt, history of JSON libraries, microservices, and schema evolution (Tokyo ver)
sbt, history of JSON libraries, microservices, and schema evolution (Tokyo ver)
Prochain SlideShare
Chargement dans…5
×

sbt, history of JSON libraries, microservices, and schema evolution (Tokyo ver)

1 040 vues

Publié le

Gave talk at 第4回Reactive System Meetup in 西新宿.

Publié dans : Logiciels
  • Identifiez-vous pour voir les commentaires

sbt, history of JSON libraries, microservices, and schema evolution (Tokyo ver)

  1. 1. sbt, history of JSON libraries, microservices, and schema evolution Eugene Yokota (@eed3si9n)
 February, 2017
  2. 2. • Scala hobbyist since 2010 • scalaxb (XML data binding) • treehugger.scala • sbt-assembly, sbt-buildinfo, etc • “learning Scalaz” / “herding Cats” • ScalaMatsuri • Lightbend/Typesafe since 2014 • tech lead of Reactive Platform team • current maintainer / tech lead of sbt who is this guy (@eed3si9n)?
  3. 3. Lightbend Production Suite
  4. 4. big picture: development at scale
  5. 5. • How do you scale a technological organization? • Goal: Sustainable development development at scale
  6. 6. • Bezos mandate (written circa 2002 before AWS) https://plus.google.com/ +RipRowan/posts/eVeouesvaVX microservice is a social stack 1. All teams will henceforth expose their data and functionality through service interfaces. 2. Teams must communicate with each other through these interfaces. 3. There will be no other form of inter-process communication allowed: no direct linking, no direct reads of another team’s data store, no shared-memory model, no back-doors whatsoever. The only communication allowed is via service interface calls over the network. 4. It doesn’t matter what technology they use. HTTP, Corba, Pubsub, custom protocols -- doesn't matter. Bezos doesn't care. 5. All service interfaces, without exception, must be designed from the ground up to be externalizable. That is to say, the team must plan and design to be able to expose the interface to developers in the outside world. No exceptions. 6. Anyone who doesn't do this will be fired.
  7. 7. Brief history of JSON libraries Eugene Yokota (@eed3si9n)
 2016
  8. 8. Brief history of JSON libraries Dispatch JSON literaljson Lift JSON sjson Spray JSON Play JSON Argonaut json4s Circe Jawn SLIP-28 JSON Rapture JSON sjson-new 2009 2010 2014
  9. 9. 2008
  10. 10. Two books
  11. 11. Programming in Scala import scala.util.parsing.combinator._ class JSON extends JavaTokenParsers { def value : Parser[Any] = obj | arr | stringLiteral | floatingPointNumber | "null" | "true" | "false" def obj : Parser[Any] = "{"~repsep(member, ",")~"}" def arr : Parser[Any] = "["~repsep(value, ",")~"]" def member: Parser[Any] = stringLiteral~":"~value } https://www.artima.com/pins1ed/combinator-parsing.html#31.4
  12. 12. • Chapter 5. Writing a library: working with JSON data Real World Haskell data JValue = JString String | JNumber Double | JBool Bool | JNull | JObject [(String, JValue)] | JArray [JValue] deriving (Eq, Ord, Show)
  13. 13. 2009
  14. 14. • https://github.com/dispatch/dispatch/commit/ 41edb939baa5c6edb4378c1bd8e1d2f10f3350f2 • Contributed by Jorge Ortiz • Parsing using parser combinator • Values stored in AST, JsValue Dispatch JSON
  15. 15. • https://github.com/jonifreeman/literaljson • Authored by Joni Freeman • Custom parser • Values stored in AST, JValue • On August 11, 2009, Joni contributed literaljson to Lift, and became Lift-JSON jonifreeman/literaljson
  16. 16. 2010
  17. 17. • https://github.com/debasishg/sjson • Authored by Debasish Ghosh • sjson: Now offers Type Class based JSON Serialization in Scala • Uses Dispatch JSON for AST sjson
  18. 18. Typeclass
  19. 19. • Allows adding capability to a class after the fact in a typesafe manner. Typeclass trait Eq[A] {   def equals(x: A, y: A): Boolean } • Eq typeclass can enable === operator to compare only the supported types, and prevent compilation of "1" === 1 • Scala Implicits : Type Classes Here I Come
  20. 20. JsonFormat trait CanRead[A] {   def reads(json: JsValue): A } trait CanWrite[A] {   def writes(a: A): JsValue } trait JsonFormat[A] extends CanRead[A] with CanWrite[A] implicit val intFormat: JsonFormat[Int] = ...
  21. 21. JsonFormat • JsonFormat typeclass allows conversion to and from an arbitrary type to a JSON AST.
  22. 22. 2011
  23. 23. • https://github.com/spray/spray-json • Authored by Mathias Dönitz • Original parser • Ported AST from Dispatch JSON and type classes from sjson spray-json
  24. 24. • https://github.com/playframework/playframework/commit/ 63448578b15dcc7bf4806878c7b3aa4c74193af6 • Started out as port of Dispatch JSON and sjson typeclasses, but quickly added its own implementations. Play JSON
  25. 25. 2012
  26. 26. • http://argonaut.io/ • Purely functional JSON library • Authored by Mark Hibberd, Tony Morris, Sean Parsons • Uses Scalaz or Cats • Very feature rich (Lenses, Cursor, History Cursor) Argonaut
  27. 27. 2013
  28. 28. • https://github.com/json4s/json4s • Fork of Lift-json. JSON library that is not strongly tied to a web framework. json4s
  29. 29. 2014
  30. 30. • https://github.com/non/jawn • Authored by Erik Osheim (@d6) • Backend-independent JSON parser Jawn
  31. 31. • http://rapture.io/mod/json • Authored by Jon Pretty • Backend-independent JSON library Rapture JSON
  32. 32. 2015
  33. 33. • https://github.com/travisbrown/circe • Authored by Travis Brown • Port of Argonaut Circe
  34. 34. 2016
  35. 35. • https://github.com/mdedetrich/scala-json-ast • Authored by Matthew de Detrich • Aiming to be the common JSON AST Scala JSON AST (SLIP-28)
  36. 36. • https://github.com/eed3si9n/sjson-new • Backend-independent typeclass based JSON codec • No macros sjson-new
  37. 37. Brief history of JSON libraries Dispatch JSON literaljson Lift JSON sjson Spray JSON Play JSON Argonaut json4s Circe Jawn SLIP-28 JSON Rapture JSON sjson-new 2009 2010 2014
  38. 38. Contract-first approach Eugene Yokota (@eed3si9n)
 2016
  39. 39. • “Serialization” tends to start from a programming language construct, and it generates String or byte array. • Data binding starts with a contract or a schema of the wire format, and generates the binding in a programming language. • XML Schema / WSDL • Google Protocol Buffer • Apache Thrift • Apache Avro • Facebook GraphQL Serialization vs Data binding
  40. 40. • sealed traits • case classes Representing data in Scala
  41. 41. • sealed traits • case classes Representing data in Scala Cannot evolve in a binary compatible way.
  42. 42. Representing data in Scala class Greeting(name: String) {   def copy(name: String = name): Greeting = ???   def unapply(v: Greeting): Option[String] = ??? } class Greeting(name: String, x: Int) {   def copy(name: String = name, x: Int = x): Greeting = ???   def unapply(v: Greeting): Option[(String, Int)] = ??? }
  43. 43. • sealed traits • case classes • Cannot evolve in a binary compatible way. • But generating equals, hash, and toString is generally useful. Representing data in Scala
  44. 44. Contraband
  45. 45. • http://www.scala-sbt.org/contraband/ • Contraband is a description language for your datatypes and APIs, currently targeting Java and Scala. • Based on GraphQL schema. Contraband
  46. 46. Record types package com.example @target(Scala) ## Character represents the characters in Star Wars. type Character { name: String! appearsIn: [com.example.Episode]! }
  47. 47. @since annotation package com.example @target(Scala) type Greeting { value: String! x: Int @since("0.2.0") }
  48. 48. Enumeration types package com.example @target(Scala) ## Star Wars trilogy. enum Episode { NewHope Empire Jedi }
  49. 49. Interfaces package com.example @target(Scala) ## Character represents the characters in Star Wars. interface Character { name: String! appearsIn: [com.example.Episode]! friends: lazy [com.example.Character] }
  50. 50. Record type example package com.example @target(Scala) type Person { name: String! age: Int }
  51. 51. Record type example // DO NOT EDIT MANUALLY package com.example final class Person private (   val name: String,   val age: Option[Int]) extends Serializable {   override def equals(o: Any): Boolean = o match {     case x: Person => (this.name == x.name) && (this.age == x.age)     case _ => false   }   override def hashCode: Int = {     37 * (37 * (17 + name.##) + age.##)   }   override def toString: String = {
  52. 52. Record type example   override def toString: String = {     "Person(" + name + ", " + age + ")"   }   protected[this] def copy(name: String = name, age: Option[Int] = age): Person = {     new Person(name, age)   }   def withName(name: String): Person = {     copy(name = name)   }   def withAge(age: Option[Int]): Person = {     copy(age = age)   }   def withAge(age: Int): Person = {
  53. 53. Record type example object Person {   def apply(name: String, age: Option[Int]): Person = new Person(name, age)   def apply(name: String, age: Int): Person = new Person(name, Option(age)) }
  54. 54. Record type example > val x = Person("Alice", 20) > x.withAge(21)
  55. 55. Contraband can derive sjson-new codecs
  56. 56. JSON codec generation package com.example @target(Scala) type Person { name: String! age: Int }
  57. 57. JSON codec generation package generated import _root_.sjsonnew.{ deserializationError, serializationError, Builder, JsonFormat, Unbuilder } trait PersonFormats { self: sjsonnew.BasicJsonProtocol =>   implicit lazy val personFormat: JsonFormat[_root_.Person] = JsonFormat[_root_.Person] {     override def read[J](jsOpt: Option[J], unbuilder: Unbuilder[J]): _root_.Person = {       jsOpt match {         case Some(js) =>           unbuilder.beginObject(js)           val name = unbuilder.readField[String]("name")           val age = unbuilder.readField[Option[Int]]("age")           unbuilder.endObject()           _root_.Person(name)
  58. 58. JSON codec generation scala> import sjsonnew.support.scalajson.unsafe.{ Converter, CompactPrinter, Parser } scala> import com.example.codec.CustomJsonProtocol._ scala> import com.example.Person scala> val p = Person("Bob", 20) p: com.example.Person = Person(Bob, 20) scala> val j = Converter.toJsonUnsafe(p) j: scala.json.ast.unsafe.JValue = JObject([Lscala.json.ast.unsafe.JField;@6731ad72) scala> val s = CompactPrinter(j) s: String = {"name":"Bob","age":20} scala> val x = Parser.parseUnsafe(s)
  59. 59. JSON codec generation scala> val s = CompactPrinter(j) s: String = {"name":"Bob","age":20} scala> val x = Parser.parseUnsafe(s) x: scala.json.ast.unsafe.JValue = JObject([Lscala.json.ast.unsafe.JField;@7331f7f8) scala> val q = Converter.fromJsonUnsafe[Person](x) q: com.example.Person = Person(Bob, 20) scala> assert(p == q)
  60. 60. • http://www.scala-sbt.org/contraband/ • Contraband is a description language for your datatypes and APIs, currently targeting Java and Scala. • Low-tech metaprogramming (code generation) • Binary compatible evolution of pseudo case class. • Auto derivation of backend-independent JSON codec. Contraband
  61. 61. • An opportunity for datatype-generic programming. • For example, one backend is Int Contraband
  62. 62. • An opportunity for datatype-generic programming. • For example, one backend is Int (Murmurhash support) • Builder API is used to build one-way hash. Contraband

×