How to leverage schema evolution to create robust and evolutionary data microservices.

1. Vinicius Carvalho - Advisory Platform Architect @Pivotal
@vccarvalho
http://github.com/viniciusccarvalho/
Schema Evolution for
Data Microservices

2. The way we build
software

3. Servlet JSP Struts JSF Spring GWT Angular
time
Customer
DB
JSONXMLJava Serialization
Format evolution

4. What happens when
data evolves?

5. How do we handle
versioning?

6. Monolithic
architectures

7. Account
User
Product
Order
Common jar
<dependency>
<groupId>com.acme</groupId>
<artifactId>common-domain</artifactId>
<version>1.1.3</version>
</dependency>

8. RECOMMENDATION
SEARCHCATALOG
Updates?
1.1.3 1.1.1
0.9.5

9. Enterprise Service Bus
RECOMMENDATION
SEARCH CATALOGCanonical Message
Format
Haven’t we solved
this already?

10. Yes, But …

11. • Majority of ESB systems uses XML as
the Canonical Model
• XML is good for structure, but it
has no notion of evolution
• It’s heavy

12. And then there’s
this µService thing
Bounded Contexts
Contexts Maps
Aggregates
Value Objects
Anti corruption
Layer

13. Ownership
Sam Newman’s Building Microservices
Who owns this?

14. MSDN CQRS Journey ebook
CQRS?

15. Data evolution
…

16. • Data evolution is a
hard problem to
grasp
• Even in known
territories such as
traditional RDBMS
is a hard problem
to tackle

17. Schema Evolution
“The problem of evolving a data schema to
adapt it to a change in modeled reality”

18. Services Evolution
behavioral
structural
New functions are
added to the system
Information model
changes over time

19. Backward
compatibility
• Newer version can
read old version
• Challenges:
Field renaming
V1
V2

20. Forward
compatibility
• Older version can
read new version
• Challenges:
Field renaming
Field removal
V1
V2

21. Request / Response
RECOMMENDATIONUser
GET /v1/…
RECOMMENDATIONUser
GET /v2/…

22. Data Streaming

23. ▪@EnableBindings(Source.class)
▪one output
▪@EnableBindings(Sink.class)
▪one input
▪@EnableBinding(Processor.class)
▪one input and one output
▪@EnableBinding(MyOrderHandler.class)
▪custom interfaces with as many inputs and outputs
▪@EnableRxJavaProcessor
▪OOTB support for RxJava with one input and one
output
@Enable All the
things

24. Binder SPI

25. Formats
Choosing the right one

26. Structure
Adaptability
Guarantees a contract
between users of the
model
How flexible the
format is for changes
on it’s structure
BenchmarkingBecause … we love it

27. Format Structure Adaptability
CSV
Positional,
no type definition
Possible if appending
new columns
XML
flexible,
strong typed
Append, remove only via
version (no standard)
supports defaults
JSON
flexible,
untyped
Append and remove are
handled by parser
no support for defaults
Avro
flexible,
strong typed
Append, Removal
supports defaults
Version is built in

28. Payload

29. public class Sensor {
private String id;
private float temperature;
private float velocity;
private float acceleration;
private float[] accelerometer;
private float[] magneticField;
private float[] orientation;
}
How much do you
weight?

30. 0
150
300
450
600
Payload Size (bytes)
514
237
93
Avro JSON XML

31. 0
1000
2000
3000
4000
Read Write
3,4833,433
1,3001,333
Avro JSON
How fast?

33. Features
✓ Compact
✓ Strongly typed
✓ Adaptable
✓ Versioned
✓ Centralized

34. ImplementationFinally we get to see something concrete

35. Spring Cloud Stream

37. Source
spring:
cloud:
stream:
bindings:
output:
destination: sensor-topic
contentType: “avro/binary”
W
OR
KIN
PR
OG
R
E
SS
Activates the converter

38. Avro Converter
• Scans the classpath for *.avsc
files and register them
• During writes, infer the schema
from payload (SpecificDatum,
GenericDatum, Reflection)
• During reads uses message
headers to discover the schema
being used

39. Source Sink
Content-Type: avro/binary
X-Schema-Name: acme.User
Headers

40. Avro Converter
- Demo -

41. Avro Converter
Good, but …

42. Avro Converter
• Each component still needs the
avsc file
• Avro versioning only works if
both writer and reader schemas
are available
• Transmitting the schema with the
message is an overhead

43. Schema registry
• Centralized store for schemas
• Idempotent registration (same
schema payload always return the
same id)
• Compatibility test
• Schema utilization

44. Idempotent
registration
Schema
Registry
user
user
user
version: 1
user
user
version: 2

45. Schema registry
• Allows developers to check if new
schemas can break existing ones in
the registry
• BACKWARD: new schema can read old
versions
• FORWARD: Old schema can read new
version
• FULL: BACKWARD + FORWARD

46. Schema utilization
{
"registrations" : [
{"application-name":"user-producer",
"type" : "source"
},
{"application-name":"user-enricher",
"type" : "processor"
},
{"application-name":"user-filter",
"type" : "processor"
}
]
}
GET /schemas/user/{version}

47. Schema registry
- Demo -

48. Sink
Content-Type: avro/binary
X-Schema-Id: 17
Headers
Writer’s schema
spring:
cloud:
stream:
bindings:
input:
destination: sensor-topic
schema: “org.acme.Sensor”
Reader’s schema

49. Source Processor Sink
1. Register and
obtain schema id
Payload
2. Reads headers
fetch writer’s
schema
Schema
Registry
Stream
Content-Type: avro/binary
X-Schema-Id: 17
Headers

50. References
• Martin Kleppmann Schema Evolution in avro, thrift and protobufers: https://
martin.kleppmann.com/2012/12/05/schema-evolution-in-avro-protocol-buffers-thrift.html
• http://dataintensive.net/ - Martin Kleppmann
• The CQRS Journey: https://msdn.microsoft.com/en-us/library/jj554200.aspx
• Oracle Datastore schema evolution : https://docs.oracle.com/cd/NOSQL/html/
GettingStartedGuide/schemaevolution.html
• Building Microservices by Sam Newman: http://samnewman.io/books/building_microservices/
• Apache Avro: https://avro.apache.org/docs/1.7.7/gettingstartedjava.html
• https://github.com/viniciusccarvalho/schema-evolution-samples