Recommended
More Related Content
What's hot
What's hot (12)
Similar to Wealthfront's Query Engine
Similar to Wealthfront's Query Engine (20)
Recently uploaded
Recently uploaded (20)
Wealthfront's Query Engine
- 1. Wealthfront’s Query Engine Service Framework and Standardized RPC julien@wealthfront.com March 8th, 2012
- 2. Roadmap • Query Engine • Queries • Services • Remote Query InvocaKon
- 3. The Query Engine is • a plaMorm to build distributed services quickly • designed with testability in mind • powering all of Wealthfront’s backend services • running on the JVM
- 4. The Query Engine is not • a web framework • a RESTful web service – a query is a funcKon, not a resource • Java-‐specific
- 5. Queries • First-‐class ciKzens – Queries can be passed around and later invoked • Serializable – Queries can be persisted • Closed with their dependencies • Composable units of work – Queries can invoke other queries – Queries can produce other queries • Entry points into our backend services – Services can invoke queries on other services – Queries can also be invoked from the command line
- 6. Queries (concretely) • Queries are classes • Constructors define input parameters • Instances are invokable using a driver • Dependencies can be requested at run-‐Kme • InvocaKons produce a result • Queries are easy to test
- 7. UNIX Processes stdin Environment Process stdout Return code stderr
- 8. Queries Arguments Dependencies Query Result ExcepKon
- 9. Invoking a Query Query Class Arguments Query Instance Dependencies Driver Result
- 10. Invoking a Query Query Instance Dependencies Run Scoping Monitoring Retrying TransacKng InjecKng Result
- 11. Services • A collecKon of Queries – Usually with a similar purpose, e.g. all the queries related to customer management • Able to saKsfy the dependencies required by its Queries – E.g. access to the customer database, connecKon to the NASDAQ NLS feed, … • Queries can be installed in different services
- 12. Remote Query InvocaKon • Queries are remotely invoked by doing an HTTP POST request • Arguments are encoded in the HTTP request • Results are returned in the HTTP response
- 13. Request SerializaKon • Most of our services rely on the so-‐called “qp0p1” serializaKon – The simple name of the invoked query is passed with the q parameter – The nth argument is passed with the pnth parameter – Arguments are serialized and de-‐serialized to and from strings using converters
- 14. Hello(@OpKonal(“World”) String): String Instan&a&on Serializa&on new Hello(“Bob”) q=Hello&p0=Bob new Hello(null) q=Hello
- 15. [julien@glados ~]$ curl um0:8085 -‐-‐data 'q=Hello&p0=Bob’ -‐v > POST / HTTP/1.1> Content-‐Length: 14 > Content-‐Type: applicaKon/x-‐www-‐form-‐urlencoded > > q=Hello&p0=Bob < HTTP/1.1 200 OK < X-‐KC-‐TraceToken: 962dcf36-‐9b25-‐4731-‐a022-‐e054b925637c < Server: kawala < Content-‐Length: 11 Hello, Bob!
- 16. Request De-‐serializaKon • Services’ request interpreters de-‐serialize the HTTP POST requests and create instances of the query class using the specified arguments • Then, they invoke the query instance using a driver
- 17. Request De-‐serializaKon HTTP POST request Request Interpreter Query Class Arguments Query Instance Driver
- 18. Smart Clients • Queries can be remotely invoked from Java code using smart clients • Smart clients analyze the bytecode to recover the arguments passed to the query constructor and generate the HTTP POST request • Smart clients also de-‐serialize the result from the HTTP response (see “Result SerializaKon”)
- 19. Smart Clients Query Class Arguments Query Instance Smart Client HTTP POST request
- 20. The Life of a Remote Request Client Server Query Class Arguments Query Instance Smart Client HTTP POST request Request Interpreter Query Class Arguments Query Instance Driver
- 21. Constructor Requirements • In order to be analyzable, query constructors must follow strict requirements – Basically, they should only assign their arguments to fields hwps://github.com/wealthfront/kawala/wiki/InstanKators
- 22. Sugar for the Constructor • @OpKonal(“2011-‐02-‐25”) LocalDate • @OpKonal(“false”) boolean • OpKon<LocalDate> • @PosiKve int age
- 23. Early ValidaKon • Arguments are validated when instanKaKng a serialized query thanks to the converters [julien@glados ~]$ ikq um0 GetUser foo HTTP Error 400: For input string: "foo” X-‐KC-‐TraceToken: 962dcf36-‐9b25-‐4731-‐a022-‐e054b925637c Server: kawala ConnecKon: close
- 24. Result SerializaKon • Results are usually serialized as – JSON – Protobuf • The service has the responsibility to serialize the result of a query with the proper serializaKon method • Smart clients rely on the same logic to select the proper de-‐serializaKon method
- 25. Result SerializaKon Return Type Addi&onal Constraints Serializa&on ConverKble type* JSON value** T T annotated with @EnKty JSON object List<T> T annotated with @EnKty JSON array Set<T> T annotated with @EnKty JSON array T extends Message Protobuf List<T extends Message> Protobuf array * As specified in KachingMarshallers ** JSON strings are returned without surrounding quotes
- 26. The Life of a Remote InvocaKon Client Server Query Class Request Interpreter Arguments Query Class Query Instance Arguments Query Instance Smart Client Driver Result Result Serializer
- 27. Query InvocaKon Summary • Local InvocaKon QueryExecutor executor = … User user = executor.submit(new GetUser(Id.<User> of(10)); • Remote InvocaKon SmartClient<UM> um = … User user = um.invoke(new GetUser(Id.<User> of(10));