Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
Cameron Waeland
Software Engineer
Rethinking CodeGen:
IDL, Thrift, gRPC, Ohh My…
InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese an...
Presented at QCon New York
www.qconnewyork.com
Purpose of QCon
- to empower software development by facilitating the sprea...
12 Months Ago -
Adding a new API Endpoint
Lots of steps
and coordination
across the stack
Difficult to
enforce standards
Pr...
Old API
Framework Issues
• Business logic spread
across Python API and
Java backend services
• Maintenance difficult for
de...
Fast Forward 12 Months
Add a method 

to the relevant
Thrift IDL.
I'm a Compass developer and I want to add an endpoint:
1...
Cameron Waeland
14°32´55˝ N, 20°59´35˝ E
Software Engineer APIs
• Leads API development at Compass
• Developer frameworks ...
COMPASS COMPANY OVERVIEW |
2017
Our
Nationwide
Network
1600+
Agents in
30
offices across
10
major regions
East
West
Boston
...
COMPASS COMPANY OVERVIEW |
2017
7
Our Technology
The Compass technology suite
empowers agents to become trusted
advisors t...
COMPASS COMPANY OVERVIEW |
2017
Drive Toward
Market Leadership
$180M
Compass nearly tripled its
2016 revenue year-over-yea...
CodeGen
is a valuable tool
and less difficult
than you think
AGENDA
Birth of a CodeGen Framework
Code Generating an API
Going Further
Thrift to gRPC with CodeGen
What’s a Remote
Procedure Call?
• Interservice
communication over a
network
• Call remote method as
though it were local
•...
Thrift @Compass
Developed by Facebook,
open sourced via Apache
Interfaces defined using
Interface Description Language
• De...
Go Code Generator…
…or a gopher appears
—
Parallel initiative to add Go for backend services.
We were excited by Go’s:
• C...
Common Code Gen 

Misconceptions
—
• High level of complexity
• Produces messy unreadable code
• Poorly performing code
• ...
Thrift to Go Generator
—
• Used go-thrift - Thrift parser
• Library generates Abstract Syntax Trees
• ASTs provide Thrift ...
Code Generation Pipeline
—
• Built Go template files compatible with the AST
generated.
• With template files and ASTs we we...
AST Model
• Using a Parsing
Expression Grammar
(PEG) the file is broken
down into key sections
• Set of rules for string
re...
Looking at a Struct
AST
BIRTH OF CODEGEN FRAMEWORK
"Blog": {
"Name": "Blog",
"Fields": [
{
"ID": 1,
"Name": "id",
"Type": ...
Go Templates for
Generation
BIRTH OF CODEGEN FRAMEWORK
type Blog struct {
ID *int32 `thrift:"1" json:"id,omitempty" param:...
Dispelling the

Misconceptions
—
Thrift's clear syntax:
• Straightforward templates
• Create clean and performant code
• A...
AGENDA
Birth of a CodeGen Framework
Code Generating an API
Going Further
Thrift to gRPC with CodeGen
Motivation for a New
API Framework
—
• Monolithic Python app
• New endpoints built ad hoc by frontend devs
• Business logi...
Where did
we start?
• Consistency - validation,
request/response formats,
URL patterns
• Discoverability -
complete, accur...
A Common Refrain
—
How does this endpoint work?
Check the Thrift...
What does this parameter do?
Look at the Thrift...
Is ...
General Approach
—
• Thrift emphasis - should be source of truth for any API
• Now have access to a Thrift CodeGen framewo...
The Prototypes
—
• 2 reverse proxy prototypes generated from Thrift
• First would be a more traditional JSON REST API
• Se...
What We Chose
—
Decision: JSON REST API
As much as we tried to justify using GraphQL…
• Risk mitigation - easier migration...
Three Key Aspects of an 

API CodeGen Framework
Source of Truth
In our case Thrift, but could be
database model, object mo...
1. Source of Truth
—
• Already established that Thrift is our source of truth
• For an HTTP API we were just short of all ...
Annotations
• Language feature of Thrift
• Applied to services,
methods, structs and fields
• Similar to Java - control
par...
2. Request Context
—
• URL route + HTTP method to service method binding
• Endpoint authorization
• Python API style conve...
Injectors
• Introduced special Thrift
annotation for struct fields
• API will inject relevant
information into annotated
fie...
Key Benefits
—
1. Reduces logic and custom code in Python mono app.
2. Thrift files are “source of truth” for all API intera...
AGENDA
Birth of a CodeGen Framework
Code Generating an API
Going Further
Thrift to gRPC with CodeGen
Code Generating
Clients
• Improve consistency for
frontend developers
• Clients define and maintain
their own templates
• U...
Code Generating Documentation
—
• We added a documentation template
• The output is RESTful API Modeling Language
(RAML)
•...
Key Benefits
—
1. Reduces logic and custom code in Python mono app.
2. Thrift files are “source of truth” for all API intera...
AGENDA
Birth of a CodeGen Framework
Code Generating an API
Going Further
Thrift to gRPC with CodeGen
Introducing gRPC @Compass
—
• Increased commitment to microservices
• Started to re-evaluate our usage of Thrift
• Experie...
gRPC Advantages
—
• gRPC fills same role as Thrift
• RPC framework for polyglot architectures via CodeGen
• However it also...
Code Generated Migration
—
• A problem - gRPC recommends protobuf as its IDL
• Compass has a substantial number of Thrift ...
gRPC Stub Example
—
Code generated client/server extensions in target languages
public class BlogServiceGRPC extends com.u...
Where are we today?
Go API + gRPC active
in production -
mobile/web clients
Go API is
deployed as a
single service
Backend...
New Architecture
—
• All requests reverse proxied through Go API
• App services are annotated and available to clients
• C...
Where should we go next?
Future goal - deploy
an instance of API v3
per service
Service change only
affects its API v3
ins...
Future Infrastructure Diagram
Other Future Opportunities
—
Robust JavaScript (or TypeScript) SDK
• Leverage Thrift type information in our JS
• Give our...
Wrapping Up
• CodeGen is a valuable tool
• Cut 2 major time consuming steps for our devs
• We migrated to a new RPC framew...
Rapidly Growing and Hiring for multiple positions
www.compass.com/careers/jobs/
Thanks!
Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
compass-rest-grpc
Rethinking CodeGen: IDL, Thrift, gRPC, Ohh My
Rethinking CodeGen: IDL, Thrift, gRPC, Ohh My
Rethinking CodeGen: IDL, Thrift, gRPC, Ohh My
Rethinking CodeGen: IDL, Thrift, gRPC, Ohh My
Rethinking CodeGen: IDL, Thrift, gRPC, Ohh My
Prochain SlideShare
Chargement dans…5
×

Rethinking CodeGen: IDL, Thrift, gRPC, Ohh My

1 052 vues

Publié le

Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2x2wBdf.

Cameron Waeland discusses the evolution of Compass - both the technology stack and the migration process to get where they are. He presents their extensible code generation framework which is at the heart of their automatically generated REST to gRPC reverse proxy and how they've been able to leverage it to empower their client and service developers. Filmed at qconnewyork.com.

Cameron Waeland works as a Software Engineer at Compass, where he has been leading API development for the past year and a half. Prior to Compass, he worked at JW Player where he was head of platform architecture building both video player and platform solutions instrumental in driving adoption of the JW Player.

Publié dans : Technologie
  • Soyez le premier à commenter

Rethinking CodeGen: IDL, Thrift, gRPC, Ohh My

  1. 1. Cameron Waeland Software Engineer Rethinking CodeGen: IDL, Thrift, gRPC, Ohh My…
  2. 2. InfoQ.com: News & Community Site • 750,000 unique visitors/month • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • News 15-20 / week • Articles 3-4 / week • Presentations (videos) 12-15 / week • Interviews 2-3 / week • Books 1 / month Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ compass-rest-grpc
  3. 3. Presented at QCon New York www.qconnewyork.com Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide
  4. 4. 12 Months Ago - Adding a new API Endpoint Lots of steps and coordination across the stack Difficult to enforce standards Proliferation of similar APIs
  5. 5. Old API Framework Issues • Business logic spread across Python API and Java backend services • Maintenance difficult for developers • Python API coupled to backend • Numerous ad hoc endpoints • Adding an HTTP API falls to service developer INTRODUCTION
  6. 6. Fast Forward 12 Months Add a method 
 to the relevant Thrift IDL. I'm a Compass developer and I want to add an endpoint: 1. Implement the method in a backend service. 2. Run a command. 3. Get a beer. 4. a. Generate APIs b. Generate SDKs c. Generate Documentation
  7. 7. Cameron Waeland 14°32´55˝ N, 20°59´35˝ E Software Engineer APIs • Leads API development at Compass • Developer frameworks and tooling • Service development/architecture
  8. 8. COMPASS COMPANY OVERVIEW | 2017 Our Nationwide Network 1600+ Agents in 30 offices across 10 major regions East West Boston New York City The Hamptons Washington DC Miami San Francisco Santa Barbara & Montecito Los Angeles Orange County Aspen
  9. 9. COMPASS COMPANY OVERVIEW | 2017 7 Our Technology The Compass technology suite empowers agents to become trusted advisors to their clients, build their network and grow their business. We provide value through technology and data to our agents so they can help their clients make smart real estate decisions. Advanced Search Agent Mobile App Listing Strategy Markets App Network Performance Metrics Valuation Insights Deal Closer Documents Listing Presentation Showsheets Toursheets Efficiency Collections Open House App Client Portal Relationship Building
  10. 10. COMPASS COMPANY OVERVIEW | 2017 Drive Toward Market Leadership $180M Compass nearly tripled its 2016 revenue year-over-year to more than 98% Our technology and agent support has led Compass to achieve industry- leading retention of #1 #1 TOP 5 Compass is ranked as #1 overall brokerage in Washington D.C. Compass is ranked as the #1 brokerage in single-family home sales in San Francisco Compass is a top 5 brokerage by market share in New York, San Francisco, Los Angeles, Miami, D.C. and Boston
  11. 11. CodeGen is a valuable tool and less difficult than you think
  12. 12. AGENDA Birth of a CodeGen Framework Code Generating an API Going Further Thrift to gRPC with CodeGen
  13. 13. What’s a Remote Procedure Call? • Interservice communication over a network • Call remote method as though it were local • Low overhead, binary format • E.g. Thrift, GRPC, Avro HTTP typically used for external requests ADDING GO SUPPORT FOR THRIFT SERVICES
  14. 14. Thrift @Compass Developed by Facebook, open sourced via Apache Interfaces defined using Interface Description Language • Define services, methods, types, etc. • Inputs to a code generation tool • Clients and servers generated for each service in a target language. • Thrift IDL files are similar to Protobuf struct Blog { 1: optional i32 id; // The unique identifier of the blog post 2: optional i32 ownerId; // The userId of the post creator 3: optional string title; // The title of the blog post 4: optional string content; // The content of the blog post } struct ViewPostRequest { 1: optional i32 blogId; } struct ViewPostResponse { 1: optional base.ResponseStatus status; 2: optional Blog post; } struct CreatePostRequest { 1: optional i32 userId; // The user creating the new post 2: optional Blog blog; } struct CreatePostResponse { 1: optional base.ResponseStatus status; 2: optional Blog blog; } service BlogService { ViewPostResponse viewPost(1: ViewPostRequest request); CreatePostResponse createPost(1: CreatePostRequest request); } Thrift IDL • Similar to Protobuf Blog Service • Exposes a read method and a write method • Single model object • Basic request/ response structs ADDING GO SUPPORT FOR THRIFT SERVICES
  15. 15. Go Code Generator… …or a gopher appears — Parallel initiative to add Go for backend services. We were excited by Go’s: • Concurrency primitives & performance • Static typing & unambiguous syntax. But: • Support in the official Thrift project was poor. • We needed to build our own generator for Thrift -> Go
  16. 16. Common Code Gen 
 Misconceptions — • High level of complexity • Produces messy unreadable code • Poorly performing code • Not as good as handwritten code
  17. 17. Thrift to Go Generator — • Used go-thrift - Thrift parser • Library generates Abstract Syntax Trees • ASTs provide Thrift file contents in a structured way - Enums, Structs, Services, Methods, etc. • Think compilers but not as scary!
  18. 18. Code Generation Pipeline — • Built Go template files compatible with the AST generated. • With template files and ASTs we were able to render out idiomatic Go code. • Generated files are compiled into a compass-thrift binary for use by services.
  19. 19. AST Model • Using a Parsing Expression Grammar (PEG) the file is broken down into key sections • Set of rules for string recognition (think regexes) • Choice operator selects the first match • The file is represented as a tree using Go data structures. • Field & method names, type information become easily accessible. BIRTH OF CODEGEN FRAMEWORK
  20. 20. Looking at a Struct AST BIRTH OF CODEGEN FRAMEWORK "Blog": { "Name": "Blog", "Fields": [ { "ID": 1, "Name": "id", "Type": {"Name": "i32"} }, { "ID": 2, "Name": "ownerId", "Type": {"Name": "i32"} }, { "ID": 3, "Name": "title", "Type": {"Name": "string"} }, { "ID": 4, "Name": "content", "Type": {"Name": "string"} } ] } struct Blog { 1: optional i32 id; // The unique identifier of the blog post 2: optional i32 ownerId; // The userId of the post creator 3: optional string title; // The title of the blog post 4: optional string content; // The content of the blog post }
  21. 21. Go Templates for Generation BIRTH OF CODEGEN FRAMEWORK type Blog struct { ID *int32 `thrift:"1" json:"id,omitempty" param:"id" ` OwnerID *int32 `thrift:"2" json:"ownerId,omitempty" param:"ownerId" ` Title *string `thrift:"3" json:"title,omitempty" param:"title" ` Content *string `thrift:"4" json:"content,omitempty" param:"content" ` } type {{$name|Export}} struct { type {{$name|Export}} struct { {{range . -}} {{.Name|Export}} {{if .|ShouldDereference}}*{{end}}{{.|FieldType}} ` + "`thrift:"{{.ID}}{{if .Type|IsSet}},set{{end}}"" + " json:"{{.Name}},omitempty"" + " param:"{{.Name}}"" + " {{if Injectable .}}inject:"{{(Injectable .)}}"{{end -}}`" + ` {{end -}} } }
  22. 22. Dispelling the
 Misconceptions — Thrift's clear syntax: • Straightforward templates • Create clean and performant code • Add the features we want
  23. 23. AGENDA Birth of a CodeGen Framework Code Generating an API Going Further Thrift to gRPC with CodeGen
  24. 24. Motivation for a New API Framework — • Monolithic Python app • New endpoints built ad hoc by frontend devs • Business logic spread across services and Python app • Inconsistent & difficult to maintain
  25. 25. Where did we start? • Consistency - validation, request/response formats, URL patterns • Discoverability - complete, accurate and generated API documentation • Productivity - generated API clients, insights (performance, errors, availability) CODE GENERATING AN API Requirements Gathering! Group discussions/ interviews with client/service developers. Key requirements: Thrift RPC 
 Everywhere Python App REST Proxy for Backend Services Go Code Generator Available Compass Tech During Design Phase
  26. 26. A Common Refrain — How does this endpoint work? Check the Thrift... What does this parameter do? Look at the Thrift... Is there any documentation? Read the Thrift… A pattern emerges… Thrift is the source of truth
  27. 27. General Approach — • Thrift emphasis - should be source of truth for any API • Now have access to a Thrift CodeGen framework • Let's try an API framework generated directly from Thrift • HTTP to Thrift reverse proxy • Go Application - simpler to extend already generated code
  28. 28. The Prototypes — • 2 reverse proxy prototypes generated from Thrift • First would be a more traditional JSON REST API • Second would be a more experimental GraphQL API
  29. 29. What We Chose — Decision: JSON REST API As much as we tried to justify using GraphQL… • Risk mitigation - easier migration of clients • Engineer familiarity with REST • Relative immaturity of GraphQL
  30. 30. Three Key Aspects of an 
 API CodeGen Framework Source of Truth In our case Thrift, but could be database model, object model, etc. 1. Request Context 2. Deployment, Operations & Developer Tooling 3. HTTP, session info, etc. Supporting development Performance monitoring, high availability, etc.
  31. 31. 1. Source of Truth — • Already established that Thrift is our source of truth • For an HTTP API we were just short of all the info we would need • The "Aha!" moment - Thrift annotations
  32. 32. Annotations • Language feature of Thrift • Applied to services, methods, structs and fields • Similar to Java - control parser/compiler behavior • Specify arbitrary key/value metadata • Small modification to parser to support CODE GENERATING AN API service BlogService extends base.BaseService { ViewPostResponse viewPost(1: ViewPostRequest request) (api.url="/blog/:blogId", api.method="GET"); CreatePostResponse createPost(1: CreatePostRequest request) (api.url="/blog", api.method="POST", api.roles="Editor"); } (api.url="/blog") Example • Top level URL for the service • Per-method URLs including resource IDs • HTTP request methods • Endpoint permission requirements
  33. 33. 2. Request Context — • URL route + HTTP method to service method binding • Endpoint authorization • Python API style convenience features • Current user • File uploads • User permissions • HTTP headers • etc.
  34. 34. Injectors • Introduced special Thrift annotation for struct fields • API will inject relevant information into annotated fields. • (api.path) • The value of ":blogId" as defined in the path will be injected into this field. • (api.inject="userId") • ID of requesting user will injected into this field. CODE GENERATING AN API struct ViewPostRequest { 1: optional i32 blogId (api.path); } struct ViewPostResponse { 1: optional base.ResponseStatus status; 2: optional Blog post; } struct CreatePostRequest { 1: optional i32 userId (api.inject="userId"); 2: optional Blog blog; } struct CreatePostResponse { 1: optional base.ResponseStatus status; 2: optional i64 result; }
  35. 35. Key Benefits — 1. Reduces logic and custom code in Python mono app. 2. Thrift files are “source of truth” for all API interactions. 3. Consistent & predictable API. 4. Improved developer productivity

  36. 36. AGENDA Birth of a CodeGen Framework Code Generating an API Going Further Thrift to gRPC with CodeGen
  37. 37. Code Generating Clients • Improve consistency for frontend developers • Clients define and maintain their own templates • Use structure or frameworks that they want • API team doesn't need to maintain them GOING FURTHER
  38. 38. Code Generating Documentation — • We added a documentation template • The output is RESTful API Modeling Language (RAML) • Supports custom extensions and complex objects in GET. • Updated our parser to surface Thrift comments • Hosted and easily accessible to developers
  39. 39. Key Benefits — 1. Reduces logic and custom code in Python mono app. 2. Thrift files are “source of truth” for all API interactions. 3. Consistent & predictable API. 4. Automatically generated client code - clients instantly get backend changes 5. Improved developer productivity

  40. 40. AGENDA Birth of a CodeGen Framework Code Generating an API Going Further Thrift to gRPC with CodeGen
  41. 41. Introducing gRPC @Compass — • Increased commitment to microservices • Started to re-evaluate our usage of Thrift • Experienced several frustrations: • Low quality Python Thrift Server • Reliability of connections • General lack of Thrift development • Investigated gRPC by Google after it was open sourced.
  42. 42. gRPC Advantages — • gRPC fills same role as Thrift • RPC framework for polyglot architectures via CodeGen • However it also provides: • Bi-directional streaming using http/2 • Customizability - pluggable auth, tracing, load balancing and health checking • Built-in connection retrying • Efficient, idiomatic clients and servers.
  43. 43. Code Generated Migration — • A problem - gRPC recommends protobuf as its IDL • Compass has a substantial number of Thrift files • gRPC pluggability + CodeGen framework • Migrate and keep our Thrift files • Wrote our own Thrift codec which we plugged into gRPC.
  44. 44. gRPC Stub Example — Code generated client/server extensions in target languages public class BlogServiceGRPC extends com.urbancompass.common.base.BaseServiceGRPC { private static final Empty __EMPTY = new Empty(); private static final MethodDescriptor<CreatePostRequest, CreatePostResponse> __METHOD_DESCRIPTOR_CREATE_POST = MethodDescriptor.create( io.grpc.MethodDescriptor.MethodType.UNARY, generateFullMethodName("BlogService", "createPost"), new ThriftMarshaller<>(CreatePostRequest.class), new ThriftMarshaller<>(CreatePostResponse.class) ); private static final MethodDescriptor<ViewPostRequest, ViewPostResponse> __METHOD_DESCRIPTOR_VIEW_POST = MethodDescriptor.create( io.grpc.MethodDescriptor.MethodType.UNARY, generateFullMethodName("BlogService", "viewPost"), new ThriftMarshaller<>(ViewPostRequest.class), new ThriftMarshaller<>(ViewPostResponse.class) );
  45. 45. Where are we today? Go API + gRPC active in production - mobile/web clients Go API is deployed as a single service Backend service changes require regen and deployment
  46. 46. New Architecture — • All requests reverse proxied through Go API • App services are annotated and available to clients • Core/Internal services only accessed via App services
  47. 47. Where should we go next? Future goal - deploy an instance of API v3 per service Service change only affects its API v3 instance to regen Each Go API instance registers with load balancer/reverse proxy
  48. 48. Future Infrastructure Diagram
  49. 49. Other Future Opportunities — Robust JavaScript (or TypeScript) SDK • Leverage Thrift type information in our JS • Give our developers convenient static analysis Moving to Protobuf • Protobuf is the standard IDL format for gRPC • Standardize our stack - leverage more of the ecosystem 
 Evaluating GRPC-WEB • New protocol - web clients speak with gRPC using JSON over HTTP • Eventually - browsers can use native gRPC over HTTP/2 via upcoming whatwg fetch/streams API
  50. 50. Wrapping Up • CodeGen is a valuable tool • Cut 2 major time consuming steps for our devs • We migrated to a new RPC framework • Generated an API from a single source of truth • It is easier than you think • Straightforward to build • Easy to extend to new applications CodeGen at Compass has allowed us to rapidly transform our stack over the last year.
  51. 51. Rapidly Growing and Hiring for multiple positions www.compass.com/careers/jobs/ Thanks!
  52. 52. Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ compass-rest-grpc

×