Discussion on tools to work with OpenAPI

1. Slicing, Dicing, and
Linting OpenAPI
Go Conference 2018 Autumn
Nov 25, 2018
Daisuke Maki (@lestrrat) / HDE Inc.

2. • @lestrrat
• Perl/Go hacker, author, father
• Author of github.com/peco/peco
• Organizer for builderscon

3. tl;dr
• github.com/lestrrat-go/openapi
• API designed for spec consumer
• Safety + Predictability
• Tools to work with specs

4. OpenAPI

5. Who _Knows_ OpenAPI?

6. …in its entirety?

7. Specification for
REST services

8. 「んー僕もopenapiへの恨み言は、
いくらでも出てきそうだ」
「別名、SOAP 2nd generationだからなー 」
What They Are Saying
“I think I can go all night with
complaints about OpenAPI”
“It’s also known as SOAP the
Next Generation”

9. JSON or YAML

11. OpenAPI v2
OpenAPI v3

12. OpenAPI v2

13. OpenAPI v3

14. “I don’t understand any of it: We’re just pretending to understand OpenAPI"

15. My Goals
• Use Cloud Endpoints (GCP)
• Generate gRPC server/client
• Generate ES6/Golang REST client
• Write definitions in ONE file

16. Cloud Endpoints

17. gRPC
…because it’s easier to write the server

18. REST Clients
…because the web isn’t ready for gRPC-only

19. LB
openapi
endpoints-runtime
use config=2018-10-01r1
fetch configuration
(sidecar)
HTTP2 protobuf
protobuf
Endpoints config
generated
gRPC Server
generated (stub)
HTTP/1.1
JSON payload
(REST in Go or ES6)
generated
Cloud Endpoints
register
versioned configurations
config 2018-10-01r0
config 2018-10-01r1
config 2018-10-04r0
…

20. Support

21. “I wish the ecosystem/rest of the world would support v3”

22. OpenAPI 3.0.2
OAI Specification

23. OpenAPI 2.0
Google Cloud

24. hmmm…

25. !!!!
AWS

26. …

27. (look at the bright side: we’re going
to need tools to do v2 → v3)

28. Prior Art (1)

29. Gripes

30. Gripes
• Already tried tinkering w/ it
• Only generates protobuf
• Does NOT check correctness

31. Prior Art (2)

32. Gripes
• Where’s v3?
• How do I generate custom code?
• Does NOT check correctness

33. Where’s v3?

34. Sooner or later
we will need to move to v3

35. github.com/go-openapi/spec3

36. hmm…

37. Ensuring Correctness

38. Remember?

39. “I don’t understand any of it: We’re just pretending to understand OpenAPI"

40. We will make mistakes
surely
(really horrible schemas)

41. API should defend us
from making mistakes

42. everything is exported
(no defense against invalid values)
completely relies on encoding/json
(must match how encoding/json works, not necessarily how
the user wants to code)

43. required fields, but can’t tell

44. _Should_ scream FAIL
spec := openapi.Swagger{
Version: “3.0.2”, // Wrong Version
// Missing Info
// Missing Paths
}

45. bonus: Processing paths
• Need context to process leaves
• e.g. need PathItem.Path while
processing Operation objects

46. paths:
/pets:
get:
description: …
operationId: findPets
parameters:
- name: tags
in: query
required: false
…
- name: limit
in: query
required: false
type: integer
format: int32
responses:
200: …
Sample Spec
Paths object
PathItem object
Operation object
Operation object by itself does
not know the HTTP verb it’s
associated with

47. func processOperation(verb string, oper Operation) {
…
}
Currying is, meh…
• You have to remember what to curry
• API signature is no longer standardized
across methods

48. bonus: concurrent updates

49. _Should_ be safe
go func() { spec.Info.Title = “foo” }()
go func() { spec.Info.Title = “bar” }()
• Exposing always leaves a way for
users to screw up

50. Yak Shaving Time

51. github.com/lestrrat-go/openapi

52. github.com/lestrrat-go/openapi
• Model: v2 usable, v3 experimental
• Generation: gRPC, ES6+flow, Go
generation: usable
• Lint: usable

53. General Direction
• Don’t allow users to screw up
• Provide API to easily consume data
• Provide Linting/Code Generation

54. Protection from Errors

55. // github.com/lestrrat-go/openapi
type OpenAPI = Swagger
type Swagger interface { … }
spec := Swagger{} // Can’t do it
Prohibited Direct Instantiation

56. spec, err := openapi.NewSwagger([ mandatory params ]).
BasePath(…). // Optional params
…
Build() // Finally, build the object
Object Creation via Builder

57. Object Creation API
✓ Can’t instantiate invalid objects without
proper initialization
✓ Required/Optional parameters can be
distinguished visually
✓ Automatic validation

58. err := openapi.MutateSwagger(spec).
Info(…). // Any values that needs changing
…
Apply() // Finally, mutate the object
Object Mutation via Mutator

59. Object Mutation API
✓ Can’t mutate objects into invalid state
✓ Consistency can be controlled across
same spec tree

60. Data Access API

61. Data Traversal
• OpenAPI is a tree structure
• You need to traverse through the tree
• … and you need to know the context of
the current node

62. Iterating
• Giving access to raw slices and maps
could mean danger
• Allow access to objects, but no
mutation

63. for pIter := paths.Paths(); pIter.Next(); {
pathName, pathItem := pIter.Item()
for piIter := pathItem.Operations(); piIter.Next(); {
oper := piIter.Item()
}
}
Consistent Iterator API

64. Context
• HTTP verb gives context to Operation
object
• But the data is far from the Operation
object
• Currying is, meh…

65. verb := oper.Verb() // NOT part of OpenAPI spec
Utility Accessors
✓ Creation/Mutation API ensures consistency
with context
✓ No need to curry extra parameters

66. Linting

67. Linting is a MUST
• OpenAPI is too hard
• Programs should tell us of
inconsistencies

68. oalint
go get github.com/lestrrat-go/openapi/cmd/oalint
✓ Checks for semantic errors
✓ Formats output

69. finch% git diff spec
…
--- a/spec/examples/v2.0/petstore-expanded.yaml
+++ b/spec/examples/v2.0/petstore-expanded.yaml
@@ -1,4 +1,4 @@
-swagger: "2.0"
+swagger: "2.0.1"
info:
version: 1.0.0
title: Swagger Petstore
Sample Spec

70. finch% ./oalint -file spec/examples/v2.0/petstore-
expanded.yaml
2018/11/15 09:24:59 failed to parse input: failed to validate
spec: failed to visit Swagger element: swagger field must be
"2.0" (got "2.0.1")
Catches Errors!

71. Code Generation
(caveat emptor: needs polishing)

72. oagen protobuf
-package myapp
-output api.proto
-annotate
-global-option go_package=pb
api.yaml
Create gRPC protobuf

73. oagen rest client
-target=go
-directory=/path/to/myapp
api.yaml
Create Go REST Client

74. oagen restclient
-target=es6flow
-directory=/path/to/myapp
api.yaml
Create ES6(+flow) REST Client

75. # This does not yet exist
oagen v2tov3 api.yaml
(Future) v2tov3

76. Recap

77. LB
openapi
endpoints-runtime
use config=2018-10-01r1
fetch configuration
(sidecar)
HTTP2 protobuf
protobuf
Endpoints config
generated
gRPC Server
generated (stub)
HTTP/1.1
JSON payload
(REST in Go or ES6)
generated
Cloud Endpoints
register
versioned configurations
config 2018-10-01r0
config 2018-10-01r1
config 2018-10-04r0
…

78. Current Status
• Working gRPC/Go REST client
• ES6 client should work
• V3 is mostly done
• v2tov3 is still in my head

79. Questions?