Flink Forward San Francisco 2022. Next time you want to integrate with a new destination for a demo, concept or production application, the Async Sink framework will bootstrap development, allowing you to move quickly without compromise. In Flink 1.15 we introduced the Async Sink base (FLIP-171), with the goal to encapsulate common logic and allow developers to focus on the key integration code. The new framework handles things like request batching, buffering records, applying backpressure, retry strategies, and at least once semantics. It allows you to focus on your business logic, rather than spending time integrating with your downstream consumers. During the session we will dive deep into the internals to uncover how it works, why it was designed this way, and how to use it. We will code up a new sink from scratch and demonstrate how to quickly push data to a destination. At the end of this talk you will be ready to start implementing your own Flink sink using the new Async Sink framework. by Steffen Hausmann & Danny Cranmer

1. ONE SINK TO RULE THEM ALL: INTRODUCING THE NEW ASYNC SINK
© 2022, Amazon Web Services, Inc. or its affiliates.
© 2022, Amazon Web Services, Inc. or its affiliates.
One sink to rule them all
Introducing the new Async Sink
Danny Cranmer
Sr. Software Development Engineer, AWS
Steffen Hausmann
Principal Streaming Architect, AWS

2. ONE SINK TO RULE THEM ALL: INTRODUCING THE NEW ASYNC SINK
© 2022, Amazon Web Services, Inc. or its affiliates.
Apache Flink’s connector ecosystem
2
Amazon Kinesis
Data Streams
Elasticsearch
RabbitMQ
Google PubSub

3. ONE SINK TO RULE THEM ALL: INTRODUCING THE NEW ASYNC SINK
© 2022, Amazon Web Services, Inc. or its affiliates.
How to build a high quality connector
Buffering Send batch
requests
Retry failed
requests
Checkpointing Rate limiting and
Backpressure

4. ONE SINK TO RULE THEM ALL: INTRODUCING THE NEW ASYNC SINK
© 2022, Amazon Web Services, Inc. or its affiliates.
Async Sink internals
internal buffer
endpoint
message request entries
batch request
response
requeue
failed requests
checkpoint
sink to
AsyncSinkWriter
request entries -> batch request
ElementConverter
message -> request entry
AsyncSinkWriter
response -> failed requests
AsyncSink
WriterStateSerializer
4

5. ONE SINK TO RULE THEM ALL: INTRODUCING THE NEW ASYNC SINK
© 2022, Amazon Web Services, Inc. or its affiliates.
How the Async Sink manages throughput
6
endpoint
sink subtask3
sink subtask2
sink subtask1
1. Limit the number of in-flight request entries
2. If a batch request was successful, increase
the limit by a constant factor
3. If a batch request failed, cut the limit in half
Additive Increase/Multiplicative Decrease

6. ONE SINK TO RULE THEM ALL: INTRODUCING THE NEW ASYNC SINK
© 2022, Amazon Web Services, Inc. or its affiliates.
© 2022, Amazon Web Services, Inc. or its affiliates.
Let’s go build!
7

7. ONE SINK TO RULE THEM ALL: INTRODUCING THE NEW ASYNC SINK
© 2022, Amazon Web Services, Inc. or its affiliates.
Async Sink configuration
endpoint
message request entries
batch request
sink to
maxRecordSizeInBytes
8
maxInFlightRequests
maxBatchSize
maxBatchSizeInBytes
maxBufferedRequests
maxTimeInBufferMs
internal buffer

8. ONE SINK TO RULE THEM ALL: INTRODUCING THE NEW ASYNC SINK
© 2022, Amazon Web Services, Inc. or its affiliates.
Current limitations
At least
once
Ordering Thread pool
management

9. ONE SINK TO RULE THEM ALL: INTRODUCING THE NEW ASYNC SINK
© 2022, Amazon Web Services, Inc. or its affiliates.
What’s Next?
FLIP-242
Configurable
Rate Limiting
FLIP-252
Amazon DynamoDB
Sink
Wider
Community
Adoption

10. ONE SINK TO RULE THEM ALL: INTRODUCING THE NEW ASYNC SINK
© 2022, Amazon Web Services, Inc. or its affiliates.
Thank you!
© 2022, Amazon Web Services, Inc. or its affiliates.
Danny Cranmer Steffen Hausmann