Dealing with Cultural Dispersion — Stefano Lambiase — ICSE-SEIS 2024
How to Avoid Common Mistakes When Using Reactor Netty
1. How to Avoid Common Mistakes
When Using Reactor Netty
September 2–3, 2020
springone.io
session-how-to-avoid-common-mistakes-when-using-reactor-netty
1
55. 55
@Component
public class MyNettyWebServerCustomizer
implements WebServerFactoryCustomizer<NettyReactiveWebServerFactory> {
@Override
public void customize(NettyReactiveWebServerFactory factory) {
factory.addServerCustomizers(httpServer -> httpServer.wiretap(true));
}
}
Server Side - Wire Logging
56. 56
@Component
public class MyNettyWebServerCustomizer
implements WebServerFactoryCustomizer<NettyReactiveWebServerFactory> {
@Override
public void customize(NettyReactiveWebServerFactory factory) {
factory.addServerCustomizers(httpServer -> httpServer.wiretap(true));
}
}
Wire logging
Server Side - Wire Logging
57. Client Side - Wire Logging
var httpClient = HttpClient.create()
.wiretap(true);
WebClient.builder()
.clientConnector(new ReactorClientHttpConnector(httpClient))
.build();
57
58. 58
Client Side - Wire Logging
var httpClient = HttpClient.create()
.wiretap(true);
WebClient.builder()
.clientConnector(new ReactorClientHttpConnector(httpClient))
.build();
6
Wire logging
68. LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records:
Created at:
io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:363)
io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:139)
io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:114)
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:147)
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.base/java.lang.Thread.run(Thread.java:832)
Memory Leaks
68
70. LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records:
#1:
Hint: [id: 0x2c17d156, L:/127.0.0.1:62487 - R:localhost/127.0.0.1:8080] Buffered ByteBufHolder in Inbound Flux Queue
io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:86)
io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:25)
reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:357)
reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:373)
reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:686)
reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:94)
...
#2:
Hint: 'reactor.right.reactiveBridge' will handle the message from this point.
io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:86)
io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:25)
io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
...
Memory Leaks
70
71. LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records:
#1:
Hint: [id: 0x2c17d156, L:/127.0.0.1:62487 - R:localhost/127.0.0.1:8080] Buffered ByteBufHolder in Inbound Flux Queue
io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:86)
io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:25)
reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:357)
reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:373)
reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:686)
reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:94)
...
#2:
Hint: 'reactor.right.reactiveBridge' will handle the message from this point.
io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:86)
io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:25)
io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
...
Connection ID
Memory Leaks
71
72. LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records:
#1:
Hint: [id: 0x2c17d156, L:/127.0.0.1:62487 - R:localhost/127.0.0.1:8080] Buffered ByteBufHolder in Inbound Flux Queue
io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:86)
io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:25)
reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:357)
reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:373)
reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:686)
reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:94)
...
#2:
Hint: 'reactor.right.reactiveBridge' will handle the message from this point.
io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:86)
io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:25)
io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
...
Incoming Data Buffered
Memory Leaks
72
73. 10:55:23.820 ... [ctor-http-nio-9] ... [id: 0x2c17d156, L:/XXX - R:YYY] Channel acquired, now 500 active connections
and 0 inactive connections
10:55:23.820 ... [ctor-http-nio-9] ... [id: 0x2c17d156, L:/XXX - R:YYY] Handler is being applied: {uri=http://
localhost:8080/, method=GET}
10:55:23.821 ... [ctor-http-nio-9] ... [id: 0x2c17d156, L:/XXX - R:YYY] Received response (auto-read:false) : [Content-
Type=text/html;charset=UTF-8, Transfer-Encoding=chunked, Date=Sun, 30 Aug 2020 07:55:23 GMT,
Connection=close]
10:55:23.821 ... [ctor-http-nio-9] ... [id: 0x2c17d156, L:/XXX - R:YYY] Received last HTTP packet
Hint: [id: 0x2c17d156, L:/XXX - R:YYY] Buffered ByteBufHolder in Inbound Flux Queue
Memory Leaks
73
74. 10:55:23.820 ... [ctor-http-nio-9] ... [id: 0x2c17d156, L:/XXX - R:YYY] Channel acquired, now 500 active connections
and 0 inactive connections
10:55:23.820 ... [ctor-http-nio-9] ... [id: 0x2c17d156, L:/XXX - R:YYY] Handler is being applied: {uri=http://
localhost:8080/, method=GET}
10:55:23.821 ... [ctor-http-nio-9] ... [id: 0x2c17d156, L:/XXX - R:YYY] Received response (auto-read:false) : [Content-
Type=text/html;charset=UTF-8, Transfer-Encoding=chunked, Date=Sun, 30 Aug 2020 07:55:23 GMT,
Connection=close]
10:55:23.821 ... [ctor-http-nio-9] ... [id: 0x2c17d156, L:/XXX - R:YYY] Received last HTTP packet
Hint: [id: 0x2c17d156, L:/XXX - R:YYY] Buffered ByteBufHolder in Inbound Flux Queue
Memory Leaks
74
75. 10:55:23.820 ... [ctor-http-nio-9] ... [id: 0x2c17d156, L:/XXX - R:YYY] Channel acquired, now 500 active connections
and 0 inactive connections
10:55:23.820 ... [ctor-http-nio-9] ... [id: 0x2c17d156, L:/XXX - R:YYY] Handler is being applied: {uri=http://
localhost:8080/, method=GET}
10:55:23.821 ... [ctor-http-nio-9] ... [id: 0x2c17d156, L:/XXX - R:YYY] Received response (auto-read:false) : [Content-
Type=text/html;charset=UTF-8, Transfer-Encoding=chunked, Date=Sun, 30 Aug 2020 07:55:23 GMT,
Connection=close]
10:55:23.821 ... [ctor-http-nio-9] ... [id: 0x2c17d156, L:/XXX - R:YYY] Received last HTTP packet
Hint: [id: 0x2c17d156, L:/XXX - R:YYY] Buffered ByteBufHolder in Inbound Flux Queue
Memory Leaks
75
76. 76
10:55:23.820 ... [ctor-http-nio-9] ... [id: 0x2c17d156, L:/XXX - R:YYY] Channel acquired, now 500 active connections
and 0 inactive connections
10:55:23.820 ... [ctor-http-nio-9] ... [id: 0x2c17d156, L:/XXX - R:YYY] Handler is being applied: {uri=http://
localhost:8080/, method=GET}
10:55:23.821 ... [ctor-http-nio-9] ... [id: 0x2c17d156, L:/XXX - R:YYY] Received response (auto-read:false) : [Content-
Type=text/html;charset=UTF-8, Transfer-Encoding=chunked, Date=Sun, 30 Aug 2020 07:55:23 GMT,
Connection=close]
10:55:23.821 ... [ctor-http-nio-9] ... [id: 0x2c17d156, L:/XXX - R:YYY] Received last HTTP packet
Hint: [id: 0x2c17d156, L:/XXX - R:YYY] Buffered ByteBufHolder in Inbound Flux Queue
6
Memory Leaks
77. 10:55:23.820 ... [ctor-http-nio-9] ... [id: 0x2c17d156, L:/XXX - R:YYY] Channel acquired, now 500 active connections
and 0 inactive connections
10:55:23.820 ... [ctor-http-nio-9] ... [id: 0x2c17d156, L:/XXX - R:YYY] Handler is being applied: {uri=http://
localhost:8080/, method=GET}
10:55:23.821 ... [ctor-http-nio-9] ... [id: 0x2c17d156, L:/XXX - R:YYY] Received response (auto-read:false) : [Content-
Type=text/html;charset=UTF-8, Transfer-Encoding=chunked, Date=Sun, 30 Aug 2020 07:55:23 GMT,
Connection=close]
10:55:23.821 ... [ctor-http-nio-9] ... [id: 0x2c17d156, L:/XXX - R:YYY] Received last HTTP packet
Hint: [id: 0x2c17d156, L:/XXX - R:YYY] Buffered ByteBufHolder in Inbound Flux Queue
🤔
Was the incoming data
consumed !??!!
Memory Leaks
77
79. @GetMapping("/remote")
public Mono<String> remote() {
return webClient.get()
.uri("http://localhost:8080/")
.exchange()
.flatMap(response -> Mono.just(response.statusCode()
.toString()))
.timeout(Duration.ofSeconds(5));
}
🤔
Was the incoming data
consumed !??!!
Memory Leaks
79
80. Unlike retrieve(), when using exchange(), it is the responsibility of the application to
consume any response content regardless of the scenario (success, error, unexpected data,
etc). Not doing so can cause a memory leak.
https://docs.spring.io/spring/docs/current/spring-framework-reference/web-
reactive.html#webflux-client-exchange
80
Memory Leaks
85. When onStatus is used, if the response is expected to have content, then the onStatus
callback should consume it. If not, the content will be automatically drained to ensure
resources are released.
https://docs.spring.io/spring/docs/current/spring-framework-reference/web-
reactive.html#webflux-client-retrieve
85
OnStatus
86. Releasing the Data
● releaseBody()
● toBodilessEntity()
● bodyToMono(Void.class) !Closes the connection!
86
88. Server Side - Read Timeout
@Component
public class MyNettyWebServerCustomizer
implements WebServerFactoryCustomizer<NettyReactiveWebServerFactory> {
@Override
public void customize(NettyReactiveWebServerFactory factory) {
factory.addServerCustomizers(
server -> server.doOnConnection(conn -> conn.addHandlerFirst(
new ReadTimeoutHandler(50, TimeUnit.MILLISECONDS))));
}
}
88
89. Server Side - Read Timeout
@Component
public class MyNettyWebServerCustomizer
implements WebServerFactoryCustomizer<NettyReactiveWebServerFactory> {
@Override
public void customize(NettyReactiveWebServerFactory factory) {
factory.addServerCustomizers(
server -> server.doOnConnection(conn -> conn.addHandlerFirst(
new ReadTimeoutHandler(50, TimeUnit.MILLISECONDS))));
}
}
Read timeout
89
90. Server Side - Read Timeout
● Time between two requests
● Timeout for the incoming data
○ network latency
● May interfere when TLS handshake
● May interfere when sending a response
○ processing latency
90
91. Client Side - Timeouts
var provider = ConnectionProvider.builder("demo")
.maxConnections(16)
.maxIdleTime(Duration.ofSeconds(60))
.maxLifeTime(Duration.ofSeconds(60))
.build();
var httpClient = HttpClient.create()
.option(ChannelOption.CONNECT_TIMEOUT_MILLIS, 60000)
.wiretap(true)
.responseTimeout(Duration.ofSeconds(1));
WebClient.builder()
.clientConnector(new ReactorClientHttpConnector(httpClient))
.build();
91
92. Client Side - Connection Pool
var provider = ConnectionProvider.builder("demo")
.maxConnections(16)
.maxIdleTime(Duration.ofSeconds(60))
.maxLifeTime(Duration.ofSeconds(60))
.build();
var httpClient = HttpClient.create()
.option(ChannelOption.CONNECT_TIMEOUT_MILLIS, 60000)
.wiretap(true)
.responseTimeout(Duration.ofSeconds(1));
WebClient.builder()
.clientConnector(new ReactorClientHttpConnector(httpClient))
.build();
Max idle time in the pool
92
93. Client Side - Connection Pool
var provider = ConnectionProvider.builder("demo")
.maxConnections(16)
.maxIdleTime(Duration.ofSeconds(60))
.maxLifeTime(Duration.ofSeconds(60))
.build();
var httpClient = HttpClient.create()
.option(ChannelOption.CONNECT_TIMEOUT_MILLIS, 60000)
.wiretap(true)
.responseTimeout(Duration.ofSeconds(1));
WebClient.builder()
.clientConnector(new ReactorClientHttpConnector(httpClient))
.build();
Max life time for the
connection
93
94. Client Side - Establishing Connection
var provider = ConnectionProvider.builder("demo")
.maxConnections(16)
.maxIdleTime(Duration.ofSeconds(60))
.maxLifeTime(Duration.ofSeconds(60))
.build();
var httpClient = HttpClient.create()
.option(ChannelOption.CONNECT_TIMEOUT_MILLIS, 60000)
.wiretap(true)
.responseTimeout(Duration.ofSeconds(1));
WebClient.builder()
.clientConnector(new ReactorClientHttpConnector(httpClient))
.build();
Max time waiting to
establish connection
94
95. Client Side - Response Timeout
var provider = ConnectionProvider.builder("demo")
.maxConnections(16)
.maxIdleTime(Duration.ofSeconds(60))
.maxLifeTime(Duration.ofSeconds(60))
.build();
var httpClient = HttpClient.create()
.option(ChannelOption.CONNECT_TIMEOUT_MILLIS, 60000)
.wiretap(true)
.responseTimeout(Duration.ofSeconds(1));
WebClient.builder()
.clientConnector(new ReactorClientHttpConnector(httpClient))
.build();
Max time waiting for a
response
95
97. Client Side - Response Timeout
@GetMapping("/remote")
public Mono<String> remote() {
return webClient.get()
.uri("http://localhost:8080/")
.httpRequest(request ->
((HttpClientRequest) request.getNativeRequest())
.responseTimeout(Duration.ofSeconds(1)))
.retrieve()
…
}
Max time waiting for a
response
97
98. Client Side - Timeout for Mono/Flux
@GetMapping("/remote")
public Mono<String> remote() {
return webClient.get()
.uri("http://localhost:8080/")
.retrieve()
…
.timeout(…)
}
98
99. Client Side - Timeout for Mono/Flux
@GetMapping("/remote")
public Mono<String> remote() {
return webClient.get()
.uri("http://localhost:8080/")
.retrieve()
…
.timeout(…)
}
Timeout for this Mono
99
100. ● Time between two signals from this Flux/Mono
● Timeout for the incoming data
○ network latency
● May interfere when establishing connection
● May interfere when TLS handshake
● May interfere when sending a request
○ processing latency
100
Client Side - Timeout for Mono/Flux
132. Target Server - Upload Limit
@Configuration
public class WebConfig implements WebFluxConfigurer {
@Override
public void configureHttpMessageCodecs(ServerCodecConfigurer configurer) {
configurer.defaultCodecs().maxInMemorySize(512);
}
}
132
133. Target Server - Upload Limit
@Configuration
public class WebConfig implements WebFluxConfigurer {
@Override
public void configureHttpMessageCodecs(ServerCodecConfigurer configurer) {
configurer.defaultCodecs().maxInMemorySize(512);
}
}
Max in memory
configuration
133
134. Target Server - Upload Limit
@PostMapping("/echo")
public String echo(@RequestBody String body) {
return body;
}
134