SlideShare une entreprise Scribd logo
1  sur  203
Break Me If You Can
Practical Guide to Building Fault-tolerant Systems
Devoxx Ukraine, November 23, 2018
Alex Borysov, Software Engineer @ Google
Mykyta Protsenko, Software Engineer @ Netflix
Who are we?
Alex Borysov
Software Engineer @Google
Mykyta Protsenko
Software Engineer @Netflix
@aiborisov
@mykyta_p
Fault-Tolerance?
@aiborisov
@mykyta_p
Fault vs Error vs Failure
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Fault
@aiborisov
@mykyta_p
incorrect
internal
state
Picture by Bob McMillan. Public domain. See slide #200 for details.
@aiborisov
@mykyta_p
Error
@aiborisov
@mykyta_p
visibly
incorrect
behaviour
Picture by David Goehring. CC BY 2.0. See slide #200 for details.
@aiborisov
@mykyta_p
Failure
@aiborisov
@mykyta_p
main
functionality
is broken
Picture by Camerafiend. CC BY-SA 3.0. See slide #200 for details.
@aiborisov
@mykyta_p
RMS Titanic vs Miracle on the Hudson
@aiborisov
@mykyta_p
Willy Stöwer. Public domain. See slide #200 for details. By Greg Lam Pak Ng. CC BY 2.0. See slide #201 for details.
@aiborisov
@mykyta_p
RMS Titanic
@aiborisov
@mykyta_p
Fault: Hitting an iceberg
Error: Water in the hull
Failure: Sinking
Willy Stöwer. Public domain. See slide #200 for details.
@aiborisov
@mykyta_p
Miracle on the Hudson
@aiborisov
@mykyta_p
Fault: Hitting geese at 859 m
Error: Engines shut down
No Failure!
By Greg Lam Pak Ng. CC BY 2.0. See slide #201 for details.
Fault Error Failure
@aiborisov
@mykyta_p
→ →
Fault Error Failure
@aiborisov
@mykyta_p
→ →
@aiborisov
@mykyta_p
Fault Tolerance
@aiborisov
@mykyta_p
Code and Design Patterns
Product-Driven Decisions
Communication
By Greg Lam Pak Ng. CC BY 2.0. See slide #201 for details.
Dodging Geese
@aiborisov
@mykyta_p
Dodging Geese
@aiborisov
@mykyta_p
#вгеймдевенасовсем
@aiborisov
@mykyta_p
Dodging Geese Architecture
TOP-5
Geese Service
Clouds Service
Leaderboard Service
API
Gateway
@aiborisov
@mykyta_p
See slides ##200, 201 for licensing details.
@aiborisov
@mykyta_p
Dodging Geese Architecture
TOP-5
Geese Service
Clouds Service
Leaderboard Service
API
Gateway
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Dodging Geese Architecture
TOP-5
Geese Service
Leaderboard Service
API
Gateway
@aiborisov
@mykyta_p
Clouds Service
@aiborisov
@mykyta_p
Dodging Geese Architecture
TOP-5
Leaderboard Service
API
Gateway
@aiborisov
@mykyta_p
Clouds Service
Geese Service
@aiborisov
@mykyta_p
Dodging Geese Architecture
Geese Service
Clouds ServiceAPI
Gateway
@aiborisov
@mykyta_p
TOP-5
Leaderboard Service
@aiborisov
@mykyta_p
Dodging Geese Architecture
TOP-5
Geese Service
Clouds Service
Leaderboard Service
API
Gateway
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Dodging Geese Architecture
TOP-5
Geese Service
Clouds Service
Leaderboard Service
API
Gateway
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Dodging Geese Architecture
TOP-5
Geese Service
Clouds Service
Leaderboard Service
API
Gateway
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Leaderboard API (REST)
/players/<username>/score
{"name": "Jane", "score": 100}
/leaderboard/top/<n>
[{"name": "Jane", "score": 100},
{"name": "John", "score": 50}, ...]
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
gRPC Service Definitions
@aiborisov
@mykyta_p
service GeeseService {
// Return next line of geese.
rpc GetGeese (GetGeeseRequest) returns (GeeseResponse);
}
@aiborisov
@mykyta_p
gRPC Service Definitions
@aiborisov
@mykyta_p
service GeeseService {
// Return next line of geese.
rpc GetGeese (GetGeeseRequest) returns (GeeseResponse);
}
service CloudsService {
// Return next line of clouds.
rpc GetClouds (GetCloudsRequest) returns (CloudsResponse);
}
@aiborisov
@mykyta_p
service FixtureService {
// Return next line of geese and clouds.
rpc GetFixture (GetFixtureRequest) returns (FixtureResponse);
}
gRPC Gateway Service
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
service FixtureService {
// Return next line of geese and clouds.
rpc GetFixture (GetFixtureRequest) returns (FixtureResponse);
}
+ = Fixture
gRPC Gateway Service
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
public class FixtureService extends FixtureServiceImplBase {
Gateway Fixture Service
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Gateway Fixture Service
Geese Service
Clouds ServiceAPI
Gateway
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Gateway Fixture Service
Clouds ServiceAPI
Gateway
@aiborisov
@mykyta_p
Geese Service
@aiborisov
@mykyta_p
Gateway Fixture Service
Clouds ServiceAPI
Gateway
@aiborisov
@mykyta_p
Geese Service
@aiborisov
@mykyta_p
Gateway Fixture Service
API
Gateway
@aiborisov
@mykyta_p
Geese Service
Clouds Service
@aiborisov
@mykyta_p
Gateway Fixture Service
API
Gateway
@aiborisov
@mykyta_p
Geese Service
Clouds Service
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Fixture Latency =
Geese Latency
+
Clouds Latency
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Non-Blocking Calls
Don’t block
Send requests in parallel
Combine results when ready
@aiborisov
@mykyta_p
public class FixtureService extends FixtureServiceImplBase {
Gateway Service Implementation
@aiborisov
@mykyta_p
private final GeeseServiceFutureStub geeseClient = ...;
private final CloudsServiceFutureStub cloudsClient = ...;
@aiborisov
@mykyta_p
public class FixtureService extends FixtureServiceImplBase {
Gateway Service Implementation
@aiborisov
@mykyta_p
private final GeeseServiceFutureStub geeseClient = ...;
private final CloudsServiceFutureStub cloudsClient = ...;
@Override
public void getFixture(GetFixtureRequest request, StreamObserver<FixtureResponse> response) {
ListenableFuture<GeeseResponse> geese = geeseClient.getGeese(toGeese(request));
ListenableFuture<CloudsResponse> clouds = cloudsClient.getClouds(toClouds(request));
ListenableFuture<List<GeneratedMessageV3>> geeseAndClouds =
Futures.allAsList(geese, clouds);
...
@aiborisov
@mykyta_p
public class FixtureService extends FixtureServiceImplBase {
Gateway Service Implementation
@aiborisov
@mykyta_p
private final GeeseServiceFutureStub geeseClient = ...;
private final CloudsServiceFutureStub cloudsClient = ...;
@Override
public void getFixture(GetFixtureRequest request, StreamObserver<FixtureResponse> response) {
ListenableFuture<GeeseResponse> geese = geeseClient.getGeese(toGeese(request));
ListenableFuture<CloudsResponse> clouds = cloudsClient.getClouds(toClouds(request));
ListenableFuture<List<GeneratedMessageV3>> geeseAndClouds =
Futures.allAsList(geese, clouds);
...
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Slow dependencies
Slow upstream services
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Timeouts
Guaranteed latency
for integration points
@aiborisov
@mykyta_p
public class FixtureService extends FixtureServiceImplBase {
...
Gateway Service Implementation
@aiborisov
@mykyta_p
@Override
public void getFixture(GetFixtureRequest request, StreamObserver<FixtureResponse> response) {
ListenableFuture<GeeseResponse> geese =
geeseClient.getGeese(toGeese(request));
ListenableFuture<CloudsResponse> clouds =
cloudsClient.getClouds(toClouds(request));
ListenableFuture<List<GeneratedMessageV3>> geeseAndClouds =
Futures.allAsList(geese, clouds);
...
@aiborisov
@mykyta_p
public class FixtureService extends FixtureServiceImplBase {
...
Gateway Service Implementation
@aiborisov
@mykyta_p
@Override
public void getFixture(GetFixtureRequest request, StreamObserver<FixtureResponse> response) {
ListenableFuture<GeeseResponse> geese =
geeseClient.withDeadlineAfter(500, MILLISECONDS).getGeese(toGeeseRequest(request));
ListenableFuture<CloudsResponse> clouds =
cloudsClient.withDeadlineAfter(500, MILLISECONDS).getClouds(toCloudsRequest(request));
ListenableFuture<List<GeneratedMessageV3>> geeseAndClouds =
Futures.allAsList(geese, clouds);
...
@aiborisov
@mykyta_p
@Override
public void getFixture(GetFixtureRequest request, StreamObserver<FixtureResponse> response) {
ListenableFuture<GeeseResponse> geese =
geeseClient.withDeadlineAfter(500, MILLISECONDS).getGeese(toGeeseRequest(request));
ListenableFuture<CloudsResponse> clouds =
cloudsClient.withDeadlineAfter(500, MILLISECONDS).getClouds(toCloudsRequest(request));
ListenableFuture<List<GeneratedMessageV3>> geeseAndClouds =
Futures.allAsList(geese, clouds);
...
public class FixtureService extends FixtureServiceImplBase {
...
Gateway Service Implementation
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
REST: Non-Blocking Calls
CompletableFuture<List<LeaderboardEntry>> leaderboard =
httpClient
.get().uri("/top/5")
.exchange()
.timeout(Duration.ofMillis(500))
.flatMap(cr -> cr.bodyToMono(...))
.toFuture();
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
REST: Non-Blocking Calls with Timeout
CompletableFuture<List<LeaderboardEntry>> leaderboard =
httpClient
.get().uri("/top/5")
.exchange()
.timeout(Duration.ofMillis(500))
.flatMap(cr -> cr.bodyToMono(...))
.toFuture();
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Demo
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
No Geese
No Clouds
Blinking Leaderboard
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Observability
Monitoring:
QPS, latency, errors, ...
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Observability: gRPC
Monitoring:
QPS, latency, errors, ...
// OpenCensus
RpcViews.registerAllViews();
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Tracing: gRPC
GrpcTracing grpcTracing =
GrpcTracing.create(...);
ManagedChannelBuilder
...
.intercept(grpcTracing.newClientInterceptor())
.build() ;
ServerBuilder.forPort(8080)
...
.intercept(grpcTracing.newServerInterceptor())
.build();
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Tracing: gRPC
GrpcTracing grpcTracing =
GrpcTracing.create(...);
ManagedChannelBuilder
...
.intercept(grpcTracing.newClientInterceptor())
.build();
ServerBuilder.forPort(8080)
...
.intercept(grpcTracing.newServerInterceptor())
.build();
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Tracing: REST
build.gradle:
dependencies {
compile '...:spring-cloud-sleuth-zipkin'
compile '...:spring-cloud-starter-sleuth'
...
}
application.properties:
spring.zipkin.baseUrl=http://zipkin:9411/
spring.sleuth.sampler.probability=1.0
spring.sleuth.web.enabled=true
@aiborisov
@mykyta_p
Demo
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Clouds are slow
Geese are fast
Entire call fails
@aiborisov
@mykyta_p
ListenableFuture<GeeseResponse> geese =
geeseClient..getGeese(toGeese(request));
ListenableFuture<CloudsResponse> clouds =
cloudsClient.getClouds(toClouds(request));
ListenableFuture<List<GeneratedMessageV3>>
geeseAndClouds =
Futures.allAsList(geese, clouds);
...
@aiborisov
@mykyta_p
Partial Degradation
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Partial Degradation
ListenableFuture<GeeseResponse> geese =
geeseClient..getGeese(toGeese(request));
ListenableFuture<CloudsResponse> clouds =
cloudsClient.getClouds(toClouds(request));
ListenableFuture<List<GeneratedMessageV3>>
geeseAndClouds =
Futures.successfulAsList(geese, clouds);
...
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Some L-board calls fail
L-board latency is low
Scores disappear
@aiborisov
@mykyta_p
CompletableFuture<List<Leaderboard>> request() {
return httpClient
.get().uri("/top/5").exchange()
.timeout(Duration.ofMillis(500))
.flatMap(...).toFuture();
}
@aiborisov
@mykyta_p
Retries: REST
@aiborisov
@mykyta_p
CompletableFuture<List<Leaderboard>> request() {
return httpClient
.get().uri("/top/5").exchange()
.timeout(Duration.ofMillis(500))
.flatMap(...).toFuture();
}
RetryPolicy RETRY_POLICY = new RetryPolicy()
.retryOn(IOException.class)
.withMaxRetries(MAX_RETRIES);
CompletableFuture<List<Leaderboard>> top5 =
Failsafe.with(RETRY_POLICY)
...
.future(this::httpRequest);
@aiborisov
@mykyta_p
Retries: REST
@aiborisov
@mykyta_p
Demo
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Retry slow calls?
Retry failed calls?
Retry network faults?
@aiborisov
@mykyta_p
Retry Storm
Clouds ServiceAPI
Gateway
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
new RetryPolicy()
.withBackoff(
MIN_DELAY,
MAX_DELAY,
TimeUnit.MILLISECONDS, 100.0)
...
...
@aiborisov
@mykyta_p
Exponential Backoffs
@aiborisov
@mykyta_p
Failsafe
.with(RETRY_POLICY)
.withFallback(
() -> emptyLeaderboard())
...
@aiborisov
@mykyta_p
Fallbacks
@aiborisov
@mykyta_p
Failsafe
.with(RETRY_POLICY)
.withFallback(
() -> cachedLeaderboard())
...
@aiborisov
@mykyta_p
Fallbacks
@aiborisov
@mykyta_p
Retry
Fallback
Fail Fast
@aiborisov
@mykyta_p
On Error
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
High 99%ile latency
100 requests
Error probability?
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
High 99%ile latency
100 requests
Error probability:
1 – 0.99^100 = 63%
@aiborisov
@mykyta_p
Tail-Tolerance
@aiborisov
@mykyta_p
Request
200 ms deadline
@aiborisov
@mykyta_p
Tail-Tolerance
@aiborisov
@mykyta_p
Request
200 ms deadline
↓ 100 ms
@aiborisov
@mykyta_p
Tail-Tolerance
@aiborisov
@mykyta_p
Request
200 ms deadline
↓ 100 ms
Request
@aiborisov
@mykyta_p
Tail-Tolerance
@aiborisov
@mykyta_p
Request
200 ms deadline
↓ 100 ms
Request
Fastest Response
@aiborisov
@mykyta_p
High 99%ile latency
100 requests
@aiborisov
@mykyta_p
Request Hedging
@aiborisov
@mykyta_p
High 99%ile latency
100 requests
Error probability:
63% x 0.01 < 1%
@aiborisov
@mykyta_p
Request Hedging
@aiborisov
@mykyta_p
Channel geeseChannel = ManagedChannelBuilder
.forAddress(geeseHost, geesePort)
.enableRetry()
.maxHedgedAttempts(MAX_HEDGES)
.build();
GeeseServiceFutureStub geeseStub =
GeeseServiceGrpc
.newFutureStub(geeseChannel);
@aiborisov
@mykyta_p
Hedging in gRPC (soon)
@aiborisov
@mykyta_p
Channel geeseChannel = ManagedChannelBuilder
.forAddress(geeseHost, geesePort)
.enableRetry()
.maxHedgedAttempts(MAX_HEDGES)
.build();
GeeseServiceFutureStub geeseStub =
GeeseServiceGrpc
.newFutureStub(geeseChannel);
@aiborisov
@mykyta_p
Hedging in gRPC (soon)
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
High mean latency
100 requests
Error probability?
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
High mean latency
100 requests
Error probability:
1 – 0.50^100 = 99.99...%
@aiborisov
@mykyta_p
CircuitBreaker CIRCUIT_BREAKER =
new CircuitBreaker()
.withFailureThreshold(3, 5);
CompletableFuture<...> top5 =
Failsafe
.with(CIRCUIT_BREAKER)
.with(RETRY_POLICY)
...
.future(this::httpRequest);
@aiborisov
@mykyta_p
Circuit Breaker
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Error Handling
100% Error Fail Fast
Intermittent Slow Hedging
Intermittent Fast Retry
Fallback✚
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Error Handling
100% Error Fail Fast
Intermittent Slow Hedging
Intermittent Fast Retry
Fallback✚
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Client-driven deadline
Don’t process failed calls
@aiborisov
@mykyta_p
Deadlines
API
Gateway
@aiborisov
@mykyta_p
See slides ##200, 201 for licensing details.
@aiborisov
@mykyta_p
Deadlines
API
Gateway
@aiborisov
@mykyta_p
Deadline 200 ms
→
@aiborisov
@mykyta_p
Deadlines
API
Gateway
@aiborisov
@mykyta_p
Deadline 200 ms
→ Spent 120 ms
→
@aiborisov
@mykyta_p
Deadlines
API
Gateway
@aiborisov
@mykyta_p
Spent 120 ms
→ Spent 90 ms
Deadline 200 ms
→
X
@aiborisov
@mykyta_p
Deadlines
API
Gateway
@aiborisov
@mykyta_p
Spent 120 ms
→ Spent 90 ms
Deadline 200 ms
→
X
→
@aiborisov
@mykyta_p
Deadlines Propagation
API
Gateway
@aiborisov
@mykyta_p
Deadline 200 ms
→
@aiborisov
@mykyta_p
Deadline 80 ms
Deadlines Propagation
API
Gateway
@aiborisov
@mykyta_p
Deadline 200 ms
→ Spent 120 ms
→
@aiborisov
@mykyta_p
Deadline 80 ms
Deadlines Propagation
API
Gateway
@aiborisov
@mykyta_p
Spent 120 ms
→ Spent 90 ms
Deadline 200 ms
→
X
@aiborisov
@mykyta_p
Deadline 80 ms
Deadlines Propagation
API
Gateway
@aiborisov
@mykyta_p
Spent 120 ms
→ Spent 90 ms
Deadline -10 ms
Deadline 200 ms
→
X
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Throughput has limits
Exceeding limits?
@aiborisov
@mykyta_p
new ConcurrencyLimitServletFilter(
new ServletLimiterBuilder()
.partitionByHeader("GEESE_TYPE",
c -> c.assign("premium", 0.9)
.assign("free", 0.1))
.limiter(l -> l.limit(
newBuilder()
.initialLimit(1000)...);
@aiborisov
@mykyta_p
REST
@aiborisov
@mykyta_p
new ConcurrencyLimitServletFilter(
new ServletLimiterBuilder()
.partitionByHeader("GEESE_TYPE",
c -> c.assign("premium", 0.9)
.assign("free", 0.1))
.limiter(l -> l.limit(
newBuilder()
.initialLimit(1000)...);
@aiborisov
@mykyta_p
REST
@aiborisov
@mykyta_p
var limiter =
new GrpcServerLimiterBuilder()
.partitionByHeader(GEESE_TYPE)
.partition("premium", 0.9)
.partition("free", 0.1)
.limiter(l ->
l.limit(
newBuilder()
.initialLimit(1000)...);
ConcurrencyLimitServerInterceptor
.newBuilder(limiter).build();
@aiborisov
@mykyta_p
gRPC: Server
@aiborisov
@mykyta_p
var limiter =
new GrpcServerLimiterBuilder()
.partitionByHeader(GEESE_TYPE)
.partition("premium", 0.9)
.partition("free", 0.1)
.limiter(l ->
l.limit(
newBuilder()
.initialLimit(1000)...);
ConcurrencyLimitServerInterceptor
.newBuilder(limiter).build();
@aiborisov
@mykyta_p
gRPC: Server
@aiborisov
@mykyta_p
new GrpcClientLimiterBuilder()
.limit(
newBuilder()
.initialLimit(1000).build())
.blockOnLimit(false) // fail-fast
.build();
@aiborisov
@mykyta_p
gRPC: Client
@aiborisov
@mykyta_p
Demo
@aiborisov
@mykyta_p
Demo
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Monitoring
@aiborisov
@mykyta_p
APM
Service
metrics
Distributed
tracing
Business
metrics
Picture by Alex Borysov. CC BY 2.0. See slide #201 for details.
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Code and Design
Timeouts / Deadline Propagation
Retries / Hedging
Proper Fallbacks
Concurrency Limits
Load Shedding
Observability
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Request for each response
Requests don’t change
@aiborisov
@mykyta_p
Redundant Requests
@aiborisov
@mykyta_p
GeeseRequest
GeeseResponse
GeeseRequest
GeeseResponse
GeeseRequest
GeeseResponse
@aiborisov
@mykyta_p
Redundant Requests
@aiborisov
@mykyta_p
GeeseRequest
GeeseResponse
GeeseRequest
GeeseResponse
GeeseRequest
GeeseResponse
@aiborisov
@mykyta_p
Streaming
@aiborisov
@mykyta_p
GeeseRequest
GeeseResponse
GeeseResponse
GeeseResponse
@aiborisov
@mykyta_p
service GeeseService {
rpc GetGeese (GetGeeseRequest)
returns (GeeseResponse);
}
service CloudsService {
rpc GetClouds (GetCloudsRequest)
returns (CloudsResponse);
}
@aiborisov
@mykyta_p
gRPC Streaming
@aiborisov
@mykyta_p
service GeeseService {
rpc GetGeese (GetGeeseRequest)
returns (stream GeeseResponse);
}
service CloudsService {
rpc GetClouds (GetCloudsRequest)
returns (stream CloudsResponse);
}
@aiborisov
@mykyta_p
gRPC Streaming
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Server faster than client
Client cannot keep up
@aiborisov
@mykyta_p
Too Many Streaming Responses
@aiborisov
@mykyta_p
GeeseRequest
@aiborisov
@mykyta_p
Too Many Streaming Responses
@aiborisov
@mykyta_p
GeeseRequest
X
@aiborisov
@mykyta_p
Flow Control
@aiborisov
@mykyta_p
GeeseRequest
@aiborisov
@mykyta_p
Flow Control
@aiborisov
@mykyta_p
GeeseRequest
5
@aiborisov
@mykyta_p
Flow Control
@aiborisov
@mykyta_p
GeeseRequest
5
@aiborisov
@mykyta_p
Flow Control
@aiborisov
@mykyta_p
GeeseRequest
5
3
@aiborisov
@mykyta_p
Flow Control
@aiborisov
@mykyta_p
GeeseRequest
5
3
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Decople producer and
consumer
Decople failures
@aiborisov
@mykyta_p
Message-driven
Elastic
Responsive
Resilient
@aiborisov
@mykyta_p
Reactive Systems
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Per
instance
limits
@aiborisov
@mykyta_p
Door Capacity
@aiborisov
@mykyta_p
Why didn’t Rose make room for
Jack on the door?
Willy Stöwer. Public domain. See slide #200 for details.
@aiborisov
@mykyta_p
Door Capacity
@aiborisov
@mykyta_p
Why didn’t Rose make room for
Jack on the door?
“ The answer is very simple
because it says on page 147
that Jack dies “
James Cameron
Willy Stöwer. Public domain. See slide #200 for details.
@aiborisov
@mykyta_p
Capacity
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Capacity
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Autoscaling
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Prescaling
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Prescaling
@aiborisov
@mykyta_p
See slides ##200, 202 for licensing details.
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Services
break
each other
@aiborisov
@mykyta_p
$
Free and Premium?
Free
Premium
$
@aiborisov
@mykyta_p
Free and Premium Outage
Free
Premium
$
$
@aiborisov
@mykyta_p
$
$
Bulkheads
Free
Premium
$
@aiborisov
@mykyta_p
Bulkheads
Free
Premium $
$
$
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Bulkheads
By Request Type
By Client Priority
By Region
By Availability Zone
etc
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Deployments
can be
risky
@aiborisov
@mykyta_p
Exploding Whale
Engineering solution
Half a ton of dynamite
@aiborisov
@mykyta_p
Illustration by Greg Williams. CC BY 3.0. See slide #203.
@aiborisov
@mykyta_p
Exploding Whale
Engineering solution
Half a ton of dynamite
Ooops! Non-limited blast radius
Learn more at TheExplodingWhale.com
@aiborisov
@mykyta_p
Illustration by Greg Williams. CC BY 3.0. See slide #203.
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Testing
is
hard
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Willy Stöwer. Public domain. See slide #200 for details.
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
✔
Willy Stöwer. Public domain. See slide #200 for details.
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
✔
✔
Willy Stöwer. Public domain. See slide #200 for details.
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
✔
✔
✔
Willy Stöwer. Public domain. See slide #200 for details.
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
✔
✔
✔
✔
Willy Stöwer. Public domain. See slide #200 for details.
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
✔
✔
✔
✔
✔
Willy Stöwer. Public domain. See slide #200 for details.
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
✔
✔
✔
✔
✔
Willy Stöwer. Public domain. See slide #200 for details.
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Chaos Engineering
Break on purpose
Find problems
Prevent catastrophes
PrinciplesOfChaos.org
@aiborisov
@mykyta_p
Demo
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Bad user experience
Metrics are not enough
@aiborisov
@mykyta_p
Prober
TOP-5
API
Gateway
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Prober
TOP-5
API
Gateway
@aiborisov
@mykyta_p
See slides ##200, 202 for licensing details.
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Prober
Availability
Latency SLO
Response verification
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Prober
Availability
Latency SLO
Response verification
CloudProber.org
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Technical
solutions
are not enough
@aiborisov
@mykyta_p
Communication
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Communication
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Communication Channels
@aiborisov
@mykyta_p
GEESE
at 270
@aiborisov
@mykyta_p
Communication Channels
@aiborisov
@mykyta_p
GEESE
at 270
@aiborisov
@mykyta_p
GEESE
at 270
Communication Channels
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
GEESE
at 270
Communication Channels
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Postmortems
@aiborisov
@mykyta_p
Blameless
Constructive
@aiborisov
@mykyta_p
Postmortems
@aiborisov
@mykyta_p
Blameless
Constructive
Social
See slides ##200, 202, 203 for licensing details.
@aiborisov
@mykyta_p
Postmortems
@aiborisov
@mykyta_p
Timeline
Causes
Remedies
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Learn from Failure
Blameless postmortems
Alert playbooks
Incident knowledge base
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Libraries and Tools
@aiborisov
@mykyta_p
Demo: github.com/break-me-if-you-can
Failsafe: github.com/jhalterman/failsafe
Observability: opencensus.io, opentracing.io
Prober: cloudprober.org
Concurrency Limits: github.com/Netflix/concurrency-limits
Canaries: github.com/spinnaker/kayenta
@aiborisov
@mykyta_p
Demo UI
@HalloGene_
Yevgen Golubenko
Twitter: @HalloGene_
github.com/HalloGene
Picture by Yevgen Golubenko. Also see slide #203 for licensing details.
@aiborisov
@mykyta_p
Books
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Fault-Tolerance
Code & Design Patterns
Product decisions
Communication culture
@aiborisov
@mykyta_p
Please Break Me!
If you can
@aiborisov
@mykyta_p
Please Break Me!
Rate
If you can
@aiborisov
@mykyta_p
Please Break Me!
Rate Us
If you can
@aiborisov
@mykyta_p
Please Break Me!
Rate Us
If you enjoyed the talk
Or give feedback
If you can
@aiborisov
@mykyta_p
Please Break Me!
Rate Us
If you enjoyed the talk
Or give feedback
If you can
5 STARS!
@aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Images and Licensing
Images of geese, clouds, pilots, plane, arrows, cup, airport traffic control tower are property of Mykyta Protsenko and Alex Borysov, if not
stated otherwise (see below). All Rights Reserved.
Other images used:
Slide #5: commons.wikimedia.org/wiki/File:FEMA_-_16381_-_Photograph_by_Bob_McMillan_taken_on_09-28-2005_in_Texas.jpg
- Picture by Bob McMillan, the US federal government work, public domain
Slide #6: www.flickr.com/photos/carbonnyc/3290528875
- Picture by David Goehring. Attribution 2.0 Generic (CC BY 2.0): creativecommons.org/licenses/by/2.0
- changes were made
Slide #7: www.flickr.com/photos/carbonnyc/3290528875
- Picture by Camerafiend. Attribution-ShareAlike 3.0 Unported (CC BY-SA 3.0): creativecommons.org/licenses/by-sa/3.0/deed.en
- no changes were made
Slides ##8, 9, 135, 136, 155-161: commons.wikimedia.org/wiki/File:Titanic_sinking,_painting_by_Willy_St%C3%B6wer.jpg
- Willy Stöwer. Public domain work of art
@aiborisov
@mykyta_p
Images and Licensing
Slides ##8, 10, 13: www.flickr.com/photos/22608787@N00/3200086900
- Picture y Greg Lam Pak Ng. Attribution 2.0 Generic (CC BY 2.0): creativecommons.org/licenses/by/2.0
- no changes were made
Slides ##16-23, 30-34, 68, 77-80, 94-102, 116-118, 123-129, 137-141, 144-147, 171-172:
- Blue Game Boy Color by kure: piq.codeus.net/picture/31994/Blue-Game-Boy-Color
- Attribution 3.0 Unported (CC BY 3.0): creativecommons.org/licenses/by/3.0
- changes were made
Slides ##94-102:
- The Sun by Vinicius615: piq.codeus.net/picture/191706/The-Sun
- Attribution 3.0 Unported (CC BY 3.0): creativecommons.org/licenses/by/3.0
- changes were made
Slide #113:
- Picture by Alex Borysov. Attribution 2.0 Generic (CC BY 2.0): creativecommons.org/licenses/by/2.0
@aiborisov
@mykyta_p
Images and Licensing
Slide #141: piq.codeus.net/picture/254492/CVsantahat
- Santa hat for CommanderVideo, CVsantahat by anonymous
- Attribution 3.0 Unported (CC BY 3.0): creativecommons.org/licenses/by/3.0
- no changes were made
Slide #172: piq.codeus.net/picture/423109/UFO
- UFO by anonymous
- Attribution 3.0 Unported (CC BY 3.0): creativecommons.org/licenses/by/3.0
- no changes were made
Slides #186, 187: piq.codeus.net/picture/334023/beer
- beer by Investa
- Attribution 3.0 Unported (CC BY 3.0): creativecommons.org/licenses/by/3.0
- changes were made
@aiborisov
@mykyta_p
Images and Licensing
Slides #186, 187: piq.codeus.net/picture/444498/Beer-Bottle
- Beer Bottle by jacklrj
- Attribution 3.0 Unported (CC BY 3.0): creativecommons.org/licenses/by/3.0
- changes were made
Slide #191: https://piq.codeus.net/picture/330338/Deal-With-It
- Deal With It by Shiro
- Attribution 3.0 Unported (CC BY 3.0): creativecommons.org/licenses/by/3.0
- changes were made
Slides ##151, 152: https://commons.wikimedia.org/wiki/File:Whale_WikiWorld.png
- Cartoon illustration has been created by Greg Williams in cooperation with the Wikimedia Foundation
- Attribution 3.0 Unported (CC BY 3.0): creativecommons.org/licenses/by/3.0
- changes were made

Contenu connexe

Tendances

WordPress RESTful API & Amazon API Gateway - WordCamp Kansai 2016
WordPress RESTful API & Amazon API Gateway - WordCamp Kansai 2016WordPress RESTful API & Amazon API Gateway - WordCamp Kansai 2016
WordPress RESTful API & Amazon API Gateway - WordCamp Kansai 2016
崇之 清水
 

Tendances (17)

Break me if you can: practical guide to building fault-tolerant systems (with...
Break me if you can: practical guide to building fault-tolerant systems (with...Break me if you can: practical guide to building fault-tolerant systems (with...
Break me if you can: practical guide to building fault-tolerant systems (with...
 
"gRPC-Web: It’s All About Communication": Devoxx Belgium 2019
"gRPC-Web: It’s All About Communication": Devoxx Belgium 2019"gRPC-Web: It’s All About Communication": Devoxx Belgium 2019
"gRPC-Web: It’s All About Communication": Devoxx Belgium 2019
 
"gRPC-Web: It’s All About Communication": Devoxx Ukraine 2019
"gRPC-Web: It’s All About Communication": Devoxx Ukraine 2019"gRPC-Web: It’s All About Communication": Devoxx Ukraine 2019
"gRPC-Web: It’s All About Communication": Devoxx Ukraine 2019
 
"gRPC-Web: It’s All About Communication": Devoxx Ukraine 2019
"gRPC-Web:  It’s All About Communication": Devoxx Ukraine 2019"gRPC-Web:  It’s All About Communication": Devoxx Ukraine 2019
"gRPC-Web: It’s All About Communication": Devoxx Ukraine 2019
 
"Enabling Googley microservices with gRPC" Riga DevDays 2018 edition
"Enabling Googley microservices with gRPC" Riga DevDays 2018 edition"Enabling Googley microservices with gRPC" Riga DevDays 2018 edition
"Enabling Googley microservices with gRPC" Riga DevDays 2018 edition
 
Cloud Expo Europe 2022 "Break me if you can: practical guide to building faul...
Cloud Expo Europe 2022 "Break me if you can: practical guide to building faul...Cloud Expo Europe 2022 "Break me if you can: practical guide to building faul...
Cloud Expo Europe 2022 "Break me if you can: practical guide to building faul...
 
Curl with rust
Curl with rustCurl with rust
Curl with rust
 
common mistakes when using libcurl
common mistakes when using libcurlcommon mistakes when using libcurl
common mistakes when using libcurl
 
HTTP/3 is next generation HTTP
HTTP/3 is next generation HTTPHTTP/3 is next generation HTTP
HTTP/3 is next generation HTTP
 
2018 IterateConf Deconstructing and Evolving REST Security
2018 IterateConf Deconstructing and Evolving REST Security2018 IterateConf Deconstructing and Evolving REST Security
2018 IterateConf Deconstructing and Evolving REST Security
 
WordPress RESTful API & Amazon API Gateway - WordCamp Kansai 2016
WordPress RESTful API & Amazon API Gateway - WordCamp Kansai 2016WordPress RESTful API & Amazon API Gateway - WordCamp Kansai 2016
WordPress RESTful API & Amazon API Gateway - WordCamp Kansai 2016
 
Docker Docker - Docker Security - Docker
Docker Docker - Docker Security - DockerDocker Docker - Docker Security - Docker
Docker Docker - Docker Security - Docker
 
Re invent 2018 - The Evolution of AircraftML
Re invent 2018  - The Evolution of AircraftMLRe invent 2018  - The Evolution of AircraftML
Re invent 2018 - The Evolution of AircraftML
 
GraphQL 101 | An introduction to the world of GraphQL
GraphQL 101 | An introduction to the world of GraphQLGraphQL 101 | An introduction to the world of GraphQL
GraphQL 101 | An introduction to the world of GraphQL
 
Connecting to the Pulse of the Planet with the Twitter Platform
Connecting to the Pulse of the Planet with the Twitter PlatformConnecting to the Pulse of the Planet with the Twitter Platform
Connecting to the Pulse of the Planet with the Twitter Platform
 
Big Data LDN 2017: Will they blend?
Big Data LDN 2017: Will they blend?Big Data LDN 2017: Will they blend?
Big Data LDN 2017: Will they blend?
 
Building Serverless applications with Python
Building Serverless applications with PythonBuilding Serverless applications with Python
Building Serverless applications with Python
 

Similaire à Devoxx Ukraine 2018 "Break me if you can: practical guide to building fault-tolerant systems (with examples from REST and gRPC polyglot stacks)"

Baremetal deployment scale
Baremetal deployment scaleBaremetal deployment scale
Baremetal deployment scale
baremetal
 

Similaire à Devoxx Ukraine 2018 "Break me if you can: practical guide to building fault-tolerant systems (with examples from REST and gRPC polyglot stacks)" (20)

CloudExpo Frankfurt 2023 "Break me if you can: practical guide to building fa...
CloudExpo Frankfurt 2023 "Break me if you can: practical guide to building fa...CloudExpo Frankfurt 2023 "Break me if you can: practical guide to building fa...
CloudExpo Frankfurt 2023 "Break me if you can: practical guide to building fa...
 
Serverless Security: Doing Security in 100 milliseconds
Serverless Security: Doing Security in 100 millisecondsServerless Security: Doing Security in 100 milliseconds
Serverless Security: Doing Security in 100 milliseconds
 
Pairing Based Transform Cryptography (Proxy Re-Encryption - PRE)
Pairing Based Transform Cryptography (Proxy Re-Encryption - PRE)Pairing Based Transform Cryptography (Proxy Re-Encryption - PRE)
Pairing Based Transform Cryptography (Proxy Re-Encryption - PRE)
 
3h à l'assaut d'une application réactive - Devoxx 2015
3h à l'assaut d'une application réactive - Devoxx 20153h à l'assaut d'une application réactive - Devoxx 2015
3h à l'assaut d'une application réactive - Devoxx 2015
 
The Past, Present and Future of Real-Time Apps and Communications
The Past, Present and Future of Real-Time Apps and CommunicationsThe Past, Present and Future of Real-Time Apps and Communications
The Past, Present and Future of Real-Time Apps and Communications
 
Devoxx Belgium 2022 gRPC Cornerstone: HTTP/2… or HTTP/3?
Devoxx Belgium 2022 gRPC Cornerstone: HTTP/2… or HTTP/3?Devoxx Belgium 2022 gRPC Cornerstone: HTTP/2… or HTTP/3?
Devoxx Belgium 2022 gRPC Cornerstone: HTTP/2… or HTTP/3?
 
From Spring Boot 2.2 to Spring Boot 2.3 #jsug
From Spring Boot 2.2 to Spring Boot 2.3 #jsugFrom Spring Boot 2.2 to Spring Boot 2.3 #jsug
From Spring Boot 2.2 to Spring Boot 2.3 #jsug
 
Lessons learned from implementing api management in the real world
Lessons learned from implementing api management in the real worldLessons learned from implementing api management in the real world
Lessons learned from implementing api management in the real world
 
Serverless Architectural Patterns
Serverless Architectural PatternsServerless Architectural Patterns
Serverless Architectural Patterns
 
Making the most out of kubernetes audit logs
Making the most out of kubernetes audit logsMaking the most out of kubernetes audit logs
Making the most out of kubernetes audit logs
 
Testing Java Microservices Workshop
Testing Java Microservices WorkshopTesting Java Microservices Workshop
Testing Java Microservices Workshop
 
Secret Mgmt using vault DevSecOps sg Meetup
Secret Mgmt using vault DevSecOps sg MeetupSecret Mgmt using vault DevSecOps sg Meetup
Secret Mgmt using vault DevSecOps sg Meetup
 
Security in the FaaS Lane
Security in the FaaS LaneSecurity in the FaaS Lane
Security in the FaaS Lane
 
HowYourAPIBeMyAPI
HowYourAPIBeMyAPIHowYourAPIBeMyAPI
HowYourAPIBeMyAPI
 
Serverless Security: A How-to Guide @ SnowFROC 2019
Serverless Security: A How-to Guide @ SnowFROC 2019Serverless Security: A How-to Guide @ SnowFROC 2019
Serverless Security: A How-to Guide @ SnowFROC 2019
 
Deploy With Confidence
Deploy With ConfidenceDeploy With Confidence
Deploy With Confidence
 
Baremetal deployment scale
Baremetal deployment scaleBaremetal deployment scale
Baremetal deployment scale
 
Becoming a Git Master
Becoming a Git MasterBecoming a Git Master
Becoming a Git Master
 
Join Our Party: The Cloud Native Adventure Brigade (OSS 2019)
Join Our Party: The Cloud Native Adventure Brigade (OSS 2019)Join Our Party: The Cloud Native Adventure Brigade (OSS 2019)
Join Our Party: The Cloud Native Adventure Brigade (OSS 2019)
 
Serverless and serverfull - where microservices compliments serverless
Serverless and serverfull - where microservices compliments serverlessServerless and serverfull - where microservices compliments serverless
Serverless and serverfull - where microservices compliments serverless
 

Dernier

CNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of ServiceCNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of Service
giselly40
 
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slideHistor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slide
vu2urc
 
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsIAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI Solutions
Enterprise Knowledge
 

Dernier (20)

How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
CNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of ServiceCNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of Service
 
Real Time Object Detection Using Open CV
Real Time Object Detection Using Open CVReal Time Object Detection Using Open CV
Real Time Object Detection Using Open CV
 
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slideHistor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slide
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...
 
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
 
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
 
[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf
 
A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?
 
Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024
 
What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?
 
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024
 
🐬 The future of MySQL is Postgres 🐘
🐬  The future of MySQL is Postgres   🐘🐬  The future of MySQL is Postgres   🐘
🐬 The future of MySQL is Postgres 🐘
 
Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)
 
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationScaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organization
 
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsIAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI Solutions
 
Automating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps ScriptAutomating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps Script
 
Factors to Consider When Choosing Accounts Payable Services Providers.pptx
Factors to Consider When Choosing Accounts Payable Services Providers.pptxFactors to Consider When Choosing Accounts Payable Services Providers.pptx
Factors to Consider When Choosing Accounts Payable Services Providers.pptx
 
Presentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreterPresentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreter
 
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonData Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt Robison
 

Devoxx Ukraine 2018 "Break me if you can: practical guide to building fault-tolerant systems (with examples from REST and gRPC polyglot stacks)"