Talk from O'Reilly Software Architecture Conference San Jose 2019 Microservices and containers have taken the software industry by storm. Transitioning from a monolith to microservices enables you to deploy your application more frequently, independently, and reliably. However, microservice architecture has its own challenges, and it has to deal with the same problems encountered while designing distributed systems. Enter service mesh technology to the rescue. A service mesh reduces the complexity associated with microservices and provides functionality like load balancing, service discovery, traffic management, circuit breaking, telemetry, fault injection, and more. Istio is one of the best implementations of a service mesh at this point, while Kubernetes provides a platform for running microservices and automating deployment of containerized applications. Join Samir Behara to go beyond the buzz and understand microservices and service mesh technologies.

1. Building a scalable Microservice Architecture
With Kubernetes, Envoy and Istio

2. SAMIR BEHARA
System Architect, EBSCO
Samir Behara builds software solutions
using cutting edge technologies.
Has a Bachelor Degree in Computer
Science with 13 years of IT experience.
Frequent Speaker at Technical
Conferences.
Author of www.dotnetvibes.com
@samirbehara

3. Agenda
• Monolith vs Microservices
• How to break a Monolith into Microservices
• Complexities in a Microservice Architecture
• Journey from Netflix OSS to Istio Service Mesh
• The Rise of Sidecar Design Pattern
• Istio Architecture and capabilities
• How to make your microservices resilient & fault tolerant
• Service Mesh Observability

4. Monolithic Architecture
Order
Management
Service
Monolithic
Database
Large Codebase
Difficult to Scale
Longer Development Cycle
Complicated Deployments
Fixed Technology stack
Performance Issues
Tight Coupling

5. Applying Domain Driven Design

6. Monolith Architecture – Centralized Database
Order Service
Title Service
Currency Service
Pricing Service
Monolithic
Shared
Database
API
Gateway

7. Databases are private to each Microservice
Order API Pricing API

8. Deploying Monolithic Applications

9. Strangler Pattern in action

10. Transform and Eliminate Pattern
TRANSFORM CO-EXIST ELIMINATE

11. Microservices Deployments

12. Emergence of Microservices
Shorter Development Cycle
Faster Deployments
Highly Scalable
Right Technology Stack
Business Domain Driven
Resiliency & Observability
High Cohesion & Loose Coupling

13. Immutable Infrastructure
Declarative Configuration
Horizontal Scaling
Self Healing SystemsService Discovery
Decoupled Architecture
Load Balancing
Scalable Microservices with Kubernetes

14. Microservice Architecture - Challenges

15. 8 Fallacies of Distributed Computing
Fallacy Solutions
The network is reliable Automatic Retries, Message Queues
Latency is zero Caching Strategy, Bulk Requests, Deploy in AZs near client
Bandwidth is infinite Throttling Policy, Small payloads with Microservices
The network is secure Network Firewalls, Encryption, Certificates, Authentication
Topology does not change No hardcoding IP, Service Discovery Tools
There is one administrator DevOps Culture eliminates Bus Factor
Transport cost is zero Standardized protocols like JSON, Cost Calculation
The network is homogenous Circuit Breaker, Retry and Timeout Design Pattern

16. Complexities in a Microservice Architecture

17. Tracing
Service
Discovery
Dynamic
Routing
Resiliency
Load Balancing
Netflix OSS to the rescue

18. What are the issues with Netflix OSS?
ROUTING
CIRCUIT BREAKER
LOAD BALANCING
SERVICE DISCOVERY
TRACING
ROUTING
CIRCUIT BREAKER
LOAD BALANCING
SERVICE DISCOVERY
TRACING
INFRASTRUCTURE
SERVICE A SERVICE B
• Tightly coupled to the Java Platform
• Not a good fit for Polyglot Architecture
• Netflix Libraries needs to be embedded
inside each microservice along side Business
functionalities
• Increases overall Application Complexity
• Operational Complexity - Patching/Upgrades

19. Sidecar Design Pattern
Microservice A
Microservice B
Microservice C
Sidecar
Sidecar
Sidecar
Service Mesh Control Plane

20. Shared Libraries vs Service Mesh
Pricing
Service
Sidecar
Order
Service
Sidecar
Currency
Service
Sidecar
Customer
Service
Sidecar
Title Service
Sidecar
Control
Plane
Business Logic
+
Shared Libraries
Business Logic
+
Shared Libraries
Business Logic
+
Shared Libraries
Business Logic
+
Shared Libraries
Business Logic
+
Shared Libraries

21. Smart Pipes and Smart Endpoints with Service Mesh
Responsibility of network is to transfer messages
Responsibility of microservices is to handle Business Logic,
transformations, validations and process messages.
Dumb Pipes and Smart Endpoints

22. Envoy
• Envoy is a high performance Open Source Proxy designed for Cloud-Native Applications
• Envoy makes the network transparent to the applications
• Envoy is deployed as a Sidecar Proxy to every service
• All traffic in a Microservice architecture flows via the Envoy Proxy
Out of Process
Architecture
Service Discovery Load Balancing
Circuit Breakers Fault Injection Observability

23. Istio
• Platform to Connect, Secure, Control and Monitor
Services consistently.
• Open Source Service Mesh – Governed by Google & IBM
• Shifts the complexity of running a distributed
microservice architecture to the infrastructure layer
• Control Plane for service proxies like Envoy
• Platform Independent & Language agnostic

24. Istio Features
Traffic Management Policy Enforcement
Observability Security Telemetry

25. Service A Service B
Network
Service to Service Communication over Network

26. Service A Service B
Sidecar Deployment using Envoy Proxy
Envoy Proxy intercepts all network traffic flowing between applications

27. Service A Service B
Configuration Validation, Management and Distribution
Galley

28. Service A Service B
Sidecar Configuration and Traffic Management capabilities
Galley Pilot
Push config data
to sidecar proxies

29. Service A Service B
Policy Enforcement and Telemetry Collection
Galley Pilot Mixer
Policy Checks
& Telemetry

30. Service A Service B
Enable Secure Communication using mutual TLS
with built-in identity and credential management
Galley Pilot Mixer Citadel
Push TLS certificates
to sidecar proxies

31. Service A Service B
Galley Pilot Mixer Citadel
Istio Mesh Integrated Control Plane

32. Sidecar
Container
SERVICE A
Pod
Container
JVM
SERVICE B
Pod
Container
JVM
Sidecar
Container
Istio Data Plane with Envoy Sidecar

33. SERVICE A SERVICE B
Istio Architecture
PILOT CITADEL MIXER
Control Plane
Data Plane
Service Discovery
Traffic Management
Resiliency
Policy Enforcement
Telemetry
Authentication
Security
GALLEY
Configuration Validation
and Distribution
HTTP, gRPC, TCP
Security - mTLS
Pod Pod

34. Service A
Istio
Gateway
Service
Entry
Virtual
Service
Service B
Destination
Rule
User
Traffic
Istio Traffic Management

35. Traffic Routing
Envoy
Service A
Pod
Envoy
Service B
Pod
Envoy
Service B
Pod
Pod Labels -
version: v1
env: staging
Pod Labels -
version: v2
env: prodPILOT
Traffic
Routing Rules
# Route all traffic to v1 of ServiceB
kind: VirtualService
metadata:
name: serviceB
spec:
hosts:
- serviceB
http:
- route:
- destination:
host: serviceB
subset: v1

36. Canary Deployment
Envoy
Service A
Pod
Envoy
Service B
Pod
Envoy
Service B
Pod
Pod Labels -
version: v1
env: staging
Pod Labels -
version: v2
env: prod
90%
10%
PILOT
Traffic
Routing Rules
# Percentage based Traffic Split
kind: VirtualService
metadata:
name: serviceB
spec:
hosts:
- serviceB
http:
- route:
- destination:
host: serviceB
subset: v1
weight: 90
- destination:
host: serviceB
subset: v2
weight: 10

37. Dark Launches
Envoy
Service A
Pod
Envoy
Service B
Pod
Envoy
Service B
Pod
Pod Labels -
version: v1
env: staging
Pod Labels -
version: v2
env: prod
100%
Mirror
Traffic
PILOT
Traffic
Routing Rules
# Traffic Mirroring
kind: VirtualService
metadata:
name: serviceB
spec:
hosts:
- serviceB
http:
- route:
- destination:
host: serviceB
subset: v1
weight: 100
mirror:
host: serviceB
subset: v2

43. Circuit Breaker
Envoy
Service A
Pod
Envoy
Service B
Pod
Envoy
Service C
Pod
# Limits the number of concurrent
connections and requests
kind: DestinationRule
metadata:
name: serviceC
spec:
hosts:
- serviceC
trafficPolicy:
connectionPool:
http:
http1MaxPendingRequests: 10
maxRequestsPerConnection: 1
tcp:
maxConnections: 1

44. Outlier Detection
# Detect faulty instances in the
pool & remove from traffic routing
kind: DestinationRule
metadata:
name: serviceB
spec:
hosts:
- serviceB
trafficPolicy:
outlierDetection:
baseEjectionTime: 20s
consecutiveErrors: 3
interval: 10s
maxEjectionPercent: 100
Envoy
Service A
Pod
Envoy
Service B
Pod
Envoy
Service B
Pod
Pod Labels -
version: v1
env: staging
Pod Labels -
version: v2
env: staging

45. Timeout
Envoy
Service A
Pod
Envoy
Service B
Pod
Envoy
Service C
Pod
# Timeout strategy for service
communication over network
kind: VirtualService
metadata:
name: serviceB
spec:
hosts:
- serviceB
http:
- route:
- destination:
host: serviceB
timeout: 10s
Timeout:
10 sec
Timeout:
10 sec

46. Istio Retry Policy
Envoy
Service A
Pod
Envoy
Service B
Pod
# Retry strategy for service
communication over network
kind: VirtualService
metadata:
name: serviceB
spec:
hosts:
- serviceB
http:
- route:
- destination:
host: serviceB
retries:
attempts: 3
perTryTimeout: 2s
Retry: 5
5XX Error

47. Chaos Testing – Inject Delays
Envoy
Service A
Pod
Envoy
Service B
Pod
Envoy
Service B
Pod
Pod Labels -
version: v1
env: staging
Pod Labels -
version: v2
env: prod
# Create rule to delay traffic to
ServiceB v1
kind: VirtualService
metadata:
name: serviceB
spec:
hosts:
- serviceB
http:
- fault:
delay:
fixedDelay: 10s
percent: 50
route:
- destination:
host: serviceB
subset: v1
10s delay
in 50% of
requests

48. Chaos Testing – Inject Errors
Envoy
Service A
Pod
Envoy
Service B
Pod
Envoy
Service B
Pod
Pod Labels -
version: v1
env: staging
Pod Labels -
version: v2
env: prod
# Create rule to inject errors to
ServiceB v1
kind: VirtualService
metadata:
name: serviceB
spec:
hosts:
- serviceB
http:
- fault:
abort:
httpStatus: 500
percent: 50
route:
- destination:
host: serviceB
subset: v2
HTTP 500
in 50% of
requests

49. Monitoring your Microservices Architecture

50. The Three Pillars of Observability
LOGGING METRICS TRACING

51. Prometheus Architecture

52. Visualizing the Service Mesh with Kiali
• Service Mesh Observability & Configuration
• Visualize Service Communication in real time
• Displays Traffic Rates and Latencies
• Quickly identify problem areas
• Configure, Update & Validate Service Mesh

53. View the response time and request
rate of each of the microservice inside
the service mesh.

54. Visualization and Time Series Analytics
Create your own Dashboards for Monitoring and explore the service metrics
Visualization tools to help you understand your data better

55. RED Metrics for Microservices Monitoring
Rate - Number of requests per second your services are serving
Errors - Number of failed requests per second
Duration - Amount of time each request takes to fulfil a request

56. Distributed Tracing
Provides end to end visibility & insights into service requests
Used to troubleshoot latency issues in a Microservice Architecture

58. Resources
Istio blogs on dotnetvibes - https://dotnetvibes.com/?s=istio
Katacoda Interactive Learning Platform - https://www.katacoda.com
Introducing Istio Service Mesh for Microservices - By Burr Sutter and Christian Posta
https://developers.redhat.com/books/introducing-istio-service-mesh-microservices/
Red Hat Developer Blogs and Tutorials - https://developers.redhat.com/topics/service-mesh/
Istio Blogs - https://istio.io/blog/
O’Reilly Live Online Training - https://www.oreilly.com/live-training

59. Thank You
@samirbehara www.dotnetvibes.com