Observability For Modern Applications

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Director, AWS Developer Technology & Evangelism
4 March 2019
Observability for Modern Applications
Ian Massingham @IanMmmm ianm@amazon.com

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
ListenIterate
Experiment
Innovation
Flywheel
Experiments power the engine of rapid innovation

What changes do you need to make to adopt these best practices?
Serverless
No provisioning/management
Automatic scaling
Pay for value billing
Availability and resiliency
Microservices
Componentization
Business capabilities
Products not projects
Infrastructure automation
DevOps
Cultural philosophies
Cross-disciplinary teams
CI/CD
Automation tools
DEV OPS
Architectural
patterns
Operational
Model
Software
Delivery

Approaches to modern application development
• Simplify environment management
• Reduce the impact of code changes
• Automate operations
• Accelerate the delivery of new, high-quality services
• Gain insight across resources and applications
• Protect customers and the business

Approaches to modern application development
• Simplify environment management with serverless technologies
• Reduce the impact of code changes with microservice architectures
• Automate operations by modeling applications & infrastructure as code
• Accelerate the delivery of new, high-quality services with CI/CD
• Gain insight across resources and applications by enabling observability
• Protect customers and the business with end-to-end security & compliance

Microservices increase release agility
Monolithic application Microservices

Monolith

Service
Service
Service
Service
Service
Service
Service
Service
Service
Service
Service
Service

Rust
Database
DB
Database
Rust
GoNode.is
Java
Node.is
Node.is

Containers
Database
DB
Database
Containers
λContainers
VMs
Managed
Service

Proactive operations helps mitigate issues
Degraded state
Outage
Latency
Time (ms)

Observability in Control Theory
On the General Theory of Control Systems
R. E. KALMAN
Introduction
In no small measure, the great technological progress in
automatic control and communication systems during the past
two decades has depended on advances and refinements in the
mathematical study of such systems. Conversely, the growth
of technology brought forth many new problems (such as those
related to using digital computers in control, etc.) to challenge
the ingenuity and competence of research workers concerned
with theoretical questions.
Despite the appearance and effective resolution of many new
problems, our understanding of fundamental aspects of control
has remained superficial. The only basic advance so far appears
to be the theory ofinformation created by Shannon 1. The chief
significance of his work in our present interpretation is the
discovery ofgeneral' laws' underlying the process ofinformation
transmission, which are quite independent of the particular
models being considered or even the methods used for the des-
cription and analysis of these models. These results could be
compared with the' laws' of physics, with the crucial difference
that the' laws' governing man-made objects cannot be discovered
by straightforward experimentation but only by a purely abstract
analysis guided by intuition gained in observing present-day
examples of technology and economic organization. We may
thus classify Shannon's result as belonging to the pure theory
of communication and control, while everything else can be
labelled as the applied theory; this terminology reflects the well-
known distinctions between pure and applied physics or
mathematics. For reasons pointed out above, in its methodo-
logy the pure theory of communication and control closely
resembles mathematics, rather than physics; however, it is not
a. branch of mathematics because at present we cannot (yet?)
d1sregard questions of physical realizability in the study of
mathematical models.
This paper initiates study of the pure theory of control
imitating the spirit of Shannon's investigations but
using entirely different techniques. Our ultimate objective is
to answer questions of the following type: What kind and how
much information is needed to achieve a desired type ofcontrol?
What intrinsic properties characterize a given unalterable plant
as far as control is concerned?
At present only superficial answers are available to these
questions, and even then only in special cases.
Initial results presented in this Note are far from the degree
of generality of Shannon's work. By contrast, however, only
metho?s are employed here, giving some hope of
beIng able to aVOld the well-known difficulty of Shannon's
theory: methods of proof which are impractical for actually
constructing practical solutions. In fact, this paper arose
fr.om the need for a better understanding of some recently
d1scovered computation methods of control-system syn-
thesis 2-s. Another by-product of the paper is a new com-
putation method for the solution of the classical Wiener
filtering problem 7.
The organization of the paper is as follows:
16
In Section 3 we introduce the models for which a fairly
complete theory is available: dynamic systems with a finite
dimensional state space and linear transition functions (i.e.
systems obeying linear differential or difference equations).
The class of random processes considered consists of such
dynamic systems excited by an uncorrelated gaussian random
process. Other assumptions, such as stationarity, discretiza-
tion, single input/single output, etc., are made only to facilitate
the presentation and will be absent in detailed future accounts
of the theory.
In Section 4 we define the concept of controllability and show
that this is the' natural' generalization of the so-called' dead-
beat' control scheme discovered by Oldenbourg and Sartorius 21
and later rederived independently by Tsypkin22 and the author17•
We then show in Section 5 that the general problem ofoptimal
regulation is solvable if and only if the plant is completely
controllable.
In Section 6 we introduce the concept of observability and
solve the problem ofreconstructing unmeasurable state variables
from the measurable ones in the minimum possible length of
time.
We formalize the similarities between controllability and
observability in Section 7 by means of the Principle of Duality
and show that the Wiener filtering problem is the natural dual
of the problem of optimal regulation.
Section 8 is a brief discussion of possible generalizations and
currently unsolved problems of the pure theory of control.
Notation and Terminology
The reader is assumed to be familiar with elements of linear
algebra, as discussed, for instance, by Halmos 8.
Consider an n-dimensional real vector space X. A basis in
X is a set of vectors at ... , all in X such that any vector x in X
can be written uniquely as
(I)
the Xi being real numbers, the components or coordinates of x.
Vectors will be denoted throughout by small bold-face letters.
The set X* of all real-valued linear functions x* (= covec-
tors) on X. with the' natural' definition of addition and scalar
multiplication, is an n-dimensional vector space. The value of
a covector y* at any vector x is denoted by [y*, x]. We call
this the inner product of y* by x. The vector space X* has a
natural basis a*1... , a*n associated with a given basis in X;
it is defined by the requirement that
[a*j, aj] = Ojj
Using the' orthogonality relation' 2, we may write
form n
X = L [a*j, x]aj
j= t
which will be used frequently.
(2)
in the
(3)
For purposes of numerical computation, a vector may be
considered a matrix with one column and a covector a matrix
481
491
J.S.I.A.M. CONTROI
Ser. A, Vol. 1, No.
Printed in U.,q.A., 1963
MATHEMATICAL DESCRIPTION OF LINEAR
DYNAMICAL SYSTEMS*
R. E. KALMAN
Abstract. There are two different ways of describing dynamical systems: (i) by
means of state w.riables and (if) by input/output relations. The first method may be
regarded as an axiomatization of Newton’s laws of mechanics and is taken to be the
basic definition of a system.
It is then shown (in the linear case) that the input/output relations determine
only one prt of a system, that which is completely observable and completely con-
trollable. Using the theory of controllability and observability, methods are given
for calculating irreducible realizations of a given impulse-response matrix. In par-
ticular, an explicit procedure is given to determine the minimal number of state
varibles necessary to realize a given transfer-function matrix. Difficulties arising
from the use of reducible realizations are discussed briefly.
1. Introduction and summary. Recent developments in optimM control
system theory are bsed on vector differential equations as models of
physical systems. In the older literature on control theory, however, the
same systems are modeled by ransfer functions (i.e., by the Laplace trans-
forms of the differential equations relating the inputs to the outputs). Two
differet languages have arisen, both of which purport to talk about the
same problem. In the new approach, we talk about state variables, tran-
sition equations, etc., and make constant use of abstract linear algebra.
In the old approach, the key words are frequency response, pole-zero pat-
terns, etc., and the main mathematical tool is complex function theory.
Is there really a difference between the new and the old? Precisely what
are the relations between (linear) vector differential equations and transfer-
functions? In the literature, this question is surrounded by confusion [1].
This is bad. Communication between research workers and engineers is
impeded. Important results of the "old theory" are not yet fully integrated
into the new theory.
In the writer’s view--which will be argued t length in this paperthe
diiIiculty is due to insufficient appreciation of the concept of a dynamical
system. Control theory is supposed to deal with physical systems, and not
merely with mathematical objects such as a differential equation or a trans-
fer function. We must therefore pay careful attention to the relationship
between physical systems and their representation via differential equations,
transfer functions, etc.
* Received by the editors July 7, 1962 and in revised form December 9, 1962.
Presented at the Symposium on Multivariable System Theory, SIAM, November 1,
1962 at Cambridge, Massachusetts.
This research was supported in part under U. S. Air Force Contracts AF 49 (638)-382
and AF 33(616)-6952 as well as NASA Contract NASr-103.
Research Institute for Advanced Studies (RIAS), Baltimore 12, Maryland.
152
Downloaded11/11/13to152.3.159.32.RedistributionsubjecttoSIAMlicenseorcopyright;seehttp://www.siam.org/journals/ojsa.php
1961-62

Observability
In control theory, observability is a measure of
how well internal states of a system
can be inferred from knowledge
of its external outputs.
https://en.wikipedia.org/wiki/Observability

Levels of Observability
Network
Machine (HW, OS)
Application

The Three Pillars of Observability
Distributed Systems Observability by Cindy Sridharan

The Three Pillars of Observability
Event Logs Metrics Tracing
Distributed Systems Observability by Cindy Sridharan

Using Observability
Event Logs Metrics Tracing
Log aggregation
& analytics
VisualizationsAlerting

Observability on AWS
CloudWatch
Logs
CloudWatch
Metrics
AWS
X-Ray
Traces
CloudWatch
Insights
CloudWatch
Dashboard
CloudWatch
Alarms
AWS X-Ray
ServiceGraph
CloudWatch
Metric Filter

https://www.youtube.com/watch?v=rgfww8tLM0A

CloudWatch API PutMetricData
const metricData = await cloudWatch.putMetricData({
MetricData: [
{
MetricName: 'My Business Metric',
Dimensions: [
{
Name: 'Location',
Value: 'Paris'
}
],
Timestamp: new Date,
Value: 123.4
}
],
Namespace: METRIC_NAMESPACE
}).promise();
• Metric name
• Dimensions
• Timestamp
• Value
• Namespace

Add correlation IDs to logs – CloudWatch Logs + Insights

End-to-end tracing – AWS X-Ray Traces

AWS X-Ray Key Concepts
Segments
Subsegments

End-to-end tracing – AWS X-Ray Service Map

Enabling X-Ray tracing
AWS Lambda
Console
Amazon
API Gateway
Console

Enabling X-Ray tracing in your code
const AWS = require('aws-sdk');
const AWSXRay = require('aws-xray-sdk');
const AWS = AWSXRay.captureAWS(require('aws-sdk'));

Enabling X-Ray tracing in your code
const AWSXRay = require('aws-xray-sdk’);
const app = express();
app.use(AWSXRay.express.openSegment('my-segment'));
app.get('/send', function (req, res) {
res.setHeader('Content-Type', 'application/json’);
res.send('{"hello": "world"}');
});
app.use(AWSXRay.express.closeSegment());

Understand performance…
Systems Performance by Brendan Gregg

Understand performance… and latency…
Systems Performance by Brendan Gregg

Understand performance… and latency… and percentiles!
P50
P90
P99
P100

Choose the right integration patterns
✓
Decouple
and scale
distributed
systems
✓
Decouple
producers
from
subscribers
✓
Combine
multiple tasks
and manage
distributed state
Message
queue
Pub/sub
messaging
Workflows
Amazon
Simple Notification
Service (SNS)
Amazon
Simple Queue Service
(SQS)
AWS
Step Functions

What is needed
Consistent
communications
management
Complete visibility Failure isolation
and protection
Fine-grained
deployment controls

Client side traffic management
Traffic Shaping
Service discovery
Retries
Timeouts
Circuit breakers
Health checks
Routing Controls
Protocols support
Header based
Cookie based
Path based
Host based

Instrumentation options
Microservice
Container
In-process
(SDK)
Option 1

Option 1: In-process SDK
Java
Scala
Node.js
Python
C++
Django
.NET
GO
…

Instrumentation options
Microservice
Container
In-process
(SDK)
Out-of-process
(sidecar proxy)
Option 1 Option 2
Microservice
Container
Proxy

Option 2: Side-car proxy
Application CodeMicroservice
Proxy
Monitoring
Routing
Discovery
Deployment

Option 2: Side-car proxy
Proxy runs
as a container
Task or Pod
External traffic
Application
Code

Option 2: Proxy
Proxy

Why service mesh proxy
vs. Libraries or app code
Overall—migrate to microservices safer and faster
Reduce work required
by developers
Follow best practices Use any language
or platform
Simplify visibility,
troubleshooting, and
deployments

App Mesh configures every proxy
https://www.youtube.com/watch?v=GVni3ruLSe0

OSS project
Wide community support, numerous integrations
Stable and production-proven
“Graduated Project” in Cloud Native Computing Foundation
Started at Lyft in 2016
App Mesh uses Envoy proxy

Application observability
+ others
Faster
troubleshooting due
to consistent data
across services
Existing tools or
dashboards with a lot
more metrics, logs
and traces
Distinguish between
service and network
issues

Representing your sample app in App Mesh
Elastic
Load
Balancing
Microservices App Mesh
Mesh – [myapp]
Virtual
Node A
Service
Discovery
Listener Backends
Virtual
Node B
Service
Discovery
Listener Backends

Virtual Node
Virtual Node
Service
Discovery
BackendsListeners
Virtual Node
Logical representation
of runtime services
Backends
Set of destinations that this node
will communicate with (hostnames)
Service Discovery
Describes how its callers and locate this
node (DNS hostname or AWS Cloud Map*
namespace, serviced, and selectors)
Listeners
Policies to handle
incoming traffic
Ed: port, Health check*,
Circuit breaker*, Retries*

Virtual routes Destination’s virtual
router and route
Virtual router: B
HTTP routes
Match
Prefix: /
Action:
Targets
B
Route B
Virtual node
destination
+ weight
Route Name: B1
Match
Action:
Route Name: B2
Other Protocol routes

Update routes
Virtual router: B
HTTP route
targets:
prefix: /
B
B’
Route B
Virtual node
destination + weight
Route B’
New service or service
version

Mesh – [myapp]
Virtual
Node A
Service
Discovery
BackendListener
Virtual router
Domains
action:
match: /
B
B’
Service B
Service B’
Virtual
Node B’
Service
Discovery
Listener Backends
Virtual
Node B
Service
Discovery
Listener Backends

Virtual
Node B1
Mesh
Service A
Service B
Service C
Virtual
router
Virtual
router
Service D
Virtual
router

Demo

Takeaways
1. Build the instrumentation you need to understand what is happening
in your (distributed) application
2. Use technical and business metrics together to get better insights
3. Use correlation IDs in log and tracing frameworks to understand
distributed architectures (microservices)
4. Think at scale and plan for a service mesh control plane

Director, AWS Developer Technology & Evangelism
4 March 2019
Thank You - Any Questions?
Ian Massingham @IanMmmm ianm@amazon.com

Observability For Modern Applications

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Observability For Modern Applications

Similaire à Observability For Modern Applications (20)

Plus de Amazon Web Services

Plus de Amazon Web Services (20)

Observability For Modern Applications