If knowing is half the battle, having the most information available is the best way to win. Using real-time log streaming and a knowledge of the data passing through the system, metrics can provide more depth and breadth in to the goings on requests as they pass through various parts of the stack. This session will cover the difference between logging and metrics, writing JSON and Influx Line Protocol in VCL, and building out dashboards to give deeper insights (and more importantly, alerting) on requests and responses at the edge.

1. Leveraging Log Streaming to
Build the Best Dashboards, Ever
Eddie Roger
Principal Engineer, API Platform
Target
@eddieroger

7. Metrics Metrics tell the story of what’s happening right now.
Logs Logs tell you what went wrong.

8. if (resp.status >= 400) {
log {"syslog $SERVICE_ID DEST-APP :: "}
{"{"}
{" "product":"api", "}
{" "x-api-id":""} req.http.X-API-ID {"", "}
{" "api_name":""} req.http.API-Name {"", "}
{" "request_url":""} req.url {"", "}
{" "req_body_read":"} req.bytes_read {", "}
{" "resp_body_written":"} resp.bytes_written {", "}
{" "request_backend":""} req.http.Request-Backend {"", "}
{" "actual_backend_name":""} req.http.Actual-Backend-Name {"", "}
{" "actual_backend_ip":""} req.http.Actual-Backend-IP {"", "}
{" "client_ip":""} client.ip {"", "}
{" "event_timestamp":""} strftime({"%Y-%m-%dT%H:%M:%SZ"}, now) {"", "}
{" "method":""} req.request {"", "}
{" "resp_status":""} resp.status {"", "}
{" "resp_message":""} resp.response {"", "}
{" "cache_status":""} resp.http.X-Cache {"", "}
{" "elapsed_time":"} time.elapsed.msec {", "}
{" "fastly_info_state":""} fastly_info.state {"", "}
{" "fastly_pop":""} server.datacenter {"", "}
{" "fastly_region":""} server.region {"", "}
{" "fastly_server":""} server.identity {"", "}
{" "scheme_type":""} req.http.SchemeType {"""}
{"}"};
}
Only on non-OK
Where should it go?
Where did it go?
Which API?

11. Syslog

12. Metrics Metrics tell the story of what’s happening right now.
Logs Logs tell you what went wrong.

13. Metrics Metrics tell the story of what’s happening right now.
Logs Metrics Logs Metrics tell you what went wrong.

15. https://docs.influxdata.com/influxdb/v1.5/write_protocols/line_protocol_reference/
Measurement
Tag
Field
Measurement is the grouping of the data you’re collecting. Think
database table. This is required, and only one is allowed.
A tag is an attribute of the measure itself. These are useful in the
equivalent of group by or where clauses. Tags are strings.
A field is the measurement itself. This can be a float, int,
string or bool.
Timestamp When the measurement was taken, ideally in nanoseconds since
epoch.
Cardinality The number of elements in a set. In Influx, it’s the combination of
all tags in the measurement.

16. https://docs.influxdata.com/influxdb/v1.5/write_protocols/line_protocol_reference/
<measurement>[,<tag_key>=<tag_value>[,<tag_key>=<tag_value>]] <field_key>=<field_value>[,<field_key>=<field_value>] [<timestamp>]

17. cdnmetrics,product=api,environment=prod,cache_status=HIT,resp_status=200,api_name=locations-v2—,request_backend=director_Target,actual_backend=$SERVICE—ActualTargetBackend,info_state=HIT,pop=MDW,region=US-Central elapsed_time=0,time_to_first_byte=0.000 1522670899146991000

18. cdnmetrics,
product=api,
environment=prod,
cache_status=HIT,
resp_status=200,
api_name=locations-v2,
request_backend=director_Target,
actual_backend=$SERVICEID—TargetBackend1,
info_state=HIT,
pop=MDW,
region=US-Central
elapsed_time=0,
time_to_first_byte=0.000
1522670899146991000
Measurement
Tags
Fields
Timestamp

19. log {"syslog $SERVICE-ID DEST-METRICS :: "}
{"cdnmetrics,"}
{"product=api,"}
{"environment=prod,"}
{"cache_status="} resp.http.X-Cache {","}
{"resp_status="} resp.status {","}
{"api_name="} req.http.TGT-Sane-Name {","}
{"request_backend="} req.http.TGT-Backend {","}
{"actual_backend="} req.http.TGT-Actual-Backend-Name {","}
{"info_state="} fastly_info.state {","}
{"pop="} server.datacenter {","}
{"region="} server.region
{" "}
{"elapsed_time="} time.elapsed.msec {","}
{"time_to_first_byte="} time.to_first_byte
{" "}
time.end.usec {"000"};
Super duper
important
Super duper
important

31. Metrics Metrics tell the story of what’s happening right now.
Logs Metrics Logs Metrics tell you what went wrong.
Alerting Alerting tells you when you actually need to worry,
so you can calmly enjoy your morning coffee.

32. Metrics Metrics tell the story of what’s happening right now.
Logs Metrics Logs Metrics tell you what went wrong.
Alerting Alerting tells you when it’s time to worry.

35. Helpful Tips and Tricks
• No one knows your data better than you do - use that knowledge to build
your log stream.
• Cut Influx some slack - either use short windows with faster refreshes, or big
windows with slower refreshes, but not both.
• Tag smartly, but generously.
• Learn the feel of your data.
• Ease in to metrics.
• Don’t drink too much coffee.

36. Let’s Talk!
tech.target.com
@target
@eddieroger