Extending Flux to Support Other Databases and Data Stores | Adam Anthony | InfluxData

Adam Anthony
Software Engineer, Flux team
Extending Flux –
Writing Custom Data
Sources

© InfluxData. All rights reserved.
Why Custom Data Sources
import “sql”
phBalances = from(bucket: "PH_Balances")
|> range(start: now- )
|> filter(fn: (r) => r._measurement == "water_sensor" )
deviceIds = sql.from(driverName: “mysql”,
dataSourceName: “username:password@tcp(host)/dbname”,
query: "SELECT customer_name, customer_location, device_id AS devicen"+
"FROM DeviceTable WHERE customer_name = ‘InfluxData’”)
join(tables: {t : phBalances, t : deviceIds}, on: ["device"])
|> aggregateWindow(every: v.windowPeriod, fn: mean)
|> yield(name: "mean")

Overview
Pre-Demo
Flux Extension landscape
Flux Data Model
Creating tables from scratch
Source Decoder Interface
Row Reader Interface
Example: Promql Ephemeral Scraper

Pre-Demo
● Counting Table Generator
● columns, N rows
● Each row has value +=

Flux Extension Landscape
Write pure ﬂux functions
Port scalar constants and functions from a Go library
Stream transformations
– Go API for processing streams of tables
– Used when pure ﬂux cannot work
Data Sources/Sinks
– If there's a go library for it, you can connect it to Flux

SINKSOURCE
Query Pipeline
from(bucket:
"my-bucket")
range(start: -5m)
filter(fn: (r) =>
r._measurement ==
"m", r._field == "f1")
window(every:5m) mean()
to(bucket:
"my-output")
Sinks may be
● custom writes to persistent storage
● client output (e.g. CSV encoding,
JSON encoding, charts, http, etc.)

Columnar Data Stores
Logical
Joins vs. Aggregates
– Sales Lead vs. Avg Host CPU
1145 US-West nginx.svc 89%
_time Region Host CPU
1146 US-West docker.svc 99%
60%
Mem
99%
Physical
Disk + Memory locality
Cache locality
Data Requirements

Flux Data Model
An inﬁnite stream…
Of Finite tables…
Identiﬁed by the GroupKey
Time Host _field Value
10:45 H1A cpu 25.9
11:00 H1A cpu 20.0
GroupKey[Host=H1A,_field]
Time Host _field Value
11:45 H1A mem 18.0
12:00 H1A mem 34.8
GroupKey[Host=H2B,_field]
...

Stream Transformation Process
Time Host Value
10:30 H1A 25.9
10:45 H1A 20.0
GroupKey[Host=H1A]
Time Host Value
11:45 H2B 18.0
11:50 H2B 34.8
GroupKey[Host=H2B]
Time Host Value
10:00 H1A 45.9
GroupKey[Host=H1A]
Time Host Value
11:00 H2B 52.8
GroupKey[Host=H2B]
|> aggregateWindow(every: 60m, fn: sum))
For Each Table:
. Convert
incoming rows
into or more
outgoing rows
. Sort outgoing
rows into tables

CODE ALERT
Counting Table Demo:
● Written in go
● Will Learn:
○ how to create a table
○ how to install the source as a ﬂux function

Building Tables
func BuildStaticTable (keyColumn, valueColumn , key string, nrows int64, a execute.Administration ) (flux.Table, error) {
// 1. group keys help ID a table
groupKey := execute.NewGroupKeyBuilder (nil)
groupKey.AddKeyValue (keyColumn, values. NewString(key))
gk, err := groupKey.Build()
if err != nil { return nil, err }
// 2. Create a new table builder indexed by the group key.
builder := execute.NewColListTableBuilder (gk, a.Allocator())
if _, err = builder. AddCol(flux.ColMeta{Label: keyColumn, Type: flux.TString}); err != nil { return nil, err }
if _, err = builder. AddCol(flux.ColMeta{Label: valueColumn, Type: flux.TFloat}); err != nil { return nil, err }
// 3. Add a row of data by appending one value to each column.
for i := 0; i < int(nrows); i++ {
if err = builder. AppendString (0, key); err != nil { return nil, err
}
if err = builder. AppendFloat (1, float64(i)); err != nil {
return nil, err
}
}
return builder.Table()
}

Implementing Custom Data Sources
type SourceDecoder interface {
// Create a connection to a data source
Connect(ctx context.Context) error
// Fetch all data for a single, complete table
Fetch(ctx context.Context) (bool, error)
// Given data fetched above, decode it into a flux table
Decode(ctx context.Context) (flux.Table, error)
Close() error
}

A First Decoder
type StaticDecoder struct {
administration execute.Administration
keyColumn string
valueColumn string
key string
nrows int64
}
func (s *StaticDecoder) Connect(ctx context.Context) error {
return nil
}
func (s *StaticDecoder) Fetch(ctx context.Context) (bool, error) {
return false, nil
}
func (s *StaticDecoder) Decode(ctx context.Context) (flux.Table, error) {
return BuildStaticTable(s.keyColumn, s.valueColumn, s.key, s.nrows, s.administration)
}
func (s *StaticDecoder) Close() error {
return nil
}

Install A Source Decoder: OpSpec
// unique name for mapping
const FromStaticKind = "fromStatic"
// storing user params that are declared elsewhere
// op spec represents what the user has told us;
type FromStaticOpSpec struct {
nrows int64
}
func createFromStaticOpSpec(args flux.Arguments, administration *flux.Administration) (flux.OperationSpec, error) {
spec := new(FromStaticOpSpec) // reading flux.args and extracting params
var err error
if spec.nrows, err = args.GetRequiredInt("nrows"); err != nil { return nil, err }
return spec, nil
}
func newFromStaticOp() flux.OperationSpec {
return new(FromStaticOpSpec)
}
func (s *FromStaticOpSpec) Kind() flux.OperationKind {
return FromStaticKind
}
● OpSpec: Collect User Parameters and Store for Execution

Install a Source Decoder: Procedure Spec
type FromStaticProcedureSpec struct {
plan.DefaultCost
nrows int64
}
// use op spec to initialize procedure spec
func newFromStaticProcedure(qs flux.OperationSpec, pa plan.Administration) (plan.ProcedureSpec, error) {
spec, ok := qs.(*FromStaticOpSpec)
if !ok { return nil, fmt.Errorf("invalid spec type %T", qs) }
return &FromStaticProcedureSpec{nrows: spec.nrows},nil
}
func (s *FromStaticProcedureSpec) Kind() plan.ProcedureKind {
return FromStaticKind
}
func (s *FromStaticProcedureSpec) Copy() plan.ProcedureSpec {
ns := new(FromStaticProcedureSpec)
return ns
}
● procedure spec is internal representation of the entire ﬁle used by the planner

Install A Source Decoder: Create Source
// uses a procedure spec to create a source object for flux runtime
func createFromStaticSource (prSpec plan.ProcedureSpec, dsid execute.DatasetID, a execute.Administration )
(execute.Source, error) {
spec, ok := prSpec.(*FromStaticProcedureSpec)
if !ok {
return nil, fmt.Errorf("invalid spec type %T", prSpec)
}
StaticDecoder := StaticDecoder{
administration: a,
keyColumn: "T1",
valueColumn: "V1",
key: "tag1",
nrows: spec.nrows}
return execute.CreateSourceFromDecoder (&StaticDecoder, dsid, a)
}

Install A Source Decoder: Register
Constructors
func init() {
fromStaticSignature := semantic.FunctionPolySignature{
Parameters: map[string]semantic.PolyType{
"nrows": semantic.Int,
}, // user params
Required: semantic.LabelSet{"nrows"},
Return: flux.TableObjectType,
}
// tell the flux runtime about the objects that we're creating
flux.RegisterPackageValue("static", "from",
flux.FunctionValue(FromStaticKind, createFromStaticOpSpec, fromStaticSignature))
flux.RegisterOpSpec(FromStaticKind, newFromStaticOp)
plan.RegisterProcedureSpec(FromStaticKind, newFromStaticProcedure, FromStaticKind)
execute.RegisterSource(FromStaticKind, createFromStaticSource)
}

Final Steps
. flux package: flux/stdlib/static/static.flux
. Put `from.go` in same dir
. Flux root dir: run 'make'
. Build Flux/Influxdb binary

Data!

Demo Code
static.from:
https://github.com/influxdata/flux/tree/demo/influxdaysSFO2019
prometheus.scrape:
https://github.com/influxdata/flux/blob/cb2d438ac7881c794c8a2982618e81503b4f2781
/stdlib/experimental/prometheus/scrape.go

Not Covered
● Writing data out.
○ Modeled after a normal ﬂux function that sends data out as a side
effect
○ For each table:
i. Collect rows from a ﬂux.table into a write request for your data destination
ii. Send the request
● Alternate Interface: RowIterator

If you can connect to a data source in `go` you can write a
source/sink for it in flux.
● Currently, we can read from:
○ influxdb
○ mysql
○ postgres
○ BigTable
○ CSV
○ Prometheus
● Currently we can write to:
○ influxdb
○ mysql
○ postgres
○ MQTT
○ kafka
Contributions are welcome!

Extending Flux to Support Other Databases and Data Stores | Adam Anthony | InfluxData

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Extending Flux to Support Other Databases and Data Stores | Adam Anthony | InfluxData

Similaire à Extending Flux to Support Other Databases and Data Stores | Adam Anthony | InfluxData (20)

Plus de InfluxData

Plus de InfluxData (20)

Dernier

Dernier (20)

Extending Flux to Support Other Databases and Data Stores | Adam Anthony | InfluxData