A quick overview of CAVE, a managed service for monitoring infrastructure, platform, and application metrics, to provide visibility into your system's performance and operational levels. CAVE is built at GILT, using Scala, Play and Akka.
4. What is CAVE?
A monitoring system that is:
● secure
● independent
● proprietary
● open source
5. Requirements
● horizontally scalable to millions of metrics, alerts
● multi-tenant, multi-user
● extensible HTTP-based API
● flexible metric definition
● data aggregation / multiple dimensions
● flexible and extensible alert grammar
● pluggable notification delivery system
● clean user interface for graphing and dashboarding
13. Alert Grammar
Metric has name and tags (key-value pairs)
e.g.
orders [shipTo: US]
response-time [svc: svc-important, env: prod]
14. Alert Grammar
Aggregated Metric has metric, aggregator and
period of aggregation, e.g.
orders [shipTo: US].sum.5m
response-time [svc: svc-important, env: prod].p99.5m
Supported aggregators:
count, min, max, mean, mode, median, sum
stddev, p99, p999, p95, p90
15. Alert Grammar
Alert Condition contains one expression with
two terms and an operator. Each term is a
metric, an aggregated metric or a value.
e.g.
orders [shipTo: US].sum.5m < 10
orders [shipTo: US].sum.5m < ordersPredictedLow [shipTo: US]
16. Alert Grammar
An optional number of times the threshold is
broken, e.g.
response-time [svc: svc-team, env: prod].p99.5m > 3000 at least
3 times
17. Alert Grammar
Special format for missing data
e.g.
orders [shipTo: US] missing for 5m
heartbeat [svc: svc-important, env: prod] missing for 10m
18. Alert Grammar
trait AlertParser extends JavaTokenParsers {
sealed trait Source
case class ValueSource(value: Double) extends Source
case class MetricSource(
metric: String, tags: Map[String, String]) extends Source
case class AggregatedSource(
metricSource: MetricSource,
aggregator: Aggregator, duration: FiniteDuration) extends Source
sealed trait AlertEntity
case class SimpleAlert(
sourceLeft: Source, operator: Operator,
sourceRight: Source, times: Int) extends AlertEntity
case class MissingDataAlert(
metricSource: MetricSource, duration: FiniteDuration) extends AlertEntity
…
}
19. Alert Grammar
trait AlertParser extends JavaTokenParsers {
…
def valueSource: Parser[ValueSource] = decimalNumber ^^ {
case num => ValueSource(num.toDouble)
}
def word: Parser[String] = """[a-zA-Z][a-zA-Z0-9.-]*""".r
def metricTag: Parser[(String, String)] = (word <~ ":") ~ word ^^ {
case key ~ value => key -> value
}
def metricTags: Parser[Map[String, String]] = repsep(metricTag, ",") ^^ {
case list => list.toMap
}
…
}
26. Scala Collections API
case class Person(id: Int, name: String)
val list = List(Person(1, "Pawel"),
Person(2, "Val"),
Person(3, "Unknown Name"))
27. Scala Collections API
case class Person(id: Int, name: String)
val list = List(Person(1, "Pawel"),
Person(2, "Val"),
Person(3, "Unknown Name"))
list.filter(_.id > 1)
28. Scala Collections API
case class Person(id: Int, name: String)
val list = List(Person(1, "Pawel"),
Person(2, "Val"),
Person(3, "Unknown Name"))
list.filter(_.id > 1).map(_.name)
29. Scala Collections API
case class Person(id: Int, name: String)
val list = List(Person(1, "Pawel"),
Person(2, "Val"),
Person(3, "Unknown Name"))
list.filter(_.id > 1).map(_.name)
SELECT name FROM list WHERE id > 1
31. Entity mapping
/** Table description of table orgs.*/
class OrganizationsTable(tag: Tag) extends Table[OrganizationsRow](tag,"organizations") {
...
/** Database column id AutoInc, PrimaryKey */
val id: Column[Long] = column[Long]("id", O.AutoInc, O.PrimaryKey)
/** Database column name */
val name: Column[String] = column[String]("name")
/** Database column created_at */
val createdAt: Column[java.sql.Timestamp] = column[java.sql.Timestamp]("created_at")
…
/** Foreign key referencing Organizations (database name token_organization_fk) */
lazy val organizationsFk = foreignKey("token_organization_fk", organizationId,
Organizations)(r => r.id, onUpdate = ForeignKeyAction.NoAction, onDelete =
ForeignKeyAction.NoAction)
}
32. CRUD
val organizationsTable = TableQuery[OrganizationsTable]
// SELECT * FROM ORGANIZATIONS
organizationsTable.list
// SELECT * FROM ORGANIZATIONS WHERE ID > 10 OFFSET 3 LIMIT 5
organizationsTable.filter(_.id > 10).drop(3).take(5).list
// INSERT
organizationsTable += OrganizationsRow(1, "name", "email", "notificationUrl", ... , None,
None)
// UPDATE ORGANIZATIONS SET name = “new org name” WHERE ID=10
organizationsTable.filter(_.id === 10).map(_.name).update("new org name")
// DELETE FROM ORGANIZATIONS WHERE ID=10
organizationsTable.filter(_.id === 10).delete
33. Queries - JOINS
val organizationsTable = TableQuery[OrganizationsTable]
val teamsTable = TableQuery[TeamsTable]
val name = “teamName”
val result = for {
t <- teamsTable.sortBy(_.createdAt).filter(t => t.deletedAt.isEmpty)
o <- t.organization.filter(o => o.deletedAt.isEmpty && o.name === name)
} yield (t.name, o.name)
SELECT t.name, o.name FROM TEAMS t
LEFT JOIN ORGANIZATIONS o ON t.organization_id = o.id
WHERE t.deleted_at IS NULL AND o.deleted_at IS NULL AND o.name = `teamName`
ORDER BY t.created_at
34. SELECT t.name, o.name FROM TEAMS t
LEFT JOIN ORGANIZATIONS o ON t.organization_id = o.id
WHERE t.deleted_at IS NULL AND o.deleted_at IS NULL AND o.name = `teamName`
ORDER BY t.created_at
val result: List[(String, String)]
35. Connection pool and transactions
val ds = new BoneCPDataSource
val db = {
ds.setDriverClass(rdsDriver)
ds.setJdbcUrl(rdsJdbcConnectionString)
ds.setPassword(rdsPassword)
ds.setUser(rdsUser)
Database.forDataSource(ds)
}
db.withTransaction { implicit session =>
// SLICK CODE GOES HERE
}