Antoine Coetsier - billing the cloud. Presentation from CloudStack European User Group, Thursday, April 19, London.

1. #CloudstackCephDay
Billing the cloud
A tale of devops

2. #CloudstackCephDay
Antoine Coetsier
● CEO & co-founder at Exoscale
● Engineer
● Since 2011

3. #CloudstackCephDay
Credits: Pierre-Yves Ritchards
● CTO & co-founder at Exoscale
● Open Source Developer
● Monitoring & Distributed Systems Enthusiast
● @pyr

4. #CloudstackCephDay
Billing the cloud
A tale of devops

5. #CloudstackCephDay
● Our Story
● Billing resources
● Scaling methodologies
● Our approach

6. #CloudstackCephDay

7. #CloudstackCephDay

8. #CloudstackCephDay
provider "exoscale" {
api_key = "${var.exoscale_api_key}"
secret_key = "${var.exoscale_secret_key}"
}
resource "exoscale_instance" "web" {
template = "ubuntu 17.04"
disk_size = "50g"
profile = "medium"
ssh_key = "production"
}

11. #CloudstackCephDay
Infrastructure isn’t free!
(sorry)

12. #CloudstackCephDay
Business Model
● Provide cloud infrastructure
● (???)
● Profit!

15. #CloudstackCephDay
10000 mile high view

17. Quantities

18. Quantities
● 10 megabytes have been sent from
159.100.251.251 over the last minute

19. Resources

20. Resources
● Account bar started instance foo with profile large
today at 12:00
● Account bar stopped instance foo today at 12:15

21. A bit closer to reality
{:type :usage
:entity :vm
:action :create
:time #inst "2017-11-12T15:48:32.000-00:00"
:template "ubuntu-17.04"
:source :compute
:account "modern-devops"
:uuid "7a070a3d-66ff-4658-ab08-fe3cecd7c70f"
:version 1
:offering "medium"}

22. #CloudstackCephDay
Theory

23. #CloudstackCephDay
Quantities are simple

25. #CloudstackCephDay
Resources are harder

27. #CloudstackCephDay
This is per account

29. #CloudstackCephDay
Solving for all events

30. resources = {}
metering = []
def usage_metering():
for event in fetch_all_events():
uuid = event.uuid()
time = event.time()
if event.action() == 'start':
resources[uuid] = time
else:
timespan = duration(resources[uuid], time)
usage = Usage(uuid, timespan)
metering.append(usage)
return metering

31. #CloudstackCephDay
In Practice

32. #CloudstackCephDay
● This is a never-ending process
● Minute-precision billing
● Applied every hour

33. #CloudstackCephDay
● Avoid overbilling at all cost
● Avoid underbilling (we need to eat!)

34. #CloudstackCephDay
● Keep a small operational footprint

35. #CloudstackCephDay
A naive approach

36. 30 * * * * usage-metering >/dev/null 2>&1

38. #CloudstackCephDay
Advantages

39. #CloudstackCephDay
● Low operational overhead
● Simple functional boundaries
● Easy to test

40. #CloudstackCephDay
Drawbacks

41. #CloudstackCephDay
● High pressure on SQL server
● Hard to avoid overlapping jobs
● Overlaps result in longer metering intervals

42. You are in a room full of overlapping cron jobs.
You can hear the screams of a dying MySQL server.
An Oracle vendor is here.
To the West, a door is marked “Map/Reduce”
To the East, a door is marked “Stream Processing”

43. > Talk to Oracle

44. You’ve been eaten by a grue.

45. > Go West

46. #CloudstackCephDay

47. #CloudstackCephDay
● Conceptually simple
● Spreads easily
● Data locality aware processing

48. #CloudstackCephDay
● ETL
● High latency
● High operational overhead

49. > Go East

50. #CloudstackCephDay

51. #CloudstackCephDay
● Continuous computation on an unbounded stream
● Each record processed as it arrives
● Very low latency

52. #CloudstackCephDay
● Conceptually harder
● Where do we store intermediate results?
● How does data flow between computation steps?

53. #CloudstackCephDay
Deciding factors

54. #CloudstackCephDay
Our shopping list
● Operational simplicity
● Integration through our whole stack
● Room to grow

55. #CloudstackCephDay
Operational simplicity
● Experience matters
● Spark and Storm are intimidating
● Hbase & Hive discarded

56. #CloudstackCephDay
Integration
● HDFS & Kafka require simple integration
● Spark goes hand in hand with Cassandra

57. #CloudstackCephDay
Room to grow
● A ton of logs
● A ton of metrics

58. #CloudstackCephDay
Small confession
● Previously knew Kafka

59. #CloudstackCephDay

61. #CloudstackCephDay
● Publish & Subscribe
● Processing
● Store

62. #CloudstackCephDay
Publish & Subscribe
● Records are produced on topics
● Topics have a predefined number of partitions
● Records have a key which determines their
partition

63. #CloudstackCephDay
● Consumers get assigned a set of partitions
● Consumers store their last consumed offset
● Brokers own partitions, handle replication

65. #CloudstackCephDay
● Stable consumer topology
● Memory disaggregation
● Can rely on in-memory storage
● Age expiry and log compaction

66. #CloudstackCephDay

67. #CloudstackCephDay
Billing at Exoscale

71. #CloudstackCephDay
Problem solved?

72. #CloudstackCephDay
● Process crashes
● Undelivered message?
● Avoiding overbilling

73. #CloudstackCephDay
Reconciliation
● Snapshot of full inventory
● Converges stored resource state if necessary
● Handles failed deliveries as well

74. #CloudstackCephDay
Avoiding overbilling
● Reconciler acts as logical clock
● When supplying usage, attach a unique transaction ID
● Reject multiple transaction attempts on a single ID

75. #CloudstackCephDay
Avoiding overbilling
● Reconciler acts as logical clock
● When supplying usage, attach a unique transaction ID
● Reject multiple transaction attempts on a single ID

76. #CloudstackCephDay
Parting words

77. #CloudstackCephDay
Looking back
● Things stay simple (roughly 600 LoC)
● Room to grow
● Stable and resilient
● DNS, Logs, Metrics, Event Sourcing

78. #CloudstackCephDay
What about batch?
● Streaming doesn’t work for everything
● Sometimes throughput matters more than latency
● Building models in batch, applying with stream
processing

79. #CloudstackCephDay
Thanks!
Questions?
(we are hiring!)