By Satish Ramjee Principle Software Engineer - First Utility

1. Satish Ramjee
Principal Software Engineer
First Utility
June 2017
Managing Smart Meter
Data with DataStax DSE

2. • Who are First Utility
• Creating a high availability platform
• Enabling customer control and insights through analytics
• Providing a self healing platform with event history
• Monitoring and alerting across the platform
• Conclusion
Contents

3. Who are First Utility

4. First Utility
● Largest independent energy supplier
● Supply almost 1 million UK homes
● One of the first to provide Smart meters
● Provide broadband and home services as well
● Technology driven company

5. Creating a high availability platform

6. Creating a high availability platform
● National smart meter rollout
○ Aim is to have all 53 million residential premises in
England, Wales, Scotland with Smart Meters by 2020
● Energy suppliers are required to offer smart meters to all
their customers
● Our immediate need to manage a high volume of meter
reads with high availability
● Meter access devices can potentially provide data every 5-
10 seconds from the total meter estate

7. Creating a high availability platform
● Smart reads ideal for an event sourced type database
● Saw the need to add resilience, high volume and linear scaling capability into
our key information base
● Opportunity to manage business expectation through service layers
● Providing a meter agnostic business layer
“One of the things DataStax offered to us was the ability to really compress that learning curve. So we had
some of their key experts come in and spend a few weeks with us over a period of time and provide that
technical leadership” Bill Wilkins, CIO, First Utility.
http://diginomica.com/2017/02/02/first-utility-makes-smart-meters-even-smarter-datastax-platform/

8. Creating a high availability platform
store
From Monolithic to Microservices Architecture
Application Server
store
application
(instance 1)
application
(instance 1)
application
(instance 1)
Docker Container
application
(instance 2)
Application Server
Docker Container
application
(instance 1)
application
(instance 2)
single node
rdms
multiple node
distributed store

9. Creating a high availability platform
Meter Type
2
Meter Type
1
Customer
Insights
Billing
Industry
Data
Warehouse
Meter Type
n
spark
Usage View
Billing View
Industry
View
Data Stores
spark
Meter Data 1
Meter Data 2
Meter Data 3
spark

10. Creating a high availability platform
● Multiple DCs
● Appropriate
RF
● DCs split
between
Business
Critical access
and Analytics

11. Enabling Customer Control and
Insights

12. Enabling Customer Control and Insights
● Provide customers with an insightful view
● Help customers better manage their usage
● My energy comparison of usage
● Allow control of read submission frequency
● In home display reflects current tariff and usage

13. Enabling Customer Control and Insights
Meter
Customer
Insights
Meter
Preferences
Meter Data
Usage Viewread
preference/price TariffIHD
Data Storage
read

14. Enabling Customer Control and Insights

15. Enabling Customer Control and Insights

16. Enabling Customer Control and Insights

17. Providing a self healing platform

18. Providing a self healing platform
● Record commands sent to the meter
● Detect failures
● Replay commands (with retry limit)
Meter
commands
meter state
reconcile
replay
spark
Event
command
response

19. Monitoring and alerting across the
platform

20. Monitoring and alerting across the platform
● Eyes and ears
● Dashboards of platform behaviour
● Operational: is the platform working
● Business information: reporting and analytics

21. Monitoring and alerting across the platform

22. Conclusion

23. Conclusion
● Gone from read only system to ability control of meter (read/write)
● Self healing
● Real time monitoring and alerting
● Enduring solution can scale both processing and persistence
● Lessons learnt
○ Denormalise - many copies with different PK
○ Know your queries in advance
○ Reporting and aggregation - duplicate, spark or counters

24. That’s all folks