HBaseCon 2012 | Overcoming Data Deluge with HBase to Help Save the Environment - OPower
•Télécharger en tant que PPT, PDF•
0 j'aime•3,262 vues
Opower is a fast moving energy management SaaS company that collects sensor data from nearly all of the major utilities in the United States–meaning from more than 45 million American households–along with major utilities in 5 countries throughout Europe and AsiaPac. Opower manages more than 100 billion meter reads, ranging from high frequency power data (AMI), smart thermostats data, and weather data. Currently all data at Opower is stored in HBase or Hadoop (and is notably not security sensitive). This discussion will discuss Opower’s HBase architecture, highlight potential and current uses of data in HBase, share the vision of Opower’s future projects and directions, and reveal how Opower’s big data management has allowed the company to help its utility clients save enough energy to power a city of nearly 200,000 people and save utility customers more than $70 million since only 2008!
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
En vedette
En vedette (20)
Similaire à HBaseCon 2012 | Overcoming Data Deluge with HBase to Help Save the Environment - OPower
Similaire à HBaseCon 2012 | Overcoming Data Deluge with HBase to Help Save the Environment - OPower (20)
Plus de Cloudera, Inc.
Plus de Cloudera, Inc. (20)
Dernier
Dernier (20)
HBaseCon 2012 | Overcoming Data Deluge with HBase to Help Save the Environment - OPower
- 1. HBase to Save the Planet Alex Newman posix4e@apache.org Architect, Drawn to Scale Strategic Advisor, Opower
- 2. My life with HBase Drawn to Factset Cloudera Opower Scale
- 3. About Opower Opower is a customer engagement platform for the utility industry
- 4. About Opower Home energy reports Customized utility bills Energy efficiency programs for utilities
- 5. About Opower Opower runs on analytics Analytics run on Hadoop + HBase
- 6. Opower analysis relies on data from a variety of sources » Electric Utility Usage » Thermostat » Weather » Gas Utility Usage Data data data Data Data Storage & 4 Shared Energy Processing Signature Repository 3 1 2 Disaggregation OPOWER Algorithms Platform
- 7. Opower’s first architecture could not support their analytic vision MySQL Scalability? Performance? Data integration?
- 8. Opower’s first architecture could not support their analytic vision Analytic workflow instead of analytic apps: SQL -> CSV -> R -> too little, too slow
- 9. Problem #1 Data Lake Cost Usage AMI Regional AMI Sensor Data Data Lake
- 10. Problem #2 Slower and slower queries Smart-grid-scale data Lots of supporting data: weather, demographics, etc.
- 11. Problem #3 It was taking lots of “magic” Intense analytics Strange schemas Segmented queries
- 12. Hadoop + HBase at Opower Opower determined that they needed an entirely new data architecture
- 13. NexGen Architecture @ Opower
- 14. Hadoop + HBase at Opower Early success: HBase AMI
- 15. What rocked Endless, cheap scalability
- 16. What rocked The analytics team loved it!
- 17. What sucked Hard on the ops team – still trying to grok it
- 18. What sucked NoSchema p1. Creating Schema Managing MetaData Schema <=> Performance
- 19. What sucked HA Failover Snapshots
- 20. What sucked No secondary index Aggregation is slow (Rollup/OLAP) Poor Client Performance
- 21. It would be better if only … Developers were not forced to know how the data is stored, indexed, etc.
- 22. It would be better if only … There were nicer APIs and better query languages (SQL?)
- 23. It would be better if only … Version migrations were easy Hierarchical Tables
- 24. It would be better if only … Real-time tuning
- 25. It would be better if only … Did I mention HA?
- 26. In summary HBase has helped Opower achieve their analytic vision But they’ve still got a long way to go HBase still has a long way to go
- 27. Questions? Alex Newman posix4e@apache.org Architect, Drawn to Scale Strategic Advisor, Opower
Notes de l'éditeur
- Name Email Address Title
- WARNING THESE ARE MY WORDS, not FDS, Cloudera or OPower Factest 2005: - Maybe I was crazy to use it - Tens of databases 10 of query langagues, VMS moving towards commdity servers. Running into issues with scaling on environments like MySQL - They were used to code that crashed. In fact, I would say while I was on call, a service from one of the sites was down, at least once a week. Luckily they had redundancy in multiple sites, and multiple servers within those sites. The redundancy was added at a higher level, so generally, at least all of the times I remember, it was able to increase the availability and downtime wasn't actually an issue. - What was an issue was scale. - INteresting enough Hbase, even at that time, was a pretty highly available database. So what did they use it for - Time and Sales. This is the collection of all of the Quotes and Trades, for different securities. So to translate you put out quotes to buy or sell stocks at a certain price. If they overlapp, the echange registers a trade, and you just bought or sold a security. Not just stocks, but options and extremely high frequency data. - There was some value add on top of that, for calculating more complicated statistics on the fly through a home grown Web SASS thing - Cloudera: - Started off in kitchen focusing on building the packages that y’all know and love. When I entered it was all manual, when I left it was all automated. One could think of this as sortof like dev-opsie, meets, qa, meets release engineering, meets generic development - Moved into our first management tools team as a developer. Where we developed the cloudera manager. It was originally part of HUE and it became more springy. - Then I left Cloudera to be a founder in Drawn to Scale. We built a prototype and started pitching it for about 6 to 7 months. - While that was going on, I because the Lead Data Architect at OPower. And then more recently, after funding, I have returned to drawntoscale as a coder in the trenches, and have changed myself to a advisor to opower. The reason why I bring this up, is I have been working with HBase in production for about 5 years.
- Opower helps people use energy more efficiently and ultimately save money on their energy bills.it vastly improves the overall customer experience by making energy use personally relevant. - Behavioral Science (Great marketing, understanding people, great hci) - Data Science (Analytics, Data Infastructure Teams) - Lobbying (Yep we do lobbying)
- - OHow many of you get a bill - OPower White labeled websites. So this is the interface you probably use through your energy website to view how much power you use. Bill forecasting, etc. - Smart Thermostats - Gas and Electric - Social
- - Analytics is used to understand who we should be targeting - Answering questions that our customers what answered. We can help them improve customer service, improve there marketing, etc. - Justifying our own existence. (Compliance)
- - This is an old slide which doesn’t really include all the places we get data - Story about detecting broken thermostats
- - But it had it’s up - Spring and MVC provided a very clear and systematic way for developers developer systems. - It was very easy to manage from an operations perspective.
- - WE did this at FDS as well. Of course not with R, but specialized langauages. - IN fact our customers did as well, and they had a whole team of people to help customers do it.
- So here is the data sizes we have, along with the costs with traditional hadoop systems. - We were a cisco shop but we ended up going with dell, mostly because of the 3.5 inch disks. It looks like cisco is wising up to this whole hadoop thing. - These numbers are for dell. So I think this is priced out assuming a 710, then a 810 and then a 910 for the RDBMS, and 510's for hadoop.
- - A lot of this data just doesn’t work well with traditional databases. - An unnamed utility takes 3 days to mysqldump the ami data out. subsampling interpolation
- - I should warn you, i drawn almost all of my drawings in xfig so if this isn’t clear I’m sorry. - Basically the utility data has to come in from a variety of different protocols, as we integrate into the utility pipeline. It then flows into hbase, it’s validated from hbase, and then imported into our existent workflow. - Some of that data, you could imagine for instance information about user is still stored in MySQL. - All of the data is in a HIVE data lake
- All of our timeseries data in regards to high frequency data is being ported to being stored in HBase. Also soon things like bill forecasting, and a bunch of cool other stuff I probably should mention is being moved here. This includes data from the utilities, and data that users are enterring themselves. In additition thermostat data is moving here.
- - We still need to improve effeciency - We are doubling the size of the cluster this year - We have a ton of room to grow.
- - Having all of your data is a huge thing - Having a place to do m/r based R is great - No more running out of memory or being bounded to a single machine - Having a cheap scratch space
- At cloudera i thought all we needed was cfengine, snmp and syslog. Frankly that would have made ops happy. But more and more I think we made the right decision and that these tools really aren’t the right answer. JuJu looks interesting. - cloudera of course built there own tool. - access and auth