Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
Living Data: Applying Adaptable Schemas to HBaseAaron Kimball – CTO WibiData, Inc.
HBase is a nexus for your data
HBase: Schema free (unfortunately)• Cells only hold byte arrays• Column names implicitly defined by apps• Each app m...
Datatypes can get rooted in place =
Avro: Flexible schemas =
Avro decouples schemas
Every cell stores its schema (hash)
Layout table stores common schemas <column> <name>info:email</name> <description>User email address</descript...
Conclusions• Avro allows decoupled applications to: – Share the same data store – Change individual applications without...
www.wibidata.com / @wibidata Aaron Kimball – aaron@wibidata.com
HBaseCon 2012 | Living Data: Applying Adaptable Schemas to HBase - Aaron Kimball, WibiData
Prochain SlideShare
Chargement dans…5
×
Technologie
3 947 vues
25 mai 2012

HBaseCon 2012 | Living Data: Applying Adaptable Schemas to HBase - Aaron Kimball, WibiData

HBase application developers face a number of challenges: schema management is performed at the application level, decoupled components of a system can break one another in unexpected ways, less-technical users cannot easily access data, and evolving data collection and analysis needs are difficult to plan for. In this talk, we describe a schema management methodology based on Apache Avro that enables users and applications to share data in HBase in a scalable, evolvable fashion. By adopting these practices, engineers independently using the same data have guarantees on how their applications interact. As data collection needs change, applications are resilient to drift in the underlying data representation. This methodology results in a data dictionary that allows less-technical users to understand what data is available to them for analysis and inspect data using general-purpose tools (for example, export it via Sqoop to an RDBMS). And because of Avro’s cross-language capabilities, HBase’s power can reach new domains, like web apps built in Ruby.

no profile picture user

  • Identifiez-vous pour voir les commentaires

HBaseCon 2012 | Living Data: Applying Adaptable Schemas to HBase - Aaron Kimball, WibiData

  1. 1. Living Data: Applying Adaptable Schemas to HBaseAaron Kimball – CTO WibiData, Inc.
  2. 2. HBase is a nexus for your data
  3. 3. HBase: Schema free (unfortunately)• Cells only hold byte arrays• Column names implicitly defined by apps• Each app must (de)serialize values correctly• Changing a schema requires rewriting a column—and updating every reader/writer
  4. 4. Datatypes can get rooted in place =
  5. 5. Avro: Flexible schemas =
  6. 6. Avro decouples schemas
  7. 7. Every cell stores its schema (hash)
  8. 8. Layout table stores common schemas <column> <name>info:email</name> <description>User email address</description> <schema>“string”</schema> </column>• Data dictionary provides reference to engineers on different projects• Common schemas used by tools that want to enforce a “default” schema for a column (e.g., Sqoop-based exports)
  9. 9. Conclusions• Avro allows decoupled applications to: – Share the same data store – Change individual applications without downtime – Eliminates need to structurally modify data• Layout management allows: – Developers to communicate about data without using code – Data-agnostic applications to manipulate structured information
  10. 10. www.wibidata.com / @wibidata Aaron Kimball – aaron@wibidata.com

×