Kite SDK: Working with Datasets

Kite SDK: HBase Datasets
Ryan Blue, Software Engineer

What problem is Kite solving?
©2014 Cloudera, Inc. All rights reserved.
• Accessibility
• Hadoop is flexible, but low level
• Should be easy to use, without being an expert

Kite SDK
• A set of off-the-shelf tools
• Based on experience and best practices
• Lets you focus on your problem
• Helps you solve new challenges

Kite Datasets: Motivation
Focus on using your data, not managing it
• You shouldn’t have to maintain data files
• This is the first thing you need

Application
Database
Data files
Your code
Provided
Maintained by the database

Application Application
Database
Data files
Data files HBase
Your code

Application ApplicationApplication
Database
Data files
Data files
Kite Data
HBase
Data files HBase
Maintained by the Kite

Kite Datasets: Goals
• Think in terms of data, not files
• Describe your data and Kite does the right thing
• Should work consistently across the platform
• Reliable

Kite Datasets: Compatibility
Project HDFS (avro) HDFS (parquet) HBase
Flume Sink 1.0 1.0 1.0
MapReduce 1.0 1.0 1.0
Crunch 1.0 1.0 1.0
Hive 1.0 1.0 1.1
Impala 1.0 1.0 *
* depends on common HBase encoding format

Kite Datasets: What is it?
• A high-level API for data management
• Work with records and datasets
• Not files, directories, or byte arrays
• Standard descriptions for records and storage
• Schemas describe records
• Partition strategies describe layout
• Opinionated

Kite Datasets: Example
1. Describe your data
dataset obj-schema org.movielens.Rating --jar app.jar
--output rating.avsc

1. Describe your layout
dataset partition-config ts:year ts:month ts:day
--schema rating.avsc --output ymd.json

1. Describe your layout
dataset partition-config ts:year ts:month ts:day
--schema rating.avsc --output ymd.json
1. Create a dataset
dataset create ratings --schema rating.avsc
--partition-by ymd.json

datasets/
└── ratings/
├── year=1997/
│ ├── month=09/
│ │ ├── day=20/
│ │ ├── ...
│ │ └── day=30/
│ ├── month=10/
│ │ ├── day=01/
│ │ ├── ...

Kite HBase: Background
Application ApplicationApplication
Database
Data files
Data files
Kite Data
HBase
Data files HBase
Maintained by the Kite

Kite HBase: Background
• Rows identified by keys, managed by HBase
• Columns are organized as cells
• Cells are identified by column family, qualifier
• The catch: everything is a byte array
family name ...
row key last first ...
buzz@pixar.com Lightyear Buzz ...

• Uniform interaction with HBase and HDFS datasets
• Need to make keys from records
• Need configuration to map fields to cells
Kite HBase

Kite HBase: Partitioning
• Use partition strategy to define unique keys
• Kite builds the key from each record
• Kite translates keys to HBase row id bytes

• Partition strategy produces a storage key
• HDFS partitioning uses a group key
1403028411014 => (2014, 6, 17)
• HBase partitioning uses a unique key
• Grouping is done dynamically by HBase
1403028411014 => (1403028411014)

Kite HBase: Example partitioning
• Define key format from data
$ ./dataset partition-config --schema user.avsc
email:copy

• Define key format from data
email:copy
[ {
"source" : "email", "type" : "identity",
"name" : "email_copy"
} ]

email:hash[16] email:copy

email:hash[16] email:copy
[ {
"source" : "email", "type" : "hash",
"buckets" : 16, "name" : "email_hash"
}, {
"source" : "email", "type" : "identity",
"name" : "email_copy"
} ]

• Use partition strategy to define unique keys
• Kite builds the key from each record
• Kite translates keys to HBase row id bytes
• Some operations require keys

Kite HBase: Field mapping
• Configure the column family and qualifier for a field
{ "email": "buzz@pixar.com",
"firstName": "Buzz", ... }
family name ...
row key last first ...
buzz@pixar.com Lightyear Buzz ...

Kite HBase: Basic column mapping
column
{ "source": "firstName", "type": "column",
"family": "name", "qualifier": "first" }

Kite HBase: Counter mapping
column
{ "source": "firstName", "type": "column",
"family": "name", "qualifier": "first" }
counter (can be incremented)
{ "source": "visits", "type": "counter"
"family": "counts", "qualifier": "visits"}

Kite HBase: Key mapping
key (stored in the row key using identity)
{ "source": "email", "type": "key" }

{
"type" : "record",
"name" : "User",
"fields" : [ {
"name" : "email",
"type" : "string"
}, ... ]
}
Kite HBase: Example

{
"type" : "record",
"name" : "User",
"fields" : [ {
"name" : "email",
"type" : "string"
}, ... ]
}
Kite HBase: Example
[
{ "source": "email",
"type": "key" },
...
]

{
"type" : "record",
"name" : "User",
"fields" : [ {
"name" : "email",
"type" : "string"
}, ... ]
}
Kite HBase: Example
family name counts prefs
row key last first visits flash
buzz@pixar.co
m
Lightyear Buzz 315 true
[
{ "source": "email",
"type": "key" },
...
]

{
"type" : "record",
"name" : "User",
"fields" : [ {
"name" : "lastName",
"type" : "string"
}, ... ]
}
Kite HBase: Example

{
"type" : "record",
"name" : "User",
"fields" : [ {
"type" : "string"
}, ... ]
}
Kite HBase: Example
[
{ "source": "lastName",
"type": "column",
"family": "name",
"qualifier": "last" },
...
]

{
"type" : "record",
"name" : "User",
"fields" : [ {
"type" : "string"
}, ... ]
}
Kite HBase: Example
buzz@pixar.com Lightyear Buzz 315 true
[
{ "source": "lastName",
"type": "column",
"family": "name",
"qualifier": "last" },
...
]

{
"type" : "record",
"name" : "User",
"fields" : [ {
"name" : "visits",
"type" : "long"
}, ... ]
}
Kite HBase: Example
buzz@pixar.com Lightyear Buzz 315 true
[
{ "source": "visits",
"type": "counter",
"family": "counts",
"qualifier": "visits" },
...
]

• Working with a dataset in HBase does not change
• Readers / writers are backed by scans
• CLI tools work:
dataset csv-import pixar_users.csv users --use-hbase
• Additional methods on RandomAccessDataset
• get, put, delete, increment
Kite HBase: Interaction

RandomAccessDataset<User> users = ...;
Key buzzEmailKey = new Key.Builder()
.add("email", "buzz@pixar.com")
.build();
User buzz = users.get(buzzEmailKey);
buzz.addPreference("flash", true);
users.put(buzz);
Kite HBase: Interaction using keys

• Versioning and concurrency
• Additional occVersion type, like a counter
• Rejects a put if the record has changed
• Key-as-column mapping
• Stores maps or records in a column family
• Uses the key or field name as the qualifier
Kite HBase: More features

• Translation between objects and byte arrays in Kite
• Configuration to define key format
• Configuration to define how fields are stored
• Decreases the code and time required to
experiment
• Key format and column mappings are hard
• Try out configurations to find the right one
Kite HBase: Conclusion

Questions
Ryan Blue: blue@cloudera.com
Kite mailing list: cdk-dev@cloudera.org

Kite SDK: Working with Datasets

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Kite SDK: Working with Datasets

Similaire à Kite SDK: Working with Datasets (20)

Plus de Cloudera, Inc.

Plus de Cloudera, Inc. (20)

Dernier

Dernier (20)

Kite SDK: Working with Datasets