Huguk lily

Lily
Smart data at scale

IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org

big data,
big problems


MOORE vs data

» IDC says Digital Universe will be 35 data
Zettabytes by 2020
» 20% = enterprise data (structured,
moore
curated, $$$)
» Facebook, Yahoo!, Google, Rapleaf,
Amazon show us how the
remaining 80% can be monetized
» some of them even rent out their data
platform
» ... at the cost of infrastructure lock-in

1 Zettabyte = 1,000,000,000,000,000,000,000 bytes, or 1 billion terrabytes

IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 3

MOORE vs data

data

» coping with volume + need for
timeliness = parallel processing moore
» data becomes business-critical =
resilience through distributed
architectures
» Hadoop, MapReduce, HBase:
the future data platform


the CHALLENGES

» process ALL data
» process data in REAL-TIME
» derive INSIGHTS
» provide INSTANT FEEDBACK


current thinking

ETL

data
data STORE analytics
warehouse

batched, off-line, overnight

1. store and manage all YOUR data

DATA


2. store user behaviour, nearby

DATA

USER
Behavior


3. analyze usage patterns

DATA data processing

USER
Behavior


4. add domain knowledge


USER
Behavior

domain
knowledge
patterns
rules
keywords
lists
...


5. process, in real-time


recommendations
semantic augmentation
Analytics

USER
Behavior

domain
knowledge
patterns
rules
keywords
lists
...


6. augment data


recommendations
Analytics

USER
Behavior

domain
knowledge
patterns
rules
keywords
lists
...


data insights
SMARTER DATA data processing
s
relation
recommendations
Analytics

domain
knowledge
patterns
rules
keywords
lists
...


data insights
SMARTER DATA data processing
s
relation
recommendations
Analytics

domain
knowledge
patterns
rules
keywords
lists

SMART DATA, at SCALE
...

... and in real time

stories


HYPER-PERSONAL
recommendations
NEWS

TOGETHERNESS
interestingness

organisations
names
locations
brands

news aggregator
scale

up-selling
CROSS-SELLING
product
CATALOG
recommendedness
relatedness

product
families
related
activities
social graph

e-retail
real-time

competitive
innovation
patents

(dis)SIMILARITY

companies
people
materials
processes

IP research
insights

outerthought

“ The world is moving from content as a cost
to data as an opportunity. We provide the tools
and the platform to let organisations maximally
beneﬁt from the data they grow and collect. ”


Lily (now)

» Large-scale content storage, indexing and search
» Current pilots

e-retail mobile media isp e-gov ip research

» up-to now: 3 man-years investment (since Sept/2009)


Lily 1.0 (CR)

data
data STORE + warehouse + analytics

real time
}
Lily 2.0

lily USPs

» Integrated approach, one-stop-data-shop
» No more ﬂat ﬁle processing (Hadoop) ➙
interactive database (HBase) ➡ all data
» Real-time (vs. overnight)
» instant feedback loops
➡ real-time
» designed for on-line, interactive use
➡ easy
» Available in-house, SaaS possible
» Data Insights = data + customer retention


roadmap

» now: Lily 0.3
» Along the road:
» april 2011 : Lily 1.0
Lily SaaS edition
» Q3 2011
» real-time statistics + analytics
» Q2 2012 : Lily 2.0
» real-time data processing engine
» Data Insights


open source

» www.lilyproject.org
» docs.outerthought.org/lily-docs-current/


Lily Core Concepts
» storage
» HBase
» repository model
» versioning, varianting, mixins
» indexing
» mapping
» search
» SOLR


falling in love with Hbase : phase 1

» automatic scaling to large data sets
» fault-tolerance
» ﬂexible datamodel with sparse data
» commodity hardware
» efﬁcient random access
» community-based open source
» Java if possible



» need for consistency
» atomic single-row updates
» M/R for index regeneration



HBase
» datamodel with column families and cell versioning
» ordered tables with range scans
» HDFS for blob storage
» Apache


Lily Repository Model


Lily Datatypes


Mixins


Sample Lily Schema (excerpt)
  {
namespaces: {
    name: "b$name",
    /* Declaration of namespace prefixes. */
    valueType: { primitive: "STRING" },
    "org.lilyproject.bookssample": "b",
    scope: "versioned"
    "org.lilyproject.vtag": "vtag"
  },
  },
  {
fieldTypes: [
    name: "b$bio",
  {
    name: "b$title",
  },
  {
  },
    name: "vtag$last",
  {
    valueType: { primitive: "LONG" },
    name: "b$pages",
    scope: "non_versioned"
    valueType: { primitive: "INTEGER" },
  }
  ],
  },
recordTypes: [
  {
  {
    name: "b$language",
    name: "b$Book",
    fields: [
      {name: "b$title", mandatory: true },
  },
      {name: "b$pages", mandatory: false },
  {
      {name: "b$language", mandatory: false },
    name: "b$authors",
      {name: "b$authors", mandatory: false },
    valueType: { primitive: "LINK", multiValue: true },
      {name: "vtag$last", mandatory: false }
    ]
  },
  },

...


Lily Versioning


Flexible content model
» generic enough to accomodate many popular content
schemas
» HTML5, CMIS, RDF, NewsML, Dublin Core, ...
» academically veriﬁed
» not limited to ‘content applications’ only
» developer convenience
» higher level constructs
» schema reuse
» versioning, linking, ...


Lily Architecture
(deployment)


Lily Architecture
(components)


HBase RowLog Library

» need for sync/async operations
» updating of secondary indexes (i.e. tables)
» feeding of Indexer (= bridge to SOLR index maintenance)
» not: transactions
» need for distribution and durability


HBase RowLog Library
» WAL » Queue
» guaranteed execution of synchronous » triggering of async actions
actions
» e.g. (re)index (updated) record with
» call doesn’t return before secondary
SOLR back-end
action ﬁnishes
» size depends on speed of back-end
» e.g. update secondary index tables
process
» if all goes well,
size = #concurrent ops
» useful outside of Lily context as well!


The Lily Indexer

sharding towards
indexing of multiple incremental index blob content
denormalization batch index building multiple SOLR
versions of a record updating extraction
instances


Indexing configuration (SOLR)
<schema name="example" version="1.2">

<types>
[snipped: see SOLR example schema]
</types>

<fields>

<field name="@@key" type="string" indexed="true" stored="true" required="true"/>
<field name="@@id" type="string" indexed="true" stored="true" required="true"/>
<field name="@@vtag" type="string" indexed="true" stored="true" required="true"/>
<field name="@@versionless" type="string" indexed="true" stored="true" required="false"/>


<field name="title" type="text" indexed="true" stored="true" required="false"/>
<field name="authors" type="text" indexed="true" stored="true" required="false"
multiValued="true"/>
</fields>

<uniqueKey>@@key</uniqueKey>

<defaultSearchField>title</defaultSearchField>

<solrQueryParser defaultOperator="OR"/>

</schema>

Indexer configuration (Lily)
<?xml version="1.0"?>
<indexer xmlns:b="org.lilyproject.bookssample">
<cases>
<case recordType="b:Book" variant="*" vtags="last" indexVersionless="true"/>
</cases>

<indexFields>
<indexField name="title">
<value>
<field name="b:title"/>
</value>
</indexField>

<indexField name="authors">
<value>
<deref>
<follow field="b:authors"/>
<field name="b:name"/>
</deref>
</value>
</indexField>
</indexFields>

</indexer>


(opt.) Sharding configuration
{
shardingKey: {
    value: {
      source: "variantProperty",
      property: "language"
    },
    type: "string"
},

mapping: {
    type: "list",
    entries: [
      { shard: "shard1", values: ["en", "it"] },
      { shard: "shard2", values: ["nl", "de", "es"] }
    ]
}
}


Lily API

» Java (using Avro)
» http://docs.outerthought.org/lily-docs-current/g3/g1/390-lily.html

» REST (HTTP + JSON)
» http://docs.outerthought.org/lily-docs-current/g3/g2/427-lily.html

» All docs
» http://docs.outerthought.org/lily-docs-current/ext/toc/


Demo
» http://outerthought.blip.tv/ﬁle/4245615/


Lily and HBase

» adds high-level content model
» data types
» versioning
» blob storage on HDFS
» focus on sparse (efﬁcient) storage
» RowLog for synchronous cross-table updates and async
message queues


Lily and SOLR

» provides flexible mapping between HBase content
model and SOLR index fields
» interactive and batch (M/R) index maintenance
» sharding
» use(s) SOLR as-is: loose, flexible, extensible coupling
» search access via SOLR (HTTP) API


Lily and CDH

» we intend to rely on CDH-‘blessed’ versions of HBase/
HDFS/ZK
» 700 patches and testing
» next: adopting similar distribution lay-out
» since we contribute patches to ASF HBase trunk, we would
expect CDH to track closely (until HBase 1.0)
» some Lily users could be interested in ‘CDH-level’ services


goodbye

» It’s open source !
» Content Repository: available now
(Lily model + HBase + SOLR + RowLog)
» Lily 1.0 soon, will mainly focus on differentiating open
source and enterprise edition
» “HBase is wa de max maat.”


Thank you !
for your attention
for your questions

» stevenn@outerthought.org

» @stevenn


Huguk lily

Recommended

Recommended

More Related Content

Similar to Huguk lily

Similar to Huguk lily (20)

More from Skills Matter

More from Skills Matter (20)

Recently uploaded

Recently uploaded (20)

Huguk lily