Map r seattle streams meetup oct 2016

© 2016 MapR Technologies 1© 2016 MapR Technologies 1MapR Confidential
© 2016 MapR Technologies
When Your Stream is
the System of Record
Seattle Kafka Meetup Will Ochandarena
Sr Dir, Product
October 24 2016

Agenda
• Streaming System of Record - What?
• A Little About MapR Streams
• Versioning a Real-time Data Pipeline
– Demo - MapR + StreamSets

© 2016 MapR Technologies 3© 2016 MapR Technologies 3MapR Confidential © 2016 MapR Technologies
Streaming System of Record
System of Record (n): information storage system that is
the authoritative data source for a given data element or
piece of information.

Who Does This Today?
Events
Processing
DB
More
Processing
Long Term Storage

Reprocessing is Hard
Events
Processing
DB
More
Processing
Long Term Storage
?
Medium Term Storage
3d ago -> Now
1 Year ago -> ~an hour ago

Easy Fix - Streaming System of Persistence
Events
Processing
DB
More
Processing
Long Term Storage
Long Term Storage
Events

DMV_Updates
Imagine each event as a change to an entry in a database.
DL_ID City Points
0: { WillO : {City : Mountain View}, ts : 7/5/2009 04:01:01, src : dmv201 }
1: { BradA : {City : Atlanta}, ts : 5/11/2010 05:11:31, src : dmv1341 }
2: { BradA : {Points : +2}, ts : 6/22/2011 03:31:10, src : officer1213}
3: { WillO : {City : San Jose}, ts : 11/1/2012 04:01:01, src : dmv1661 }
WillO
BradA
Mountain View
Atlanta
0
0
San Jose
2
How Can a Stream Be a System of Record?

Key-Val Document Graph
Wide Column Time Series Relational
???Inserts Updates
Streams and Databases in Harmony

Which of these can be used to reconstruct the other?
DL_ID City Points
Will0 San Jose 0
BradA Atlanta 2
Which Makes a Better System of Record?

• Auditing - “how did BradA’s points get so high?”
• Lineage - “who added points to BradA license?”
• History - “where did WillO used to live?”
• Integrity - “can I trust this data hasn’t been tampered with?”
• Yup - Streams are immutable
Other Benefits of Streaming System of Record

• Infinitely persisted events
• A way to query your persisted stream data
• An integrated security model across data services
What Do I Need For This to Work?
• Applied Streaming System of Record @ Liaison Blog

About MapR & MapR Streams

MapR Streams:
Global Pub-sub Event Streaming System for Big Data
Producers publish billions of events/sec
to a topic in a stream.
Events persisted and immediately
delivered to all consumers, guaranteed.
Tie together geo-dispersed clusters.
Worldwide.
Standard real-time API (Kafka).
Integrates with Spark Streaming, Storm,
Apex, and Flink.
Direct data access (OJAI API) from
analytics frameworks.
To
pi
c
Stream
TopicProducers Consumers
Remote sites and consumers
Batch analytics

Streams Offers a Durable,
Persistent System of Record
[
{“Topic1Part0Seq5001”: {
“timestamp” : 1456246886,
“topic” : “Topic1”,
“partition” : 0,
“producer” : “wochanda”,
“offset” : 5001,
“key” : “MsgKey”,
“data” : {...}
},
{“Topic2Part0Seq5002”: { … } },
…
]
● Reliable
● Secure
● Immutable
● Auditable
● Replayable

Streams Enables Global Applications and Analytics
Provides
● Arbitrary topology of thousands of clusters
● Automatic loop prevention
● DNS-based discovery
● Globally synchronized message offsets
and consumer cursors
Enables
● Global applications & data collection
● Producer & consumer failover
● Analysis/filtering/aggregation at the edge
● “Occasional” connections
Producers
Consumers

Fun Facts
MapR Streams
Converged Global Scale
Secure & Multi-Tenant
Single cluster for files,
tables, and streams. Global, IoT-scale “fabrics”
with failover.
Tenant-owned streams,
logical grouping of topics
and messages.
Authentication,
authorization, encryption.
Unified policy with all
other platform services.
Infinite “system of
record” persistence.
Metadata tracked
internally, no
dependencies on ZK.
Consumers, topics scale
into millions.

Open Source Engines & Tools Commercial Engines & Applications
DataProcessing
Web-Scale Storage
MapR-FS MapR-DB
Search and
Others
Global Namespace | No Single Point of Failure | Data Protection | Multi-tenancy | Workload Management
Multi Temperature | Global Multi Datacenter | High Performance Low Latency | Security | Management & Monitoring
MapR Streams
Cloud and
Managed
Services
Search and
Others
UnifiedManagementandMonitoring
Search and
Others
Event StreamingDatabase
Custom
Apps
HDFS API POSIX, NFS HBase API JSON API Kafka API
MapR Converged Data Platform
MapR Data Platform Services
Commodity Hardware/Storage, Clouds, & Containers

Versioning a Real-time Data Pipeline

Challenges of a Streaming App Developer
Pre-Production
Streaming System
Database Hadoop Cluster
App Environment
events
logs
events2
logs2
v2
v2 /clicks /clicks2
... ...
... ...

Challenges with Versioning
Post-Production
Input Data App Logic Output Data+ =
Output Streams
Database Tables
Logs, Metrics
What if you deploy a
new version of your
application?
What happens
to all of this?

Example: Versioning in Production
45 40 60 30 37 39 72 79 60
Input_Stream
45 35 70
Output_Stream
Calculate_Mean_3
Time Value
00:00:00 70
00:00:05 35
00:00:10 45
Output_Table
Calculate_Mean_3Calculate_Median_3

© 2016 MapR Technologies 22© 2016 MapR Technologies 22MapR Confidential Calculate_Mean_3 Volume
Versioning with Converged App Volumes
45 40 60 30 37 39 72 79 60
Input_Stream
35 70
Output_Stream
Calculate_Mean_3
Time Value
00:00:00 70
00:00:05 35
00:00:10
Output_Table
Calculate_Mean_3Calculate_Median_3
Calculate_Median_3 Volume
Time Value
00:00:00 72
00:00:05 37
00:00:10 45
45 37 72
Output_Stream
Output_Table

Versioning & A/B Testing
80%
10%
10%
A
B
C

DEMO - MapR & Streamsets
Versioning a Production Data Pipeline
Rupal Shah - Streamsets

StreamSets Data Collector™
Adaptable Pipelines -> Efficiency
❑ Intent-driven ingest (minimal schema specification).
❑ Data drift handling.
Pipeline KPIs -> Visibility
❑ Real-time stage, edge and bad data metrics.
❑ Alerts via profiling, sampling and threshold-based rules.
Containerized Architecture -> Agility
❑ Flexible deployment: edge, cluster, embedded, pipeline,
pub/sub
❑ Zero-downtime upgrades due to logical component
isolation.
StreamSets Data Collector™ is open source software for building and deploying individual any-
to-any ingest pipelines in the face of data drift.

StreamSets Dataflow Performance
Manager™
StreamSets Dataflow Performance
Manager (DPM™) provides a single
pane of glass to map, measure and
master big data in motion.
MASTER
Availability & Accuracy
Proactive Remediation
MEASURE
Any Path
Any Time
MAP
Dataflow Lineage
Live Data Architecture

…helping you put data technology to work
● Find answers
● Ask technical questions
● Join on-demand training course discussions
● Follow release announcements
● Share and vote on product ideas
● Find Meetup and event listings
Connect with fellow Apache
Hadoop and Spark professionals
community.mapr.com

Backup

bit.ly/tbd
Find my slides & other related materials to this talk here:
or search:

Map r seattle streams meetup oct 2016

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

En vedette

En vedette (20)

Similaire à Map r seattle streams meetup oct 2016

Similaire à Map r seattle streams meetup oct 2016 (20)

Plus de Nitin Kumar

Plus de Nitin Kumar (16)

Dernier

Dernier (20)

Map r seattle streams meetup oct 2016