Apache Spark is the engine powering many data-driven use cases, from data engineering to data science and machine learning applications. At QuantumBlack, Spark is considered a key technology and used in a number of client engagements, from a Data Engineering, Data Science and Platform Engineering point of view. This talk will be around the lessons learned after running successfully Apache Spark workloads in production in the cloud for a number of years. As public cloud adoption grows in the enterprise, more and more organizations are choosing to run Apache Spark workloads on cloud infrastructure. While the cloud presents many benefits, there are a number of challenges that aren’t obvious until you start and require sometimes different approaches or thinking. This talk will look into a few different areas, starting with the Jigsaw pieces you face with Open Source software, balancing a platform for stability along with allowing innovation. The talk will then look at approaches used to combat the not so obvious challenges and trade-offs of using cloud scalable storage backends for storing/retrieving data. Finally, there’ll be a section on the considerations needed for reliability and manageability of robust analytic pipelines.

1. Running Spark in Production
in the Cloud is not easy
Nayur Khan
#SAISEnt12

2. 2All content copyright © 2017 QuantumBlack, a McKinsey company
Who am I?
Nayur Khan
Head of Platform Engineering

3. All content copyright © 2017 QuantumBlack, a McKinsey company
Born and proven in F1
where the smallest margins are
the difference between winning
and losing
Data has emerged as a
fundamental element of
competitive advantage
3All content copyright © 2017 QuantumBlack, a McKinsey company
QuantumBlack
“Exploit Data, Analytics and Design to
help our clients be the best they can be”

4. All content copyright © 2018 QuantumBlack, a McKinsey company
Advanced
Industries
Financial
Services
Healthcare
Infra-
structure
Telecoms Natural
Resources
Sport Consumer
Not just Formula One…

5. 1 – Background
2 – Observations of using Cloud Object Storage
3 – Design To Operate
4 – Pipelines!

6. Background
1

7. 7All content copyright © 2017 QuantumBlack, a McKinsey company
Some of the Platforms we use
Nerve Live
Powered By

8. All content copyright © 2018 QuantumBlack, a McKinsey company
Typical Flow for production
Apply Advanced
Analytics and
ML to features
Leverage unconnected, unstructured
data by preparing and linking data and
creating features
Advanced
analytics
User modules
Analytics Platform
Raw Data
...
Better & more
consistent decision
making
Prepared Data
Cleaned,
Enriched and
Linked
Infrastructure / Hosting
• Data Ingested into a Raw storage area
• Data is Prepared
• Cleansed
• Enriched
• Linked
• etc
• Models are run
• User Tools consume outputs
• Rich custom built tools
• Self service BI tools
2
3
4
1
1 2 3 4

9. All content copyright © 2018 QuantumBlack, a McKinsey company
Analytic Modules Analytic Modules
Zookeeper
Mesos / Marathon
Master Servers
Slave Servers
Nerve Live Analytics Cluster (AWS)
Zookeeper
Mesos / Marathon
Zookeeper
Mesos / Marathon
Mesos Slave / Chronos
Analytic Modules
Mesos Slave / Chronos
Analytic Modules
Spark AnalyticsSpark Analytics
…
Data Buckets
• Mesos based Spark cluster
• Open Source versions
• Data stored mainly in S3
• Raw
• Prepared
• Features
• Models
• Data published at end of Pipeline (RDS)
• Spark Analytic workloads
• Scheduled via Marathon or Chronos
Example of physical Architecture of Nerve Live in AWS
1
2
3
4
Published
Data
Storage
ServicesServices
Logs Metastore

10. All content copyright © 2018 QuantumBlack, a McKinsey company
Spark-Base-2.3.1 Image
Dependencies
+ Config
+ Config
O/S Image
Java-Base Image
• Use Docker
• Provide a Spark-Base-2.x.x Image to teams to
package code into
2
1
Example of how we package Spark Jobs

11. All content copyright © 2018 QuantumBlack, a McKinsey company
Spark-Base-2.3.1 Image
Dependencies
+ Config
+ Config
O/S Image
Java-Base Image
App Image
+ ConfigSpark Code
• Use Docker
• Provide a Spark-Base-2.x.x Image to teams to
package code into
• Teams package up their code into provided
image
• Can tell Spark to tell Mesos which Image to use
for executors
spark.mesos.executor.docker.image
• Bonus - We can run different versions of Spark
workloads in the cluster at same time!
2
3
1
4
Example of how we package Spark Jobs
5

12. Observations of using Cloud
Object Storage
2

13. All content copyright © 2018 QuantumBlack, a McKinsey company
Background – File Storage
/
foo/ a.csv
Directory Physical
b.csv
c.avrobar/
temp/
emp/
any/
File Storage
• Organize (file) data using
hierarchical structure (folder & files)
• Allows
• Traverse a path
• Quick to “list” files in a directory
• Quick to ”rename” files
1
2
+
+
+

14. All content copyright © 2018 QuantumBlack, a McKinsey company
Background – Object Storage
Object Storage
Key Physical
/foo/a.csv
/bar/b.csv
/bar/c.avro
• No concept of organsiation
Keys à Data
• Unlike File Storage
• Slow to “list” items in a “directory”
• Slow to ”rename” files
• Use REST calls
1
2
-
-
-

15. All content copyright © 2018 QuantumBlack, a McKinsey company
First Observation – The “Intent” to Read Data
// Define a dataframe
val df = spark.read.format( “csv” )
.schema( csvSchema ).load( “s3a://...” )
// Do some work...
# Custom log4j properties
...
log4j.logger.com.amazonaws.request=DEBUG
...

16. All content copyright © 2018 QuantumBlack, a McKinsey company
First Observation – The “Intent” to Read Data
Spark Code Datasource API
// Define a dataframe
val df = spark.read.format( “csv” )
.schema( csvSchema ).load( “s3a://...” )
// Do some work...
Hadoop Libs AWS Libs
load( path )
S3
HTTPS:// HEAD / GET
…
(depends on number of files/partitions)
s
ms
key
REST
# Custom log4j properties
...
log4j.logger.com.amazonaws.request=DEBUG
...

17. All content copyright © 2018 QuantumBlack, a McKinsey company
First Observation – The “Intent” to Read Data – How many HTTPS calls?
With Schema
// Define a dataframe
val df = spark.read.format( “csv” )
.schema( csvSchema ).load( “s3a://...” )
// Do some work
Spark Versions
Spark 2.1 - spark-2.1.0-hadoop2.7
Spark 2.2 - spark-2.2.1-hadoop2.7
Spark 2.3 - spark-2.3.1-hadoop2.7

18. All content copyright © 2018 QuantumBlack, a McKinsey company
First Observation – The “Intent” to Read Data – How many HTTPS calls?
With Schema
// Define a dataframe
val df = spark.read.format( “csv” )
.schema( csvSchema ).load( “s3a://...” )
// Do some work
7
6
4
6
5
4
1 1
0
0
2
4
6
8
2.1 2.2 2.3
Path to 1 x CSV File
TOTAL
HEAD
GET
Spark Versions
Spark 2.1 - spark-2.1.0-hadoop2.7
Spark 2.2 - spark-2.2.1-hadoop2.7
Spark 2.3 - spark-2.3.1-hadoop2.7

19. All content copyright © 2018 QuantumBlack, a McKinsey company
First Observation – The “Intent” to Read Data – How many HTTPS calls?
With Schema
// Define a dataframe
val df = spark.read.format( “csv” )
.schema( csvSchema ).load( “s3a://...” )
// Do some work
7
6
4
6
5
4
1 1
0
0
2
4
6
8
2.1 2.2 2.3
Path to 1 x CSV File
TOTAL
HEAD
GET
15
11
14
9
7
9
6
4 5
0
5
10
15
20
2.1 2.2 2.3
Path to 200 x CSV Files
TOTAL
HEAD
GET
Spark Versions
Spark 2.1 - spark-2.1.0-hadoop2.7
Spark 2.2 - spark-2.2.1-hadoop2.7
Spark 2.3 - spark-2.3.1-hadoop2.7

20. All content copyright © 2018 QuantumBlack, a McKinsey company
First Observation – The “Intent” to Read Data – How many HTTPS calls?
With Schema vs. Infer Schema
// Define a dataframe
val df = spark.read.format( “csv” )
.schema( csvSchema ).load( “s3a://...” )
// Do some work
// Define a dataframe
val df = spark.read.format( “csv” )
.option( “inferSchema”, true )
.load( “s3a://...” )
// Do some work
15
11
14
9
7
9
6
4 5
0
5
10
15
20
2.1 2.2 2.3
Path to 200 x CSV Files
TOTAL
HEAD
GET
7
6
4
6
5
4
1 1
0
0
2
4
6
8
2.1 2.2 2.3
Path to 1 x CSV File
TOTAL
HEAD
GET
Spark Versions
Spark 2.1 - spark-2.1.0-hadoop2.7
Spark 2.2 - spark-2.2.1-hadoop2.7
Spark 2.3 - spark-2.3.1-hadoop2.7

21. All content copyright © 2018 QuantumBlack, a McKinsey company
First Observation – The “Intent” to Read Data – How many HTTPS calls?
With Schema vs. Infer Schema
// Define a dataframe
val df = spark.read.format( “csv” )
.schema( csvSchema ).load( “s3a://...” )
// Do some work
// Define a dataframe
val df = spark.read.format( “csv” )
.option( “inferSchema”, true )
.load( “s3a://...” )
// Do some work
13
14 14
10 10 10
3
4 4
0
5
10
15
2.1 2.2 2.3
Path to 1 x CSV File
TOTAL
HEAD
GET
15
11
14
9
7
9
6
4 5
0
5
10
15
20
2.1 2.2 2.3
Path to 200 x CSV Files
TOTAL
HEAD
GET
7
6
4
6
5
4
1 1
0
0
2
4
6
8
2.1 2.2 2.3
Path to 1 x CSV File
TOTAL
HEAD
GET
Spark Versions
Spark 2.1 - spark-2.1.0-hadoop2.7
Spark 2.2 - spark-2.2.1-hadoop2.7
Spark 2.3 - spark-2.3.1-hadoop2.7

22. All content copyright © 2018 QuantumBlack, a McKinsey company
First Observation – The “Intent” to Read Data – How many HTTPS calls?
With Schema
7
6
4
6
5
4
1 1
0
0
2
4
6
8
2.1 2.2 2.3
Path to 1 x CSV File
TOTAL
HEAD
GET
vs. Infer Schema
13
14 14
10 10 10
3
4 4
0
5
10
15
2.1 2.2 2.3
Path to 1 x CSV File
TOTAL
HEAD
GET
15
11
14
9
7
9
6
4 5
0
5
10
15
20
2.1 2.2 2.3
Path to 200 x CSV Files
TOTAL
HEAD
GET
817
613
1082
610
408
643
207 205
439
0
200
400
600
800
1000
1200
2.1 2.2 2.3
Path to 200 x CSV Files
TOTAL
HEAD
GET
// Define a dataframe
val df = spark.read.format( “csv” )
.schema( csvSchema ).load( “s3a://...” )
// Do some work
// Define a dataframe
val df = spark.read.format( “csv” )
.option( “inferSchema”, true )
.load( “s3a://...” )
// Do some work
Spark Versions
Spark 2.1 - spark-2.1.0-hadoop2.7
Spark 2.2 - spark-2.2.1-hadoop2.7
Spark 2.3 - spark-2.3.1-hadoop2.7

23. All content copyright © 2018 QuantumBlack, a McKinsey company
Second Observation – Writing Data – why does it take so long?
// Define a dataframe
df.write.format( “parquet” ).save( “s3a://...” )
// Do some work...
# Custom log4j properties
...
log4j.logger.com.amazonaws.request=DEBUG
...

24. All content copyright © 2018 QuantumBlack, a McKinsey company
Second Observation – Writing Data – why does it take so long?
Spark Code Datasource API Hadoop Libs AWS Libs
save( path )
S3
HTTPS:// PUT / HEAD / GET / DELETE
…
s
ms
key
REST
// Define a dataframe
df.write.format( “parquet” ).save( “s3a://...” )
// Do some work...
# Custom log4j properties
...
log4j.logger.com.amazonaws.request=DEBUG
...
…
(depends on number of files/partitions)

25. All content copyright © 2018 QuantumBlack, a McKinsey company
Second Observation – Writing Data – How many HTTPS calls?
Default
// Define a dataframe
df.write.format( “parquet” ).save( “s3a://...” )
// Do some work...
Spark Versions
Spark 2.1 - spark-2.1.0-hadoop2.7
Spark 2.2 - spark-2.2.1-hadoop2.7
Spark 2.3 - spark-2.3.1-hadoop2.7

26. All content copyright © 2018 QuantumBlack, a McKinsey company
Second Observation – Writing Data – How many HTTPS calls?
Default
165
151 152
103 95 96
53 47 47
7 7 72 2 2
0
50
100
150
200
2.1 2.2 2.3
1 Partition
TOTAL
HEAD
GET
PUT
DELETE
// Define a dataframe
df.write.format( “parquet” ).save( “s3a://...” )
// Do some work...
Spark Versions
Spark 2.1 - spark-2.1.0-hadoop2.7
Spark 2.2 - spark-2.2.1-hadoop2.7
Spark 2.3 - spark-2.3.1-hadoop2.7

27. All content copyright © 2018 QuantumBlack, a McKinsey company
Second Observation – Writing Data – How many HTTPS calls?
Default
165
151 152
103 95 96
53 47 47
7 7 72 2 2
0
50
100
150
200
2.1 2.2 2.3
1 Partition
TOTAL
HEAD
GET
PUT
DELETE
// Define a dataframe
df.write.format( “parquet” ).save( “s3a://...” )
// Do some work...
1092 1078
381
697 689
244
323 317
114
52 52 1720 20 6
0
200
400
600
800
1000
1200
2.1 2.2 2.3
10 Partitions
TOTAL
HEAD
GET
PUT
DELETE
Spark Versions
Spark 2.1 - spark-2.1.0-hadoop2.7
Spark 2.2 - spark-2.2.1-hadoop2.7
Spark 2.3 - spark-2.3.1-hadoop2.7

28. All content copyright © 2018 QuantumBlack, a McKinsey company
Second Observation – Writing Data – How many HTTPS calls?
Default
165
151 152
103 95 96
53 47 47
7 7 72 2 2
0
50
100
150
200
2.1 2.2 2.3
1 Partition
TOTAL
HEAD
GET
PUT
DELETE
// Define a dataframe
df.write.format( “parquet” ).save( “s3a://...” )
// Do some work...
1092 1078
381
697 689
244
323 317
114
52 52 1720 20 6
0
200
400
600
800
1000
1200
2.1 2.2 2.3
10 Partitions
TOTAL
HEAD
GET
PUT
DELETE
Go Faster
spark.conf.set(
“spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version”, 2)
df.write.format( “parquet” ).save( “s3a://...” )
Spark Versions
Spark 2.1 - spark-2.1.0-hadoop2.7
Spark 2.2 - spark-2.2.1-hadoop2.7
Spark 2.3 - spark-2.3.1-hadoop2.7

29. All content copyright © 2018 QuantumBlack, a McKinsey company
Second Observation – Writing Data – How many HTTPS calls?
Default
165
151 152
103 95 96
53 47 47
7 7 72 2 2
0
50
100
150
200
2.1 2.2 2.3
1 Partition
TOTAL
HEAD
GET
PUT
DELETE
// Define a dataframe
df.write.format( “parquet” ).save( “s3a://...” )
// Do some work...
126
112 113
79
71 72
40 34 34
5 5 52 2 2
0
20
40
60
80
100
120
140
2.1 2.2 2.3
1 Partition
TOTAL
HEAD
GET
PUT
DELETE
1092 1078
381
697 689
244
323 317
114
52 52 1720 20 6
0
200
400
600
800
1000
1200
2.1 2.2 2.3
10 Partitions
TOTAL
HEAD
GET
PUT
DELETE
Go Faster
spark.conf.set(
“spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version”, 2)
df.write.format( “parquet” ).save( “s3a://...” )
Spark Versions
Spark 2.1 - spark-2.1.0-hadoop2.7
Spark 2.2 - spark-2.2.1-hadoop2.7
Spark 2.3 - spark-2.3.1-hadoop2.7

30. All content copyright © 2018 QuantumBlack, a McKinsey company
Second Observation – Writing Data – How many HTTPS calls?
Default
165
151 152
103 95 96
53 47 47
7 7 72 2 2
0
50
100
150
200
2.1 2.2 2.3
1 Partition
TOTAL
HEAD
GET
PUT
DELETE
// Define a dataframe
df.write.format( “parquet” ).save( “s3a://...” )
// Do some work...
126
112 113
79
71 72
40 34 34
5 5 52 2 2
0
20
40
60
80
100
120
140
2.1 2.2 2.3
1 Partition
TOTAL
HEAD
GET
PUT
DELETE
1092 1078
381
697 689
244
323 317
114
52 52 1720 20 6
0
200
400
600
800
1000
1200
2.1 2.2 2.3
10 Partitions
TOTAL
HEAD
GET
PUT
DELETE
738 724
272
475 467
176
220 214
81
32 32 1111 11 4
0
200
400
600
800
2.1 2.2 2.3
10 Partitions
TOTAL
HEAD
GET
PUT
DELETE
Go Faster
spark.conf.set(
“spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version”, 2)
df.write.format( “parquet” ).save( “s3a://...” )
Spark Versions
Spark 2.1 - spark-2.1.0-hadoop2.7
Spark 2.2 - spark-2.2.1-hadoop2.7
Spark 2.3 - spark-2.3.1-hadoop2.7

31. All content copyright © 2018 QuantumBlack, a McKinsey company
Conclusion
• Cloud object storage has tradeoffs
• Gives incredible storage capacity and scalability
• However, do not expect same performance/
or characteristics as a file system
• There are lots of configuration settings that
can affect your performance with reading/writing
to cloud object storage
Note: Some are Hadoop version specific!
1
2
3
4
spark.hadoop.fs.s3a.fast.upload=true
spark.hadoop.fs.s3a.connection.maximum=xxx
spark.hadoop.fs.s3a.multipart.size=xxx
spark.hadoop.fs.s3a.attempts.maximum=xxx
spark.hadoop.fs.s3a.multipart.threshold=xxx
spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version=2
etc
...

32. Design to Operate
3

33. All content copyright © 2018 QuantumBlack, a McKinsey company
Design to Operate – Choices
• Our teams have faced choices for
Loading / Saving data
• What URI scheme to use?
─ i.e. S3://… S3n://… S3a://…, JDBC:// etc
• What file format?
─ i.e. Parquet, Avro, CSV, etc
• What compression to use?
─ i.e. Snappy, LZO, Gzip, etc
• What additional options….
─ i.e. inferSchema, mergeSchema, pushDownPredicate…etc
• Even what path to load/save data
1
?
? ?

34. All content copyright © 2018 QuantumBlack, a McKinsey company
Design to Operate – Choices
• Our teams have faced choices for
Loading / Saving data
• What URI scheme to use?
─ i.e. S3://… S3n://… S3a://…, JDBC:// etc
• What file format?
─ i.e. Parquet, Avro, CSV, etc
• What compression to use?
─ i.e. Snappy, LZO, Gzip, etc
• What additional options….
─ i.e. inferSchema, mergeSchema, pushDownPredicate…etc
• Even what path to load/save data
• Choices end up being “embedded” into code
1
2
val path = "s3n://my-bucket/csv-data/people"
val peopleDF =
spark.read.format( "csv" )
.option( "sep", ";" )
.option( "inferSchema", "true" )
.option( "header", "true" )
.load( path )
...
?
? ?

35. All content copyright © 2018 QuantumBlack, a McKinsey company
Design to Operate – Challenges
• Different teams working in same cluster
• Working on different use cases
• Likely that Spark Jobs are running with different settings in
same cluster
1

36. All content copyright © 2018 QuantumBlack, a McKinsey company
Design to Operate – Challenges
• Different teams working in same cluster
• Working on different use cases
• Likely that Spark Jobs are running with different settings in
same cluster
1
• Those that build may not be the ones that end up
running/operating Spark Jobs
• Support Ops
• Or Client IT teams
2

37. All content copyright © 2018 QuantumBlack, a McKinsey company
Design to Operate – Challenges
• Different teams working in same cluster
• Working on different use cases
• Likely that Spark Jobs are running with different settings in
same cluster
1
• Those that build may not be the ones that end up
running/operating Spark Jobs
• Support Ops
• Or Client IT teams
• Difficult for those operating to understand
• What configuration settings
• What does it depend on? i.e. Data it needs
• What depends on it? i.e. Data it writes
• How can I change things quickly?
3
2
?

38. All content copyright © 2018 QuantumBlack, a McKinsey company
Design to Operate – Our Approach
• Borrow ideas from Devops world
• "Strict separation of config from code”
1
Dev Ops

39. All content copyright © 2018 QuantumBlack, a McKinsey company
Design to Operate – Our Approach
Spark Code
Nerve Load/Save Library
+ Config
• Borrow ideas from Devops world
• "Strict separation of config from code”
• Custom library and config for loading and saving2
1
USES
USES
"system" : ”bank”,
"table": "customers",
”type": ”input",
”description": ”A Table with banking customer details.",
"format": {
"type": ”csv",
"compression": "snappy",
"partitions": "20”
"mode": "overwrite"
}
"location": {
"type": "s3",
"properties": { ... }
}
"schema": [
{
"name": "id",
"type": "int",
"doc": "User Id"
},
...

40. All content copyright © 2018 QuantumBlack, a McKinsey company
Design to Operate – Our Approach
Spark Code
Nerve Load/Save Library
+ Config
• Borrow ideas from Devops world
• "Strict separation of config from code”
• Custom library and config for loading and saving
• For Loading / Saving data
• What URI scheme to use?
─ i.e. S3://… S3n://… S3a://…, JDBC:// etc
• What format?
─ i.e. Parquet, Avro, CSV, etc
• What compression to use?
─ i.e. Snappy, LZO, Gzip, etc
• What additional options….
─ i.e. inferSchema, mergeSchema, overwrite or append….
• Even what path to load/save data
2
3
1
USES
USES
"system" : ”bank”,
"table": "customers",
”type": ”input",
”description": ”A Table with banking customer details.",
"format": {
"type": ”csv",
"compression": "snappy",
"partitions": "20”
"mode": "overwrite"
}
"location": {
"type": "s3",
"properties": { ... }
}
"schema": [
{
"name": "id",
"type": "int",
"doc": "User Id"
},
...

41. All content copyright © 2018 QuantumBlack, a McKinsey company
Design to Operate – Our Approach
// Load a dataframe
val df = nerve.load( “system”, “table” )
// Save a dataframe
nerve.save( df, “system”, “table” )
Spark Code
Nerve Load/Save Library
+ Config
• Engineers use this library in their code
• Allows Engineers to focus on business logic and
not worry about concerns such as:
• Data formats
• Compressions
• Schemas
• Partitions
• Weird optimisation settings
• Allows teams that focus on operating/running to
focus on
• Consistency
• Runtime optimisations
2
3 USES
USES
1

42. Pipelines!
4

43. All content copyright © 2018 QuantumBlack, a McKinsey company
Pipelines
• Pipelines mean different things
to different people
• Ultimately, its about the
Flow of data
• We need to
• Monitor
• Understand
• React
• It looks simple,
but in fact its not
1
2
4
3
Ingest Prepare Analytics Publish
Flow
Flow

44. All content copyright © 2018 QuantumBlack, a McKinsey company
Pipelines – Example
Events
Sources
Raw Prepared Published

45. All content copyright © 2018 QuantumBlack, a McKinsey company
Pipelines – Example
Sources
Raw Prepared Published
Events

46. All content copyright © 2018 QuantumBlack, a McKinsey company
Pipelines – Example
Sources
Raw Prepared Published
Events

47. All content copyright © 2018 QuantumBlack, a McKinsey company
Pipelines – Example
Sources
Raw Prepared Published
Events

48. All content copyright © 2018 QuantumBlack, a McKinsey company
Pipelines – Example
Sources
Raw Prepared Published
Events

49. All content copyright © 2018 QuantumBlack, a McKinsey company
Pipelines – Example
Sources
Raw Prepared Published
Events

50. All content copyright © 2018 QuantumBlack, a McKinsey company
Pipelines – Example
Sources
Raw Prepared Published
Events

51. All content copyright © 2018 QuantumBlack, a McKinsey company
Pipelines – Example – Agreed Schedule
Sources
Raw Prepared Published
0500
• Agree a schedule with
client when to send data
1
Events

52. All content copyright © 2018 QuantumBlack, a McKinsey company
Pipelines – Example – Agreed Schedule
Sources
Raw Prepared Published
0500 0700
• Agree a schedule with
client when to send data
• Configure schedule to kick
off Spark Jobs
1
2
Events

53. All content copyright © 2018 QuantumBlack, a McKinsey company
Pipelines – Example – Agreed Schedule
Sources
Raw Prepared Published
0500 0700
• Agree a schedule with
client when to send data
• Configure schedule to kick
off Spark Jobs
• Subsequent Spark Jobs
kick off when previous
have completed
1
2
3
Events

54. All content copyright © 2018 QuantumBlack, a McKinsey company
Pipelines – Example – Agreed Schedule
Sources
Raw Prepared Published
0500 0700 0900
• Agree a schedule with
client when to send data
• Configure schedule to kick
off Spark Jobs
• Subsequent Spark Jobs
kick off when previous
have completed
1
2
3
Events

55. All content copyright © 2018 QuantumBlack, a McKinsey company
Pipelines – Example – Challenge 1 – Untimely Data
Sources
Raw Prepared Published
0500 0700 0900
• Data failed to be sent up on
time
• Impact to rest of the
Pipeline
• Impact to end user!
1
2
3
Events

56. All content copyright © 2018 QuantumBlack, a McKinsey company
Pipelines – Example – Challenge 2 – Bad Data
Sources
Raw Prepared Published
0500 0700 0900
• Bad/Unexpected Data
• Schema change
• Change in Business rules
• Data Quality Issue
• Impact to Pipeline
• Impact to end user!
1
2
3
Events

57. All content copyright © 2018 QuantumBlack, a McKinsey company
Pipelines – Example – Challenge 3 – Jobs Taking Longer
Sources
Raw Prepared Published
0500 0700 0900
• Spark Job’s taking longer
and longer
• Impact to Pipeline
• Impact to end user!
1
2
3
Events

58. All content copyright © 2018 QuantumBlack, a McKinsey company
Pipelines – Example - Challenges
Sources
Raw Prepared Published
0500
Events
• We need to
• Monitor
• Understand
• React
• It looks simple,
but in fact its not
1
2
0700 0900

59. All content copyright © 2018 QuantumBlack, a McKinsey company
Pipelines – Example – Our Approach
Sources
Raw Prepared Published
• Monitor all Raw sources for
timely data
1
Monitoring
Events
+

60. All content copyright © 2018 QuantumBlack, a McKinsey company
Pipelines – Example – Our Approach
Sources
Raw Prepared
• Monitor all Raw sources for
timely data
• Monitor quality of data
• Config driven
• Agreed business rules
1
Monitoring
Events
+
Metadata
2
Published

61. All content copyright © 2018 QuantumBlack, a McKinsey company
Pipelines – Example – Our Approach
Sources
Raw
• Monitor all Raw sources for
timely data
• Monitor quality of data
• Config driven
• Agreed business rules
• Monitor Spark Metrics
• At least how long
• Additional Metrics incl:
─ Intent to load data J
─ Saving data J
1
Monitoring
Events
+
Metadata
2
3
PublishedPrepared

62. 1 – Observations of using Cloud Object Storage
2 – Design To Operate
3 – Pipelines!
Recap

63. 1 – Observations of using Cloud Object Storage
2 – Design To Operate
3 – Pipelines!
Lets Talk!
Recap