Submit Search
Upload
Spark Summit EU talk by Jorg Schad
•
2 likes
•
848 views
Spark Summit
Follow
No One Puts Spark in the Container
Read less
Read more
Data & Analytics
Report
Share
Report
Share
1 of 45
Download now
Download to read offline
Recommended
Spark Summit EU talk by Steve Loughran
Spark Summit EU talk by Steve Loughran
Spark Summit
Supporting Highly Multitenant Spark Notebook Workloads with Craig Ingram and ...
Supporting Highly Multitenant Spark Notebook Workloads with Craig Ingram and ...
Spark Summit
Using Spark with Tachyon by Gene Pang
Using Spark with Tachyon by Gene Pang
Spark Summit
Spark Summit EU talk by Berni Schiefer
Spark Summit EU talk by Berni Schiefer
Spark Summit
Spark Summit EU talk by Debasish Das and Pramod Narasimha
Spark Summit EU talk by Debasish Das and Pramod Narasimha
Spark Summit
Recipes for Running Spark Streaming Applications in Production-(Tathagata Das...
Recipes for Running Spark Streaming Applications in Production-(Tathagata Das...
Spark Summit
State of Spark in the cloud (Spark Summit EU 2017)
State of Spark in the cloud (Spark Summit EU 2017)
Nicolas Poggi
Productionizing Spark and the Spark Job Server
Productionizing Spark and the Spark Job Server
Evan Chan
Recommended
Spark Summit EU talk by Steve Loughran
Spark Summit EU talk by Steve Loughran
Spark Summit
Supporting Highly Multitenant Spark Notebook Workloads with Craig Ingram and ...
Supporting Highly Multitenant Spark Notebook Workloads with Craig Ingram and ...
Spark Summit
Using Spark with Tachyon by Gene Pang
Using Spark with Tachyon by Gene Pang
Spark Summit
Spark Summit EU talk by Berni Schiefer
Spark Summit EU talk by Berni Schiefer
Spark Summit
Spark Summit EU talk by Debasish Das and Pramod Narasimha
Spark Summit EU talk by Debasish Das and Pramod Narasimha
Spark Summit
Recipes for Running Spark Streaming Applications in Production-(Tathagata Das...
Recipes for Running Spark Streaming Applications in Production-(Tathagata Das...
Spark Summit
State of Spark in the cloud (Spark Summit EU 2017)
State of Spark in the cloud (Spark Summit EU 2017)
Nicolas Poggi
Productionizing Spark and the Spark Job Server
Productionizing Spark and the Spark Job Server
Evan Chan
High Performance Enterprise Data Processing with Apache Spark with Sandeep Va...
High Performance Enterprise Data Processing with Apache Spark with Sandeep Va...
Spark Summit
Next CERN Accelerator Logging Service with Jakub Wozniak
Next CERN Accelerator Logging Service with Jakub Wozniak
Spark Summit
Analyzing IOT Data in Apache Spark Across Data Centers and Cloud with NetApp ...
Analyzing IOT Data in Apache Spark Across Data Centers and Cloud with NetApp ...
Databricks
Low Latency Execution For Apache Spark
Low Latency Execution For Apache Spark
Jen Aman
A Developer’s View into Spark's Memory Model with Wenchen Fan
A Developer’s View into Spark's Memory Model with Wenchen Fan
Databricks
Hadoop Query Performance Smackdown
Hadoop Query Performance Smackdown
DataWorks Summit
Leverage Mesos for running Spark Streaming production jobs by Iulian Dragos a...
Leverage Mesos for running Spark Streaming production jobs by Iulian Dragos a...
Spark Summit
Towards True Elasticity of Spark-(Michael Le and Min Li, IBM)
Towards True Elasticity of Spark-(Michael Le and Min Li, IBM)
Spark Summit
Spark and Spark Streaming at Netfix-(Kedar Sedekar and Monal Daxini, Netflix)
Spark and Spark Streaming at Netfix-(Kedar Sedekar and Monal Daxini, Netflix)
Spark Summit
Mobility insights at Swisscom - Understanding collective mobility in Switzerland
Mobility insights at Swisscom - Understanding collective mobility in Switzerland
François Garillot
Apache Spark on Supercomputers: A Tale of the Storage Hierarchy with Costin I...
Apache Spark on Supercomputers: A Tale of the Storage Hierarchy with Costin I...
Databricks
Apache Spark Performance is too hard. Let's make it easier
Apache Spark Performance is too hard. Let's make it easier
Databricks
Spark Summit EU talk by Ruben Pulido and Behar Veliqi
Spark Summit EU talk by Ruben Pulido and Behar Veliqi
Spark Summit
Reactive Streams, Linking Reactive Application To Spark Streaming
Reactive Streams, Linking Reactive Application To Spark Streaming
Spark Summit
Accelerating Shuffle: A Tailor-Made RDMA Solution for Apache Spark with Yuval...
Accelerating Shuffle: A Tailor-Made RDMA Solution for Apache Spark with Yuval...
Spark Summit
Using Apache Spark in the Cloud—A Devops Perspective with Telmo Oliveira
Using Apache Spark in the Cloud—A Devops Perspective with Telmo Oliveira
Spark Summit
Spark on Mesos
Spark on Mesos
Jen Aman
Tuning and Monitoring Deep Learning on Apache Spark
Tuning and Monitoring Deep Learning on Apache Spark
Databricks
Apache Flink vs Apache Spark - Reproducible experiments on cloud.
Apache Flink vs Apache Spark - Reproducible experiments on cloud.
Shelan Perera
Performance Troubleshooting Using Apache Spark Metrics
Performance Troubleshooting Using Apache Spark Metrics
Databricks
Lessons Learned from Dockerizing Spark Workloads: Spark Summit East talk by T...
Lessons Learned from Dockerizing Spark Workloads: Spark Summit East talk by T...
Spark Summit
Spark Summit EU talk by Patrick Baier and Stanimir Dragiev
Spark Summit EU talk by Patrick Baier and Stanimir Dragiev
Spark Summit
More Related Content
What's hot
High Performance Enterprise Data Processing with Apache Spark with Sandeep Va...
High Performance Enterprise Data Processing with Apache Spark with Sandeep Va...
Spark Summit
Next CERN Accelerator Logging Service with Jakub Wozniak
Next CERN Accelerator Logging Service with Jakub Wozniak
Spark Summit
Analyzing IOT Data in Apache Spark Across Data Centers and Cloud with NetApp ...
Analyzing IOT Data in Apache Spark Across Data Centers and Cloud with NetApp ...
Databricks
Low Latency Execution For Apache Spark
Low Latency Execution For Apache Spark
Jen Aman
A Developer’s View into Spark's Memory Model with Wenchen Fan
A Developer’s View into Spark's Memory Model with Wenchen Fan
Databricks
Hadoop Query Performance Smackdown
Hadoop Query Performance Smackdown
DataWorks Summit
Leverage Mesos for running Spark Streaming production jobs by Iulian Dragos a...
Leverage Mesos for running Spark Streaming production jobs by Iulian Dragos a...
Spark Summit
Towards True Elasticity of Spark-(Michael Le and Min Li, IBM)
Towards True Elasticity of Spark-(Michael Le and Min Li, IBM)
Spark Summit
Spark and Spark Streaming at Netfix-(Kedar Sedekar and Monal Daxini, Netflix)
Spark and Spark Streaming at Netfix-(Kedar Sedekar and Monal Daxini, Netflix)
Spark Summit
Mobility insights at Swisscom - Understanding collective mobility in Switzerland
Mobility insights at Swisscom - Understanding collective mobility in Switzerland
François Garillot
Apache Spark on Supercomputers: A Tale of the Storage Hierarchy with Costin I...
Apache Spark on Supercomputers: A Tale of the Storage Hierarchy with Costin I...
Databricks
Apache Spark Performance is too hard. Let's make it easier
Apache Spark Performance is too hard. Let's make it easier
Databricks
Spark Summit EU talk by Ruben Pulido and Behar Veliqi
Spark Summit EU talk by Ruben Pulido and Behar Veliqi
Spark Summit
Reactive Streams, Linking Reactive Application To Spark Streaming
Reactive Streams, Linking Reactive Application To Spark Streaming
Spark Summit
Accelerating Shuffle: A Tailor-Made RDMA Solution for Apache Spark with Yuval...
Accelerating Shuffle: A Tailor-Made RDMA Solution for Apache Spark with Yuval...
Spark Summit
Using Apache Spark in the Cloud—A Devops Perspective with Telmo Oliveira
Using Apache Spark in the Cloud—A Devops Perspective with Telmo Oliveira
Spark Summit
Spark on Mesos
Spark on Mesos
Jen Aman
Tuning and Monitoring Deep Learning on Apache Spark
Tuning and Monitoring Deep Learning on Apache Spark
Databricks
Apache Flink vs Apache Spark - Reproducible experiments on cloud.
Apache Flink vs Apache Spark - Reproducible experiments on cloud.
Shelan Perera
Performance Troubleshooting Using Apache Spark Metrics
Performance Troubleshooting Using Apache Spark Metrics
Databricks
What's hot
(20)
High Performance Enterprise Data Processing with Apache Spark with Sandeep Va...
High Performance Enterprise Data Processing with Apache Spark with Sandeep Va...
Next CERN Accelerator Logging Service with Jakub Wozniak
Next CERN Accelerator Logging Service with Jakub Wozniak
Analyzing IOT Data in Apache Spark Across Data Centers and Cloud with NetApp ...
Analyzing IOT Data in Apache Spark Across Data Centers and Cloud with NetApp ...
Low Latency Execution For Apache Spark
Low Latency Execution For Apache Spark
A Developer’s View into Spark's Memory Model with Wenchen Fan
A Developer’s View into Spark's Memory Model with Wenchen Fan
Hadoop Query Performance Smackdown
Hadoop Query Performance Smackdown
Leverage Mesos for running Spark Streaming production jobs by Iulian Dragos a...
Leverage Mesos for running Spark Streaming production jobs by Iulian Dragos a...
Towards True Elasticity of Spark-(Michael Le and Min Li, IBM)
Towards True Elasticity of Spark-(Michael Le and Min Li, IBM)
Spark and Spark Streaming at Netfix-(Kedar Sedekar and Monal Daxini, Netflix)
Spark and Spark Streaming at Netfix-(Kedar Sedekar and Monal Daxini, Netflix)
Mobility insights at Swisscom - Understanding collective mobility in Switzerland
Mobility insights at Swisscom - Understanding collective mobility in Switzerland
Apache Spark on Supercomputers: A Tale of the Storage Hierarchy with Costin I...
Apache Spark on Supercomputers: A Tale of the Storage Hierarchy with Costin I...
Apache Spark Performance is too hard. Let's make it easier
Apache Spark Performance is too hard. Let's make it easier
Spark Summit EU talk by Ruben Pulido and Behar Veliqi
Spark Summit EU talk by Ruben Pulido and Behar Veliqi
Reactive Streams, Linking Reactive Application To Spark Streaming
Reactive Streams, Linking Reactive Application To Spark Streaming
Accelerating Shuffle: A Tailor-Made RDMA Solution for Apache Spark with Yuval...
Accelerating Shuffle: A Tailor-Made RDMA Solution for Apache Spark with Yuval...
Using Apache Spark in the Cloud—A Devops Perspective with Telmo Oliveira
Using Apache Spark in the Cloud—A Devops Perspective with Telmo Oliveira
Spark on Mesos
Spark on Mesos
Tuning and Monitoring Deep Learning on Apache Spark
Tuning and Monitoring Deep Learning on Apache Spark
Apache Flink vs Apache Spark - Reproducible experiments on cloud.
Apache Flink vs Apache Spark - Reproducible experiments on cloud.
Performance Troubleshooting Using Apache Spark Metrics
Performance Troubleshooting Using Apache Spark Metrics
Viewers also liked
Lessons Learned from Dockerizing Spark Workloads: Spark Summit East talk by T...
Lessons Learned from Dockerizing Spark Workloads: Spark Summit East talk by T...
Spark Summit
Spark Summit EU talk by Patrick Baier and Stanimir Dragiev
Spark Summit EU talk by Patrick Baier and Stanimir Dragiev
Spark Summit
Spark Summit EU talk by Oscar Castaneda
Spark Summit EU talk by Oscar Castaneda
Spark Summit
Spark Summit EU talk by Javier Aguedes
Spark Summit EU talk by Javier Aguedes
Spark Summit
Microservice-based software architecture
Microservice-based software architecture
ArangoDB Database
Spark Summit EU talk by Josef Habdank
Spark Summit EU talk by Josef Habdank
Spark Summit
The Spark (R)evolution in The Netherlands
The Spark (R)evolution in The Netherlands
Spark Summit
Spark Summit EU talk by Heiko Korndorf
Spark Summit EU talk by Heiko Korndorf
Spark Summit
Spark Summit EU talk by Erwin Datema and Roeland van Ham
Spark Summit EU talk by Erwin Datema and Roeland van Ham
Spark Summit
Democratizing AI with Apache Spark
Democratizing AI with Apache Spark
Spark Summit
Spark Summit EU talk by William Benton
Spark Summit EU talk by William Benton
Spark Summit
Spark Summit EU talk by Sudeep Das and Aish Faenton
Spark Summit EU talk by Sudeep Das and Aish Faenton
Spark Summit
Spark Summit EU talk by Luc Bourlier
Spark Summit EU talk by Luc Bourlier
Spark Summit
Spark Summit EU talk by Reza Karimi
Spark Summit EU talk by Reza Karimi
Spark Summit
Spark Summit EU talk by Dean Wampler
Spark Summit EU talk by Dean Wampler
Spark Summit
Spark Summit EU talk by Sital Kedia
Spark Summit EU talk by Sital Kedia
Spark Summit
MmmooOgle: From Big Data to Decisions for Dairy Cows
MmmooOgle: From Big Data to Decisions for Dairy Cows
Spark Summit
Spark Summit EU talk by Ted Malaska
Spark Summit EU talk by Ted Malaska
Spark Summit
The Next AMPLab: Real-Time, Intelligent, and Secure Computing
The Next AMPLab: Real-Time, Intelligent, and Secure Computing
Spark Summit
Simplifying Big Data Applications with Apache Spark 2.0
Simplifying Big Data Applications with Apache Spark 2.0
Spark Summit
Viewers also liked
(20)
Lessons Learned from Dockerizing Spark Workloads: Spark Summit East talk by T...
Lessons Learned from Dockerizing Spark Workloads: Spark Summit East talk by T...
Spark Summit EU talk by Patrick Baier and Stanimir Dragiev
Spark Summit EU talk by Patrick Baier and Stanimir Dragiev
Spark Summit EU talk by Oscar Castaneda
Spark Summit EU talk by Oscar Castaneda
Spark Summit EU talk by Javier Aguedes
Spark Summit EU talk by Javier Aguedes
Microservice-based software architecture
Microservice-based software architecture
Spark Summit EU talk by Josef Habdank
Spark Summit EU talk by Josef Habdank
The Spark (R)evolution in The Netherlands
The Spark (R)evolution in The Netherlands
Spark Summit EU talk by Heiko Korndorf
Spark Summit EU talk by Heiko Korndorf
Spark Summit EU talk by Erwin Datema and Roeland van Ham
Spark Summit EU talk by Erwin Datema and Roeland van Ham
Democratizing AI with Apache Spark
Democratizing AI with Apache Spark
Spark Summit EU talk by William Benton
Spark Summit EU talk by William Benton
Spark Summit EU talk by Sudeep Das and Aish Faenton
Spark Summit EU talk by Sudeep Das and Aish Faenton
Spark Summit EU talk by Luc Bourlier
Spark Summit EU talk by Luc Bourlier
Spark Summit EU talk by Reza Karimi
Spark Summit EU talk by Reza Karimi
Spark Summit EU talk by Dean Wampler
Spark Summit EU talk by Dean Wampler
Spark Summit EU talk by Sital Kedia
Spark Summit EU talk by Sital Kedia
MmmooOgle: From Big Data to Decisions for Dairy Cows
MmmooOgle: From Big Data to Decisions for Dairy Cows
Spark Summit EU talk by Ted Malaska
Spark Summit EU talk by Ted Malaska
The Next AMPLab: Real-Time, Intelligent, and Secure Computing
The Next AMPLab: Real-Time, Intelligent, and Secure Computing
Simplifying Big Data Applications with Apache Spark 2.0
Simplifying Big Data Applications with Apache Spark 2.0
Similar to Spark Summit EU talk by Jorg Schad
DOD 2016 - Jörg Schad - Nobody Puts Java in the Conainer
DOD 2016 - Jörg Schad - Nobody Puts Java in the Conainer
PROIDEA
Jörg Schad - NO ONE PUTS Java IN THE CONTAINER - Codemotion Milan 2017
Jörg Schad - NO ONE PUTS Java IN THE CONTAINER - Codemotion Milan 2017
Codemotion
No one puts java in the container
No one puts java in the container
kensipe
Computer Vision Powered by Heterogeneous System Architecture (HSA) by Dr. Ha...
Computer Vision Powered by Heterogeneous System Architecture (HSA) by Dr. Ha...
AMD Developer Central
Why you’re going to fail running java on docker!
Why you’re going to fail running java on docker!
Red Hat Developers
Docker Introduction, and what's new in 0.9 — Docker Palo Alto at RelateIQ
Docker Introduction, and what's new in 0.9 — Docker Palo Alto at RelateIQ
Jérôme Petazzoni
Docker Introduction + what is new in 0.9
Docker Introduction + what is new in 0.9
Jérôme Petazzoni
Docker and friends at Linux Days 2014 in Prague
Docker and friends at Linux Days 2014 in Prague
tomasbart
Docker and-containers-for-development-and-deployment-scale12x
Docker and-containers-for-development-and-deployment-scale12x
rkr10
Containerized Data Persistence on Mesos
Containerized Data Persistence on Mesos
Joe Stein
Performance analysis in a multitenant cloud environment Using Hadoop Cluster ...
Performance analysis in a multitenant cloud environment Using Hadoop Cluster ...
Orgad Kimchi
Introduction to Docker (as presented at December 2013 Global Hackathon)
Introduction to Docker (as presented at December 2013 Global Hackathon)
Jérôme Petazzoni
The Proto-Burst Buffer: Experience with the flash-based file system on SDSC's...
The Proto-Burst Buffer: Experience with the flash-based file system on SDSC's...
Glenn K. Lockwood
Salting new ground one man ops from scratch
Salting new ground one man ops from scratch
Jay Harrison
ContainerDays Boston 2015: "CoreOS: Building the Layers of the Scalable Clust...
ContainerDays Boston 2015: "CoreOS: Building the Layers of the Scalable Clust...
DynamicInfraDays
10 things i wish i'd known before using spark in production
10 things i wish i'd known before using spark in production
Paris Data Engineers !
OSDC 2017 | Open POWER for the data center by Werner Fischer
OSDC 2017 | Open POWER for the data center by Werner Fischer
NETWAYS
OSDC 2017 - Werner Fischer - Open power for the data center
OSDC 2017 - Werner Fischer - Open power for the data center
NETWAYS
OSDC 2017 | Linux Performance Profiling and Monitoring by Werner Fischer
OSDC 2017 | Linux Performance Profiling and Monitoring by Werner Fischer
NETWAYS
Fully fault tolerant real time data pipeline with docker and mesos
Fully fault tolerant real time data pipeline with docker and mesos
Rahul Kumar
Similar to Spark Summit EU talk by Jorg Schad
(20)
DOD 2016 - Jörg Schad - Nobody Puts Java in the Conainer
DOD 2016 - Jörg Schad - Nobody Puts Java in the Conainer
Jörg Schad - NO ONE PUTS Java IN THE CONTAINER - Codemotion Milan 2017
Jörg Schad - NO ONE PUTS Java IN THE CONTAINER - Codemotion Milan 2017
No one puts java in the container
No one puts java in the container
Computer Vision Powered by Heterogeneous System Architecture (HSA) by Dr. Ha...
Computer Vision Powered by Heterogeneous System Architecture (HSA) by Dr. Ha...
Why you’re going to fail running java on docker!
Why you’re going to fail running java on docker!
Docker Introduction, and what's new in 0.9 — Docker Palo Alto at RelateIQ
Docker Introduction, and what's new in 0.9 — Docker Palo Alto at RelateIQ
Docker Introduction + what is new in 0.9
Docker Introduction + what is new in 0.9
Docker and friends at Linux Days 2014 in Prague
Docker and friends at Linux Days 2014 in Prague
Docker and-containers-for-development-and-deployment-scale12x
Docker and-containers-for-development-and-deployment-scale12x
Containerized Data Persistence on Mesos
Containerized Data Persistence on Mesos
Performance analysis in a multitenant cloud environment Using Hadoop Cluster ...
Performance analysis in a multitenant cloud environment Using Hadoop Cluster ...
Introduction to Docker (as presented at December 2013 Global Hackathon)
Introduction to Docker (as presented at December 2013 Global Hackathon)
The Proto-Burst Buffer: Experience with the flash-based file system on SDSC's...
The Proto-Burst Buffer: Experience with the flash-based file system on SDSC's...
Salting new ground one man ops from scratch
Salting new ground one man ops from scratch
ContainerDays Boston 2015: "CoreOS: Building the Layers of the Scalable Clust...
ContainerDays Boston 2015: "CoreOS: Building the Layers of the Scalable Clust...
10 things i wish i'd known before using spark in production
10 things i wish i'd known before using spark in production
OSDC 2017 | Open POWER for the data center by Werner Fischer
OSDC 2017 | Open POWER for the data center by Werner Fischer
OSDC 2017 - Werner Fischer - Open power for the data center
OSDC 2017 - Werner Fischer - Open power for the data center
OSDC 2017 | Linux Performance Profiling and Monitoring by Werner Fischer
OSDC 2017 | Linux Performance Profiling and Monitoring by Werner Fischer
Fully fault tolerant real time data pipeline with docker and mesos
Fully fault tolerant real time data pipeline with docker and mesos
More from Spark Summit
FPGA-Based Acceleration Architecture for Spark SQL Qi Xie and Quanfu Wang
FPGA-Based Acceleration Architecture for Spark SQL Qi Xie and Quanfu Wang
Spark Summit
VEGAS: The Missing Matplotlib for Scala/Apache Spark with DB Tsai and Roger M...
VEGAS: The Missing Matplotlib for Scala/Apache Spark with DB Tsai and Roger M...
Spark Summit
Apache Spark Structured Streaming Helps Smart Manufacturing with Xiaochang Wu
Apache Spark Structured Streaming Helps Smart Manufacturing with Xiaochang Wu
Spark Summit
Improving Traffic Prediction Using Weather Data with Ramya Raghavendra
Improving Traffic Prediction Using Weather Data with Ramya Raghavendra
Spark Summit
A Tale of Two Graph Frameworks on Spark: GraphFrames and Tinkerpop OLAP Artem...
A Tale of Two Graph Frameworks on Spark: GraphFrames and Tinkerpop OLAP Artem...
Spark Summit
No More Cumbersomeness: Automatic Predictive Modeling on Apache Spark Marcin ...
No More Cumbersomeness: Automatic Predictive Modeling on Apache Spark Marcin ...
Spark Summit
Apache Spark and Tensorflow as a Service with Jim Dowling
Apache Spark and Tensorflow as a Service with Jim Dowling
Spark Summit
Apache Spark and Tensorflow as a Service with Jim Dowling
Apache Spark and Tensorflow as a Service with Jim Dowling
Spark Summit
MMLSpark: Lessons from Building a SparkML-Compatible Machine Learning Library...
MMLSpark: Lessons from Building a SparkML-Compatible Machine Learning Library...
Spark Summit
Powering a Startup with Apache Spark with Kevin Kim
Powering a Startup with Apache Spark with Kevin Kim
Spark Summit
Improving Traffic Prediction Using Weather Datawith Ramya Raghavendra
Improving Traffic Prediction Using Weather Datawith Ramya Raghavendra
Spark Summit
Hiding Apache Spark Complexity for Fast Prototyping of Big Data Applications—...
Hiding Apache Spark Complexity for Fast Prototyping of Big Data Applications—...
Spark Summit
How Nielsen Utilized Databricks for Large-Scale Research and Development with...
How Nielsen Utilized Databricks for Large-Scale Research and Development with...
Spark Summit
Spline: Apache Spark Lineage not Only for the Banking Industry with Marek Nov...
Spline: Apache Spark Lineage not Only for the Banking Industry with Marek Nov...
Spark Summit
Goal Based Data Production with Sim Simeonov
Goal Based Data Production with Sim Simeonov
Spark Summit
Preventing Revenue Leakage and Monitoring Distributed Systems with Machine Le...
Preventing Revenue Leakage and Monitoring Distributed Systems with Machine Le...
Spark Summit
Getting Ready to Use Redis with Apache Spark with Dvir Volk
Getting Ready to Use Redis with Apache Spark with Dvir Volk
Spark Summit
Deduplication and Author-Disambiguation of Streaming Records via Supervised M...
Deduplication and Author-Disambiguation of Streaming Records via Supervised M...
Spark Summit
MatFast: In-Memory Distributed Matrix Computation Processing and Optimization...
MatFast: In-Memory Distributed Matrix Computation Processing and Optimization...
Spark Summit
Indicium: Interactive Querying at Scale Using Apache Spark, Zeppelin, and Spa...
Indicium: Interactive Querying at Scale Using Apache Spark, Zeppelin, and Spa...
Spark Summit
More from Spark Summit
(20)
FPGA-Based Acceleration Architecture for Spark SQL Qi Xie and Quanfu Wang
FPGA-Based Acceleration Architecture for Spark SQL Qi Xie and Quanfu Wang
VEGAS: The Missing Matplotlib for Scala/Apache Spark with DB Tsai and Roger M...
VEGAS: The Missing Matplotlib for Scala/Apache Spark with DB Tsai and Roger M...
Apache Spark Structured Streaming Helps Smart Manufacturing with Xiaochang Wu
Apache Spark Structured Streaming Helps Smart Manufacturing with Xiaochang Wu
Improving Traffic Prediction Using Weather Data with Ramya Raghavendra
Improving Traffic Prediction Using Weather Data with Ramya Raghavendra
A Tale of Two Graph Frameworks on Spark: GraphFrames and Tinkerpop OLAP Artem...
A Tale of Two Graph Frameworks on Spark: GraphFrames and Tinkerpop OLAP Artem...
No More Cumbersomeness: Automatic Predictive Modeling on Apache Spark Marcin ...
No More Cumbersomeness: Automatic Predictive Modeling on Apache Spark Marcin ...
Apache Spark and Tensorflow as a Service with Jim Dowling
Apache Spark and Tensorflow as a Service with Jim Dowling
Apache Spark and Tensorflow as a Service with Jim Dowling
Apache Spark and Tensorflow as a Service with Jim Dowling
MMLSpark: Lessons from Building a SparkML-Compatible Machine Learning Library...
MMLSpark: Lessons from Building a SparkML-Compatible Machine Learning Library...
Powering a Startup with Apache Spark with Kevin Kim
Powering a Startup with Apache Spark with Kevin Kim
Improving Traffic Prediction Using Weather Datawith Ramya Raghavendra
Improving Traffic Prediction Using Weather Datawith Ramya Raghavendra
Hiding Apache Spark Complexity for Fast Prototyping of Big Data Applications—...
Hiding Apache Spark Complexity for Fast Prototyping of Big Data Applications—...
How Nielsen Utilized Databricks for Large-Scale Research and Development with...
How Nielsen Utilized Databricks for Large-Scale Research and Development with...
Spline: Apache Spark Lineage not Only for the Banking Industry with Marek Nov...
Spline: Apache Spark Lineage not Only for the Banking Industry with Marek Nov...
Goal Based Data Production with Sim Simeonov
Goal Based Data Production with Sim Simeonov
Preventing Revenue Leakage and Monitoring Distributed Systems with Machine Le...
Preventing Revenue Leakage and Monitoring Distributed Systems with Machine Le...
Getting Ready to Use Redis with Apache Spark with Dvir Volk
Getting Ready to Use Redis with Apache Spark with Dvir Volk
Deduplication and Author-Disambiguation of Streaming Records via Supervised M...
Deduplication and Author-Disambiguation of Streaming Records via Supervised M...
MatFast: In-Memory Distributed Matrix Computation Processing and Optimization...
MatFast: In-Memory Distributed Matrix Computation Processing and Optimization...
Indicium: Interactive Querying at Scale Using Apache Spark, Zeppelin, and Spa...
Indicium: Interactive Querying at Scale Using Apache Spark, Zeppelin, and Spa...
Recently uploaded
VIP High Profile Call Girls Amravati Aarushi 8250192130 Independent Escort Se...
VIP High Profile Call Girls Amravati Aarushi 8250192130 Independent Escort Se...
Suhani Kapoor
Brighton SEO | April 2024 | Data Storytelling
Brighton SEO | April 2024 | Data Storytelling
Neil Barnes
VIP High Class Call Girls Jamshedpur Anushka 8250192130 Independent Escort Se...
VIP High Class Call Girls Jamshedpur Anushka 8250192130 Independent Escort Se...
Suhani Kapoor
VIP Call Girls Service Charbagh { Lucknow Call Girls Service 9548273370 } Boo...
VIP Call Girls Service Charbagh { Lucknow Call Girls Service 9548273370 } Boo...
Call Girls In Delhi Whatsup 9873940964 Enjoy Unlimited Pleasure
Carero dropshipping via API with DroFx.pptx
Carero dropshipping via API with DroFx.pptx
olyaivanovalion
04242024_CCC TUG_Joins and Relationships
04242024_CCC TUG_Joins and Relationships
ccctableauusergroup
VIP Call Girls Service Miyapur Hyderabad Call +91-8250192130
VIP Call Girls Service Miyapur Hyderabad Call +91-8250192130
Suhani Kapoor
Delhi Call Girls Punjabi Bagh 9711199171 ☎✔👌✔ Whatsapp Hard And Sexy Vip Call
Delhi Call Girls Punjabi Bagh 9711199171 ☎✔👌✔ Whatsapp Hard And Sexy Vip Call
shivangimorya083
(PARI) Call Girls Wanowrie ( 7001035870 ) HI-Fi Pune Escorts Service
(PARI) Call Girls Wanowrie ( 7001035870 ) HI-Fi Pune Escorts Service
ranjana rawat
Log Analysis using OSSEC sasoasasasas.pptx
Log Analysis using OSSEC sasoasasasas.pptx
JohnnyPlasten
B2 Creative Industry Response Evaluation.docx
B2 Creative Industry Response Evaluation.docx
Stephen266013
代办国外大学文凭《原版美国UCLA文凭证书》加州大学洛杉矶分校毕业证制作成绩单修改
代办国外大学文凭《原版美国UCLA文凭证书》加州大学洛杉矶分校毕业证制作成绩单修改
atducpo
Delhi 99530 vip 56974 Genuine Escort Service Call Girls in Kishangarh
Delhi 99530 vip 56974 Genuine Escort Service Call Girls in Kishangarh
9953056974 Low Rate Call Girls In Saket, Delhi NCR
BabyOno dropshipping via API with DroFx.pptx
BabyOno dropshipping via API with DroFx.pptx
olyaivanovalion
BigBuy dropshipping via API with DroFx.pptx
BigBuy dropshipping via API with DroFx.pptx
olyaivanovalion
Schema on read is obsolete. Welcome metaprogramming..pdf
Schema on read is obsolete. Welcome metaprogramming..pdf
Lars Albertsson
VidaXL dropshipping via API with DroFx.pptx
VidaXL dropshipping via API with DroFx.pptx
olyaivanovalion
Industrialised data - the key to AI success.pdf
Industrialised data - the key to AI success.pdf
Lars Albertsson
Smarteg dropshipping via API with DroFx.pptx
Smarteg dropshipping via API with DroFx.pptx
olyaivanovalion
Data-Analysis for Chicago Crime Data 2023
Data-Analysis for Chicago Crime Data 2023
ymrp368
Recently uploaded
(20)
VIP High Profile Call Girls Amravati Aarushi 8250192130 Independent Escort Se...
VIP High Profile Call Girls Amravati Aarushi 8250192130 Independent Escort Se...
Brighton SEO | April 2024 | Data Storytelling
Brighton SEO | April 2024 | Data Storytelling
VIP High Class Call Girls Jamshedpur Anushka 8250192130 Independent Escort Se...
VIP High Class Call Girls Jamshedpur Anushka 8250192130 Independent Escort Se...
VIP Call Girls Service Charbagh { Lucknow Call Girls Service 9548273370 } Boo...
VIP Call Girls Service Charbagh { Lucknow Call Girls Service 9548273370 } Boo...
Carero dropshipping via API with DroFx.pptx
Carero dropshipping via API with DroFx.pptx
04242024_CCC TUG_Joins and Relationships
04242024_CCC TUG_Joins and Relationships
VIP Call Girls Service Miyapur Hyderabad Call +91-8250192130
VIP Call Girls Service Miyapur Hyderabad Call +91-8250192130
Delhi Call Girls Punjabi Bagh 9711199171 ☎✔👌✔ Whatsapp Hard And Sexy Vip Call
Delhi Call Girls Punjabi Bagh 9711199171 ☎✔👌✔ Whatsapp Hard And Sexy Vip Call
(PARI) Call Girls Wanowrie ( 7001035870 ) HI-Fi Pune Escorts Service
(PARI) Call Girls Wanowrie ( 7001035870 ) HI-Fi Pune Escorts Service
Log Analysis using OSSEC sasoasasasas.pptx
Log Analysis using OSSEC sasoasasasas.pptx
B2 Creative Industry Response Evaluation.docx
B2 Creative Industry Response Evaluation.docx
代办国外大学文凭《原版美国UCLA文凭证书》加州大学洛杉矶分校毕业证制作成绩单修改
代办国外大学文凭《原版美国UCLA文凭证书》加州大学洛杉矶分校毕业证制作成绩单修改
Delhi 99530 vip 56974 Genuine Escort Service Call Girls in Kishangarh
Delhi 99530 vip 56974 Genuine Escort Service Call Girls in Kishangarh
BabyOno dropshipping via API with DroFx.pptx
BabyOno dropshipping via API with DroFx.pptx
BigBuy dropshipping via API with DroFx.pptx
BigBuy dropshipping via API with DroFx.pptx
Schema on read is obsolete. Welcome metaprogramming..pdf
Schema on read is obsolete. Welcome metaprogramming..pdf
VidaXL dropshipping via API with DroFx.pptx
VidaXL dropshipping via API with DroFx.pptx
Industrialised data - the key to AI success.pdf
Industrialised data - the key to AI success.pdf
Smarteg dropshipping via API with DroFx.pptx
Smarteg dropshipping via API with DroFx.pptx
Data-Analysis for Chicago Crime Data 2023
Data-Analysis for Chicago Crime Data 2023
Spark Summit EU talk by Jorg Schad
1.
Nobody puts Spark
in the Container Jörg Schad & Ken Sipe Mesosphere
2.
3.
© 2016 Mesosphere,
Inc. All Rights Reserved. 3
4.
5.
© 2016 Mesosphere,
Inc. All Rights Reserved. 5 Jörg Schad Distributed Systems Engineer, Mesosphere @joerg_schad Ken Sipe Distributed Applications Engineer, Mesosphere @KenSipe
6.
© 2016 Mesosphere,
Inc. All Rights Reserved. ! Datacenter-wide services to power your apps ! Turnkey installation and lifecycle management 6 DC/OS Universe DC/OS Any Infrastructure ! Container operations & big data operations ! Security, fault tolerance & high availability ! Open Source (ASL2.0) ! Based on Apache Mesos ! Production proven at scale ! Requires only a modern linux distro (windows coming soon) ! Hybrid Datacenter DATACENTER OPERATING SYSTEM (DC/OS) Datacenter Operating System (DC/OS) Distributed Systems Kernel (Mesos) Big Data + Analytics EnginesMicroservices ( containers) Streaming Batch Machine Learning Analytics Functions & Logic Search Time Series SQL / NoSQL Databases Modern App Components Any Infrastructure (Physical, Virtual, Cloud)
7.
© Gerard Julien/ Containers
8.
© 2016 Mesosphere,
Inc. All Rights Reserved. 8 Write Once Run Any Where
9.
© 2016 Mesosphere,
Inc. All Rights Reserved. 9 !I can get a shell on it (through SSH or otherwise) !It "feels" like a VM: !own process space !own network interface !can install packages !can run services High level: it appears* like a lightweight VM
10.
© 2016 Mesosphere,
Inc. All Rights Reserved. 10 !It's not like a VM: uses the host kernel can't boot a different OS !It's just a bunch of processes visible on the host machine !(contrast with VMs which are opaque) Low level: it's actually chroot on steroids
11.
© 2016 Mesosphere,
Inc. All Rights Reserved. 11 Containers vs Virtual Machines
12.
© 2016 Mesosphere,
Inc. All Rights Reserved. 12 $ ps faux USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.0 0.2 33636 2960 ? Ss Oct17 0:00 /sbin/init ... root 12972 0.0 3.9 757236 40704 ? Ssl 01:55 0:18 /usr/bin/dockerd --raw-logs root 12981 0.0 0.9 299096 9384 ? Ssl 01:55 0:01 _ docker-containerd -l unix:///var/run/docker/libcontainerd/docker- root 13850 0.0 0.4 199036 4180 ? Sl 01:58 0:00 _ docker-containerd-shim 2f86cbc34/var/run/docker/l root 13867 0.0 0.2 31752 2884 ? Ss 01:58 0:00 | _ nginx: master process nginx -g daemon off; sshd 13889 0.0 0.1 32144 1664 ? S 01:58 0:00 | _ nginx: worker process root 17642 0.0 0.4 199036 4188 ? Sl 11:54 0:00 _ docker-containerd-shim /var/run/docker/l root 17661 99.2 0.0 1172 4 ? Rs 11:54 23:37 | _ md5sum /dev/urandom root 18340 0.0 0.4 199036 4144 ? Sl 12:16 0:00 _ docker-containerd-shim 4121c64749262112b /var/run/docker/l vagrant 18353 0.0 0.0 1164 4 ? Ss 12:16 0:00 _ sleep 1000 docker run -d nginx:1.10
13.
© 2016 Mesosphere,
Inc. All Rights Reserved. 13 !Weaker isolation in containers !Containers run near-native speed CPU/IO !Containers launch in around 0.1 second (libcontainer) !Less storage and memory overhead Differences between containers and virtual machines
14.
© 2016 Mesosphere,
Inc. All Rights Reserved. 14 Isolation
15.
© 2016 Mesosphere,
Inc. All Rights Reserved. 15 (LINUX) KERNEL LAYER FS CGROUPS NAMESPACES LIBCONTAINER DOCKER
16.
© 2016 Mesosphere,
Inc. All Rights Reserved. 16 Namespaces provide isolated views: • pid (processes) • net (network interfaces, routing...) • ipc (System V IPC) • mnt (mount points, filesystems) • uts (hostname) • user (UIDs) Control groups control resources: • cpu (CPU shares) • cpuacct • cpuset (limit processes to a CPU) • memory (swap, dirty pages) • blkio (throttle reads/writes) • devices • net_cls, net_prio: control packet class and priority • freezer Namespaces VS. Cgroups
17.
© 2016 Mesosphere,
Inc. All Rights Reserved. 17 Control Groups
18.
© 2016 Mesosphere,
Inc. All Rights Reserved. 18 !Resource metering and limiting !memory !CPU !block I/O !network* !device node (/dev/*) access control !freezer Control groups
19.
© 2016 Mesosphere,
Inc. All Rights Reserved. 19 • /sys/fs/cgroup • Each subsystem (memory, CPU...) has a hierarchy (tree) • Each process belongs to exactly 1 node in each hierarchy • Each hierarchy starts with 1 node (the root) • Each node = group of processes (sharing the same resources) Control groups - Generalities
20.
© 2016 Mesosphere,
Inc. All Rights Reserved. 20 DC/OS on CoreOS
21.
© 2016 Mesosphere,
Inc. All Rights Reserved. 21 cpu/ !"" batch # !"" bitcoins # # %"" 42 # %"" hadoop # !"" 210 # %"" 98 %"" realtime !"" nginx # !"" 21 # !"" 22 # %"" 23 !"" postgres # %"" 404 %"" redis %"" 2343 memory/ !"" 21 !"" 210 !"" 22 !"" 23 !"" 42 !"" 98 %"" databases !"" 2343 %"" 404
22.
© 2016 Mesosphere,
Inc. All Rights Reserved. 22 CINF* $ sudo cinf 4026532194 PID PPID NAME STATE THREADS CGROUPS 13867 13850 nginx S (sleeping) 1 11:hugetlb:/docker/2f86cbc34a4d823be149935fa9a6dc176d161cebc719c60c7f95986c62ea7032 10:perf_event:/docker/2f86cbc34a4d823be149935fa9a6dc176d161cebc719c60c7f95986c62ea7032 9:blkio:/docker/2f86cbc34a4d823be149935fa9a6dc176d161cebc719c60c7f95986c62ea7032 8:freezer:/docker/2f86cbc34a4d823be149935fa9a6dc176d161cebc719c60c7f95986c62ea7032 7:devices:/docker/2f86cbc34a4d823be149935fa9a6dc176d161cebc719c60c7f95986c62ea7032 6:memory:/docker/2f86cbc34a4d823be149935fa9a6dc176d161cebc719c60c7f95986c62ea7032 5:cpuacct:/docker/2f86cbc34a4d823be149935fa9a6dc176d161cebc719c60c7f95986c62ea7032 4:cpu:/docker/2f86cbc34a4d823be149935fa9a6dc176d161cebc719c60c7f95986c62ea7032 3:cpuset:/docker/2f86cbc34a4d823be149935fa9a6dc176d161cebc719c60c7f95986c62ea7032 2:name=systemd:/docker/2f86cbc34a4d823be149935fa9a6dc176d161cebc719c60c7f95986c62ea7032 *https://github.com/mhausenblas/cinf
23.
© 2016 Mesosphere,
Inc. All Rights Reserved. 23 !Metrics: swap, total rss, # pages in/out !Keeps track of pages used by each group: !file (read/write/mmap from block devices) !anonymous (stack, heap, anonymous mmap) !active (recently accessed) !inactive (candidate for eviction) Memory cgroup: accounting
24.
© 2016 Mesosphere,
Inc. All Rights Reserved. 24 •Each group can have hard and soft limits •Soft limits are not enforced •Hard limits will trigger a per-group OOM killer •No OutOfMemoryError •Limits can be set for physical, kernel, total memory Memory cgroup: limits docker run -it --rm -m 128m fedora bash
25.
© 2016 Mesosphere,
Inc. All Rights Reserved. 25 !Metrics: cpuacct.stats user | system !Limitations based on type !CPU Shares !CPU Sets Cpu cgroup
26.
© 2016 Mesosphere,
Inc. All Rights Reserved. 26 !Priority Weighting across all the cores !default value is 1024 CPU Shares docker run -it --rm -c 512 stress …
27.
© 2016 Mesosphere,
Inc. All Rights Reserved. 27 !sudo cgcreate -g cpu:A !sudo cgcreate -g cpu:B !cgroup A: sudo cgset -r cpu.shares=768 A 75% !cgroup B: sudo cgset -r cpu.shares=256 B 25% CPU Shares
28.
© 2016 Mesosphere,
Inc. All Rights Reserved. 28 !Pin groups to specific CPU(s) !Reserve CPUs for specific apps !Avoid processes bouncing between CPUs !Also relevant for NUMA systems CPU Sets docker run -it -cpuset=0,4,6 stress
29.
© 2016 Mesosphere,
Inc. All Rights Reserved. 29 Namespaces
30.
© 2016 Mesosphere,
Inc. All Rights Reserved. 30 !Provide processes with their own view of the system !Multiple namespaces: !pid, net, mnt, uts, ipc, user !Each process is in one namespace of each type Namespaces
31.
© 2016 Mesosphere,
Inc. All Rights Reserved. 31 !Processes within a PID namespace only see processes in the same PID namespace !Each PID namespace has its own numbering (starting at 1) !When PID 1 goes away, the whole namespace is killed Pid namespace
32.
© 2016 Mesosphere,
Inc. All Rights Reserved. 32 Lets Talk Java
33.
© 2016 Mesosphere,
Inc. All Rights Reserved. 33 !Java Language + Java Specification + Java Runtime Java
34.
© 2016 Mesosphere,
Inc. All Rights Reserved. 34 !Native JRE !Heap !Perm / meta !JIT bytecode !JNI !NIO !Threads Java Memory Impact
35.
© 2016 Mesosphere,
Inc. All Rights Reserved. 35 From Perm to Metaspace
36.
© 2016 Mesosphere,
Inc. All Rights Reserved. 36 !JIT compiler threads !HotSpot thresholds and optimizations !Sets the default # threads for GC !Number of thread in the common fork-join pool !and more… JRE initializations based on core count
37.
© 2016 Mesosphere,
Inc. All Rights Reserved. 37 Bring it together!
38.
© 2016 Mesosphere,
Inc. All Rights Reserved. 38 !JDK 7/8 - resources from sysconf sysconf(_SC_NPROCESSORS_ONLN); !JDK 9 - sched_getaffinity !accounts for cpusets Where Java Gets it’s CPU Information
39.
© 2016 Mesosphere,
Inc. All Rights Reserved. 39 !CPUSET !pin to specific CPUs !Runtime.getRuntime().availableProcessors(); == # cores assigned* Java with CPU Set docker run -ti --cpuset=0,4,6 …
40.
© 2016 Mesosphere,
Inc. All Rights Reserved. 40 !CPU Share !Priority Weighting across all the cores !Runtime.getRuntime().availableProcessors(); == # cores on node Java with CPU Share docker run -ti -c 512 …
41.
© 2016 Mesosphere,
Inc. All Rights Reserved. 41 !Land on a 32 core box !32 cores are seen by the JRE !32 threads set by default for ForkJoinPool Java and CPU Shares
42.
© 2016 Mesosphere,
Inc. All Rights Reserved. 42 !“But memory constraints are far more problematic and may not even be queryable in general. !If there are no API's to tell the VM the real resource story what is the VM supposed to do? I don't have any answers to that.” !“When the environment lies to the VM about what is available it makes it very hard for the VM to try to adjust.” How about memory?
43.
© 2016 Mesosphere,
Inc. All Rights Reserved. 43 !“The good thing about docker containers (and some other like containers) is that they don’t hide the underlying hardware from processes like VM technology does.” !“The bad thing about docker containers (and some other like containers) is that they don’t hide the underlying hardware from processes like VM technology does.” Conclusion Kirk Pepperdine
44.
© 2016 Mesosphere,
Inc. All Rights Reserved. 44 Thank You! Learn more by visiting dcos.io and mesosphere.com
45.
© 2016 Mesosphere,
Inc. All Rights Reserved. 45 !Mar 10 17:42:39 ip-10-0-1-114.us-west-2.compute.internal mesos-slave[1190]: I0310 17:42:39.848748 1199 status_update_manager.cpp:824] Checkpointing ACK for status update TASK_RUNNING (UUID: 8d13fbb9-b02a-45da-9b52-5393ce8f0746) for task task.datanode.datanode1.1457631756250 of framework d83631ed-34 !Mar 10 17:42:41 ip-10-0-1-114.us-west-2.compute.internal mesos-slave[1190]: I0310 17:42:41.561954 1200 mem.cpp:625] OOM notifier is triggered for container 6461cafd-3962-4022-a070-f6e26488dd94 !Mar 10 17:42:41 ip-10-0-1-114.us-west-2.compute.internal mesos-slave[1190]: I0310 17:42:41.562047 1200 mem.cpp:644] OOM detected for container 6461cafd-3962-4022-a070- f6e26488dd94 !Mar 10 17:42:41 ip-10-0-1-114.us-west-2.compute.internal mesos-slave[1190]: I0310 17:42:41.566249 1200 mem.cpp:685] Memory limit exceeded: Requested: 2080MB Maximum Used: 2080MB journalctl -f _TRANSPORT=kernel
Download now