Cascalog at Strange Loop

Presentation of Cascalog at Strange Loop on October 15th, 2010. http://github.com/nathanmarz/cascalog

Technology

Cascalog
Data processing on Hadoop without the hassle

Nathan Marz
BackType
@nathanmarz

What is Cascalog?

Cascalog Variables and logic
Abstraction

Cascading Tuples, data workﬂows

Key/value pairs,
MapReduce aggregation

Cascalog’s components

Cascading (the job execution engine)
+
Datalog (basis of the API design)
+
Clojure (the host programming language)

Clojure

• General purpose programming language
• Dialect of Lisp that compiles to Java bytecode

Clojure
• “Programmable programming language”:
Easy to build Domain Speciﬁc Languages
(DSL) in Clojure

Clojure examples
Clojure code Result
(+ 1 2 3) 6
(> 20 18) true

(defn incr [x] (+ 1 x)) 4
(incr 3)

Cascalog basics

Deﬁne and
execute a query

Cascalog basics

Where to
emit results

Deﬁne and
execute a query

Cascalog basics

Where to
emit results

Output variables
Deﬁne and
execute a query

Cascalog basics

Where to “Predicates”: constrain
emit results the output variables

Output variables
Deﬁne and
execute a query

Predicates

Fields can be constants or variables

Predicates

Fields can be constants or variables

Variables are preﬁxed with ? or !

Predicates
• Functions
• Filters
• Aggregators
• Generators: ﬁnite sources of tuples

Join example

Joins are an implementation detail

Why another query
language for Hadoop?

Existing tools cause too much

Accidental Complexity

Accidental complexity

Complexity caused by the tool used
to solve a problem rather than the
problem itself

Accidental complexity

• Distinct query languages cause accidental
complexity
• Example: SQL injection

Query language

• We want:
• Ability to abstract
• Ability to compose

Abstraction

Clojure function that returns a subquery

Abstraction

Deﬁning and using custom operation

Composability

Dynamic query with parameterized operation

Composability

expands to

Using a predicate macro

Contrast to Pig

“Average” is 300 lines of code in Pig

Optimized aggregators
in Cascalog

Implementation of count and sum

Composability

Value normalization example #1

Composability

Value normalization example #2

Composability

For each id:
select value with the biggest timestamp

Value normalization algorithm

Composability

Implementing value normalization

Composability

Using value normalization

Try Cascalog yourself!
Project Page
http://www.github.com/nathanmarz/cascalog

Introductory Tutorial
http://nathanmarz.com/blog/introducing-cascalog/

5 minutes to install Clojure, Hadoop, and
Cascalog locally! See project README

BackType is hiring

Think Cascalog’s cool?
Come build amazing software at BackType.

http://www.backtype.com/jobs

Questions?

Follow me on Twitter at @nathanmarz
nathan.marz@gmail.com

What's hot

Low Latency Execution For Apache Spark

Jen Aman

PySpark Best Practices

Cloudera, Inc.

Scalable Data Science with SparkR: Spark Summit East talk by Felix Cheung

Bobby Evans and Tom Graves, the engineering leads for Spark and Storm development at Yahoo will talk about how these technologies are used on Yahoo's grids and reasons why to use one or the other. Bobby Evans is the low latency data processing architect at Yahoo. He is a PMC member on many Apache projects including Storm, Hadoop, Spark, and Tez. His team is responsible for delivering Storm as a service to all of Yahoo and maintaining Spark on Yarn for Yahoo (Although Tom really does most of that work). Tom Graves a Senior Software Engineer on the Platform team at Yahoo. He is an Apache PMC member on Hadoop, Spark, and Tez. His team is responsible for delivering and maintaining Spark on Yarn for Yahoo.

Yahoo compares Storm and Spark

Chicago Hadoop Users Group

Towards True Elasticity of Spark-(Michael Le and Min Li, IBM)

Towards Benchmaking Modern Distruibuted Systems-(Grace Huang, Intel)

As the data world undergoes its cambrian explosion phase our data tools need to become more advanced to keep pace. Deep Learning has emerged as a key tool in the non-linear arms race of machine learning. In this session we will take a look at how we parallelize Deep Belief Networks in Deep Learning on Hadoop’s next generation YARN framework with Iterative Reduce. We’ll also look at some real world examples of processing data with Deep Learning such as image classification and natural language processing.

Hadoop Summit 2014 - San Jose - Introduction to Deep Learning on Hadoop

Josh Patterson

High Performance Cloud Computing

Amazon Web Services

Spark Summit EU talk by Nimbus Goehausen

Ray (https://github.com/ray-project/ray) is a framework developed at UC Berkeley and maintained by Anyscale for building distributed AI applications. Over the last year, the broader machine learning ecosystem has been rapidly adopting Ray as the primary framework for distributed execution. In this talk, we will overview how libraries such as Horovod (https://horovod.ai/), XGBoost, and Hugging Face Transformers, have integrated with Ray. We will then showcase how Uber leverages Ray and these ecosystem integrations to simplify critical production workloads at Uber. This is a joint talk between Anyscale and Uber.

Ray and Its Growing Ecosystem

Nomad is a modern cluster manager by HashiCorp, designed for both long-lived services and short-lived batch processing workloads. The Nomad team has been working to bring a native integration between Nomad and Apache Spark. By running Spark jobs on Nomad, both Spark developers and the engineering organization benefit. Nomad’s architecture allows it to have an incredibly high scheduling throughput. To demonstrate this, HashiCorp scheduled 1 million containers in less than five minutes. That speed means that large Spark workloads can be immediately placed, minimizing job runtime and job start latencies. For an organization, Nomad offers many benefits. Since Nomad was designed for both batch and services, a single cluster can service both an organization’s Spark workload and all service-oriented jobs. That, coupled with the fact that Nomad uses bin-packing to place multiple jobs on each machine, means that organizations can achieve higher density. Which saves money and makes capacity planning easier. In the future, Nomad will also have the ability to enforce quotas and apply chargebacks, allowing multi-tenant clusters to be easily managed. To further increase the performance of Spark on Nomad, HashiCorp would like to ingest HDFS locality information to place the compute by the data.

Homologous Apache Spark Clusters Using Nomad with Alex Dadgar

Use of Spark MLib for Predicting the Offlining of Digital Media-(Christopher ...

We all dread “Lost task” and “Container killed by YARN for exceeding memory limits” messages in our scaled-up spark yarn applications. Even answering the question “How much memory did my application use?” is surprisingly tricky in the distributed yarn environment. Sqrrl has developed a testing framework for observing vital statistics of spark jobs including executor-by-executor memory and CPU usage over time for both the JDK and python portions of pyspark yarn containers. This talk will detail the methods we use to collect, store, and report spark yarn resource usage. This information has proved to be invaluable for performance and regression testing of the spark jobs in Sqrrl Enterprise.

Monitoring the Dynamic Resource Usage of Scala and Python Spark Jobs in Yarn:...

There’s a growing number of data scientists that use R as their primary language. While the SparkR API has made tremendous progress since release 1.6, with major advancements in Apache Spark 2.0 and 2.1, it can be difficult for traditional R programmers to embrace the Spark ecosystem. In this session, Zaidi will discuss the sparklyr package, which is a feature-rich and tidy interface for data science with Spark, and will show how it can be coupled with Microsoft R Server and extended with it’s lower-level API to become a full, first-class citizen of Spark. Learn how easy it is to go from single-threaded, memory-bound R functions to multi-threaded, multi-node, out-of-memory applications that can be deployed in a distributed cluster environment with minimal amount of code changes. You’ll also get best practices for reproducibility and performance by looking at a real-world case study of default risk classification and prediction entirely through R and Spark.

Extending the R API for Spark with sparklyr and Microsoft R Server with Ali Z...

This talk highlights major improvements in Machine Learning (ML) targeted for Apache Spark 2.0. The MLlib 2.0 release focuses on ease of use for data science—both for casual and power users. We will discuss 3 key improvements: persisting models for production, customizing Pipelines, and improvements to models and APIs critical to data science. (1) MLlib simplifies moving ML models to production by adding full support for model and Pipeline persistence. Individual models—and entire Pipelines including feature transformations—can be built on one Spark deployment, saved, and loaded onto other Spark deployments for production and serving. (2) Users will find it much easier to implement custom feature transformers and models. Abstractions automatically handle input schema validation, as well as persistence for saving and loading models. (3) For statisticians and data scientists, MLlib has doubled down on Generalized Linear Models (GLMs), which are key algorithms for many use cases. MLlib now supports more GLM families and link functions, handles corner cases more gracefully, and provides more model statistics. Also, expanded language APIs allow data scientists using Python and R to call many more algorithms. Finally, we will demonstrate these improvements live and show how they facilitate getting started with ML on Spark, customizing implementations, and moving to production.

Apache Spark MLlib 2.0 Preview: Data Science and Production

Building applications that can read and analyze a wide variety of data may change the way we do science and make business decisions. However, building such applications is challenging: real world data is expressed in natural language, images, or other “dark” data formats which are fraught with imprecision and ambiguity and so are difficult for machines to understand. This talk will describe Snorkel, whose goal is to make routine Dark Data and other prediction tasks dramatically easier. At its core, Snorkel focuses on a key bottleneck in the development of machine learning systems: the lack of large training datasets. In Snorkel, a user implicitly creates large training sets by writing simple programs that label data, instead of performing manual feature engineering or tedious hand-labeling of individual data items. We’ll provide a set of tutorials that will allow folks to write Snorkel applications that use Spark. Snorkel is open source on github and available from Snorkel.Stanford.edu.

Snorkel: Dark Data and Machine Learning with Christopher Ré

Jen Aman

A Spark Framework For < $100, < 1 Hour, Accurate Personalized DNA Analy...

Apache Spark has emerged over the past year as the imminent successor to Hadoop MapReduce. Spark can process data in memory at very high speed, while still be able to spill to disk if required. Spark’s powerful, yet flexible API allows users to write complex applications very easily without worrying about the internal workings and how the data gets processed on the cluster. Spark comes with an extremely powerful Streaming API to process data as it is ingested. Spark Streaming integrates with popular data ingest systems like Apache Flume, Apache Kafka, Amazon Kinesis etc. allowing users to process data as it comes in. In this talk, Hari will discuss the basics of Spark Streaming, its API and its integration with Flume, Kafka and Kinesis. Hari will also discuss a real-world example of a Spark Streaming application, and how code can be shared between a Spark application and a Spark Streaming application. Each stage of the application execution will be presented, which can help understand practices while writing such an application. Hari will finally discuss how to write a custom application and a custom receiver to receive data from other systems.

Real Time Data Processing Using Spark Streaming

Hari Shreedharan

Spark and Spark Streaming at Netfix-(Kedar Sedekar and Monal Daxini, Netflix)

Redis-ML is a Redis module for high performance, real-time serving of Spark-ML models. It allows users to train large complex models in Spark, and then store and query the models directly on Redis clusters. The high throughput and low latency of Redis-ML allows users to perform heavy classification operations in real time while using a minimal number of servers. This unique architecture enables significant savings in resources compared to current commonly used methods, without loss in precision or server performance. This session will demonstrate how to build a production-level recommendation system from the ground up using Spark-ML and Redis-ML. It will also describe performance and accuracy benchmarks, comparing the results with current standard methods.

Building a Large Scale Recommendation Engine with Spark and Redis-ML with Sha...

What's hot (20)

Low Latency Execution For Apache Spark

PySpark Best Practices

Scalable Data Science with SparkR: Spark Summit East talk by Felix Cheung

Yahoo compares Storm and Spark

Towards True Elasticity of Spark-(Michael Le and Min Li, IBM)

Towards Benchmaking Modern Distruibuted Systems-(Grace Huang, Intel)

Hadoop Summit 2014 - San Jose - Introduction to Deep Learning on Hadoop

High Performance Cloud Computing

Spark Summit EU talk by Nimbus Goehausen

Ray and Its Growing Ecosystem

Homologous Apache Spark Clusters Using Nomad with Alex Dadgar

Use of Spark MLib for Predicting the Offlining of Digital Media-(Christopher ...

Monitoring the Dynamic Resource Usage of Scala and Python Spark Jobs in Yarn:...

Extending the R API for Spark with sparklyr and Microsoft R Server with Ali Z...

Apache Spark MLlib 2.0 Preview: Data Science and Production

Snorkel: Dark Data and Machine Learning with Christopher Ré

A Spark Framework For < $100, < 1 Hour, Accurate Personalized DNA Analy...

Real Time Data Processing Using Spark Streaming

Spark and Spark Streaming at Netfix-(Kedar Sedekar and Monal Daxini, Netflix)

Building a Large Scale Recommendation Engine with Spark and Redis-ML with Sha...

Viewers also liked

Your Code is Wrong

Become Efficient or Die: The Story of BackType

The inherent complexity of stream processing

Demystifying Data Engineering

The Secrets of Building Realtime Big Data Systems

The right architecture is key for any IT project. This is especially the case for big data projects, where there are no standard architectures which have proven their suitability over years. This session discusses the different Big Data Architectures which have evolved over time, including traditional Big Data Architecture, Streaming Analytics architecture as well as Lambda and Kappa architecture and presents the mapping of components from both Open Source as well as the Oracle stack onto these architectures.

Big Data Architecture

Guido Schmutz

The Epistemology of Software Engineering

Using Simplicity to Make Hard Big Data Problems Easy

Storm

Runaway complexity in Big Data... and a plan to stop it

Big Data (volume) and real-time information processing (velocity) are two important aspects of Big Data systems. At first sight, these two aspects seem to be incompatible. Are traditional software architectures still the right choice? Do we need new, revolutionary architectures to tackle the requirements of Big Data? This presentation discusses the idea of the so-called lambda architecture for Big Data, which acts on the assumption of a bisection of the data-processing: in a batch-phase a temporally bounded, large dataset is processed either through traditional ETL or MapReduce. In parallel, a real-time, online processing is constantly calculating the values of the new data coming in during the batch phase. The combination of the two results, batch and online processing is giving the constantly up-to-date view. This talk presents how such an architecture can be implemented using Oracle products such as Oracle NoSQL, Hadoop and Oracle Event Processing as well as some selected products from the Open Source Software community. While this session mostly focuses on the software architecture of BigData and FastData systems, some lessons learned in the implementation of such a system are presented as well.

Lambda architecture for real time big data

Trieu Nguyen

Big Data and Fast Data - Lambda Architecture in Action

Guido Schmutz

Cascalog at Hadoop Day

Creating a GraphQL API is more and more common for PHP developers. The task can seem complex but there are a lot of tools to help. In this talk given at AFUP Paris PHP Meetup, I'm presenting GraphQL and why it is important. Then, I'm having a look at the existing libraries in PHP. Finally, I'm diving in the details of GraphQLite; a library that creates a GraphQL schema by analyzing your PHP code.

Cascalog at May Bay Area Hadoop User Group

Cascalog

Cascalog workshop

Cascading

Viewers also liked (17)

Your Code is Wrong

Become Efficient or Die: The Story of BackType

The inherent complexity of stream processing

Demystifying Data Engineering

The Secrets of Building Realtime Big Data Systems

Big Data Architecture

The Epistemology of Software Engineering

Using Simplicity to Make Hard Big Data Problems Easy

Storm

Runaway complexity in Big Data... and a plan to stop it

Lambda architecture for real time big data

Big Data and Fast Data - Lambda Architecture in Action

Cascalog at Hadoop Day

Cascalog at May Bay Area Hadoop User Group

Cascalog

Cascalog workshop

Cascading

Similar to Cascalog at Strange Loop

Yahoo! Hadoop User Group - May Meetup - Extraordinarily rapid and robust data...

Hadoop User Group

BDM25 - Spark runtime internal

David Lauzon

Enterprise Data Workflows with Cascading and Windows Azure HDInsight

Paco Nathan

PHP, the GraphQL ecosystem and GraphQLite

JEAN-GUILLAUME DUJARDIN

In the Hadoop world, writing Map Reduce is always a painful task especially when you want to create complex workflows with multiple Map Reduce jobs and having complex dependencies between them. There are high level languages like Pig latin or Hive which make writing Map Reduce jobs easy. But if you want to write complex logic, you need to write custom functions in Java, which makes testing and debugging difficult. This is where Cascading makes a developer's life easy. Everything is written in Java with ease of writing Map Reduce in high level language, similar to Pig or Hive. Once the logic is written, it can be easily be tested by running in stand-alone mode since everything is in Java. Not only that, Cascading provides Hadoop (or any other framework) independent APIs, which means workflows written in Cascading can be executed on multiple frameworks without any code change as long as Cascading connector is available

Building Complex Data Workflows with Cascading on Hadoop

Gagan Agrawal

Rafael Bagmanov «Scala in a wild enterprise»

e-Legion

AestasIT - Internal DSLs in Scala

Dmitry Buzdin

Buildingsocialanalyticstoolwithmongodb

MongoDB APAC

Boost your APIs with GraphQL 1.0

Otávio Santana

GraphQL-ify your APIs - Devoxx UK 2021

Soham Dasgupta

Introduction to Designing and Building Big Data Applications

Cloudera, Inc.

A talk given at ACM SIGMOD 2018 in support of the paper <a href="https://arxiv.org/abs/1802.10233"> Calcite: A Foundational Framework for Optimized Query Processing Over Heterogeneous Data Sources</a>. Apache Calcite is a foundational software framework that provides query processing, optimization, and query language support to many popular open-source data processing systems such as Apache Hive, Apache Storm, Apache Flink, Druid, and MapD. Calcite's architecture consists of a modular and extensible query optimizer with hundreds of built-in optimization rules, a query processor capable of processing a variety of query languages, an adapter architecture designed for extensibility, and support for heterogeneous data models and stores (relational, semi-structured, streaming, and geospatial). This flexible, embeddable, and extensible architecture is what makes Calcite an attractive choice for adoption in big-data frameworks. It is an active project that continues to introduce support for the new types of data sources, query languages, and approaches to query processing and optimization.

Apache Calcite: A Foundational Framework for Optimized Query Processing Over ...

Julian Hyde

ETL to ML: Use Apache Spark as an end to end tool for Advanced Analytics

Miklos Christine

Interactive Java Support to your tool -- The JShell API and Architecture

JavaDayUA

Big Data Processing with .NET and Spark (SQLBits 2020)

Michael Rys

Speaker: Hari Shreedharan Data Day Texas 2015 Apache Spark has emerged over the past year as the imminent successor to Hadoop MapReduce. Spark can process data in memory at very high speed, while still be able to spill to disk if required. Spark’s powerful, yet flexible API allows users to write complex applications very easily without worrying about the internal workings and how the data gets processed on the cluster. Spark comes with an extremely powerful Streaming API to process data as it is ingested. Spark Streaming integrates with popular data ingest systems like Apache Flume, Apache Kafka, Amazon Kinesis etc. allowing users to process data as it comes in. In this talk, Hari will discuss the basics of Spark Streaming, its API and its integration with Flume, Kafka and Kinesis. Hari will also discuss a real-world example of a Spark Streaming application, and how code can be shared between a Spark application and a Spark Streaming application. Each stage of the application execution will be presented, which can help understand practices while writing such an application. Hari will finally discuss how to write a custom application and a custom receiver to receive data from other systems.

Real Time Data Processing using Spark Streaming | Data Day Texas 2015

Cloudera, Inc.

Johnny Miller – Cassandra + Spark = Awesome- NoSQL matters Barcelona 2014

NoSQLmatters

Apache Spark is the next big data processing tool for Data Scientist. As seen on the recent StackOverflow analysis, it's the hottest big data technology on their site! In this talk, I'll use the PySpark interface to leverage the speed and performance of Apache Spark. I'll focus on the end to end workflow for getting data into a distributed platform, and leverage Spark to process the data for advanced analytics. I'll discuss the popular Spark APIs used for data preparation, SQL analysis, and ML algorithms. I'll explain the performance differences between Scala and Python, and how Spark has bridged the gap in performance. I'll focus on PySpark as the interface to the platform, and walk through a demo to showcase the APIs. Talk Overview: Spark's Architecture. What's out now and what's in Spark 2.0Spark APIs: Most common APIs used by Spark Common misconceptions and proper techniques for using Spark. Demo: Walk through ETL of the Reddit dataset. SparkSQL Analytics + Visualizations of the Dataset using MatplotLibSentiment Analysis on Reddit Comments

The Nitty Gritty of Advanced Analytics Using Apache Spark in Python

Miklos Christine

Since its introduction in Spark 1.4, SparkR has received contributions from both the Spark community and the R community. In this talk, we will summarize recent community efforts on extending SparkR for scalable advanced analytics. We start with the computation of summary statistics on distributed datasets, including single-pass approximate algorithms. Then we demonstrate MLlib machine learning algorithms that have been ported to SparkR and compare them with existing solutions on R, e.g., generalized linear models, classification and clustering algorithms. We also show how to integrate existing R packages with SparkR to accelerate existing R workflows.

Recent Developments In SparkR For Advanced Analytics