Map Reduce v2 and YARN - CHUG - 20120604

•

27 j'aime•3,865 vues

Chicago Hadoop Users Group

View the accompanying video on vimeo: https://vimeo.com/43474797

An Introduction to
MapReduce 2 and
YARN

Tom White, Cloudera
@tom_e_white
June 4, 2012
Chicago HUG
Tuesday, June 5, 2012

Road Trip

Tuesday, June 5, 2012

About me
• Apache Hadoop Committer,
PMC Member, Apache
Member

• Engineer at Cloudera
working on core Hadoop

• Founder of Apache Whirr

• Author of “Hadoop: The
Deﬁnitive Guide”

• http://hadoopbook.com

Tuesday, June 5, 2012

First, whatʼs
MapReduce 1?

Tuesday, June 5, 2012

Tuesday, June 5, 2012

Whatʼs wrong with
MR1?

Tuesday, June 5, 2012

Motivation 1

• Scaling >4000 nodes
• Fewer, larger clusters

Tuesday, June 5, 2012

Motivation 2

• HA of Job Tracker
• Large, complex state

Tuesday, June 5, 2012

Motivation 3

• Poor resource utilization
• Slots in MR1 are for either
map or reduce

Tuesday, June 5, 2012

Yet Another Resource Negotiator

Tuesday, June 5, 2012

Tuesday, June 5, 2012

Tuesday, June 5, 2012

Node Manager
is a generalized Task Tracker
• Task Tracker
• ﬁxed number of map or reduce
slots
• Node Manager
• containers with variable resource
limits

Tuesday, June 5, 2012

Tuesday, June 5, 2012

Tuesday, June 5, 2012

MR is user space
YARN is kernel

Tuesday, June 5, 2012

Bonus Apps

• Distributed shell
• MPI (MAPREDUCE-2911)

• Master-worker (MAPREDUCE-3315)

• Apache Giraph, Hama

Tuesday, June 5, 2012

Tuesday, June 5, 2012

Tuesday, June 5, 2012

Old API ≠ MR1
New API ≠ MR2

Tuesday, June 5, 2012

Old API New API
o.a.h.mapred o.a.h.mapreduce

MR1 ✓ ✓

MR2 ✓ ✓

Tuesday, June 5, 2012

Tuesday, June 5, 2012

Try out MR2

• Apache Hadoop 2.0.0-alpha
• hadoop.apache.org
• CDH4 and Cloudera Manager
• cloudera.com
• Cloud - Apache Whirr
Tuesday, June 5, 2012

MR1
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>1.0.3</version>
</dependency>

MR2
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.0.0-alpha</version>
</dependency>

Tuesday, June 5, 2012

TODO

• Still alpha status
• Performance tuning
• Usability bug ﬁxes
• RM recovery
• Security in MR2 not complete

Tuesday, June 5, 2012

Questions?

Tuesday, June 5, 2012

Contenu connexe

En vedette

An Introduction to MapReduce 2 and YARN

An Introduction to MapReduce 2 and YARN

An Introduction to MapReduce 2 and YARN

Introduction to MapReduce

Introduction to MapReduce

Introduction to MapReduce

Chicago Hadoop Users Group

Inefficient data workloads are all too common across enterprises - causing costly delays, breakages, hard-to-maintain complexity, and ultimately lost productivity. For a typical enterprise with multiple data warehouses, thousands of reports, and hundreds of thousands of ETL jobs being executed every day, this loss of productivity is a real problem. Add to all of this the complex handwritten SQL queries, and there can be nearly a million queries executed every month that desperately need to be optimized, especially to take advantage of the benefits of Apache Hadoop. How can enterprises dig through their workloads and inefficiencies to easily see which are the best fit for Hadoop and what’s the fastest path to get there? Cloudera Navigator Optimizer is the solution - analyzing existing SQL workloads to provide instant insights into your workloads and turns that into an intelligent optimization strategy so you can unlock peak performance and efficiency with Hadoop. As the newest addition to Cloudera’s enterprise Hadoop platform, and now available in limited beta, Navigator Optimizer has helped customers profile over 1.5 million queries and ultimately save millions by optimizing for Hadoop.

Hadoop for the Data Scientist: Spark in Cloudera 5.5

Hadoop for the Data Scientist: Spark in Cloudera 5.5

Hadoop for the Data Scientist: Spark in Cloudera 5.5

Apache Spark Operations

Apache Spark Operations

Apache Spark Operations

Realtime BigData Step by Step mit Lambda, Kafka, Storm und Hadoop

Realtime BigData Step by Step mit Lambda, Kafka, Storm und Hadoop

Realtime BigData Step by Step mit Lambda, Kafka, Storm und Hadoop

Valentin Zacharias

Spark on YARN: The Road Ahead

Spark on YARN: The Road Ahead

Spark on YARN: The Road Ahead

This is the presentation from the "Discover HDP 2.1: Apache Hadoop 2.4.0, YARN & HDFS" webinar on May 28, 2014. Rohit Bahkshi, a senior product manager at Hortonworks, and Vinod Vavilapalli, PMC for Apache Hadoop, discuss an overview of YARN in HDFS and new features in HDP 2.1. Those new features include: HDFS extended ACLs, HTTPs wire encryption, HDFS DataNode caching, resource manager high availability, application timeline server, and capacity scheduler pre-emption.

Discover HDP 2.1: Apache Hadoop 2.4.0, YARN & HDFS

Discover HDP 2.1: Apache Hadoop 2.4.0, YARN & HDFS

Discover HDP 2.1: Apache Hadoop 2.4.0, YARN & HDFS

The vortex of change continues all around us – inside the company, with our customers and partners. A new norm is upon us. Business models are being turned upside down – the hunters now the hunted, global equalization – size is no longer a guarantee of success. The innovative survive and thrive…the nervous and slow go under...what does all this change means for you? Find out how does Intel’s strengths help our customers in this world of change.

The Vortex of Change - Digital Transformation (Presented by Intel)

The Vortex of Change - Digital Transformation (Presented by Intel)

The Vortex of Change - Digital Transformation (Presented by Intel)

Using Big Data to Transform Your Customer’s Experience - Part 1

Using Big Data to Transform Your Customer’s Experience - Part 1

Using Big Data to Transform Your Customer’s Experience - Part 1 

At the StampedeCon 2015 Big Data Conference: YARN enables Hadoop to move beyond just pure batch processing. With that multiple workloads and tenants now must be able to share a single infrastructure for data processing. Features of the Capacity Scheduler enable resource sharing among multiple tenants in a fair manner with elastic queues to maximize utilization. This talk will focus on the features of the Capacity Scheduler that enable Multi-Tenancy and how resource sharing can be rebalanced using features like Preemption.

Apache Hadoop YARN – Multi-Tenancy, Capacity Scheduler & Preemption - Stamped...

Apache Hadoop YARN – Multi-Tenancy, Capacity Scheduler & Preemption - Stamped...

Apache Hadoop YARN – Multi-Tenancy, Capacity Scheduler & Preemption - Stamped...

[225]yarn 기반의 deep learning application cluster 구축 김제민

[225]yarn 기반의 deep learning application cluster 구축 김제민

[225]yarn 기반의 deep learning application cluster 구축 김제민

Top 5 IoT Use Cases

Top 5 IoT Use Cases

Top 5 IoT Use Cases

Part 2: Cloudera’s Operational Database: Unlocking New Benefits in the Cloud

Part 2: Cloudera’s Operational Database: Unlocking New Benefits in the Cloud

Part 2: Cloudera’s Operational Database: Unlocking New Benefits in the Cloud

Install Apache Hadoop for Development/Production

Install Apache Hadoop for Development/Production

Install Apache Hadoop for Development/Production

Apache Spark & Hadoop : Train-the-trainer

Apache Spark & Hadoop : Train-the-trainer

Apache Spark & Hadoop : Train-the-trainer

Inefficient data workloads are all too common across enterprises - causing costly delays, breakages, hard-to-maintain complexity, and ultimately lost productivity. For a typical enterprise with multiple data warehouses, thousands of reports, and hundreds of thousands of ETL jobs being executed every day, this loss of productivity is a real problem. Add to all of this the complex handwritten SQL queries, and there can be nearly a million queries executed every month that desperately need to be optimized, especially to take advantage of the benefits of Apache Hadoop. How can enterprises dig through their workloads and inefficiencies to easily see which are the best fit for Hadoop and what’s the fastest path to get there? Cloudera Navigator Optimizer is the solution - analyzing existing SQL workloads to provide instant insights into your workloads and turns that into an intelligent optimization strategy so you can unlock peak performance and efficiency with Hadoop. As the newest addition to Cloudera’s enterprise Hadoop platform, and now available in limited beta, Navigator Optimizer has helped customers profile over 1.5 million queries and ultimately save millions by optimizing for Hadoop.

Hive, Impala, and Spark, Oh My: SQL-on-Hadoop in Cloudera 5.5

Hive, Impala, and Spark, Oh My: SQL-on-Hadoop in Cloudera 5.5

Hive, Impala, and Spark, Oh My: SQL-on-Hadoop in Cloudera 5.5

HDFS Internals

Hadoop Internals (2.3.0 or later)

Hadoop Internals (2.3.0 or later)

Hadoop Internals (2.3.0 or later)

Using Big Data to Drive Customer 360

Using Big Data to Drive Customer 360

Using Big Data to Drive Customer 360

En vedette (19)

An Introduction to MapReduce 2 and YARN

An Introduction to MapReduce 2 and YARN

An Introduction to MapReduce 2 and YARN

Introduction to MapReduce

Introduction to MapReduce

Introduction to MapReduce

Hadoop for the Data Scientist: Spark in Cloudera 5.5

Hadoop for the Data Scientist: Spark in Cloudera 5.5

Hadoop for the Data Scientist: Spark in Cloudera 5.5

Apache Spark Operations

Apache Spark Operations

Apache Spark Operations

Realtime BigData Step by Step mit Lambda, Kafka, Storm und Hadoop

Realtime BigData Step by Step mit Lambda, Kafka, Storm und Hadoop

Realtime BigData Step by Step mit Lambda, Kafka, Storm und Hadoop

Spark on YARN: The Road Ahead

Spark on YARN: The Road Ahead

Spark on YARN: The Road Ahead

Discover HDP 2.1: Apache Hadoop 2.4.0, YARN & HDFS

Discover HDP 2.1: Apache Hadoop 2.4.0, YARN & HDFS

Discover HDP 2.1: Apache Hadoop 2.4.0, YARN & HDFS

The Vortex of Change - Digital Transformation (Presented by Intel)

The Vortex of Change - Digital Transformation (Presented by Intel)

The Vortex of Change - Digital Transformation (Presented by Intel)

Using Big Data to Transform Your Customer’s Experience - Part 1

Using Big Data to Transform Your Customer’s Experience - Part 1

Using Big Data to Transform Your Customer’s Experience - Part 1 

Apache Hadoop YARN – Multi-Tenancy, Capacity Scheduler & Preemption - Stamped...

Apache Hadoop YARN – Multi-Tenancy, Capacity Scheduler & Preemption - Stamped...

Apache Hadoop YARN – Multi-Tenancy, Capacity Scheduler & Preemption - Stamped...

[225]yarn 기반의 deep learning application cluster 구축 김제민

[225]yarn 기반의 deep learning application cluster 구축 김제민

[225]yarn 기반의 deep learning application cluster 구축 김제민

Top 5 IoT Use Cases

Top 5 IoT Use Cases

Top 5 IoT Use Cases

Part 2: Cloudera’s Operational Database: Unlocking New Benefits in the Cloud

Part 2: Cloudera’s Operational Database: Unlocking New Benefits in the Cloud

Part 2: Cloudera’s Operational Database: Unlocking New Benefits in the Cloud

Install Apache Hadoop for Development/Production

Install Apache Hadoop for Development/Production

Install Apache Hadoop for Development/Production

Apache Spark & Hadoop : Train-the-trainer

Apache Spark & Hadoop : Train-the-trainer

Apache Spark & Hadoop : Train-the-trainer

Hive, Impala, and Spark, Oh My: SQL-on-Hadoop in Cloudera 5.5

Hive, Impala, and Spark, Oh My: SQL-on-Hadoop in Cloudera 5.5

Hive, Impala, and Spark, Oh My: SQL-on-Hadoop in Cloudera 5.5

HDFS Internals

Hadoop Internals (2.3.0 or later)

Hadoop Internals (2.3.0 or later)

Hadoop Internals (2.3.0 or later)

Using Big Data to Drive Customer 360

Using Big Data to Drive Customer 360

Using Big Data to Drive Customer 360

Similaire à Map Reduce v2 and YARN - CHUG - 20120604

In this short talk, Ishow some great ways you can use Jenkins to supercharge both the development AND deployment of your Ruby projects. Finally, I will demonstrate how you can extend the Jenkins CI server (which is written in Java) with nothing Ruby code. If you aren't getting the most out of your CI tools, or you aren't using a CI tool at all, then this is for you! Implementing a proper CI is a game changer to the way you and your team develop software.

Ruby CI with Jenkins

Ruby CI with Jenkins

Ruby CI with Jenkins

"We should just call it testing - when it's not exploratory testing it's not real testing anyway" -Twitter, Summer 2011 Lately many professional testers have started to make a clear distinctions between thing that we call testing (like TDD and BDD) and what they consider testing - referring to TDD and BDD mostly as checking. And actually I – and I would think many of you as well – have seen projects with a test coverage of 80% and more that still fail to meet the clients' needs. Even though they meet the specifications perfectly. This points to some value that could be added to techniques like BDD and TDD by embracing the ideas from people like James Marcus Bach, Paul Carvalho and Michael Bolton. After giving an overview of current trends in the testing community like ET (exploratory testing) and ATDD (Acceptance Test Driven Development) this session will try to do exactly that: discuss the - often missing - intersection between BDD and exploratory testing and suggest ways to fill it.

Michael mahlberg exploratory-testing-the_missing_half_of_bdd

Michael mahlberg exploratory-testing-the_missing_half_of_bdd

Michael mahlberg exploratory-testing-the_missing_half_of_bdd

Michael Mahlberg

Introduction to Apache Pig

Introduction to Apache Pig

Introduction to Apache Pig

The Future of Apache CloudStack (Not So Cloudy) (Collab 2012)

The Future of Apache CloudStack (Not So Cloudy) (Collab 2012)

The Future of Apache CloudStack (Not So Cloudy) (Collab 2012)

Chiradeep Vittal

High Availability in YARN

High Availability in YARN

High Availability in YARN

Real-Time Queries in Hadoop w/ Cloudera Impala

Real-Time Queries in Hadoop w/ Cloudera Impala

Real-Time Queries in Hadoop w/ Cloudera Impala

Data Science London

Hadoop-2 @ eBay

Hadoop-2 @ eBay

Hadoop-2 @ eBay

DataWorks Summit

Similaire à Map Reduce v2 and YARN - CHUG - 20120604 (7)

Ruby CI with Jenkins

Ruby CI with Jenkins

Ruby CI with Jenkins

Michael mahlberg exploratory-testing-the_missing_half_of_bdd

Michael mahlberg exploratory-testing-the_missing_half_of_bdd

Michael mahlberg exploratory-testing-the_missing_half_of_bdd

Introduction to Apache Pig

Introduction to Apache Pig

Introduction to Apache Pig

The Future of Apache CloudStack (Not So Cloudy) (Collab 2012)

The Future of Apache CloudStack (Not So Cloudy) (Collab 2012)

The Future of Apache CloudStack (Not So Cloudy) (Collab 2012)

High Availability in YARN

High Availability in YARN

High Availability in YARN

Real-Time Queries in Hadoop w/ Cloudera Impala

Real-Time Queries in Hadoop w/ Cloudera Impala

Real-Time Queries in Hadoop w/ Cloudera Impala

Hadoop-2 @ eBay

Hadoop-2 @ eBay

Hadoop-2 @ eBay

Plus de Chicago Hadoop Users Group

Kinetica master chug_9.12

Kinetica master chug_9.12

Kinetica master chug_9.12

Chicago Hadoop Users Group

Chug dl presentation

Chug dl presentation

Chug dl presentation

Chicago Hadoop Users Group

Bobby Evans and Tom Graves, the engineering leads for Spark and Storm development at Yahoo will talk about how these technologies are used on Yahoo's grids and reasons why to use one or the other. Bobby Evans is the low latency data processing architect at Yahoo. He is a PMC member on many Apache projects including Storm, Hadoop, Spark, and Tez. His team is responsible for delivering Storm as a service to all of Yahoo and maintaining Spark on Yarn for Yahoo (Although Tom really does most of that work). Tom Graves a Senior Software Engineer on the Platform team at Yahoo. He is an Apache PMC member on Hadoop, Spark, and Tez. His team is responsible for delivering and maintaining Spark on Yarn for Yahoo.

Yahoo compares Storm and Spark

Yahoo compares Storm and Spark

Yahoo compares Storm and Spark

Chicago Hadoop Users Group

Jim Scott, CHUG co-founder and Director, Enterprise Strategy and Architecture for MapR presents "Using Apache Drill". This presentation was given on August 13th, 2014 at the Nokia office in Chicago, IL. Jim has held positions running Operations, Engineering, Architecture and QA teams. He has worked in the Consumer Packaged Goods, Digital Advertising, Digital Mapping, Chemical and Pharmaceutical industries. His work with high-throughput computing at Dow Chemical was a precursor to more standardized big data concepts like Hadoop. Apache Drill brings the power of standard ANSI:SQL 2003 to your desktop and your clusters. It is like AWK for Hadoop. Drill supports querying schemaless systems like HBase, Cassandra and MongoDB. Use standard JDBC and ODBC APIs to use Drill from your custom applications. Leveraging an efficient columnar storage format, an optimistic execution engine and a cache-conscious memory layout, Apache Drill is blazing fast. Coordination, query planning, optimization, scheduling, and execution are all distributed throughout nodes in a system to maximize parallelization. This presentation contains live demonstrations. The video can be found here: http://vimeo.com/chug/using-apache-drill

Using Apache Drill

Using Apache Drill

Using Apache Drill

Chicago Hadoop Users Group

John Leach Co-Founder and CTO of Splice Machine with 15+ years software development and machine learning experience will discuss how to use HBase co-processors to build an ANSI-99 SQL database with 1) parallelization of SQL execution plans, 2) ACID transactions with snapshot isolation and 3) consistent secondary indexing. Transactions are critical in traditional RDBMSs because they ensure reliable updates across multiple rows and tables. Most operational applications require transactions, but even analytics systems use transactions to reliably update secondary indexes after a record insert or update. In the Hadoop ecosystem, HBase is a key-value store with real-time updates, but it does not have multi-row, multi-table transactions, secondary indexes or a robust query language like SQL. Combining SQL with a full transactional model over HBase opens a whole new set of OLTP and OLAP use cases for Hadoop that was traditionally reserved for RDBMSs like MySQL or Oracle. However, a transactional HBase system has the advantage of scaling out with commodity servers, leading to a 5x-10x cost savings over traditional databases like MySQL or Oracle. HBase co-processors, introduced in release 0.92, provide a flexible and high-performance framework to extend HBase. In this talk, we show how we used HBase co-processors to support a full ANSI SQL RDBMS without modifying the core HBase source. We will discuss how endpoint transactions are used to serialize SQL execution plans over to regions so that computation is local to where the data is stored. Additionally, we will show how observer co-processors simultaneously support both transactions and secondary indexing. The talk will also discuss how Splice Machine extended the work of Google Percolator, Yahoo Labs’ OMID, and the University of Waterloo on distributed snapshot isolation for transactions. Lastly, performance benchmarks will be provided, including full TPC-C and TPC-H results that show how Hadoop/HBase can be a replacement of traditional RDBMS solutions. To view the accompanying slide deck: http://www.slideshare.net/ChicagoHUG/

Using HBase Co-Processors to Build a Distributed, Transactional RDBMS - Splic...

Using HBase Co-Processors to Build a Distributed, Transactional RDBMS - Splic...

Using HBase Co-Processors to Build a Distributed, Transactional RDBMS - Splic...

Chicago Hadoop Users Group

The Spark software stack includes a core data-processing engine, an interface for interactive querying, Spark-streaming for streaming data analysis, and growing libraries for machine-learning and graph analysis. Spark is quickly establishing itself as a leading environment for doing fast, iterative in-memory and streaming analysis. This talk will give an introduction to the Spark stack, explain how Spark has lightening fast results, and how it complements Apache Hadoop.

Meet Spark

Chicago Hadoop Users Group

Choosing the Right Big Data Architecture for your Business

Choosing the Right Big Data Architecture for your Business

Choosing the Right Big Data Architecture for your Business

Chicago Hadoop Users Group

Jeff Sposetti of Ambari discusses the Apache project Ambari used to help deploy and provision Hadoop Clusters - Ambari Overview and the Community - Ambari Architecture - Provisioning Clusters and Services -Standard Services: HDFS, YARN, MR2, Hive, new Services: Storm, Falcon - Management and Monitoring Capabilities -Nagios and Ganglia Integration - Key Innovation Features -Ambari Stacks providing dynamic service lifecycle -Ambari BluePrints powering Savannah OpenStack -Ambari Views enabling custom UI development

An Overview of Ambari

An Overview of Ambari

An Overview of Ambari

Chicago Hadoop Users Group

Hadoop and Big Data Security

Hadoop and Big Data Security

Hadoop and Big Data Security

Chicago Hadoop Users Group

Advanced Oozie

Chicago Hadoop Users Group

Scalding for Hadoop

Scalding for Hadoop

Scalding for Hadoop

Chicago Hadoop Users Group

Adam Gugliciello, a 15-year veteran in Software Engineering and Systems Architecture specializes in highly available, parallel systems. In this session, we will answer common questions and demonstrate use cases on how Hadoop and Datameer help with asset management and risk management, fraud detection and data security. This presentation was given on January 24th at the CME Group's offices in Chicago, IL.

Financial Data Analytics with Hadoop

Financial Data Analytics with Hadoop

Financial Data Analytics with Hadoop

Chicago Hadoop Users Group

Everything you wanted to know, but were afraid to ask about Oozie

Everything you wanted to know, but were afraid to ask about Oozie

Everything you wanted to know, but were afraid to ask about Oozie

Chicago Hadoop Users Group

An Introduction to Impala – Low Latency Queries for Apache Hadoop

An Introduction to Impala – Low Latency Queries for Apache Hadoop

An Introduction to Impala – Low Latency Queries for Apache Hadoop

Chicago Hadoop Users Group

HCatalog: Table Management for Hadoop - CHUG - 20120917

HCatalog: Table Management for Hadoop - CHUG - 20120917

HCatalog: Table Management for Hadoop - CHUG - 20120917

Chicago Hadoop Users Group

Hadoop in a Windows Shop - CHUG - 20120416

Hadoop in a Windows Shop - CHUG - 20120416

Hadoop in a Windows Shop - CHUG - 20120416

Chicago Hadoop Users Group

Running R on Hadoop - CHUG - 20120815

Running R on Hadoop - CHUG - 20120815

Running R on Hadoop - CHUG - 20120815

Chicago Hadoop Users Group

Avro - More Than Just a Serialization Framework - CHUG - 20120416

Avro - More Than Just a Serialization Framework - CHUG - 20120416

Avro - More Than Just a Serialization Framework - CHUG - 20120416

Chicago Hadoop Users Group

Plus de Chicago Hadoop Users Group (18)

Kinetica master chug_9.12

Kinetica master chug_9.12

Kinetica master chug_9.12

Chug dl presentation

Chug dl presentation

Chug dl presentation

Yahoo compares Storm and Spark

Yahoo compares Storm and Spark

Yahoo compares Storm and Spark

Using Apache Drill

Using Apache Drill

Using Apache Drill

Using HBase Co-Processors to Build a Distributed, Transactional RDBMS - Splic...

Using HBase Co-Processors to Build a Distributed, Transactional RDBMS - Splic...

Using HBase Co-Processors to Build a Distributed, Transactional RDBMS - Splic...

Meet Spark

Choosing the Right Big Data Architecture for your Business

Choosing the Right Big Data Architecture for your Business

Choosing the Right Big Data Architecture for your Business

An Overview of Ambari

An Overview of Ambari

An Overview of Ambari

Hadoop and Big Data Security

Hadoop and Big Data Security

Hadoop and Big Data Security

Advanced Oozie

Scalding for Hadoop

Scalding for Hadoop

Scalding for Hadoop

Financial Data Analytics with Hadoop

Financial Data Analytics with Hadoop

Financial Data Analytics with Hadoop

Everything you wanted to know, but were afraid to ask about Oozie

Everything you wanted to know, but were afraid to ask about Oozie

Everything you wanted to know, but were afraid to ask about Oozie

An Introduction to Impala – Low Latency Queries for Apache Hadoop

An Introduction to Impala – Low Latency Queries for Apache Hadoop

An Introduction to Impala – Low Latency Queries for Apache Hadoop

HCatalog: Table Management for Hadoop - CHUG - 20120917

HCatalog: Table Management for Hadoop - CHUG - 20120917

HCatalog: Table Management for Hadoop - CHUG - 20120917

Hadoop in a Windows Shop - CHUG - 20120416

Hadoop in a Windows Shop - CHUG - 20120416

Hadoop in a Windows Shop - CHUG - 20120416

Running R on Hadoop - CHUG - 20120815

Running R on Hadoop - CHUG - 20120815

Running R on Hadoop - CHUG - 20120815

Avro - More Than Just a Serialization Framework - CHUG - 20120416

Avro - More Than Just a Serialization Framework - CHUG - 20120416

Avro - More Than Just a Serialization Framework - CHUG - 20120416

Dernier

With more memory available, system performance of three Dell devices increased, which can translate to a better user experience Conclusion When your system has plenty of RAM to meet your needs, you can efficiently access the applications and data you need to finish projects and to-do lists without sacrificing time and focus. Our test results show that with more memory available, three Dell PCs delivered better performance and took less time to complete the Procyon Office Productivity benchmark. These advantages translate to users being able to complete workflows more quickly and multitask more easily. Whether you need the mobility of the Latitude 5440, the creative capabilities of the Precision 3470, or the high performance of the OptiPlex Tower Plus 7010, configuring your system with more RAM can help keep processes running smoothly, enabling you to do more without compromising performance.

Boost PC performance: How more available memory can improve productivity

Boost PC performance: How more available memory can improve productivity

Boost PC performance: How more available memory can improve productivity

Principled Technologies

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Join our latest Connector Corner webinar to discover how UiPath Integration Service revolutionizes API-centric automation in a 'Quote to Cash' process—and how that automation empowers businesses to accelerate revenue generation. A comprehensive demo will explore connecting systems, GenAI, and people, through powerful pre-built connectors designed to speed process cycle times. Speakers: James Dickson, Senior Software Engineer Charlie Greenberg, Host, Product Marketing Manager

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

presentation ICT roal in 21st century education

presentation ICT roal in 21st century education

presentation ICT roal in 21st century education

Partners Life - Insurer Innovation Award 2024

Partners Life - Insurer Innovation Award 2024

Partners Life - Insurer Innovation Award 2024

The Digital Insurer

Top 10 Most Downloaded Games on Play Store in 2024

Top 10 Most Downloaded Games on Play Store in 2024

Top 10 Most Downloaded Games on Play Store in 2024

SynarionITSolutions

This presentations targets students or working professionals. You may know Google for search, YouTube, Android, Chrome, and Gmail, but did you know Google has many developer tools, platforms & APIs? This comprehensive yet still high-level overview outlines the most impactful tools for where to run your code, store & analyze your data. It will also inspire you as to what's possible. This talk is 50 minutes in length.

Powerful Google developer tools for immediate impact! (2023-24 C)

Powerful Google developer tools for immediate impact! (2023-24 C)

Powerful Google developer tools for immediate impact! (2023-24 C)

Axa Assurance Maroc - Insurer Innovation Award 2024

Axa Assurance Maroc - Insurer Innovation Award 2024

Axa Assurance Maroc - Insurer Innovation Award 2024

The Digital Insurer

GenAI Risks & Security Meetup 01052024.pdf

GenAI Risks & Security Meetup 01052024.pdf

GenAI Risks & Security Meetup 01052024.pdf

Tata AIG General Insurance Company - Insurer Innovation Award 2024

Tata AIG General Insurance Company - Insurer Innovation Award 2024

Tata AIG General Insurance Company - Insurer Innovation Award 2024

The Digital Insurer

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Juan lago vázquez

MySQL Webinar, presented on the 25th of April, 2024. Summary: MySQL solutions enable the deployment of diverse Database Architectures tailored to specific needs, including High Availability, Disaster Recovery, and Read Scale-Out. With MySQL Shell's AdminAPI, administrators can seamlessly set up, manage, and monitor these solutions, ensuring efficiency and ease of use in their administration. MySQL Router, on the other hand, provides transparent routing from the application traffic to the backend servers in the architectures, requiring minimal configuration. Completely built in-house and supported by Oracle, these solutions have been adopted by enterprises of all sizes for their business-critical applications. In this presentation, we'll delve into various database architecture solutions to help you choose the right one based on your business requirements. Focusing on technical details and the latest features to maximize the potential of these solutions.

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

MINDCTI Revenue Release Quarter One 2024

MINDCTI Revenue Release Quarter One 2024

MINDCTI Revenue Release Quarter One 2024

A Domino Admins Adventures (Engage 2024)

A Domino Admins Adventures (Engage 2024)

A Domino Admins Adventures (Engage 2024)

Gabriella Davis

Effective data discovery is crucial for maintaining compliance and mitigating risks in today's rapidly evolving privacy landscape. However, traditional manual approaches often struggle to keep pace with the growing volume and complexity of data. Join us for an insightful webinar where industry leaders from TrustArc and Privya will share their expertise on leveraging AI-powered solutions to revolutionize data discovery. You'll learn how to: - Effortlessly maintain a comprehensive, up-to-date data inventory - Harness code scanning insights to gain complete visibility into data flows leveraging the advantages of code scanning over DB scanning - Simplify compliance by leveraging Privya's integration with TrustArc - Implement proven strategies to mitigate third-party risks Our panel of experts will discuss real-world case studies and share practical strategies for overcoming common data discovery challenges. They'll also explore the latest trends and innovations in AI-driven data management, and how these technologies can help organizations stay ahead of the curve in an ever-changing privacy landscape.

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

The Digital Insurer

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

The Digital Insurer

Boost Fertility New Invention Ups Success Rates.pdf

Boost Fertility New Invention Ups Success Rates.pdf

Boost Fertility New Invention Ups Success Rates.pdf

sudhanshuwaghmare1

Webinar Recording: https://www.panagenda.com/webinars/why-teams-call-analytics-is-critical-to-your-entire-business Nothing is as frustrating and noticeable as being in an important call and being unable to see or hear the other person. Not surprising then, that issues with Teams calls are among the most common problems users call their helpdesk for. Having in depth insight into everything relevant going on at the user’s device, local network, ISP and Microsoft itself during the call is crucial for good Microsoft Teams Call quality support. To ensure a quick and adequate solution and to ensure your users get the most out of their Microsoft 365. But did you know that ‘bad calls’ are also an excellent indicator of other problems arising? Precisely because it is so noticeable!? Like the canary in the mine, bad calls can be early indicators of problems. Problems that might otherwise not have been noticed for a while but can have a big impact on productivity and satisfaction. Join this session by Christoph Adler to learn how true Microsoft Teams call quality analytics helped other organizations troubleshoot bad calls and identify and fix problems that impacted Teams calls or the use of Microsoft365 in general. See what it can do to keep your users happy and productive! In this session we will cover - Why CQD data alone is not enough to troubleshoot call problems - The importance of attributing call problems to the right call participant - What call quality analytics can do to help you quickly find, fix-, and prevent problems - Why having retrospective detailed insights matters - Real life examples of how others have used Microsoft Teams call quality monitoring to problem shoot problems with their ISP, network, device health and more.

Why Teams call analytics are critical to your entire business

Why Teams call analytics are critical to your entire business

Why Teams call analytics are critical to your entire business

Data Cloud, More than a CDP by Matt Robison

Data Cloud, More than a CDP by Matt Robison

Data Cloud, More than a CDP by Matt Robison

Anna Loughnan Colquhoun

Dernier (20)

Boost PC performance: How more available memory can improve productivity

Boost PC performance: How more available memory can improve productivity

Boost PC performance: How more available memory can improve productivity

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

presentation ICT roal in 21st century education

presentation ICT roal in 21st century education

presentation ICT roal in 21st century education

Partners Life - Insurer Innovation Award 2024

Partners Life - Insurer Innovation Award 2024

Partners Life - Insurer Innovation Award 2024

Top 10 Most Downloaded Games on Play Store in 2024

Top 10 Most Downloaded Games on Play Store in 2024

Top 10 Most Downloaded Games on Play Store in 2024

Powerful Google developer tools for immediate impact! (2023-24 C)

Powerful Google developer tools for immediate impact! (2023-24 C)

Powerful Google developer tools for immediate impact! (2023-24 C)

Axa Assurance Maroc - Insurer Innovation Award 2024

Axa Assurance Maroc - Insurer Innovation Award 2024

Axa Assurance Maroc - Insurer Innovation Award 2024

GenAI Risks & Security Meetup 01052024.pdf

GenAI Risks & Security Meetup 01052024.pdf

GenAI Risks & Security Meetup 01052024.pdf

Tata AIG General Insurance Company - Insurer Innovation Award 2024

Tata AIG General Insurance Company - Insurer Innovation Award 2024

Tata AIG General Insurance Company - Insurer Innovation Award 2024

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

MINDCTI Revenue Release Quarter One 2024

MINDCTI Revenue Release Quarter One 2024

MINDCTI Revenue Release Quarter One 2024

A Domino Admins Adventures (Engage 2024)

A Domino Admins Adventures (Engage 2024)

A Domino Admins Adventures (Engage 2024)

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

Boost Fertility New Invention Ups Success Rates.pdf

Boost Fertility New Invention Ups Success Rates.pdf

Boost Fertility New Invention Ups Success Rates.pdf

Why Teams call analytics are critical to your entire business

Why Teams call analytics are critical to your entire business

Why Teams call analytics are critical to your entire business

Data Cloud, More than a CDP by Matt Robison

Data Cloud, More than a CDP by Matt Robison

Data Cloud, More than a CDP by Matt Robison

Map Reduce v2 and YARN - CHUG - 20120604

1. An Introduction to MapReduce 2 and YARN Tom White, Cloudera @tom_e_white June 4, 2012 Chicago HUG Tuesday, June 5, 2012

2. Road Trip Tuesday, June 5, 2012

3. About me • Apache Hadoop Committer, PMC Member, Apache Member • Engineer at Cloudera working on core Hadoop • Founder of Apache Whirr • Author of “Hadoop: The Deﬁnitive Guide” • http://hadoopbook.com Tuesday, June 5, 2012

4. First, whatʼs MapReduce 1? Tuesday, June 5, 2012

5. Tuesday, June 5, 2012

6. Whatʼs wrong with MR1? Tuesday, June 5, 2012

7. Motivation 1 • Scaling >4000 nodes • Fewer, larger clusters Tuesday, June 5, 2012

8. Motivation 2 • HA of Job Tracker • Large, complex state Tuesday, June 5, 2012

9. Motivation 3 • Poor resource utilization • Slots in MR1 are for either map or reduce Tuesday, June 5, 2012

10. Yet Another Resource Negotiator Tuesday, June 5, 2012

11. Tuesday, June 5, 2012

12. Tuesday, June 5, 2012

13. Node Manager is a generalized Task Tracker • Task Tracker • ﬁxed number of map or reduce slots • Node Manager • containers with variable resource limits Tuesday, June 5, 2012

14. Tuesday, June 5, 2012

15. Tuesday, June 5, 2012

16. MR is user space YARN is kernel Tuesday, June 5, 2012

17. Bonus Apps • Distributed shell • MPI (MAPREDUCE-2911) • Master-worker (MAPREDUCE-3315) • Apache Giraph, Hama Tuesday, June 5, 2012

18. Tuesday, June 5, 2012

19. Tuesday, June 5, 2012

20. Old API ≠ MR1 New API ≠ MR2 Tuesday, June 5, 2012

21. Old API New API o.a.h.mapred o.a.h.mapreduce MR1 ✓ ✓ MR2 ✓ ✓ Tuesday, June 5, 2012

22. Tuesday, June 5, 2012

23. Try out MR2 • Apache Hadoop 2.0.0-alpha • hadoop.apache.org • CDH4 and Cloudera Manager • cloudera.com • Cloud - Apache Whirr Tuesday, June 5, 2012

24. MR1 <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-client</artifactId> <version>1.0.3</version> </dependency> MR2 <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-client</artifactId> <version>2.0.0-alpha</version> </dependency> Tuesday, June 5, 2012

25. TODO • Still alpha status • Performance tuning • Usability bug ﬁxes • RM recovery • Security in MR2 not complete Tuesday, June 5, 2012

26. Questions? Tuesday, June 5, 2012