HTrace: Tracing in HBase and HDFS (HBase Meetup)

•

16 likes•15,797 views

HTrace: Tracing in HBase and HDFS HBase Meetup on the Night before Strata/HW @ Google in Chelsea 2014/10/15 NTT DATA Masatake Iwasaki (OSS Professional Services)

Technology

Presentation slide for HBase Meetup on the Night before Strata/HW @ Google in Chelsea
Copyright © 2014 NTT DATA Corporation
HTrace:
Tracing in HBase and HDFS
10/15/2014
Masatake Iwasaki

What is HTrace?
https://github.com/cloudera/htrace

Tracing tool for parallel distributed system
like Google's Dapper

Effective for finding bottleneck
Effective for code analysis
Low Overhead
Copyright © 2014 NTT DATA Corporation 2

Tracing spans
Span represents traced processing unit and time
Spans have parent-‐‑‒child relationship
Passing tracing info along with RPC

Time
Span Atrace id: 12345
node 1parent: root
Span Btrace id: 12345
RPC
node 2parent: A
RPCRPC
Span Ctrace id: 12345Span D
node 3parent: B
parent: B
trace id: 12345
Copyright © 2014 NTT DATA Corporation 3

Starting root span
org.htrace.Trace#startSpan create span
new trace ID is assigned when root span starts
FsShell shell = new FsShell();!
conf.setQuietMode(false);!
shell.setConf(conf);!
int res = 0;!
SpanReceiverHost.getInstance(new HdfsConfiguration());!
TraceScope ts = null;!
try {!
ts = Trace.startSpan(FsShell, Sampler.ALWAYS);!
res = ToolRunner.run(shell, argv);!
} finally {!
shell.close();!
if (ts != null) ts.close();!
}
Copyright © 2014 NTT DATA Corporation 4

Starting passive span
Starting child span only when there is parent
For server-‐‑‒side custom tracing span

if (Trace.isTracing()) {!
traceScope = Trace.startSpan(method.getName());!
}
Copyright © 2014 NTT DATA Corporation 5

$Passing tracing info along with RPC RPC header has optional field for tracing RPC with tracing info start span on server-‐‑‒side message RequestHeader {! optional uint32 call_id = 1;! optional RPCTInfo trace_info = 2;! optional string method_name = 3;! ...! }! ! message RPCTInfo {! optional int64 trace_id = 1;! optional int64 parent_id = 2;! }! Copyright © 2014 NTT DATA Corporation 6$

Span receivers
Each process loads receiver module
Receivers receive spans from in-‐‑‒process queue
Receivers send spans to collector asynchronously

SpanReceiver
Server
SpanReceiver
RPC
SpanReceiver
Client
Server
Collector/Sink
RPC
Tracing Spans
Copyright © 2014 NTT DATA Corporation 7

Passing tracing info between threads
Ongoing tracing span is stored in ThreadLocal
You need to pass tracing info between threads

if (header.hasTraceInfo()) {!
// If the incoming RPC included tracing info, always continue the trace TraceInfo parentSpan = new TraceInfo(header.getTraceInfo().getTraceId(),!
header.getTraceInfo().getParentId());!
traceSpan = Trace.startSpan(rpcRequest.toString(), parentSpan).detach();!
}!
Call call = new Call(header.getCallId(), header.getRetryCount(),!
rpcRequest, this, ProtoUtil.convert(header.getRpcKind()),!
header.getClientId().toByteArray(), traceSpan);!
!
...!
!
if (call.traceSpan != null) {!
traceScope = Trace.continueSpan(call.traceSpan);!
}!

Copyright © 2014 NTT DATA Corporation 8

JIRAs
Already available in
HBase (HBASE-‐‑‒6449)
HDFS (HDFS-‐‑‒5274)

Working on
YARN (YARN-‐‑‒1418)
Copyright © 2014 NTT DATA Corporation 9

Configurations
Setting receiver class impl turns on tracing
Each receiver impl has its own additional confs
!
property!
namehbase.trace.spanreceiver.classes/name!
valueorg.htrace.impl.HBaseSpanReceiver/value!
/property !
property!
namehbase.htrace.hbase.collector-quorum/name!
value127.0.0.1/value!
/property

Copyright © 2014 NTT DATA Corporation 10

Tracing from HBase shell
trace command start/stop tracing span
# You need configuration on client node
!
$ hbase shell!
trace 'start'!
create 'test', 'f'!
trace 'stop'
Copyright © 2014 NTT DATA Corporation 11

Example: Creating table in HBase
Copyright © 2014 NTT DATA Corporation 12

Example: Tracing of putting 200MB of a file to HDFS
Copyright © 2014 NTT DATA Corporation 13

Example: Tracing of getting 200MB of a file from HDFS
Copyright © 2014 NTT DATA Corporation 14

Example: Zipkin UI
Copyright © 2014 NTT DATA Corporation 15

Todo
Adding granular tracing spans
Sampling and filtering spans
Dynamic reconfiguration (HDFS-‐‑‒6956)
sink and viewer with less dependency
Copyright © 2014 NTT DATA Corporation 16

Copyright © 2011 NTT DATA Corporation
Copyright © 2014 NTT DATA Corporation

What's hot

Introduction to Spark with Python

Gokhan Atil

Meta/Facebook's database serving social workloads is running on top of MyRocks (MySQL on RocksDB). This means our performance and reliability depends a lot on RocksDB. Not just MyRocks, but also we have other important systems running on top of RocksDB. We have learned many lessons from operating and debugging RocksDB at scale. In this session, we will offer an overview of RocksDB, key differences from InnoDB, and share a few interesting lessons learned from production.

RocksDB Performance and Reliability Practices

Yoshinori Matsunobu

Stream Processing with Flink and Stream Sharing

confluent

Intro to HBase

alexbaranau

Zstandard is a fast compression algorithm which you can use in Apache Spark in various way. In this talk, I briefly summarized the evolution history of Apache Spark in this area and four main use cases and the benefits and the next steps: 1) ZStandard can optimize Spark local disk IO by compressing shuffle files significantly. This is very useful in K8s environments. It’s beneficial not only when you use `emptyDir` with `memory` medium, but also it maximizes OS cache benefit when you use shared SSDs or container local storage. In Spark 3.2, SPARK-34390 takes advantage of ZStandard buffer pool feature and its performance gain is impressive, too. 2) Event log compression is another area to save your storage cost on the cloud storage like S3 and to improve the usability. SPARK-34503 officially switched the default event log compression codec from LZ4 to Zstandard. 3) Zstandard data file compression can give you more benefits when you use ORC/Parquet files as your input and output. Apache ORC 1.6 supports Zstandardalready and Apache Spark enables it via SPARK-33978. The upcoming Parquet 1.12 will support Zstandard compression. 4) Last, but not least, since Apache Spark 3.0, Zstandard is used to serialize/deserialize MapStatus data instead of Gzip. There are more community works to utilize Zstandard to improve Spark. For example, Apache Avro community also supports Zstandard and SPARK-34479 aims to support Zstandard in Spark’s avro file format in Spark 3.2.0.

The Rise of ZStandard: Apache Spark/Parquet/ORC/Avro

Databricks

Dynamodb Presentation

advaitdeo

How to understand and analyze Apache Hive query execution plan for performanc...

DataWorks Summit/Hadoop Summit

Amazon Elastcsearch Service 소개 및 활용 방법 (윤석찬)

Amazon Web Services Korea

Speaker: Matthias J. Sax, Software Engineer, Confluent ksqlDB is a distributed event streaming database system that allows users to express SQL queries over relational tables and event streams. The project was released by Confluent in 2017 and is hosted on Github and developed with an open-source spirit. ksqlDB is built on top of Apache Kafka®, a distributed event streaming platform. In this talk, we discuss ksqlDB’s architecture that is influenced by Apache Kafka and its stream processing library, Kafka Streams. We explain how ksqlDB executes continuous queries while achieving fault tolerance and high vailability. Furthermore, we explore ksqlDB’s streaming SQL dialect and the different types of supported queries. Matthias J. Sax is a software engineer at Confluent working on ksqlDB. He mainly contributes to Kafka Streams, Apache Kafka's stream processing library, which serves as ksqlDB's execution engine. Furthermore, he helps evolve ksqlDB's "streaming SQL" language. In the past, Matthias also contributed to Apache Flink and Apache Storm and he is an Apache committer and PMC member. Matthias holds a Ph.D. from Humboldt University of Berlin, where he studied distributed data stream processing systems. https://db.cs.cmu.edu/events/quarantine-db-talk-2020-confluent-ksqldb-a-stream-relational-database-system/

ksqlDB: A Stream-Relational Database System

confluent

Originally, Hadoop was used as a batch analytics tool; however, this is rapidly changing, as applications move towards real-time processing and streaming. Amazon Elastic MapReduce has made running Hadoop in the cloud easier and more accessible than ever. Each day, tens of thousands of Hadoop clusters are run on the Amazon Elastic MapReduce infrastructure by users of every size — from university students to Fortune 50 companies. We recently launched Amazon Kinesis – a managed service for real-time processing of high volume, streaming data. Amazon Kinesis enables a new class of big data applications which can continuously analyze data at any volume and throughput, in real-time. Adi will discuss each service, dive into how customers are adopting the services for different use cases, and share emerging best practices. Learn how you can architect Amazon Kinesis and Amazon Elastic MapReduce together to create a highly scalable real-time analytics solution which can ingest and process terabytes of data per hour from hundreds of thousands of different concurrent sources. Forever change how you process web site click-streams, marketing and financial transactions, social media feeds, logs and metering data, and location-tracking events.

Real-time Streaming and Querying with Amazon Kinesis and Amazon Elastic MapRe...

Amazon Web Services

Pegasus In Depth (2018/10)

涛吴

Complex event processing (CEP) is about identifying business opportunities and threats in real time by detecting patterns in data and taking appropriate automated action. Example business use cases for CEP include location-based marketing, smart inventories, targeted ads, Wi-Fi offloading, fraud detection, churn prediction, fleet management, predictive maintenance, security incident event management, and many more. While Spark Streaming provides a distributed resilient framework for ingesting events in real time, effort is still needed to build CEP applications. This is because CEP use cases require correlation of events, which in turn requires us to treat every incoming event as a discrete occurrence in time. Spark Streaming treats the entire batch of events as single occurrence. Many CEP use cases also require alerts to be fired even when there is no incoming event. An example of such use case is to fire an alert when an order-shipped event is NOT received within the SLA times following an order-received event. At Oracle we have adopted a few neat techniques like running continuous query engines as long running tasks, using empty batches as triggers, etc. to bring complex event processing to Spark Streaming. Join us to learn more on CEP for Spark, the fastest growing data processing platform in the world. Speakers Prabhu Thukkaram, Senior Director, Product Development, Oracle Hoyong Park, Architect, Oracle

Bringing complex event processing to Spark streaming

DataWorks Summit

Apache HBase is an open source distributed data-store capable of managing billions of rows of semi-structured data across large clusters of commodity hardware. HBase provides real-time random read-write access as well as integration with Hadoop MapReduce, Hive, and Pig for batch analysis. In this talk, Todd will provide an introduction to the capabilities and characteristics of HBase, comparing and contrasting it with traditional database systems. He will also introduce its architecture and data model, and present some example use cases.

Chicago Data Summit: Apache HBase: An Introduction

Cloudera, Inc.

Strongly Consistent Global Indexes for Apache Phoenix

YugabyteDB

Debunking the Myths of HDFS Erasure Coding Performance

DataWorks Summit/Hadoop Summit

Apache Phoenix and HBase: Past, Present and Future of SQL over HBase

DataWorks Summit/Hadoop Summit

Hive on Spark の設計指針を読んでみた

Recruit Technologies

PHP5.5新機能「ジェネレータ」初心者入門

kwatch

Time-Series Apache HBase

HBaseCon

Getting Started with Confluent Schema Registry

confluent

What's hot (20)

Introduction to Spark with Python

RocksDB Performance and Reliability Practices

Stream Processing with Flink and Stream Sharing

Intro to HBase

The Rise of ZStandard: Apache Spark/Parquet/ORC/Avro

Dynamodb Presentation

How to understand and analyze Apache Hive query execution plan for performanc...

Amazon Elastcsearch Service 소개 및 활용 방법 (윤석찬)

ksqlDB: A Stream-Relational Database System

Real-time Streaming and Querying with Amazon Kinesis and Amazon Elastic MapRe...

Pegasus In Depth (2018/10)

Bringing complex event processing to Spark streaming

Chicago Data Summit: Apache HBase: An Introduction

Strongly Consistent Global Indexes for Apache Phoenix

Debunking the Myths of HDFS Erasure Coding Performance

Apache Phoenix and HBase: Past, Present and Future of SQL over HBase

Hive on Spark の設計指針を読んでみた

PHP5.5新機能「ジェネレータ」初心者入門

Time-Series Apache HBase

Getting Started with Confluent Schema Registry

Similar to HTrace: Tracing in HBase and HDFS (HBase Meetup)

Spark Summit EU talk by Debasish Das and Pramod Narasimha

Spark Summit

Case study ap log collector

Jyun-Yao Huang

Genomics Is Not Special: Towards Data Intensive Biology

Uri Laserson

Querying Network Packet Captures with Spark and Drill

Vince Gonzalez

From the Hadoop Summit 2015 Session with Tomer Shiran. To deliver real-time impact from big data, organizations must evolve beyond traditional analytic approaches to support a new class of agile, distributed applications. Real-time Hadoop overcomes batch programs reliant on data transformations and schema management. This session highlights how leading organizations are leveraging Hadoop and NoSQL to merge analytics and production data to make adjustments while business is happening to optimize revenue, mitigate risk and reduce operational costs. Details include how companies have achieved real-time impact on their business, collapsed data silos, and automated in-line analytics with operational data for immediate impact.

Real Time and Big Data – It’s About Time

MapR Technologies

Real Time and Big Data – It’s About Time

DataWorks Summit

If you are running Apache Spark in cloud environments, Object Stores —such as Amazon S3 or Azure WASB— are a core part of your system. What you can’t do is treat them like “just another filesystem” —do that and things will, eventually, go horribly wrong. This talk looks at the object stores in the cloud infrastructures, including underlying architectures., compares them to what a “real filesystem” is expected to do and shows how to use object stores efficiently and safely as sources of and destinations of data. It goes into depth on recent “S3a” work, showing how including improvements in performance, security, functionality and measurement —and demonstrating how to use make best use of it from a spark application. If you are planning to deploy Spark in cloud, or doing so today: this is information you need to understand. The performance of you code and integrity of your data depends on it.

Spark and Object Stores —What You Need to Know: Spark Summit East talk by Ste...

Spark Summit

Apache Spark and Object Stores

Steve Loughran

Python is one of the most popular programming languages for advanced analytics, data science, machine learning, and deep learning. One of Python’s greatest assets is its extensive set of libraries, such as Numpy, Pandas, Scikit-learn, Theano, TensorFlow, Keras, and so on. Apache Spark is becoming the core component for big data processing and playing important role to help data scientists solve complicated problems. It has a great significance and strong demand to integrate Spark with the extremely rich Python ecosystems to handle challenges in artificial intelligence. In the latest Spark 2.3, some very exciting features were put in, for example: vectorized UDF in PySpark, which leverages Apache Arrow to provide high performance interoperability between Spark and Pandas/Numpy; Image format in dataFrame/dataset, which can improve Spark and TensorFlow (or other deep learning libraries) interoperability; high-efficiency parallel modeling tuning with Spark MLlib, etc. In this talk, we'll share best practice on real use cases and hands-on experiences to illustrate the power of these new features and bring more discussions on this topic. Speaker: Yanbo Liang, Staff Software Engineer, Hortonworks

Apache Spark 2.3 boosts advanced analytics and deep learning with Python

DataWorks Summit

20150704 benchmark and user experience in sahara weiting

Wei Ting Chen

Sql saturday pig session (wes floyd) v2

Wes Floyd

For the first time, Hortonworks Data Platform ships with Apache Storm for processing stream data in Hadoop. In this presentation, Himanshu Bari, Hortonworks senior product manager, and Taylor Goetz, Hortonworks engineer and committer to Apache Storm, cover Storm and stream processing in HDP 2.1: + Key requirements of a streaming solution and common use cases + An overview of Apache Storm + Q & A

Discover HDP2.1: Apache Storm for Stream Data Processing in Hadoop

Hortonworks

Get most out of Spark on YARN

DataWorks Summit

Visual Mapping of Clickstream Data

DataWorks Summit

Fraud Detection using Hadoop

hadooparchbook

Virtual training Intro to InfluxDB & Telegraf

InfluxData

Spark mhug2

Joseph Niemiec

Gruter_TECHDAY_2014_03_ApacheTajo (in Korean)

Gruter

EKAW - Publishing with Triple Pattern Fragments

Ruben Taelman

Architecting a Fraud Detection Application with Hadoop

DataWorks Summit

Similar to HTrace: Tracing in HBase and HDFS (HBase Meetup) (20)

Spark Summit EU talk by Debasish Das and Pramod Narasimha

Case study ap log collector

Genomics Is Not Special: Towards Data Intensive Biology

Querying Network Packet Captures with Spark and Drill

Real Time and Big Data – It’s About Time

Spark and Object Stores —What You Need to Know: Spark Summit East talk by Ste...

Apache Spark and Object Stores

Apache Spark 2.3 boosts advanced analytics and deep learning with Python

20150704 benchmark and user experience in sahara weiting

Sql saturday pig session (wes floyd) v2

Discover HDP2.1: Apache Storm for Stream Data Processing in Hadoop

Get most out of Spark on YARN

Visual Mapping of Clickstream Data

Fraud Detection using Hadoop

Virtual training Intro to InfluxDB & Telegraf

Spark mhug2

Gruter_TECHDAY_2014_03_ApacheTajo (in Korean)

EKAW - Publishing with Triple Pattern Fragments

Architecting a Fraud Detection Application with Hadoop

Recently uploaded

Building Digital Trust in a Digital Economy Veronica Tan, Director - Cyber Security Agency of Singapore Apidays Singapore 2024: Connecting Customers, Business and Technology (April 17 & 18, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

apidays

What is a good lead in your organisation? Which leads are priority? What happens to leads? When sales and marketing give different answers to these questions, or perhaps aren't sure of the answers at all, frustrations build and opportunities are left on the table. Join us for an illuminating session with Cian McLoughlin, HubSpot Principal Customer Success Manager, as we look at that crucial piece of the customer journey in which leads are transferred from marketing to sales.

04-2024-HHUG-Sales-and-Marketing-Alignment.pptx

HampshireHUG

The 7 Things I Know About Cyber Security After 25 Years | April 2024

Rafal Los

2024: Domino Containers - The Next Step. News from the Domino Container commu...

Martijn de Jong

Data Cloud, More than a CDP by Matt Robison

Anna Loughnan Colquhoun

presentation ICT roal in 21st century education

jfdjdjcjdnsjd

BooK Now Call us at +918448380779 to hire a gorgeous and seductive call girl for sex. Take a Delhi Escort Service. The help of our escort agency is mostly meant for men who want sexual Indian Escorts In Delhi NCR. It should be noted that any impersonator will get 100 attention from our Young Girls Escorts in Delhi. They will assume the position of reliable allies. VIP Call Girl With Original Photos Book Tonight +918448380779 Our Cheap Price 1 Hour not available 2 Hours 5000 Full Night 8000 TAG: Call Girls in Delhi, Noida, Gurgaon, Ghaziabad, Connaught Place, Greater Kailash Delhi, Lajpat Nagar Delhi, Mayur Vihar Delhi, Chanakyapuri Delhi, New Friends Colony Delhi, Majnu Ka Tilla, Karol Bagh, Malviya Nagar, Saket, Khan Market, Noida Sector 18, Noida Sector 76, Noida Sector 51, Gurgaon Mg Road, Iffco Chowk Gurgaon, Rajiv Chowk Gurgaon All Delhi Ncr Free Home Deliver

08448380779 Call Girls In Friends Colony Women Seeking Men

Delhi Call girls

How to convert PDF to text with Nanonets

naman860154

Enterprise Knowledge’s Urmi Majumder, Principal Data Architecture Consultant, and Fernando Aguilar Islas, Senior Data Science Consultant, presented "Driving Behavioral Change for Information Management through Data-Driven Green Strategy" on March 27, 2024 at Enterprise Data World (EDW) in Orlando, Florida. In this presentation, Urmi and Fernando discussed a case study describing how the information management division in a large supply chain organization drove user behavior change through awareness of the carbon footprint of their duplicated and near-duplicated content, identified via advanced data analytics. Check out their presentation to gain valuable perspectives on utilizing data-driven strategies to influence positive behavioral shifts and support sustainability initiatives within your organization. In this session, participants gained answers to the following questions: - What is a Green Information Management (IM) Strategy, and why should you have one? - How can Artificial Intelligence (AI) and Machine Learning (ML) support your Green IM Strategy through content deduplication? - How can an organization use insights into their data to influence employee behavior for IM? - How can you reap additional benefits from content reduction that go beyond Green IM?

Driving Behavioral Change for Information Management through Data-Driven Gree...

Enterprise Knowledge

Discord is a free app offering voice, video, and text chat functionalities, primarily catering to the gaming community. It serves as a hub for users to create and join servers tailored to their interests. Discord’s ecosystem comprises servers, each functioning as a distinct online community with its own channels dedicated to specific topics or activities. Users can engage in text-based discussions, voice calls, or video chats within these channels. Understanding Discord Servers Discord servers are virtual spaces where users congregate to interact, share content, and build communities. Servers may revolve around gaming, hobbies, interests, or fandoms, providing a platform for like-minded individuals to connect. Communication Features Discord offers a range of communication tools, including text channels for messaging, voice channels for real-time audio conversations, and video channels for face-to-face interactions. These features facilitate seamless communication and collaboration. What Does NSFW Mean? The acronym NSFW stands for “Not Safe For Work,” indicating content that may be inappropriate for professional or public settings. NSFW Content NSFW content encompasses material that is sexually explicit, violent, or otherwise graphic in nature. It often includes nudity, profanity, or depictions of sensitive topics.

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

UK Journal

Partners Life - Insurer Innovation Award 2024

The Digital Insurer

Handwritten Text Recognition for manuscripts and early printed texts

Maria Levchenko

The Raspberry Pi 5 was announced on October 2023. This new version of the popular embedded device comes with a new iteration of Broadcom’s VideoCore GPU platform, and was released with a fully open source driver stack, developed by Igalia. The presentation will discuss some of the major changes required to support this new Video Core iteration, the challenges we faced in the process and the solutions we provided in order to deliver conformant OpenGL ES and Vulkan drivers. The talk will also cover the next steps for the open source Raspberry Pi 5 graphics stack. (c) Embedded Open Source Summit 2024 April 16-18, 2024 Seattle, Washington (US) https://events.linuxfoundation.org/embedded-open-source-summit/ https://eoss24.sched.com/event/1aBEx

Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...

Igalia

Scaling API-first – The story of a global engineering organization

Radu Cotescu

GenAI Risks & Security Meetup 01052024.pdf

lior mazor

With more memory available, system performance of three Dell devices increased, which can translate to a better user experience Conclusion When your system has plenty of RAM to meet your needs, you can efficiently access the applications and data you need to finish projects and to-do lists without sacrificing time and focus. Our test results show that with more memory available, three Dell PCs delivered better performance and took less time to complete the Procyon Office Productivity benchmark. These advantages translate to users being able to complete workflows more quickly and multitask more easily. Whether you need the mobility of the Latitude 5440, the creative capabilities of the Precision 3470, or the high performance of the OptiPlex Tower Plus 7010, configuring your system with more RAM can help keep processes running smoothly, enabling you to do more without compromising performance.

Boost PC performance: How more available memory can improve productivity

Principled Technologies

Imagine a world where information flows as swiftly as thought itself, making decision-making as fluid as the data driving it. Every moment is critical, and the right tools can significantly boost your organization’s performance. The power of real-time data automation through FME can turn this vision into reality. Aimed at professionals eager to leverage real-time data for enhanced decision-making and efficiency, this webinar will cover the essentials of real-time data and its significance. We’ll explore: FME’s role in real-time event processing, from data intake and analysis to transformation and reporting An overview of leveraging streams vs. automations FME’s impact across various industries highlighted by real-life case studies Live demonstrations on setting up FME workflows for real-time data Practical advice on getting started, best practices, and tips for effective implementation Join us to enhance your skills in real-time data automation with FME, and take your operational capabilities to the next level.

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

Safe Software

08448380779 Call Girls In Greater Kailash - I Women Seeking Men

Delhi Call girls

Heather Hedden, Senior Consultant at Enterprise Knowledge, presented “The Role of Taxonomy and Ontology in Semantic Layers” at a webinar hosted by Progress Semaphore on April 16, 2024. Taxonomies at their core enable effective tagging and retrieval of content, and combined with ontologies they extend to the management and understanding of related data. There are even greater benefits of taxonomies and ontologies to enhance your enterprise information architecture when applying them to a semantic layer. A survey by DBP-Institute found that enterprises using a semantic layer see their business outcomes improve by four times, while reducing their data and analytics costs. Extending taxonomies to a semantic layer can be a game-changing solution, allowing you to connect information silos, alleviate knowledge gaps, and derive new insights. Hedden, who specializes in taxonomy design and implementation, presented how the value of taxonomies shouldn’t reside in silos but be integrated with ontologies into a semantic layer. Learn about: - The essence and purpose of taxonomies and ontologies in information and knowledge management; - Advantages of semantic layers leveraging organizational taxonomies; and - Components and approaches to creating a semantic layer, including the integration of taxonomies and ontologies

The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf

Enterprise Knowledge

In this session, we will delve into strategic approaches for optimizing knowledge management within Microsoft 365, amidst the evolving landscape of Copilot. From leveraging automatic metadata classification and permission governance with SharePoint Premium, to unlocking Viva Engage for the cultivation of knowledge and communities, you will gain actionable insights to bolster your organization's knowledge-sharing initiatives. In this session, we will also explore how to facilitate solutions to enable your employees to find answers and expertise within Microsoft 365. You will leave equipped with practical techniques and a deeper understanding of how there is more to effective knowledge management than just enabling Copilot, but building actual solutions to prepare the knowledge that Copilot and your employees can use.

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

Drew Madelung

Recently uploaded (20)

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

04-2024-HHUG-Sales-and-Marketing-Alignment.pptx

The 7 Things I Know About Cyber Security After 25 Years | April 2024

2024: Domino Containers - The Next Step. News from the Domino Container commu...

Data Cloud, More than a CDP by Matt Robison

presentation ICT roal in 21st century education

08448380779 Call Girls In Friends Colony Women Seeking Men

How to convert PDF to text with Nanonets

Driving Behavioral Change for Information Management through Data-Driven Gree...

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

Partners Life - Insurer Innovation Award 2024

Handwritten Text Recognition for manuscripts and early printed texts

Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...

Scaling API-first – The story of a global engineering organization

GenAI Risks & Security Meetup 01052024.pdf

Boost PC performance: How more available memory can improve productivity

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

08448380779 Call Girls In Greater Kailash - I Women Seeking Men

The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

HTrace: Tracing in HBase and HDFS (HBase Meetup)

2. What is HTrace? https://github.com/cloudera/htrace Tracing tool for parallel distributed system like Google's Dapper Effective for finding bottleneck Effective for code analysis Low Overhead Copyright © 2014 NTT DATA Corporation 2

3. Tracing spans Span represents traced processing unit and time Spans have parent-‐‑‒child relationship Passing tracing info along with RPC Time Span Atrace id: 12345 node 1parent: root Span Btrace id: 12345 RPC node 2parent: A RPCRPC Span Ctrace id: 12345Span D node 3parent: B parent: B trace id: 12345 Copyright © 2014 NTT DATA Corporation 3

4. Starting root span org.htrace.Trace#startSpan create span new trace ID is assigned when root span starts FsShell shell = new FsShell();! conf.setQuietMode(false);! shell.setConf(conf);! int res = 0;! SpanReceiverHost.getInstance(new HdfsConfiguration());! TraceScope ts = null;! try {! ts = Trace.startSpan(FsShell, Sampler.ALWAYS);! res = ToolRunner.run(shell, argv);! } finally {! shell.close();! if (ts != null) ts.close();! } Copyright © 2014 NTT DATA Corporation 4

5. Starting passive span Starting child span only when there is parent For server-‐‑‒side custom tracing span if (Trace.isTracing()) {! traceScope = Trace.startSpan(method.getName());! } Copyright © 2014 NTT DATA Corporation 5

6. Passing tracing info along with RPC RPC header has optional field for tracing RPC with tracing info start span on server-‐‑‒side message RequestHeader {! optional uint32 call_id = 1;! optional RPCTInfo trace_info = 2;! optional string method_name = 3;! ...! }! ! message RPCTInfo {! optional int64 trace_id = 1;! optional int64 parent_id = 2;! }! Copyright © 2014 NTT DATA Corporation 6

7. Span receivers Each process loads receiver module Receivers receive spans from in-‐‑‒process queue Receivers send spans to collector asynchronously SpanReceiver Server SpanReceiver RPC SpanReceiver Client Server Collector/Sink RPC Tracing Spans Copyright © 2014 NTT DATA Corporation 7

8. Passing tracing info between threads Ongoing tracing span is stored in ThreadLocal You need to pass tracing info between threads if (header.hasTraceInfo()) {! // If the incoming RPC included tracing info, always continue the trace TraceInfo parentSpan = new TraceInfo(header.getTraceInfo().getTraceId(),! header.getTraceInfo().getParentId());! traceSpan = Trace.startSpan(rpcRequest.toString(), parentSpan).detach();! }! Call call = new Call(header.getCallId(), header.getRetryCount(),! rpcRequest, this, ProtoUtil.convert(header.getRpcKind()),! header.getClientId().toByteArray(), traceSpan);! ! ...! ! if (call.traceSpan != null) {! traceScope = Trace.continueSpan(call.traceSpan);! }! Copyright © 2014 NTT DATA Corporation 8

10. Configurations Setting receiver class impl turns on tracing Each receiver impl has its own additional confs ! property! namehbase.trace.spanreceiver.classes/name! valueorg.htrace.impl.HBaseSpanReceiver/value! /property ! property! namehbase.htrace.hbase.collector-quorum/name! value127.0.0.1/value! /property Copyright © 2014 NTT DATA Corporation 10

11. Tracing from HBase shell trace command start/stop tracing span # You need configuration on client node ! $ hbase shell! trace 'start'! create 'test', 'f'! trace 'stop' Copyright © 2014 NTT DATA Corporation 11

HTrace: Tracing in HBase and HDFS (HBase Meetup)

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to HTrace: Tracing in HBase and HDFS (HBase Meetup)

Similar to HTrace: Tracing in HBase and HDFS (HBase Meetup) (20)

More from NTT DATA OSS Professional Services

More from NTT DATA OSS Professional Services (20)

Recently uploaded

Recently uploaded (20)

HTrace: Tracing in HBase and HDFS (HBase Meetup)