Use case and integration of ClickHouse with Apache Superset & Dremio

•

2 j'aime•1,359 vues

Altinity Ltd

Presentation by Rahul Sharma to the SF Bay Area 10 September 2020 meetup.

Logiciels

Use case and Integration of Clickhouse
with Apache Superset & Dremio
Rahul Sharma, India
Lead Consultant – GIS Solution / Former Research Scientist/Mentor / Speaker
ISRO/ MoH&FW / BMGF / MoWR
rahul.rrsc@gmail.com, skype: @rahulsmartgis
https://www.linkedin.com/in/rahul-sharma-a3595a14/

Use Case: Health Census Data
4000 + Variables
700+ Tables ( 1 TB
Size)
ODK is free and open-source software that
helps millions of people collect data quickly,
accurately, offline, and at scale. The
software is in active use in every country in
the world and is supported by a large and
helpful community.
1 Physical Location & Photo
2 Basic Details
3 Infrastructure
4 Services Offered/Facilities & Equipment Available
5 Human Resource Available at Various Levels
6 Other Support Services
7 Health Programmes
8 Quality Control
9 Technology
10 TB Diagnosis & Treatment Services
Data
Collection
Tool

Major Role of Clickhouse - Data Reporting
Inject
Data
ARP Connector
Clickhouse-client driver package/ Pandas
Multiple data
sources

Data Visualization Platform – Dremio/Superset
• Data Lake that can connect to multiple databases, even no SQL
• Join and query original databases at enhanced speeds
• Common SQL Queries
• Through ARP Connector dremio is able to communicate with
clickhouse.(github: nhrr-india)
• it is in experimental mode due to performance issue.
• Easy to Slice, Dice and Visualize data
• 40 Plus charts
• Map based visualizations
Apache Superset
(incubating) is a
modern, enterprise-
ready business
intelligence web
application
Dremio's Data Lake
Engine delivers
lightning fast query
speed and a self-
service semantic layer
operating directly
against your data lake
storage.

Sample Dashboard using Apache Superset
Note: Sample Data used

Recommandé

Building an open data platform with apache icebergAlluxio, Inc.

Architecting Snowflake for High Concurrency and High PerformanceSamanthaBerlant

Using Apache Arrow, Calcite, and Parquet to Build a Relational CacheDremio Corporation

Fast analytics kudu to druidWorapol Alex Pongpech, PhD

Apache Iceberg: An Architectural Look Under the CoversScyllaDB

Architecting a datalakeLaurent Leturgez

Amazon S3 Best Practice and Tuning for Hadoop/Spark in the CloudNoritaka Sekiyama

Snowflake Datawarehouse ArchitecturingIshan Bhawantha Hewanayake

Recommandé

Building an open data platform with apache icebergAlluxio, Inc.

Architecting Snowflake for High Concurrency and High PerformanceSamanthaBerlant

Using Apache Arrow, Calcite, and Parquet to Build a Relational CacheDremio Corporation

Fast analytics kudu to druidWorapol Alex Pongpech, PhD

Apache Iceberg: An Architectural Look Under the CoversScyllaDB

Architecting a datalakeLaurent Leturgez

Amazon S3 Best Practice and Tuning for Hadoop/Spark in the CloudNoritaka Sekiyama

Snowflake Datawarehouse ArchitecturingIshan Bhawantha Hewanayake

Databricks Delta Lake and Its BenefitsDatabricks

Hive User Meeting August 2009 Facebookragho

The Top Five Mistakes Made When Writing Streaming Applications with Mark Grov...Databricks

Zero to Snowflake Presentation Brett VanderPlaats

Iceberg + Alluxio for Fast Data AnalyticsAlluxio, Inc.

Modularized ETL Writing with Apache SparkDatabricks

Cloud dw benchmark using tpd-ds( Snowflake vs Redshift vs EMR Hive )SANG WON PARK

Delta lake and the delta architectureAdam Doyle

Presto Summit 2018 - 09 - Netflix Icebergkbajda

Oracle Database performance tuning using oratopSandesh Rao

OSA Con 2022 - Apache Iceberg_ An Architectural Look Under the Covers - Alex ...Altinity Ltd

What is in a Lucene index?lucenerevolution

Data Lineage with Apache Airflow using Marquez Willy Lulciuc

Elastic Stack IntroductionVikram Shinde

Presto best practices for Cluster admins, data engineers and analystsShubham Tagra

Polyglot persistence @ netflix (CDE Meetup) Roopa Tangirala

Cassandra at eBay - Cassandra Summit 2012Jay Patel

What I learnt: Elastic search & Kibana : introduction, installtion & configur...Rahul K Chauhan

Apache Iceberg - A Table Format for Hige Analytic DatasetsAlluxio, Inc.

Snowflake OverviewSnowflake Computing

Enabling the Future of Healthcare Through Integration and Interoperability: V...Rahul Neel Mani

Improving practitioner decision making capabilities with data and analytics v1Ali Khan

Contenu connexe

Tendances

Databricks Delta Lake and Its BenefitsDatabricks

Hive User Meeting August 2009 Facebookragho

The Top Five Mistakes Made When Writing Streaming Applications with Mark Grov...Databricks

Zero to Snowflake Presentation Brett VanderPlaats

Iceberg + Alluxio for Fast Data AnalyticsAlluxio, Inc.

Modularized ETL Writing with Apache SparkDatabricks

Cloud dw benchmark using tpd-ds( Snowflake vs Redshift vs EMR Hive )SANG WON PARK

Delta lake and the delta architectureAdam Doyle

Presto Summit 2018 - 09 - Netflix Icebergkbajda

Oracle Database performance tuning using oratopSandesh Rao

OSA Con 2022 - Apache Iceberg_ An Architectural Look Under the Covers - Alex ...Altinity Ltd

What is in a Lucene index?lucenerevolution

Data Lineage with Apache Airflow using Marquez Willy Lulciuc

Elastic Stack IntroductionVikram Shinde

Presto best practices for Cluster admins, data engineers and analystsShubham Tagra

Polyglot persistence @ netflix (CDE Meetup) Roopa Tangirala

Cassandra at eBay - Cassandra Summit 2012Jay Patel

What I learnt: Elastic search & Kibana : introduction, installtion & configur...Rahul K Chauhan

Apache Iceberg - A Table Format for Hige Analytic DatasetsAlluxio, Inc.

Snowflake OverviewSnowflake Computing

Tendances (20)

Databricks Delta Lake and Its Benefits

Hive User Meeting August 2009 Facebook

The Top Five Mistakes Made When Writing Streaming Applications with Mark Grov...

Zero to Snowflake Presentation

Iceberg + Alluxio for Fast Data Analytics

Modularized ETL Writing with Apache Spark

Cloud dw benchmark using tpd-ds( Snowflake vs Redshift vs EMR Hive )

Delta lake and the delta architecture

Presto Summit 2018 - 09 - Netflix Iceberg

Oracle Database performance tuning using oratop

OSA Con 2022 - Apache Iceberg_ An Architectural Look Under the Covers - Alex ...

What is in a Lucene index?

Data Lineage with Apache Airflow using Marquez

Elastic Stack Introduction

Presto best practices for Cluster admins, data engineers and analysts

Polyglot persistence @ netflix (CDE Meetup)

Cassandra at eBay - Cassandra Summit 2012

What I learnt: Elastic search & Kibana : introduction, installtion & configur...

Apache Iceberg - A Table Format for Hige Analytic Datasets

Snowflake Overview

Similaire à Use case and integration of ClickHouse with Apache Superset & Dremio

Enabling the Future of Healthcare Through Integration and Interoperability: V...Rahul Neel Mani

Improving practitioner decision making capabilities with data and analytics v1Ali Khan

Brief on Linked Data for U.S. EPA's Chief Data Scientist3 Round Stones

Brief on Linked Data at U.S. EPA to Chief Data ScientistBernadette Hyland-Wood

3 Round Stones Briefing to U.S. EPA's Chief Data Scientist on Open DataBernadette Hyland-Wood

Batter Up! Advanced Sports Analytics with R and StormRevolution Analytics

Building Real-Time Data Pipeline for Diabetes Medication Recommender System U...Databricks

Big Data Tools: A Deep Dive into Essential ToolsFredReynolds2

Thesis blending big data and cloud -epilepsy global data research and inform...Anup Singh

The Data Operating System: Changing the Digital Trajectory of HealthcareHealth Catalyst

Moving Health Care Analytics to Hadoop to Build a Better Predictive ModelDataWorks Summit

The Data Operating System: Changing the Digital Trajectory of HealthcareDale Sanders

Big DataBen Duan

Building an Intelligent Biobank to Power Research Decision-MakingDenodo

SAP HANA in Healthcare: Real-Time Big Data AnalysisSAP Technology

Tackle healthcare interoperability challenges and improve transitions of care v3Perficient, Inc.

Big Data at Geisinger Health System: Big Wins in a Short TimeDataWorks Summit

Getting to What Matters: Accelerating Your Path Through the Big Data Lifecycl...Hortonworks

Shanghai health bureau_big_data_healthcarexband

Big data: Descoberta de conhecimento em ambientes de big data e computação na...Rio Info

Similaire à Use case and integration of ClickHouse with Apache Superset & Dremio (20)

Enabling the Future of Healthcare Through Integration and Interoperability: V...

Improving practitioner decision making capabilities with data and analytics v1

Brief on Linked Data for U.S. EPA's Chief Data Scientist

Brief on Linked Data at U.S. EPA to Chief Data Scientist

3 Round Stones Briefing to U.S. EPA's Chief Data Scientist on Open Data

Batter Up! Advanced Sports Analytics with R and Storm

Building Real-Time Data Pipeline for Diabetes Medication Recommender System U...

Big Data Tools: A Deep Dive into Essential Tools

Thesis blending big data and cloud -epilepsy global data research and inform...

The Data Operating System: Changing the Digital Trajectory of Healthcare

Moving Health Care Analytics to Hadoop to Build a Better Predictive Model

The Data Operating System: Changing the Digital Trajectory of Healthcare

Big Data

Building an Intelligent Biobank to Power Research Decision-Making

SAP HANA in Healthcare: Real-Time Big Data Analysis

Tackle healthcare interoperability challenges and improve transitions of care v3

Big Data at Geisinger Health System: Big Wins in a Short Time

Getting to What Matters: Accelerating Your Path Through the Big Data Lifecycl...

Shanghai health bureau_big_data_healthcare

Big data: Descoberta de conhecimento em ambientes de big data e computação na...

Plus de Altinity Ltd

Building an Analytic Extension to MySQL with ClickHouse and Open Source.pptxAltinity Ltd

Cloud Native ClickHouse at Scale--Using the Altinity Kubernetes Operator-2022...Altinity Ltd

Building an Analytic Extension to MySQL with ClickHouse and Open SourceAltinity Ltd

Fun with ClickHouse Window Functions-2021-08-19.pdfAltinity Ltd

Cloud Native Data Warehouses - Intro to ClickHouse on Kubernetes-2021-07.pdfAltinity Ltd

Building High Performance Apps with Altinity Stable Builds for ClickHouse | A...Altinity Ltd

Application Monitoring using Open Source - VictoriaMetrics & Altinity ClickHo...Altinity Ltd

Own your ClickHouse data with Altinity.Cloud Anywhere-2023-01-17.pdfAltinity Ltd

ClickHouse ReplacingMergeTree in Telecom AppsAltinity Ltd

Adventures with the ClickHouse ReplacingMergeTree EngineAltinity Ltd

Building a Real-Time Analytics Application with Apache Pulsar and Apache PinotAltinity Ltd

Altinity Webinar: Introduction to Altinity.Cloud-Platform for Real-Time Data.pdfAltinity Ltd

OSA Con 2022 - What Data Engineering Can Learn from Frontend Engineering - Pe...Altinity Ltd

OSA Con 2022 - Welcome to OSA CON Version 2022 - Robert Hodges - Altinity.pdfAltinity Ltd

OSA Con 2022 - Using ClickHouse Database to Power Analytics and Customer Enga...Altinity Ltd

OSA Con 2022 - Tips and Tricks to Keep Your Queries under 100ms with ClickHou...Altinity Ltd

OSA Con 2022 - The Open Source Analytic Universe, Version 2022 - Robert Hodge...Altinity Ltd

OSA Con 2022 - Switching Jaeger Distributed Tracing to ClickHouse to Enable A...Altinity Ltd

OSA Con 2022 - Streaming Data Made Easy - Tim Spann & David Kjerrumgaard - St...Altinity Ltd

OSA Con 2022 - State of Open Source Databases - Peter Zaitsev - Percona.pdfAltinity Ltd

Plus de Altinity Ltd (20)

Building an Analytic Extension to MySQL with ClickHouse and Open Source.pptx

Cloud Native ClickHouse at Scale--Using the Altinity Kubernetes Operator-2022...

Building an Analytic Extension to MySQL with ClickHouse and Open Source

Fun with ClickHouse Window Functions-2021-08-19.pdf

Cloud Native Data Warehouses - Intro to ClickHouse on Kubernetes-2021-07.pdf

Building High Performance Apps with Altinity Stable Builds for ClickHouse | A...

Application Monitoring using Open Source - VictoriaMetrics & Altinity ClickHo...

Own your ClickHouse data with Altinity.Cloud Anywhere-2023-01-17.pdf

ClickHouse ReplacingMergeTree in Telecom Apps

Adventures with the ClickHouse ReplacingMergeTree Engine

Building a Real-Time Analytics Application with Apache Pulsar and Apache Pinot

Altinity Webinar: Introduction to Altinity.Cloud-Platform for Real-Time Data.pdf

OSA Con 2022 - What Data Engineering Can Learn from Frontend Engineering - Pe...

OSA Con 2022 - Welcome to OSA CON Version 2022 - Robert Hodges - Altinity.pdf

OSA Con 2022 - Using ClickHouse Database to Power Analytics and Customer Enga...

OSA Con 2022 - Tips and Tricks to Keep Your Queries under 100ms with ClickHou...

OSA Con 2022 - The Open Source Analytic Universe, Version 2022 - Robert Hodge...

OSA Con 2022 - Switching Jaeger Distributed Tracing to ClickHouse to Enable A...

OSA Con 2022 - Streaming Data Made Easy - Tim Spann & David Kjerrumgaard - St...

OSA Con 2022 - State of Open Source Databases - Peter Zaitsev - Percona.pdf

Dernier

VTU technical seminar 8Th Sem on Scikit-learnAmarnathKambale

Pharm-D Biostatistics and Research methodologyAnusha Are

Right Money Management App For Your Financial GoalsJhone kinadey

call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️Delhi Call girls

Exploring the Best Video Editing App.pdfproinshot.com

HR Software Buyers Guide in 2024 - HRSoftware.comFatema Valibhai

A Secure and Reliable Document Management System is Essential.docxComplianceQuest1

%in Bahrain+277-882-255-28 abortion pills for sale in Bahrainmasabamasaba

Unveiling the Tech Salsa of LAMs with Janus in Real-Time ApplicationsAlberto González Trastoy

Software Quality Assurance Interview QuestionsArshad QA

How To Troubleshoot Collaboration Apps for the Modern Connected WorkerThousandEyes

%in kempton park+277-882-255-28 abortion pills for sale in kempton park masabamasaba

Payment Gateway Testing Simplified_ A Step-by-Step Guide for Beginners.pdfkalichargn70th171

Optimizing AI for immediate response in Smart CCTVshikhaohhpro

The Guide to Integrating Generative AI into Unified Continuous Testing Platfo...kalichargn70th171

%in ivory park+277-882-255-28 abortion pills for sale in ivory park masabamasaba

%in Midrand+277-882-255-28 abortion pills for sale in midrandmasabamasaba

8257 interfacing 2 in microprocessor for btech studentsHimanshiGarg82

%in tembisa+277-882-255-28 abortion pills for sale in tembisamasabamasaba

TECUNIQUE: Success Stories: IT Service providermohitmore19

Dernier (20)

VTU technical seminar 8Th Sem on Scikit-learn

Pharm-D Biostatistics and Research methodology

Right Money Management App For Your Financial Goals

call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️

Exploring the Best Video Editing App.pdf

HR Software Buyers Guide in 2024 - HRSoftware.com

A Secure and Reliable Document Management System is Essential.docx

%in Bahrain+277-882-255-28 abortion pills for sale in Bahrain

Unveiling the Tech Salsa of LAMs with Janus in Real-Time Applications

Software Quality Assurance Interview Questions

How To Troubleshoot Collaboration Apps for the Modern Connected Worker

%in kempton park+277-882-255-28 abortion pills for sale in kempton park

Payment Gateway Testing Simplified_ A Step-by-Step Guide for Beginners.pdf

Optimizing AI for immediate response in Smart CCTV

The Guide to Integrating Generative AI into Unified Continuous Testing Platfo...

%in ivory park+277-882-255-28 abortion pills for sale in ivory park

%in Midrand+277-882-255-28 abortion pills for sale in midrand

8257 interfacing 2 in microprocessor for btech students

%in tembisa+277-882-255-28 abortion pills for sale in tembisa

TECUNIQUE: Success Stories: IT Service provider

Use case and integration of ClickHouse with Apache Superset & Dremio

1. Use case and Integration of Clickhouse with Apache Superset & Dremio Rahul Sharma, India Lead Consultant – GIS Solution / Former Research Scientist/Mentor / Speaker ISRO/ MoH&FW / BMGF / MoWR rahul.rrsc@gmail.com, skype: @rahulsmartgis https://www.linkedin.com/in/rahul-sharma-a3595a14/

2. Use Case: Health Census Data 4000 + Variables 700+ Tables ( 1 TB Size) ODK is free and open-source software that helps millions of people collect data quickly, accurately, offline, and at scale. The software is in active use in every country in the world and is supported by a large and helpful community. 1 Physical Location & Photo 2 Basic Details 3 Infrastructure 4 Services Offered/Facilities & Equipment Available 5 Human Resource Available at Various Levels 6 Other Support Services 7 Health Programmes 8 Quality Control 9 Technology 10 TB Diagnosis & Treatment Services Data Collection Tool

3. Major Role of Clickhouse - Data Reporting Inject Data ARP Connector Clickhouse-client driver package/ Pandas Multiple data sources

4. Data Visualization Platform – Dremio/Superset • Data Lake that can connect to multiple databases, even no SQL • Join and query original databases at enhanced speeds • Common SQL Queries • Through ARP Connector dremio is able to communicate with clickhouse.(github: nhrr-india) • it is in experimental mode due to performance issue. • Easy to Slice, Dice and Visualize data • 40 Plus charts • Map based visualizations Apache Superset (incubating) is a modern, enterprise- ready business intelligence web application Dremio's Data Lake Engine delivers lightning fast query speed and a self- service semantic layer operating directly against your data lake storage.

5. Sample Dashboard using Apache Superset Note: Sample Data used

6. 6 Thanks! Q&A