SlideShare une entreprise Scribd logo
1  sur  10
Télécharger pour lire hors ligne
On HBase Integration Testing
  Andrew Purtell, Hadoop Group at Trend Micro
             apurtell@apache.org
Introducing Apache Bigtop




                            (and other custom
                            distros)
Introducing Apache Bigtop
A new project in incubation with committers from
 Cloudera, Ubuntu, and others

The goal is to do for Hadoop what Debian did for
  Linux

Build and packaging infrastructure
  Makefile based (yes!)
  RPM and DEB packaging targets

Deployment infrastructure
  Puppet

Integration test infrastructure
  iTest
Introducing Apache Bigtop
A new project in incubation with committers from
 Cloudera, Ubuntu, and others

The goal is to do for Hadoop what Debian did for
  Linux

Build and packaging infrastructure
  Makefile based (yes!)
  RPM and DEB packaging targets

Deployment infrastructure
  Puppet

Integration test infrastructure
  iTest
                   This seems like it could be useful
iTest
A framework for integration testing

Provides a high-level abstraction for querying and
  driving OS level packaging systems during full
  system integration tests
  Install package
  Run tests
  Check results
  Remove package
  Start/stop SysV services

Transparent shell-outs written in Java and Groovy
  Convenient to write tests that are scripting (Groovy) layered
    over command line invocations ("hadoop jar ...")

Helper functions for unpacking data from JAR files,
 filesystem paths, classpath, etc.
HBASE-6201 (and friends)
From Enis Söztutar

Add ability to run all integration/system tests locally or
 on a given cluster

  With 'mvn verify' though Stack suggests this hasn't been
    possible to date using Maven machinery alone

  Or, by importing and using iTest (and exposing this also to
    BigTop) – I personally think this is promising

                                         MiniCluster


        'mvn verify'       iTest


                                         Actual cluster
HBASE-6201 (and friends)
Convert candidate unit tests that run on a full
 MiniCluster into integration tests that can run on a
 real cluster

  TestAcidGuarantees
  TestAtomicOperation
  TestRegionBalancing
  TestFullLogReconstruction
  TestMasterFailover
  TestImportExport
  TestRollingRestart
  TestMultiVersions
  TestKeepDeletes
  TestFromClientSide
  TestSplitTransactionOnCluster
  ... and more
HBASE-6201 (and friends)
Add more tests, especially, long running ingestion
 tests and chaos monkey

  LoadTestTool

  YCSB

  GoraCI

  Resurrect Gremlins? https://github.com/toddlipcon/gremlins
Going beyond HBASE-6201
Add full system application test cases

  I have a Heritrix / hbase-writer based webtable thingy ?

  JGit backed by HBase https://github.com/apurtell/jgit-hbase ?

  Canned queries for Titan (http://thinkaurelius.github.com/titan/ ) or
    something that uses Apache Giraph's HBase storage
    adapter ?

  Canned data and scripts for Apache Pig and Hive ?

Add Kerberos setup and usage helper functions to
 iTest
  so then we can finesse integration testing with security
    enabled at the Hadoop, HBase, and/or ZooKeeper layers
End. Discuss.

Contenu connexe

Tendances

OpenStack Tempest and REST API testing
OpenStack Tempest and REST API testingOpenStack Tempest and REST API testing
OpenStack Tempest and REST API testingopenstackindia
 
OpenShift, Docker, Kubernetes: The next generation of PaaS
OpenShift, Docker, Kubernetes: The next generation of PaaSOpenShift, Docker, Kubernetes: The next generation of PaaS
OpenShift, Docker, Kubernetes: The next generation of PaaSGraham Dumpleton
 
How to Achieve Canary Deployment on Kubernetes
How to Achieve Canary Deployment on KubernetesHow to Achieve Canary Deployment on Kubernetes
How to Achieve Canary Deployment on KubernetesHanLing Shen
 
Make stateful apps in Kubernetes a no brainer with Pure Storage and GitOps
Make stateful apps in Kubernetes a no brainer with Pure Storage and GitOpsMake stateful apps in Kubernetes a no brainer with Pure Storage and GitOps
Make stateful apps in Kubernetes a no brainer with Pure Storage and GitOpsWeaveworks
 
Tempest scenariotests 20140512
Tempest scenariotests 20140512Tempest scenariotests 20140512
Tempest scenariotests 20140512Masayuki Igawa
 
Travis CI: Fun and easy CI for your Plone packages
Travis CI: Fun and easy CI for your Plone packagesTravis CI: Fun and easy CI for your Plone packages
Travis CI: Fun and easy CI for your Plone packagesNejc Zupan
 
Использование Docker в CI / Александр Акбашев (HERE Technologies)
Использование Docker в CI / Александр Акбашев (HERE Technologies)Использование Docker в CI / Александр Акбашев (HERE Technologies)
Использование Docker в CI / Александр Акбашев (HERE Technologies)Ontico
 
DevStack: Learn OpenStack by Running OpenStack
DevStack: Learn OpenStack by Running OpenStackDevStack: Learn OpenStack by Running OpenStack
DevStack: Learn OpenStack by Running OpenStackEverett Toews
 
OpenStack DevStack Configuration localrc local.conf Tutorial
OpenStack DevStack Configuration localrc local.conf TutorialOpenStack DevStack Configuration localrc local.conf Tutorial
OpenStack DevStack Configuration localrc local.conf TutorialSaju Madhavan
 
Tectonic Summit 2016: The Origins of Kubernetes
Tectonic Summit 2016: The Origins of KubernetesTectonic Summit 2016: The Origins of Kubernetes
Tectonic Summit 2016: The Origins of KubernetesCoreOS
 
Dockerin10mins
Dockerin10minsDockerin10mins
Dockerin10minsDawood M.S
 
머신 중심에서 애플리케이션 중심으로 불변의 인프라스트럭처 개념 이해
머신 중심에서 애플리케이션 중심으로 불변의 인프라스트럭처 개념 이해머신 중심에서 애플리케이션 중심으로 불변의 인프라스트럭처 개념 이해
머신 중심에서 애플리케이션 중심으로 불변의 인프라스트럭처 개념 이해Opennaru, inc.
 
Whose Job Is It Anyway? Kubernetes, CRI, & Container Runtimes
Whose Job Is It Anyway? Kubernetes, CRI, & Container RuntimesWhose Job Is It Anyway? Kubernetes, CRI, & Container Runtimes
Whose Job Is It Anyway? Kubernetes, CRI, & Container RuntimesPhil Estes
 
Introduction to CircleCI
Introduction to CircleCIIntroduction to CircleCI
Introduction to CircleCIHungWei Chiu
 
Immutable infrastructure 介紹與實做:以 kolla 為例
Immutable infrastructure 介紹與實做:以 kolla 為例Immutable infrastructure 介紹與實做:以 kolla 為例
Immutable infrastructure 介紹與實做:以 kolla 為例kao kuo-tung
 
제4회 한국IBM과 함께하는 난공불락 오픈소스 인프라 세미나-Ranchers
제4회 한국IBM과 함께하는 난공불락 오픈소스 인프라 세미나-Ranchers제4회 한국IBM과 함께하는 난공불락 오픈소스 인프라 세미나-Ranchers
제4회 한국IBM과 함께하는 난공불락 오픈소스 인프라 세미나-RanchersTommy Lee
 
Openstack Rally - Benchmark as a Service. Openstack Meetup India. Ananth/Rahul.
Openstack Rally - Benchmark as a Service. Openstack Meetup India. Ananth/Rahul.Openstack Rally - Benchmark as a Service. Openstack Meetup India. Ananth/Rahul.
Openstack Rally - Benchmark as a Service. Openstack Meetup India. Ananth/Rahul.Rahul Krishna Upadhyaya
 

Tendances (20)

OpenStack Tempest and REST API testing
OpenStack Tempest and REST API testingOpenStack Tempest and REST API testing
OpenStack Tempest and REST API testing
 
OpenShift, Docker, Kubernetes: The next generation of PaaS
OpenShift, Docker, Kubernetes: The next generation of PaaSOpenShift, Docker, Kubernetes: The next generation of PaaS
OpenShift, Docker, Kubernetes: The next generation of PaaS
 
How to Achieve Canary Deployment on Kubernetes
How to Achieve Canary Deployment on KubernetesHow to Achieve Canary Deployment on Kubernetes
How to Achieve Canary Deployment on Kubernetes
 
Make stateful apps in Kubernetes a no brainer with Pure Storage and GitOps
Make stateful apps in Kubernetes a no brainer with Pure Storage and GitOpsMake stateful apps in Kubernetes a no brainer with Pure Storage and GitOps
Make stateful apps in Kubernetes a no brainer with Pure Storage and GitOps
 
Releasing a Distribution in the Age of DevOps.
Releasing a Distribution in the Age of DevOps. Releasing a Distribution in the Age of DevOps.
Releasing a Distribution in the Age of DevOps.
 
Tempest scenariotests 20140512
Tempest scenariotests 20140512Tempest scenariotests 20140512
Tempest scenariotests 20140512
 
Travis CI: Fun and easy CI for your Plone packages
Travis CI: Fun and easy CI for your Plone packagesTravis CI: Fun and easy CI for your Plone packages
Travis CI: Fun and easy CI for your Plone packages
 
Использование Docker в CI / Александр Акбашев (HERE Technologies)
Использование Docker в CI / Александр Акбашев (HERE Technologies)Использование Docker в CI / Александр Акбашев (HERE Technologies)
Использование Docker в CI / Александр Акбашев (HERE Technologies)
 
DevStack: Learn OpenStack by Running OpenStack
DevStack: Learn OpenStack by Running OpenStackDevStack: Learn OpenStack by Running OpenStack
DevStack: Learn OpenStack by Running OpenStack
 
Arquillian
ArquillianArquillian
Arquillian
 
OpenStack DevStack Configuration localrc local.conf Tutorial
OpenStack DevStack Configuration localrc local.conf TutorialOpenStack DevStack Configuration localrc local.conf Tutorial
OpenStack DevStack Configuration localrc local.conf Tutorial
 
Tectonic Summit 2016: The Origins of Kubernetes
Tectonic Summit 2016: The Origins of KubernetesTectonic Summit 2016: The Origins of Kubernetes
Tectonic Summit 2016: The Origins of Kubernetes
 
Dockerin10mins
Dockerin10minsDockerin10mins
Dockerin10mins
 
머신 중심에서 애플리케이션 중심으로 불변의 인프라스트럭처 개념 이해
머신 중심에서 애플리케이션 중심으로 불변의 인프라스트럭처 개념 이해머신 중심에서 애플리케이션 중심으로 불변의 인프라스트럭처 개념 이해
머신 중심에서 애플리케이션 중심으로 불변의 인프라스트럭처 개념 이해
 
Whose Job Is It Anyway? Kubernetes, CRI, & Container Runtimes
Whose Job Is It Anyway? Kubernetes, CRI, & Container RuntimesWhose Job Is It Anyway? Kubernetes, CRI, & Container Runtimes
Whose Job Is It Anyway? Kubernetes, CRI, & Container Runtimes
 
Introduction to CircleCI
Introduction to CircleCIIntroduction to CircleCI
Introduction to CircleCI
 
Immutable infrastructure 介紹與實做:以 kolla 為例
Immutable infrastructure 介紹與實做:以 kolla 為例Immutable infrastructure 介紹與實做:以 kolla 為例
Immutable infrastructure 介紹與實做:以 kolla 為例
 
제4회 한국IBM과 함께하는 난공불락 오픈소스 인프라 세미나-Ranchers
제4회 한국IBM과 함께하는 난공불락 오픈소스 인프라 세미나-Ranchers제4회 한국IBM과 함께하는 난공불락 오픈소스 인프라 세미나-Ranchers
제4회 한국IBM과 함께하는 난공불락 오픈소스 인프라 세미나-Ranchers
 
Helm intro
Helm introHelm intro
Helm intro
 
Openstack Rally - Benchmark as a Service. Openstack Meetup India. Ananth/Rahul.
Openstack Rally - Benchmark as a Service. Openstack Meetup India. Ananth/Rahul.Openstack Rally - Benchmark as a Service. Openstack Meetup India. Ananth/Rahul.
Openstack Rally - Benchmark as a Service. Openstack Meetup India. Ananth/Rahul.
 

Similaire à On HBase Integration Testing

Hortonworks HBase Meetup Presentation
Hortonworks HBase Meetup PresentationHortonworks HBase Meetup Presentation
Hortonworks HBase Meetup PresentationHortonworks
 
Testing the Enterprise Layers - the A, B, C's of Integration Testing - Aslak ...
Testing the Enterprise Layers - the A, B, C's of Integration Testing - Aslak ...Testing the Enterprise Layers - the A, B, C's of Integration Testing - Aslak ...
Testing the Enterprise Layers - the A, B, C's of Integration Testing - Aslak ...JAXLondon2014
 
Deploying Hadoop-based Bigdata Environments
Deploying Hadoop-based Bigdata Environments Deploying Hadoop-based Bigdata Environments
Deploying Hadoop-based Bigdata Environments buildacloud
 
Deploying Hadoop-Based Bigdata Environments
Deploying Hadoop-Based Bigdata EnvironmentsDeploying Hadoop-Based Bigdata Environments
Deploying Hadoop-Based Bigdata EnvironmentsPuppet
 
A DevOps guide to Kubernetes
A DevOps guide to KubernetesA DevOps guide to Kubernetes
A DevOps guide to KubernetesPaul Czarkowski
 
Testing Cassandra Guarantees under Diverse Failure Modes with Jepsen
Testing Cassandra Guarantees under Diverse Failure Modes with JepsenTesting Cassandra Guarantees under Diverse Failure Modes with Jepsen
Testing Cassandra Guarantees under Diverse Failure Modes with Jepsenjkni
 
DataStax: Testing Cassandra Guarantees Under Diverse Failure Modes With Jepsen
DataStax: Testing Cassandra Guarantees Under Diverse Failure Modes With JepsenDataStax: Testing Cassandra Guarantees Under Diverse Failure Modes With Jepsen
DataStax: Testing Cassandra Guarantees Under Diverse Failure Modes With JepsenDataStax Academy
 
Docker Tips And Tricks at the Docker Beijing Meetup
Docker Tips And Tricks at the Docker Beijing MeetupDocker Tips And Tricks at the Docker Beijing Meetup
Docker Tips And Tricks at the Docker Beijing MeetupJérôme Petazzoni
 
Migrating to a Bazel-based CI System: 6 Learnings - Or Shachar
Migrating to a Bazel-based CI System: 6 Learnings - Or ShacharMigrating to a Bazel-based CI System: 6 Learnings - Or Shachar
Migrating to a Bazel-based CI System: 6 Learnings - Or ShacharWix Engineering
 
Testing Java Web Apps With Selenium
Testing Java Web Apps With SeleniumTesting Java Web Apps With Selenium
Testing Java Web Apps With SeleniumMarakana Inc.
 
Building JBoss AS 7 for Fedora
Building JBoss AS 7 for FedoraBuilding JBoss AS 7 for Fedora
Building JBoss AS 7 for Fedorawolfc71
 
Kubernetes or OpenShift - choosing your container platform for Dev and Ops
Kubernetes or OpenShift - choosing your container platform for Dev and OpsKubernetes or OpenShift - choosing your container platform for Dev and Ops
Kubernetes or OpenShift - choosing your container platform for Dev and OpsTomasz Cholewa
 
stackconf 2020 | Enterprise CI/CD Integration Testing Environments Done Right...
stackconf 2020 | Enterprise CI/CD Integration Testing Environments Done Right...stackconf 2020 | Enterprise CI/CD Integration Testing Environments Done Right...
stackconf 2020 | Enterprise CI/CD Integration Testing Environments Done Right...NETWAYS
 
Corwin on Containers
Corwin on ContainersCorwin on Containers
Corwin on ContainersCorwin Brown
 
Apache Bigtop: a crash course in deploying a Hadoop bigdata management platform
Apache Bigtop: a crash course in deploying a Hadoop bigdata management platformApache Bigtop: a crash course in deploying a Hadoop bigdata management platform
Apache Bigtop: a crash course in deploying a Hadoop bigdata management platformrhatr
 
Igalia Focus and Goals 2020 (2019 WebKit Contributors Meeting)
Igalia Focus and Goals 2020 (2019 WebKit Contributors Meeting)Igalia Focus and Goals 2020 (2019 WebKit Contributors Meeting)
Igalia Focus and Goals 2020 (2019 WebKit Contributors Meeting)Igalia
 
Django deployment with PaaS
Django deployment with PaaSDjango deployment with PaaS
Django deployment with PaaSAppsembler
 
Kubernetes for the PHP developer
Kubernetes for the PHP developerKubernetes for the PHP developer
Kubernetes for the PHP developerPaul Czarkowski
 

Similaire à On HBase Integration Testing (20)

Hortonworks HBase Meetup Presentation
Hortonworks HBase Meetup PresentationHortonworks HBase Meetup Presentation
Hortonworks HBase Meetup Presentation
 
Testing the Enterprise Layers - the A, B, C's of Integration Testing - Aslak ...
Testing the Enterprise Layers - the A, B, C's of Integration Testing - Aslak ...Testing the Enterprise Layers - the A, B, C's of Integration Testing - Aslak ...
Testing the Enterprise Layers - the A, B, C's of Integration Testing - Aslak ...
 
Deploying Hadoop-based Bigdata Environments
Deploying Hadoop-based Bigdata Environments Deploying Hadoop-based Bigdata Environments
Deploying Hadoop-based Bigdata Environments
 
Deploying Hadoop-Based Bigdata Environments
Deploying Hadoop-Based Bigdata EnvironmentsDeploying Hadoop-Based Bigdata Environments
Deploying Hadoop-Based Bigdata Environments
 
A DevOps guide to Kubernetes
A DevOps guide to KubernetesA DevOps guide to Kubernetes
A DevOps guide to Kubernetes
 
Testing Cassandra Guarantees under Diverse Failure Modes with Jepsen
Testing Cassandra Guarantees under Diverse Failure Modes with JepsenTesting Cassandra Guarantees under Diverse Failure Modes with Jepsen
Testing Cassandra Guarantees under Diverse Failure Modes with Jepsen
 
DataStax: Testing Cassandra Guarantees Under Diverse Failure Modes With Jepsen
DataStax: Testing Cassandra Guarantees Under Diverse Failure Modes With JepsenDataStax: Testing Cassandra Guarantees Under Diverse Failure Modes With Jepsen
DataStax: Testing Cassandra Guarantees Under Diverse Failure Modes With Jepsen
 
Docker Tips And Tricks at the Docker Beijing Meetup
Docker Tips And Tricks at the Docker Beijing MeetupDocker Tips And Tricks at the Docker Beijing Meetup
Docker Tips And Tricks at the Docker Beijing Meetup
 
2012 09-08-josug-jeff
2012 09-08-josug-jeff2012 09-08-josug-jeff
2012 09-08-josug-jeff
 
Migrating to a Bazel-based CI System: 6 Learnings - Or Shachar
Migrating to a Bazel-based CI System: 6 Learnings - Or ShacharMigrating to a Bazel-based CI System: 6 Learnings - Or Shachar
Migrating to a Bazel-based CI System: 6 Learnings - Or Shachar
 
PaaSing Your Code Around
PaaSing Your Code AroundPaaSing Your Code Around
PaaSing Your Code Around
 
Testing Java Web Apps With Selenium
Testing Java Web Apps With SeleniumTesting Java Web Apps With Selenium
Testing Java Web Apps With Selenium
 
Building JBoss AS 7 for Fedora
Building JBoss AS 7 for FedoraBuilding JBoss AS 7 for Fedora
Building JBoss AS 7 for Fedora
 
Kubernetes or OpenShift - choosing your container platform for Dev and Ops
Kubernetes or OpenShift - choosing your container platform for Dev and OpsKubernetes or OpenShift - choosing your container platform for Dev and Ops
Kubernetes or OpenShift - choosing your container platform for Dev and Ops
 
stackconf 2020 | Enterprise CI/CD Integration Testing Environments Done Right...
stackconf 2020 | Enterprise CI/CD Integration Testing Environments Done Right...stackconf 2020 | Enterprise CI/CD Integration Testing Environments Done Right...
stackconf 2020 | Enterprise CI/CD Integration Testing Environments Done Right...
 
Corwin on Containers
Corwin on ContainersCorwin on Containers
Corwin on Containers
 
Apache Bigtop: a crash course in deploying a Hadoop bigdata management platform
Apache Bigtop: a crash course in deploying a Hadoop bigdata management platformApache Bigtop: a crash course in deploying a Hadoop bigdata management platform
Apache Bigtop: a crash course in deploying a Hadoop bigdata management platform
 
Igalia Focus and Goals 2020 (2019 WebKit Contributors Meeting)
Igalia Focus and Goals 2020 (2019 WebKit Contributors Meeting)Igalia Focus and Goals 2020 (2019 WebKit Contributors Meeting)
Igalia Focus and Goals 2020 (2019 WebKit Contributors Meeting)
 
Django deployment with PaaS
Django deployment with PaaSDjango deployment with PaaS
Django deployment with PaaS
 
Kubernetes for the PHP developer
Kubernetes for the PHP developerKubernetes for the PHP developer
Kubernetes for the PHP developer
 

Dernier

Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfDaniel Santiago Silva Capera
 
Spring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdfSpring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdfAnna Loughnan Colquhoun
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...Aggregage
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataSafe Software
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 
RAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AIRAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AIUdaiappa Ramachandran
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UbiTrack UK
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfDianaGray10
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...DianaGray10
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IES VE
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostMatt Ray
 
Introduction to Quantum Computing
Introduction to Quantum ComputingIntroduction to Quantum Computing
Introduction to Quantum ComputingGDSC PJATK
 

Dernier (20)

Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
 
Spring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdfSpring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdf
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 
RAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AIRAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AI
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
 
Introduction to Quantum Computing
Introduction to Quantum ComputingIntroduction to Quantum Computing
Introduction to Quantum Computing
 

On HBase Integration Testing

  • 1. On HBase Integration Testing Andrew Purtell, Hadoop Group at Trend Micro apurtell@apache.org
  • 2. Introducing Apache Bigtop (and other custom distros)
  • 3. Introducing Apache Bigtop A new project in incubation with committers from Cloudera, Ubuntu, and others The goal is to do for Hadoop what Debian did for Linux Build and packaging infrastructure Makefile based (yes!) RPM and DEB packaging targets Deployment infrastructure Puppet Integration test infrastructure iTest
  • 4. Introducing Apache Bigtop A new project in incubation with committers from Cloudera, Ubuntu, and others The goal is to do for Hadoop what Debian did for Linux Build and packaging infrastructure Makefile based (yes!) RPM and DEB packaging targets Deployment infrastructure Puppet Integration test infrastructure iTest This seems like it could be useful
  • 5. iTest A framework for integration testing Provides a high-level abstraction for querying and driving OS level packaging systems during full system integration tests Install package Run tests Check results Remove package Start/stop SysV services Transparent shell-outs written in Java and Groovy Convenient to write tests that are scripting (Groovy) layered over command line invocations ("hadoop jar ...") Helper functions for unpacking data from JAR files, filesystem paths, classpath, etc.
  • 6. HBASE-6201 (and friends) From Enis Söztutar Add ability to run all integration/system tests locally or on a given cluster With 'mvn verify' though Stack suggests this hasn't been possible to date using Maven machinery alone Or, by importing and using iTest (and exposing this also to BigTop) – I personally think this is promising MiniCluster 'mvn verify' iTest Actual cluster
  • 7. HBASE-6201 (and friends) Convert candidate unit tests that run on a full MiniCluster into integration tests that can run on a real cluster TestAcidGuarantees TestAtomicOperation TestRegionBalancing TestFullLogReconstruction TestMasterFailover TestImportExport TestRollingRestart TestMultiVersions TestKeepDeletes TestFromClientSide TestSplitTransactionOnCluster ... and more
  • 8. HBASE-6201 (and friends) Add more tests, especially, long running ingestion tests and chaos monkey LoadTestTool YCSB GoraCI Resurrect Gremlins? https://github.com/toddlipcon/gremlins
  • 9. Going beyond HBASE-6201 Add full system application test cases I have a Heritrix / hbase-writer based webtable thingy ? JGit backed by HBase https://github.com/apurtell/jgit-hbase ? Canned queries for Titan (http://thinkaurelius.github.com/titan/ ) or something that uses Apache Giraph's HBase storage adapter ? Canned data and scripts for Apache Pig and Hive ? Add Kerberos setup and usage helper functions to iTest so then we can finesse integration testing with security enabled at the Hadoop, HBase, and/or ZooKeeper layers