ConcourseCI love Minio

•

2 likes•1,844 views

Due to some design decision with ConcourseCI there is no sharing between JOB, so Object storage is a good way of achivig this. Minio is an OSS Object storage with a great coverage of S3 api.

Technology

30/11/2016
Etourneau Gwenn
@The_shinji62
ConcourseCI
Minio

About me
Etourneau Gwenn
Sr Solution Architect
Bef.: Platform Architect
https://github.com/shinji62
https://twitter.com/the_shinji62

Pivotal
• Spring Framework
• RabbitMQ
• Concourse
• Cloud Foundry
• Pivotal Tracker
• Gemﬁre

Agenda
• Concourse
• Minio
• Minio for Concourse
• Q&A

ConcourseCI
• Do not allow snowﬂake
• Isolation between job
• Non share volume / storage / cache

Jobs 1 Jobs 2
ﬁle_out ﬁle_out
X
Concourse

Concourse
• Concourse support S3
• S3 on premise is not easy
• RiackCS, leoFS, Swift are hard
• S3 compatibility so so…

Minio
• Open Source
• Object Storage written in Go
• Distributed ? Not really more like replication
• S3 compatible
• Data protection using Bit rots , Erasure Code
• Single go binary easy to start

Minio limitation
• Do not scale-out or scale back
• Not multi-tenant, user are the same for the same pool
• number of nodes =>4 and <= 16
• Still early phase (>5 commit /day on master)

Minio and Concourse
• Support S3 native resource from concourse
Jobs 1
Jobs 2
ﬁle_out
ﬁle_out
ﬁle_out

Minio bosh release
• Deploy a pool of distributed minio in minutes
• On any IaaS (supported by bosh)

Minio bosh release
• Use bosh v2 links
• Easy to spin-off
• well, you need to learn bosh
a little….

Demo
• Deploy Minio Cluster
• Set Concourse Pipeline

Resources
https://github.com/shinji62/minio-dist-boshrelease
https://github.com/cloudfoundry-community/minio-boshrelease
• Minio bosh release
• Minio
• https://github.com/minio/minio
• https://minio.io/

Resources
• ConcourseCI
• https://concourse.ci

Slack
https://concourseci.slack.com/
#general

What's hot

Offensive Security basics part 1wharpreet

Data ops in practiceLars Albertsson

[Easy] LastPass Password Manager TutorialAna Uy

Super Easy Memory ForensicsIIJ

Operating System ForensicsArunJS5

Data and AI summit: data pipelines observability with open lineageJulien Le Dem

Parrot Drones HijackingPriyanka Aash

Snowflake: The Good, the Bad, and the UglyTyler Wishnoff

Threat Hunting for Command and Control ActivitySqrrl

Finding attacks with these 6 eventsMichael Gough

NTXISSACSC4 - Layered Security / Defense in DepthNorth Texas Chapter of the ISSA

Lecture 9 and 10 comp forensics 09 10-18 file systemAlchemist095

nullcon 2011 - Penetration Testing a Biometric Systemn|u - The Open Security Community

iOS Forensics Tjylen Veselyj

Data Privatisation, Data Anonymisation, Data Pseudonymisation and Differentia...Alan McSweeney

Data Pipline Observability meetup Omid Vahdaty

Matomo External Dashboards & Data Visualisation.pdfMichael Weber

Modernize Solutions with SharePoint & the Power PlatformJonathan Schultz

Threat Hunting Procedures and Measurement MatriceVishal Kumar

Data management principlesFiddy Prasetiya

What's hot (20)

Offensive Security basics part 1

Data ops in practice

[Easy] LastPass Password Manager Tutorial

Super Easy Memory Forensics

Operating System Forensics

Data and AI summit: data pipelines observability with open lineage

Parrot Drones Hijacking

Snowflake: The Good, the Bad, and the Ugly

Threat Hunting for Command and Control Activity

Finding attacks with these 6 events

NTXISSACSC4 - Layered Security / Defense in Depth

Lecture 9 and 10 comp forensics 09 10-18 file system

nullcon 2011 - Penetration Testing a Biometric System

iOS Forensics

Data Privatisation, Data Anonymisation, Data Pseudonymisation and Differentia...

Data Pipline Observability meetup

Matomo External Dashboards & Data Visualisation.pdf

Modernize Solutions with SharePoint & the Power Platform

Threat Hunting Procedures and Measurement Matrice

Data management principles

Viewers also liked

Minio hearts GoGarima Kapoor

Monitor Cloud Foundry and Bosh with PrometheusGwenn Etourneau

Cloud Foundry Meetup Tokyo #1 Route serviceGwenn Etourneau

Route service-pcf-techmeetupGwenn Etourneau

Bosh 2-0-reloadedGwenn Etourneau

Concourse updatesGwenn Etourneau

Team Support in Concourse CI 2.0 #concourse_tokyoToshiaki Maki

Concourse and DatabaseGwenn Etourneau

Ceph and Storage Management with openATTIC, Ceph Tech Talks 2016-06-23Lenz Grimmer

Lattice yapc-slideshareGwenn Etourneau

Demo Pivotal Circle Of CodeGwenn Etourneau

ConcourseCi DockerimageGwenn Etourneau

[RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB ProjectRakuten Group, Inc.

从玩具到生产 - Cloud Foundry 上海峰会2015年Duncan Johnston-Watt

ConcourseCi overviewGwenn Etourneau

ECSとSpotFleetで新規ビジネスのトライアルYu Sudo

Concourse CI Meetup DemoToshiaki Maki

Pivotal Cloud Foundry: Building a diverse geo-architecture for Cloud Native A...DataStax Academy

Database Security for PCI DSSOhyama Masanori

Cloud Foundry Technical Overviewcornelia davis

Viewers also liked (20)

Minio hearts Go

Monitor Cloud Foundry and Bosh with Prometheus

Cloud Foundry Meetup Tokyo #1 Route service

Route service-pcf-techmeetup

Bosh 2-0-reloaded

Concourse updates

Team Support in Concourse CI 2.0 #concourse_tokyo

Concourse and Database

Ceph and Storage Management with openATTIC, Ceph Tech Talks 2016-06-23

Lattice yapc-slideshare

Demo Pivotal Circle Of Code

ConcourseCi Dockerimage

[RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project

从玩具到生产 - Cloud Foundry 上海峰会2015年

ConcourseCi overview

ECSとSpotFleetで新規ビジネスのトライアル

Concourse CI Meetup Demo

Pivotal Cloud Foundry: Building a diverse geo-architecture for Cloud Native A...

Database Security for PCI DSS

Cloud Foundry Technical Overview

Similar to ConcourseCI love Minio

Prototyping like it is 2022 Michael Yagudaev

Docker Meetup Tokyo #23 - Zenko Open Source Multi-Cloud Data Controller - Lau...Laure Vergeron

Engage 2020: Hello are you listening, There is stream for everythingFrank van der Linden

Managing your user data with Sitecore xDBRuud van Falier

Cloud anti-patternsMallika Iyer

How to start WebGL easily?誠人堀口

GitYervand Aghababyan

APIs in production - we built it, can we fix it?Martin Gutenbrunner

Lessons in Open Source from the MongooseJS ODMValeri Karpov

February 13th, 2014 - Unicon IAM Webinar UpdateMisagh Moayyed

Kotti 紹介Yahoo!デベロッパーネットワーク

2014 Q4 IAM Open Source Support Program UpdateJohn Gasper

Stackato v6Jonas Brømsø

Qt Creator Bootcampaccount inactive

S2GX 2012 - Spring Projects InfrastructureGunnar Hillert

Spring Projects InfrastructureRoy Clarkson

Ruby and Distributed Storage SystemsSATOSHI TAGOMORI

Tooling for the JavaScript Eramartinlippert

Ready? Bootstrap! Go! (CFUG Belgium 24 04-2012)Guust Nieuwenhuis

Similar to ConcourseCI love Minio (20)

Prototyping like it is 2022

Docker Meetup Tokyo #23 - Zenko Open Source Multi-Cloud Data Controller - Lau...

Engage 2020: Hello are you listening, There is stream for everything

Managing your user data with Sitecore xDB

Cloud anti-patterns

How to start WebGL easily?

Git

APIs in production - we built it, can we fix it?

Lessons in Open Source from the MongooseJS ODM

February 13th, 2014 - Unicon IAM Webinar Update

Kotti 紹介

2014 Q4 IAM Open Source Support Program Update

Stackato v6

Qt Creator Bootcamp

S2GX 2012 - Spring Projects Infrastructure

Spring Projects Infrastructure

Ruby and Distributed Storage Systems

Tooling for the JavaScript Era

Ready? Bootstrap! Go! (CFUG Belgium 24 04-2012)

Recently uploaded

The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdfEnterprise Knowledge

From Event to Action: Accelerate Your Decision Making with Real-Time AutomationSafe Software

🐬 The future of MySQL is Postgres 🐘RTylerCroy

Powerful Google developer tools for immediate impact! (2023-24 C)wesley chun

08448380779 Call Girls In Diplomatic Enclave Women Seeking MenDelhi Call girls

08448380779 Call Girls In Civil Lines Women Seeking MenDelhi Call girls

Breaking the Kubernetes Kill Chain: Host Path MountPuma Security, LLC

The 7 Things I Know About Cyber Security After 25 Years | April 2024Rafal Los

Factors to Consider When Choosing Accounts Payable Services Providers.pptxKatpro Technologies

TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc

Tata AIG General Insurance Company - Insurer Innovation Award 2024The Digital Insurer

How to convert PDF to text with Nanonetsnaman860154

IAC 2024 - IA Fast Track to Search Focused AI SolutionsEnterprise Knowledge

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024The Digital Insurer

Scaling API-first – The story of a global engineering organizationRadu Cotescu

08448380779 Call Girls In Friends Colony Women Seeking MenDelhi Call girls

A Call to Action for Generative AI in 2024Results

GenCyber Cyber Security Day PresentationMichael W. Hawkins

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Miguel Araújo

The Codex of Business Writing Software for Real-World Solutions 2.pptxMalak Abu Hammad

Recently uploaded (20)

The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

🐬 The future of MySQL is Postgres 🐘

Powerful Google developer tools for immediate impact! (2023-24 C)

08448380779 Call Girls In Diplomatic Enclave Women Seeking Men

08448380779 Call Girls In Civil Lines Women Seeking Men

Breaking the Kubernetes Kill Chain: Host Path Mount

The 7 Things I Know About Cyber Security After 25 Years | April 2024

Factors to Consider When Choosing Accounts Payable Services Providers.pptx

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

Tata AIG General Insurance Company - Insurer Innovation Award 2024

How to convert PDF to text with Nanonets

IAC 2024 - IA Fast Track to Search Focused AI Solutions

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

Scaling API-first – The story of a global engineering organization

08448380779 Call Girls In Friends Colony Women Seeking Men

A Call to Action for Generative AI in 2024

GenCyber Cyber Security Day Presentation

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

The Codex of Business Writing Software for Real-World Solutions 2.pptx

ConcourseCI love Minio

1. 30/11/2016 Etourneau Gwenn @The_shinji62 ConcourseCI Minio

2. About me Etourneau Gwenn Sr Solution Architect Bef.: Platform Architect https://github.com/shinji62 https://twitter.com/the_shinji62

3. Pivotal • Spring Framework • RabbitMQ • Concourse • Cloud Foundry • Pivotal Tracker • Gemﬁre

4. Meetup

5. Agenda • Concourse • Minio • Minio for Concourse • Q&A

6. ConcourseCI

7. ConcourseCI • Do not allow snowﬂake • Isolation between job • Non share volume / storage / cache

8. Jobs 1 Jobs 2 ﬁle_out ﬁle_out X Concourse

9. Concourse • Concourse support S3 • S3 on premise is not easy • RiackCS, leoFS, Swift are hard • S3 compatibility so so…

10. Minio

11. Minio • Open Source • Object Storage written in Go • Distributed ? Not really more like replication • S3 compatible • Data protection using Bit rots , Erasure Code • Single go binary easy to start

12. Minio limitation • Do not scale-out or scale back • Not multi-tenant, user are the same for the same pool • number of nodes =>4 and <= 16 • Still early phase (>5 commit /day on master)

13. Minio 1 disk by node

14. Minio Many disk by nodes

15. Minio and Concourse • Support S3 native resource from concourse Jobs 1 Jobs 2 file_out file_out file_out

16. Minio bosh release

17. Minio bosh release • Deploy a pool of distributed minio in minutes • On any IaaS (supported by bosh)

18. Minio bosh release • Use bosh v2 links • Easy to spin-off • well, you need to learn bosh a little….

19. Minio 1 disk by node

20. Demo

21. Demo • Deploy Minio Cluster • Set Concourse Pipeline

22.

23. Resources https://github.com/shinji62/minio-dist-boshrelease https://github.com/cloudfoundry-community/minio-boshrelease • Minio bosh release • Minio • https://github.com/minio/minio • https://minio.io/

24. Resources • ConcourseCI • https://concourse.ci