SlideShare une entreprise Scribd logo
1  sur  20
Télécharger pour lire hors ligne
1
Andrea Fuggetta
Sr. Software Engineer, Procter & Gamble
@andreafuggetta
Elastic at Procter &
Gamble:
A Network Story
2
Agenda
What are we talking about?
Who we are1
Challenges3
The road so far4
What’s next?5
Problem statement2
3
Who we are
• Founded in 1837 (180+ years)
• Superior quality products
• More than 180 countries
• http://www.pg.com/
Making every day more than ordinary
4
Who we are (cont’d)
• Andrea Fuggetta
• Cincinnati (OH)
• Sr. Software Engineer
• Network Automation
Making every day more than ordinary
5
Problem statement
• Insourcing
• Where is our data?
• What data do we have?
• What are my processes and use cases?
• How do we aggregate the data?
• How do we leverage the data?
• Where is our data!?
Where did we start?
6
I have not failed. I've just found
10,000 ways that won't work
Thomas Edison
7
Problem statement
• So many choices
• Trial and error
• What is Elasticsearch?
• Demo sessions
• First visualizations
• Interest from customers
Where to start?
8
It works on my machine
Every programmer out there
9
Challenges
• Find the processes that govern your data
• Find the people responsible for them
• Find the technology to support the business and use cases
Where to start?
10
Challenges
• How much data do we expect?
‒ Do you know all our data?
‒ Throughput
• Infrastructure
‒ Cloud or DC
‒ Compute power
‒ Memory
‒ Storage
• What product?
‒ Single server
‒ Local cluster
‒ Elastic Cloud
‒ ECE
Scaling – Hosting – Product
11
Challenges
• Define the data sources
‒ Network metrics
• Data flow
‒ 500MB/day
• Infrastructure
‒ Cloud
• What product?
‒ 3 node cluster
‒ Installed manually on VMs
Solutions?
12
“The important
thing is to
never stop
questioning…”
Albert Einstein
13
The road so far
• Define the data sources
‒ Network devices’ syslogs and metrics
• Data flow
‒ 500MB/day ~6TB/day
• Infrastructure
‒ Cloud
• What product?
‒ Elastic Cloud Enterprise (ECE)
• How?
Current state
14
The road so far
Architecture
15
The road so far
Automation
GitHub
Azure
DevOps
Terraform
Ansible
Elastic ECE
Logstash
16
VMs
Supporting ECE
and Logstash
pipelines.
Hot/Warm/Cold
lifecycle
Some numbers
Terabytes
Data coming from
network devices
i.e. Firewall
syslogs
Teams
Currently
leveraging the
solution
42 6 6
17
Results
• Prevented downtime and potential issues
• Increased knowledge of our data
• One destination for logs and metrics
• Easier troubleshooting and forensics
• Increased scalability and mobility
• All in less than 1 year
Long road ahead
18
Results
• Move infrastructure from Azure to AWS
‒ Load balancers
‒ Kafka-like queue (Kinesis)
‒ Virtual Machines
‒ Storage
‒ Monitoring
‒ Installing and configuring software (ECE, Logstash)
• Half day
Examples
19
What’s next?
• More customers – more data
‒ Information Security (SIEM)
‒ Data Science (Search, aggregation, analysis)
• ML
‒ Anomalies detection
• Cloud data and logs
‒ Function beats
‒ Custom ingestion pipelines
• Alerts and actions
‒ Anomalies trigger alerts and scripts to self-heal
• Canvas
‒ Executive views
‒ Hallway monitors
Near future
20
Thank you

Contenu connexe

Tendances

Tendances (20)

University of Oxford: building a next generation SIEM
University of Oxford: building a next generation SIEMUniversity of Oxford: building a next generation SIEM
University of Oxford: building a next generation SIEM
 
Keynote
KeynoteKeynote
Keynote
 
Better Search and Business Analytics at Southern Glazer’s Wine & Spirits
Better Search and Business Analytics at Southern Glazer’s Wine & SpiritsBetter Search and Business Analytics at Southern Glazer’s Wine & Spirits
Better Search and Business Analytics at Southern Glazer’s Wine & Spirits
 
Logging, Metrics, and APM: The Operations Trifecta
Logging, Metrics, and APM: The Operations TrifectaLogging, Metrics, and APM: The Operations Trifecta
Logging, Metrics, and APM: The Operations Trifecta
 
Hunting for Evil with the Elastic Stack
Hunting for Evil with the Elastic StackHunting for Evil with the Elastic Stack
Hunting for Evil with the Elastic Stack
 
What’s Evolving in the Elastic Stack
What’s Evolving in the Elastic StackWhat’s Evolving in the Elastic Stack
What’s Evolving in the Elastic Stack
 
Machine Learning for Anomaly Detection, Time Series Modeling, and More
Machine Learning for Anomaly Detection, Time Series Modeling, and MoreMachine Learning for Anomaly Detection, Time Series Modeling, and More
Machine Learning for Anomaly Detection, Time Series Modeling, and More
 
Building a reliable and cost effect logging system at Box
Building a reliable and cost effect logging system at Box Building a reliable and cost effect logging system at Box
Building a reliable and cost effect logging system at Box
 
Industrial production process visualization with the Elastic Stack in real-ti...
Industrial production process visualization with the Elastic Stack in real-ti...Industrial production process visualization with the Elastic Stack in real-ti...
Industrial production process visualization with the Elastic Stack in real-ti...
 
Elasticsearch on Azure
Elasticsearch on AzureElasticsearch on Azure
Elasticsearch on Azure
 
Turning Evidence into Insights: How NCIS Leverages Elastic
Turning Evidence into Insights: How NCIS Leverages Elastic Turning Evidence into Insights: How NCIS Leverages Elastic
Turning Evidence into Insights: How NCIS Leverages Elastic
 
The evolution of the big data platform @ Netflix (OSCON 2015)
The evolution of the big data platform @ Netflix (OSCON 2015)The evolution of the big data platform @ Netflix (OSCON 2015)
The evolution of the big data platform @ Netflix (OSCON 2015)
 
Infrastructure monitoring made easy, from ingest to insight
Infrastructure monitoring made easy, from ingest to insightInfrastructure monitoring made easy, from ingest to insight
Infrastructure monitoring made easy, from ingest to insight
 
Logging, Metrics, and APM: The Operations Trifecta (P)
Logging, Metrics, and APM: The Operations Trifecta (P)Logging, Metrics, and APM: The Operations Trifecta (P)
Logging, Metrics, and APM: The Operations Trifecta (P)
 
Solving Hybrid Cloud Data Replication with Apache Cassandra
Solving Hybrid Cloud Data Replication with Apache CassandraSolving Hybrid Cloud Data Replication with Apache Cassandra
Solving Hybrid Cloud Data Replication with Apache Cassandra
 
Using Azure Databricks, Structured Streaming, and Deep Learning Pipelines to ...
Using Azure Databricks, Structured Streaming, and Deep Learning Pipelines to ...Using Azure Databricks, Structured Streaming, and Deep Learning Pipelines to ...
Using Azure Databricks, Structured Streaming, and Deep Learning Pipelines to ...
 
Architectural Best Practices to Master + Pitfalls to Avoid (P)
Architectural Best Practices to Master + Pitfalls to Avoid (P) Architectural Best Practices to Master + Pitfalls to Avoid (P)
Architectural Best Practices to Master + Pitfalls to Avoid (P)
 
T-Mobile and Elastic
T-Mobile and ElasticT-Mobile and Elastic
T-Mobile and Elastic
 
Divide & Conquer - Logging Architecture in Distributed Ecosystems with Elasti...
Divide & Conquer - Logging Architecture in Distributed Ecosystems with Elasti...Divide & Conquer - Logging Architecture in Distributed Ecosystems with Elasti...
Divide & Conquer - Logging Architecture in Distributed Ecosystems with Elasti...
 
Security sizing meetup
Security sizing meetupSecurity sizing meetup
Security sizing meetup
 

Similaire à Elastic at Procter & Gamble: A Network Story

Séminaire Big Data Alter Way - Elasticsearch - octobre 2014
Séminaire Big Data Alter Way - Elasticsearch - octobre 2014Séminaire Big Data Alter Way - Elasticsearch - octobre 2014
Séminaire Big Data Alter Way - Elasticsearch - octobre 2014
ALTER WAY
 
The challenges of live events scalability
The challenges of live events scalabilityThe challenges of live events scalability
The challenges of live events scalability
Guy Tomer
 

Similaire à Elastic at Procter & Gamble: A Network Story (20)

Lessons Learned Replatforming A Large Machine Learning Application To Apache ...
Lessons Learned Replatforming A Large Machine Learning Application To Apache ...Lessons Learned Replatforming A Large Machine Learning Application To Apache ...
Lessons Learned Replatforming A Large Machine Learning Application To Apache ...
 
Monitoring Half a Million ML Models, IoT Streaming Data, and Automated Qualit...
Monitoring Half a Million ML Models, IoT Streaming Data, and Automated Qualit...Monitoring Half a Million ML Models, IoT Streaming Data, and Automated Qualit...
Monitoring Half a Million ML Models, IoT Streaming Data, and Automated Qualit...
 
Games Industry Analytics Forum 2 - Plumbee
Games Industry Analytics Forum 2 - PlumbeeGames Industry Analytics Forum 2 - Plumbee
Games Industry Analytics Forum 2 - Plumbee
 
The world is not black and white – Impact of decisions over the lifetime of a...
The world is not black and white – Impact of decisions over the lifetime of a...The world is not black and white – Impact of decisions over the lifetime of a...
The world is not black and white – Impact of decisions over the lifetime of a...
 
Correlation does not mean causation
Correlation does not mean causationCorrelation does not mean causation
Correlation does not mean causation
 
Tools and best practices for sustainable software
Tools and best practices for sustainable softwareTools and best practices for sustainable software
Tools and best practices for sustainable software
 
Tools and best practices for sustainable software.pdf
Tools and best practices for sustainable software.pdfTools and best practices for sustainable software.pdf
Tools and best practices for sustainable software.pdf
 
Tools and best practices for sustainable software.pdf
Tools and best practices for sustainable software.pdfTools and best practices for sustainable software.pdf
Tools and best practices for sustainable software.pdf
 
MongoDB.local Atlanta: MongoDB @ Sensus: Xylem IoT and MongoDB
MongoDB.local Atlanta: MongoDB @ Sensus: Xylem IoT and MongoDBMongoDB.local Atlanta: MongoDB @ Sensus: Xylem IoT and MongoDB
MongoDB.local Atlanta: MongoDB @ Sensus: Xylem IoT and MongoDB
 
From Pipelines to Refineries: scaling big data applications with Tim Hunter
From Pipelines to Refineries: scaling big data applications with Tim HunterFrom Pipelines to Refineries: scaling big data applications with Tim Hunter
From Pipelines to Refineries: scaling big data applications with Tim Hunter
 
Séminaire Big Data Alter Way - Elasticsearch - octobre 2014
Séminaire Big Data Alter Way - Elasticsearch - octobre 2014Séminaire Big Data Alter Way - Elasticsearch - octobre 2014
Séminaire Big Data Alter Way - Elasticsearch - octobre 2014
 
Don't build a data science team
Don't build a data science teamDon't build a data science team
Don't build a data science team
 
Big Data Rampage
Big Data RampageBig Data Rampage
Big Data Rampage
 
How we integrate Machine Learning Algorithms into our IT Platform at Outfittery
How we integrate Machine Learning Algorithms into our IT Platform at OutfitteryHow we integrate Machine Learning Algorithms into our IT Platform at Outfittery
How we integrate Machine Learning Algorithms into our IT Platform at Outfittery
 
Artificial Intelligence (ML - DL)
Artificial Intelligence (ML - DL)Artificial Intelligence (ML - DL)
Artificial Intelligence (ML - DL)
 
PXL Data Engineering Workshop By Selligent
PXL Data Engineering Workshop By Selligent PXL Data Engineering Workshop By Selligent
PXL Data Engineering Workshop By Selligent
 
GOTO Night: Decision Making Based on Machine Learning
GOTO Night: Decision Making Based on Machine LearningGOTO Night: Decision Making Based on Machine Learning
GOTO Night: Decision Making Based on Machine Learning
 
AEMP Connect 2021 Can AI Solve Construction Telematics Overload Problem? Ode...
AEMP Connect 2021  Can AI Solve Construction Telematics Overload Problem? Ode...AEMP Connect 2021  Can AI Solve Construction Telematics Overload Problem? Ode...
AEMP Connect 2021 Can AI Solve Construction Telematics Overload Problem? Ode...
 
How we integrate Machine Learning Algorithms into our IT Platform at Outfitte...
How we integrate Machine Learning Algorithms into our IT Platform at Outfitte...How we integrate Machine Learning Algorithms into our IT Platform at Outfitte...
How we integrate Machine Learning Algorithms into our IT Platform at Outfitte...
 
The challenges of live events scalability
The challenges of live events scalabilityThe challenges of live events scalability
The challenges of live events scalability
 

Plus de Elasticsearch

Plus de Elasticsearch (20)

An introduction to Elasticsearch's advanced relevance ranking toolbox
An introduction to Elasticsearch's advanced relevance ranking toolboxAn introduction to Elasticsearch's advanced relevance ranking toolbox
An introduction to Elasticsearch's advanced relevance ranking toolbox
 
From MSP to MSSP using Elastic
From MSP to MSSP using ElasticFrom MSP to MSSP using Elastic
From MSP to MSSP using Elastic
 
Cómo crear excelentes experiencias de búsqueda en sitios web
Cómo crear excelentes experiencias de búsqueda en sitios webCómo crear excelentes experiencias de búsqueda en sitios web
Cómo crear excelentes experiencias de búsqueda en sitios web
 
Te damos la bienvenida a una nueva forma de realizar búsquedas
Te damos la bienvenida a una nueva forma de realizar búsquedas Te damos la bienvenida a una nueva forma de realizar búsquedas
Te damos la bienvenida a una nueva forma de realizar búsquedas
 
Tirez pleinement parti d'Elastic grâce à Elastic Cloud
Tirez pleinement parti d'Elastic grâce à Elastic CloudTirez pleinement parti d'Elastic grâce à Elastic Cloud
Tirez pleinement parti d'Elastic grâce à Elastic Cloud
 
Comment transformer vos données en informations exploitables
Comment transformer vos données en informations exploitablesComment transformer vos données en informations exploitables
Comment transformer vos données en informations exploitables
 
Plongez au cœur de la recherche dans tous ses états.
Plongez au cœur de la recherche dans tous ses états.Plongez au cœur de la recherche dans tous ses états.
Plongez au cœur de la recherche dans tous ses états.
 
Modernising One Legal Se@rch with Elastic Enterprise Search [Customer Story]
Modernising One Legal Se@rch with Elastic Enterprise Search [Customer Story]Modernising One Legal Se@rch with Elastic Enterprise Search [Customer Story]
Modernising One Legal Se@rch with Elastic Enterprise Search [Customer Story]
 
An introduction to Elasticsearch's advanced relevance ranking toolbox
An introduction to Elasticsearch's advanced relevance ranking toolboxAn introduction to Elasticsearch's advanced relevance ranking toolbox
An introduction to Elasticsearch's advanced relevance ranking toolbox
 
Welcome to a new state of find
Welcome to a new state of findWelcome to a new state of find
Welcome to a new state of find
 
Building great website search experiences
Building great website search experiencesBuilding great website search experiences
Building great website search experiences
 
Keynote: Harnessing the power of Elasticsearch for simplified search
Keynote: Harnessing the power of Elasticsearch for simplified searchKeynote: Harnessing the power of Elasticsearch for simplified search
Keynote: Harnessing the power of Elasticsearch for simplified search
 
Cómo transformar los datos en análisis con los que tomar decisiones
Cómo transformar los datos en análisis con los que tomar decisionesCómo transformar los datos en análisis con los que tomar decisiones
Cómo transformar los datos en análisis con los que tomar decisiones
 
Explore relève les défis Big Data avec Elastic Cloud
Explore relève les défis Big Data avec Elastic Cloud Explore relève les défis Big Data avec Elastic Cloud
Explore relève les défis Big Data avec Elastic Cloud
 
Comment transformer vos données en informations exploitables
Comment transformer vos données en informations exploitablesComment transformer vos données en informations exploitables
Comment transformer vos données en informations exploitables
 
Transforming data into actionable insights
Transforming data into actionable insightsTransforming data into actionable insights
Transforming data into actionable insights
 
Opening Keynote: Why Elastic?
Opening Keynote: Why Elastic?Opening Keynote: Why Elastic?
Opening Keynote: Why Elastic?
 
Empowering agencies using Elastic as a Service inside Government
Empowering agencies using Elastic as a Service inside GovernmentEmpowering agencies using Elastic as a Service inside Government
Empowering agencies using Elastic as a Service inside Government
 
The opportunities and challenges of data for public good
The opportunities and challenges of data for public goodThe opportunities and challenges of data for public good
The opportunities and challenges of data for public good
 
Enterprise search and unstructured data with CGI and Elastic
Enterprise search and unstructured data with CGI and ElasticEnterprise search and unstructured data with CGI and Elastic
Enterprise search and unstructured data with CGI and Elastic
 

Dernier

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
?#DUbAI#??##{{(☎️+971_581248768%)**%*]'#abortion pills for sale in dubai@
 

Dernier (20)

What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?
 
Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024
 
GenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationGenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day Presentation
 
Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherStrategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a Fresher
 
Advantages of Hiring UIUX Design Service Providers for Your Business
Advantages of Hiring UIUX Design Service Providers for Your BusinessAdvantages of Hiring UIUX Design Service Providers for Your Business
Advantages of Hiring UIUX Design Service Providers for Your Business
 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivity
 
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonData Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt Robison
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
 
Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024
 
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
🐬 The future of MySQL is Postgres 🐘
🐬  The future of MySQL is Postgres   🐘🐬  The future of MySQL is Postgres   🐘
🐬 The future of MySQL is Postgres 🐘
 
[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)
 
The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024
 
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
 
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...
 
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
 

Elastic at Procter & Gamble: A Network Story

  • 1. 1 Andrea Fuggetta Sr. Software Engineer, Procter & Gamble @andreafuggetta Elastic at Procter & Gamble: A Network Story
  • 2. 2 Agenda What are we talking about? Who we are1 Challenges3 The road so far4 What’s next?5 Problem statement2
  • 3. 3 Who we are • Founded in 1837 (180+ years) • Superior quality products • More than 180 countries • http://www.pg.com/ Making every day more than ordinary
  • 4. 4 Who we are (cont’d) • Andrea Fuggetta • Cincinnati (OH) • Sr. Software Engineer • Network Automation Making every day more than ordinary
  • 5. 5 Problem statement • Insourcing • Where is our data? • What data do we have? • What are my processes and use cases? • How do we aggregate the data? • How do we leverage the data? • Where is our data!? Where did we start?
  • 6. 6 I have not failed. I've just found 10,000 ways that won't work Thomas Edison
  • 7. 7 Problem statement • So many choices • Trial and error • What is Elasticsearch? • Demo sessions • First visualizations • Interest from customers Where to start?
  • 8. 8 It works on my machine Every programmer out there
  • 9. 9 Challenges • Find the processes that govern your data • Find the people responsible for them • Find the technology to support the business and use cases Where to start?
  • 10. 10 Challenges • How much data do we expect? ‒ Do you know all our data? ‒ Throughput • Infrastructure ‒ Cloud or DC ‒ Compute power ‒ Memory ‒ Storage • What product? ‒ Single server ‒ Local cluster ‒ Elastic Cloud ‒ ECE Scaling – Hosting – Product
  • 11. 11 Challenges • Define the data sources ‒ Network metrics • Data flow ‒ 500MB/day • Infrastructure ‒ Cloud • What product? ‒ 3 node cluster ‒ Installed manually on VMs Solutions?
  • 12. 12 “The important thing is to never stop questioning…” Albert Einstein
  • 13. 13 The road so far • Define the data sources ‒ Network devices’ syslogs and metrics • Data flow ‒ 500MB/day ~6TB/day • Infrastructure ‒ Cloud • What product? ‒ Elastic Cloud Enterprise (ECE) • How? Current state
  • 14. 14 The road so far Architecture
  • 15. 15 The road so far Automation GitHub Azure DevOps Terraform Ansible Elastic ECE Logstash
  • 16. 16 VMs Supporting ECE and Logstash pipelines. Hot/Warm/Cold lifecycle Some numbers Terabytes Data coming from network devices i.e. Firewall syslogs Teams Currently leveraging the solution 42 6 6
  • 17. 17 Results • Prevented downtime and potential issues • Increased knowledge of our data • One destination for logs and metrics • Easier troubleshooting and forensics • Increased scalability and mobility • All in less than 1 year Long road ahead
  • 18. 18 Results • Move infrastructure from Azure to AWS ‒ Load balancers ‒ Kafka-like queue (Kinesis) ‒ Virtual Machines ‒ Storage ‒ Monitoring ‒ Installing and configuring software (ECE, Logstash) • Half day Examples
  • 19. 19 What’s next? • More customers – more data ‒ Information Security (SIEM) ‒ Data Science (Search, aggregation, analysis) • ML ‒ Anomalies detection • Cloud data and logs ‒ Function beats ‒ Custom ingestion pipelines • Alerts and actions ‒ Anomalies trigger alerts and scripts to self-heal • Canvas ‒ Executive views ‒ Hallway monitors Near future