Informatique CDC & la Suite Elastic : une seule suite pour de multiples usages

•

0 j'aime•874 vues

Découvrez comment ICDC utilise la Suite Elastic pour répondre à différents besoins avec une seule suite technologique, que ce soit pour des problématiques business analytics, search ou d’observabilité.

Informatique CDC & la Suite Elastic
Une Seule Suite pour de Multiples Usages
Fabien BALIGAND, Elastic Support Lead
Elastic{ON} Tour France
02/07/2020

2
Qui suis-je ?
Fabien BALIGAND
• Contributeur Open Source sur Logstash,
Elasticsearch & Kibana
• Créateur de 3 plugins Elastic :
‒ kibana-enhanced-table
‒ logstash-ﬁlter-aggregate
‒ vscode-logstash-editor
• Elastic Certiﬁed Engineer
• Elastic Support Lead à Informatique CDC
@fbaligand
@fbaligand

3
BANCAIRE RETRAITES
INTERET
GENERAL
Informatique CDC
Filière IT de la Caisse des Dépôts et Consignations

5
Contexte : Refonte intégrale du SI bancaire
Problème : Manque de visibilité

8
Apports de la suite Elastic
• Analyse ﬁne des performances (bench & prod)
• Analyse/Visualisation des problèmes
• Statistiques sur la charge et l’utilisation de l’application
• Analyse précise d’une transaction ou d’un parcours utilisateur

9
Apports inattendus
Une transparence totale : Prod / Etudes / MOA
Adoption unanime de Kibana par les utilisateurs

10
Tableaux de bord opérationnels
Statistiques de charge & performance de la banque en ligne

11
Tableaux de bord métier
Indicateurs métier d’utilisation de l’application, créés par la MOA

12
Stockage disque
Quelques chiﬀres
documents
par jour
Historique
250G 30M 90J

13
Histoire des usages Elastic @ Informatique CDC
Log Analytics
Business Analytics2
Search3
BAM4
APM5
Primary Database6
2014
Now
1

17
CICLADE : enjeux projet
• Loi Eckert : demande de l’état
• Permettre aux français de retrouver leurs comptes inactifs
• Près de 4milliards d’euros
• 8millions de comptes inactifs

18
Pourquoi Elasticsearch ?
Open Source Basé sur Lucene
Personnalisation
de la recherche
Scalable dès sa conception

19
CICLADE : pic de charge au JT de TF1
Jusqu’à 1600 requêtes/s

26
Elastic APM : un ROI imbattable
• Pas de beaux logs à générer
• Pas de pipeline Logstash à développer
• Pas de tableaux de bords Kibana à construire
→ ROI : économie de temps et d’argent pour l’Informatique CDC

28
Socle BAM
• Objectifs
‒ Monter un socle permettant de gérer les différents besoins de suivi d’activité métier
‒ Mutualiser les coûts et l’infrastructure
• 6 modules en production
• 9 modules en cours
• Pierre angulaire de la surveillance des processus métier

29
Socle BAM : architecture multi-tenant
LOGSTASH KIBANAELASTICSEARCHSOURCES
BAM 1
BAM 2
BAM 3
BAM 4
Instance dédiée
BAM 1
Instance dédiée
BAM 2
Instance dédiée
BAM 3
Instance dédiée
BAM 4
Indices
bam1-*
Indices
bam2-*
Indices
bam3-*
Indices
bam4-*
Espace
BAM 1
Espace
BAM 2
Espace
BAM 3
Espace
BAM 4

30
Socle BAM : premier module (SPOT)
Pilotage Opérationnel de la Trésorerie

31
Socle BAM : module SURF
Surveillance des ﬂux ﬁnanciers de bout en bout

32
Socle BAM : module SPIDOC
Suivi de la conformité des dossiers clients

34
Cas d’usages atypiques
Classification de
courriers postaux
Moteur de chatbot
Marche en double
prod → bench
Geocoding &
Reverse Geocoding

35
Perspectives
Socle
d’Observabilité
ECK
Reporting
intégré
• Métriques
• Logs
• APM
• Cross Cluster Search
• Référentiels
• AI Ops
• Centralisation des logs
sur notre socle Kubernetes
• Site internet mêlant
applicatif & dashboards
Kibana

Recommandé

Grafana Loki: like Prometheus, but for Logs

Marco Pracucci

Loki is a horizontally-scalable, highly-available log aggregation system inspired by Prometheus. It is designed to be very cost-effective and easy to operate, as it does not index the contents of the logs, but rather labels for each log stream. In this talk, we will introduce Loki, its architecture and the design trade-offs in an approachable way. We’ll both cover Loki and Promtail, the agent used to scrape local logs to push to Loki, including the Prometheus-style service discovery used to dynamically discover logs and attach metadata from applications running in a Kubernetes cluster. Finally, we’ll show how to query logs with Grafana using LogQL - the Loki query language - and the latest Grafana features to easily build dashboards mixing metrics and logs.

【第26回Elasticsearch勉強会】Logstashとともに振り返る、やっちまった事例ごった煮

Hibino Hisashi

Linking Metrics to Logs using Loki

Knoldus Inc.

Loki is an open source logging aggregation system that indexes the metadata of logs rather than the full contents. It consists of several microservices including the distributor, ingester, query frontend, and querier. The distributor routes logs to the ingesters which store the data in chunks in object storage. The querier handles log queries. Promtail is an agent that can be deployed to scrape logs from files and systemd on servers and ship them to Loki with labels for indexing. Compared to other logging solutions, Loki stores data more cost efficiently and is optimized for scaling.

Road to (Enterprise) Observability

Christoph Engelbert

OpenStackをコマンドで攻める！構築・運用とトラブル解決 - OpenStack最新情報セミナー 2014年6月

VirtualTech Japan Inc.

DockerCon参加報告 (`docker build`が30倍以上速くなる話など)

Akihiro Suda

Lightmapping for beginners- Unite Copenhagen 2019

Unity Technologies

OSMC 2022 | OpenTelemetry 101 by Dotan Horovit s.pdf

NETWAYS

This document provides an overview of OpenTelemetry, including: - OpenTelemetry is an observability framework that assists in generating and capturing telemetry data from cloud-native software across traces, metrics, and logs. - It includes vendor-agnostic APIs, SDKs, and tools for generating, collecting, and exporting telemetry data to analysis tools. - OpenTelemetry has reached general availability for tracing and is in release candidate for metrics, with client libraries available for many popular programming languages.

Recommandé

Grafana Loki: like Prometheus, but for Logs

Marco Pracucci

【第26回Elasticsearch勉強会】Logstashとともに振り返る、やっちまった事例ごった煮

Hibino Hisashi

Linking Metrics to Logs using Loki

Knoldus Inc.

Road to (Enterprise) Observability

Christoph Engelbert

OpenStackをコマンドで攻める！構築・運用とトラブル解決 - OpenStack最新情報セミナー 2014年6月

VirtualTech Japan Inc.

DockerCon参加報告 (`docker build`が30倍以上速くなる話など)

Akihiro Suda

Lightmapping for beginners- Unite Copenhagen 2019

Unity Technologies

OSMC 2022 | OpenTelemetry 101 by Dotan Horovit s.pdf

NETWAYS

NGINX Back to Basic 2 Part 2 (Japanese Webinar)

NGINX, Inc.

もう知らずにはいられないGitOpsをArgoCDで学ぶ【WESEEK Tech Conf #3】

WESEEKWESEEK

This document summarizes an ArgoCD presentation about implementing GitOps. Some key points: - ArgoCD was used because it has a great UI, self-healing capabilities, and supports SSO and RBAC. - GitOps aims to synchronize infrastructure as code in a Git repository with a cluster. ArgoCD helps achieve this through continuous delivery triggered by Git changes. - ArgoCD's architecture has three main components: the API server, repository server, and application controller. The application controller monitors for Git changes and syncs applications. - An example Application CR was shown to define the Git repository, target revision, destination cluster and namespace to deploy the application to.

Docker超入門

VirtualTech Japan Inc.

Terraform modules restructured

Ami Mahloof

Distributed Tracing for Kafka with OpenTelemetry with Daniel Kim | Kafka Summ...

HostedbyConfluent

With the increasing complexity that comes with adopting distributed architectures like Kafka, it is difficult to identify and triage performance anomalies. Implementing Distributed Tracing technologies like OpenTelemetry can add context to the messages flowing through your system, allowing us to visualize the data flow to pinpoint bottlenecks, bugs, and other performance issues. However, implementing end-to-end distributed tracing can be challenging for multi-request and asynchronous processes built on Kafka. This talk will walk through how to use OpenTelemetry to tell the full story of a request as it travels through your Kafka producer, queue, and consumer. First, we will learn how context propagation works in OpenTelemetry with W3C and B3 protocols. Then, we will go through the problems faced when instrumenting asynchronous requests. Finally, we will go through the context propagation implementation of the OpenTelemetry Kafka exporter, tracing multi-request and asynchronous processes.

Object Detection Beyond Mask R-CNN and RetinaNet I

Wanjin Yu

This document summarizes a tutorial on object detection beyond RetinaNet and Mask R-CNN. It discusses challenges in object detection including the backbone network, detection head, pretraining, handling scale variations, large batch sizes, detecting objects in crowds, and neural architecture search. It also introduces recent works that aim to address these challenges, such as DetNet, Light Head R-CNN, Objects365 pretraining, SFace for scale, MegDet for batch size, CrowdHuman benchmark for crowds, and NAS approaches. The document concludes that further improving object detection requires focusing on details and that continued progress will significantly benefit computer vision applications.

WTF is GitOps and Why You Should Care?

Weaveworks

Slides from OpenSource101.com Talk (https://opensource101.com/sessions/wtf-is-gitops-why-should-you-care/) If you’re interested in learning more about Cloud Native Computing or are already in the Kubernetes community you may have heard the term GitOps. It’s become a bit of a buzzword, but it’s so much more! The benefits of GitOps are real – they bring you security, reliability, velocity and more! And the project that started it all was Flux – a CNCF Incubating project developed and later donated by Weaveworks (the GitOps company who coined the term). Pinky will share from personal experience why GitOps has been an essential part of achieving a best-in-class delivery and platform team. Pinky will give a brief overview of definitions, CNCF-based principles, and Flux’s capabilities: multi-tenancy, multi-cluster, (multi-everything!), for apps and infra, and more. Pinky will cover a little of Flux’s microservices architecture and how the various components deliver this robust, secure, and trusted open source solution. Through the components of the Flux project, users today are enjoying compatibility with Helm, Jenkins, Terraform, Prometheus, and more as well as with cloud providers such as AWS, Azure, Google Cloud, and more. Join us for this informative session and get all of your GitOps questions answered by an end user in the community! Speaker: Priyanka (aka “Pinky”) is a Developer Experience Engineer at Weaveworks. She has worked on a multitude of topics including front end development, UI automation for testing and API development. Previously she was a software developer at State Farm where she was on the delivery engineering team working on GitOps enablement. She was instrumental in the multi-tenancy migration to utilize Flux for an internal Kubernetes offering. Outside of work, Priyanka enjoys hanging out with her husband and two rescue dogs as well as traveling around the globe.

KubernetesのRBACを掘り下げてみる（Kubernetes Novice Tokyo #17 発表資料）

NTT DATA Technology & Innovation

Kubernetes extensibility

Docker, Inc.

Kubernetes is designed to be an extensible system. But what is the vision for Kubernetes Extensibility? Do you know the difference between webhooks and cloud providers, or between CRI, CSI, and CNI? In this talk we will explore what extension points exist, how they have evolved, and how to use them to make the system do new and interesting things. We’ll give our vision for how they will probably evolve in the future, and talk about the sorts of things we expect the broader Kubernetes ecosystem to build with them.

PR-146: CornerNet detecting objects as paired keypoints

jaewon lee

The document summarizes the CornerNet object detection method. CornerNet detects objects as pairs of top-left and bottom-right corners using a convolutional neural network. It introduces corner pooling to better localize corners and achieves state-of-the-art performance among single-stage detectors. The method formulates object detection as an association problem between corners using embeddings and outperforms other detectors on standard benchmarks with an average inference time of 244ms per image.

Aaron Mildenstein - Using Logstash with Zabbix

Zabbix

Logstash is a terrific tool for capturing, filtering, parsing and enriching data from a number of sources—including logs, of course. But Logstash is also able to capture from many other sources, including social media streams, databases, and many more. Data streams like these are a potential gold mine for Zabbix trending and alerting of all kinds. In this talk Aaron Mildensten will provide an overview of how to configure and integrate Logstash with Zabbix to: * capture data * parse data events into key/value pairs * associate an event with the time-stamp provided by the data * generate metrics from the data * output these values to Zabbix, with the associated time-stamp Zabbix Conference 2015

Operator Framework Overview

Rob Szumski

Temperatura Zabbix Procedimento Temper Usb

Thiago Santos

Multi-Clusters Made Easy with Liqo: Getting Rid of Your Clusters Keeping Them...

KCDItaly

Many companies are experiencing a dramatic increase in the number of their Kubernetes clusters, for reasons such as geographical/legislative constraints, data/service replication, etc. However, when the number of clusters increases, the complexity of deploying apps, managing the entire multi-cluster infrastructure, and keeping its state under control, becomes rapidly an unmanageable problem. A possible solution is Liqo, an open-source project that simplifies the creation of multi-cluster topologies by replicating the Kubernetes “cattle” model also to clusters. Liqo creates a virtual cluster that spans multiple real clusters, either on-prem or managed (AKS, EKS, GKE), and instantiates the desired applications seamlessly in the appropriate cluster. This talk will discuss the potentials and roadblocks of this vision and highlight how Liqo brings multi- cluster transparency to the users.

Kubernetes Service Account As Multi-Cloud Identity / Cloud Native Security Co...

Preferred Networks

Relic's FX System

Daniel Barrero

Prometheus monitoring from outside of Kubernetes  〜どうして我々はKubernetes上のPromet...

whywaita

Gain Better Observability with OpenTelemetry and InfluxDB

InfluxData

Many developers and DevOps engineers have become aware of using their observability data to gain greater insights into their infrastructure systems. InfluxDB is the purpose-built time series database used to collect metrics and gain observability into apps, servers, containers, and networks. Developers use InfluxDB to improve the quality and efficiency of their CI/CD pipelines. Start using InfluxDB to aggregate infrastructure and application performance monitoring metrics to enable better anomaly detection, root-cause analysis, and alerting. This session will demonstrate how to record metrics, logs, and traces with one library — OpenTelemetry — and store them in one open source time series database — InfluxDB. Zoe will demonstrate how easy it is to set up the OpenTelemetry Operator for Kubernetes and to store and analyze your data in InfluxDB.

UnboundとNSDの紹介　BIND9との比較編

hdais

君にもできる! にゅーとろん君になってみよー！！「Neutronになって理解するOpenStack Net - OpenStack最新情報セミナー ...

VirtualTech Japan Inc.

- デモのゴール --- Neutronが作るネットワークを、複数のコマンドラインを組み合わせることで再現する --- インスタンスに見立てたインターフェースから、外部ネットワークに見立てたインターフェースへのPing疎通を確認する --- 2テナント構成本資料の5章以外のドキュメントはこちらにあります。 http://www.slideshare.net/VirtualTech-JP/20140929-rev-r

Système d’Information à l’Apec : un nouveau coeur de métier mis en place avec...

Ippon

#OSSPARIS19 - Stream processing : de la base de données classique au streamin...

Paris Open Source Summit

#Data management & #Blockchain - Track - Data : from stream processing to data lakes and data science Toute organisation a un système d’information réparti sur un nombre d’applications plus ou moins important. Aujourd’hui elle cherche à obtenir une vision homogène, fédérée, quasi temps réel de son activité, si bien que nous avons des demandes client pour lesquelles il est question de fournir aux métiers une information « instantanée » et non plus vielle de 5 minutes ! Capturer les changements dans les bases de données, le CDC, cela existe depuis longtemps, mais avec une mise en œuvre ô combien complexe, et couteuse en performance ! Debezium est une plateforme distribuée et open source dédié au CDC. Sa simplicité de mise en œuvre, son ouverture (open source, multi bases de données, multi plateformes de streaming…), ses performances la rendent unique et la placent telle une brique incontournable du système d’information. Avec Debezium, les métiers vont enfin avoir des indicateurs de leurs processus instantanément, couvrant ainsi de nombreux cas d’usage (personnalisation client, audit, vision client 360° instantanée, monitoring temps réel...) Notre talk a pour but de faire connaître Debezium, une couche manquante à Apache Kafka pour mieux dialoguer avec les systèmes d’information actuels (legacy) et apporter du « streaming » aux bases de données classiques.

Contenu connexe

Tendances

NGINX Back to Basic 2 Part 2 (Japanese Webinar)

NGINX, Inc.

もう知らずにはいられないGitOpsをArgoCDで学ぶ【WESEEK Tech Conf #3】

WESEEKWESEEK

Docker超入門

VirtualTech Japan Inc.

Terraform modules restructured

Ami Mahloof

Distributed Tracing for Kafka with OpenTelemetry with Daniel Kim | Kafka Summ...

HostedbyConfluent

Object Detection Beyond Mask R-CNN and RetinaNet I

Wanjin Yu

WTF is GitOps and Why You Should Care?

Weaveworks

KubernetesのRBACを掘り下げてみる（Kubernetes Novice Tokyo #17 発表資料）

NTT DATA Technology & Innovation

Kubernetes extensibility

Docker, Inc.

PR-146: CornerNet detecting objects as paired keypoints

jaewon lee

Aaron Mildenstein - Using Logstash with Zabbix

Zabbix

Operator Framework Overview

Rob Szumski

Temperatura Zabbix Procedimento Temper Usb

Thiago Santos

Multi-Clusters Made Easy with Liqo: Getting Rid of Your Clusters Keeping Them...

KCDItaly

Kubernetes Service Account As Multi-Cloud Identity / Cloud Native Security Co...

Preferred Networks

Relic's FX System

Daniel Barrero

Prometheus monitoring from outside of Kubernetes  〜どうして我々はKubernetes上のPromet...

whywaita

Gain Better Observability with OpenTelemetry and InfluxDB

InfluxData

UnboundとNSDの紹介　BIND9との比較編

hdais

君にもできる! にゅーとろん君になってみよー！！「Neutronになって理解するOpenStack Net - OpenStack最新情報セミナー ...

VirtualTech Japan Inc.

Tendances (20)

NGINX Back to Basic 2 Part 2 (Japanese Webinar)

もう知らずにはいられないGitOpsをArgoCDで学ぶ【WESEEK Tech Conf #3】

Docker超入門

Terraform modules restructured

Distributed Tracing for Kafka with OpenTelemetry with Daniel Kim | Kafka Summ...

Object Detection Beyond Mask R-CNN and RetinaNet I

WTF is GitOps and Why You Should Care?

KubernetesのRBACを掘り下げてみる（Kubernetes Novice Tokyo #17 発表資料）

Kubernetes extensibility

PR-146: CornerNet detecting objects as paired keypoints

Aaron Mildenstein - Using Logstash with Zabbix

Operator Framework Overview

Temperatura Zabbix Procedimento Temper Usb

Multi-Clusters Made Easy with Liqo: Getting Rid of Your Clusters Keeping Them...

Kubernetes Service Account As Multi-Cloud Identity / Cloud Native Security Co...

Relic's FX System

Prometheus monitoring from outside of Kubernetes  〜どうして我々はKubernetes上のPromet...

Gain Better Observability with OpenTelemetry and InfluxDB

UnboundとNSDの紹介　BIND9との比較編

君にもできる! にゅーとろん君になってみよー！！「Neutronになって理解するOpenStack Net - OpenStack最新情報セミナー ...

Similaire à Informatique CDC & la Suite Elastic : une seule suite pour de multiples usages

Système d’Information à l’Apec : un nouveau coeur de métier mis en place avec...

Ippon

#OSSPARIS19 - Stream processing : de la base de données classique au streamin...

Paris Open Source Summit

Plateformes et infrastructure infonuagique natif de ville de Montréall

CloudOps2005

La Suite Elastic à BPCE IT: Centre de service Elastic et Security Analytics

Informatique CDC & la Suite Elastic : une seule suite pour de multiples usages

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Informatique CDC & la Suite Elastic : une seule suite pour de multiples usages

Similaire à Informatique CDC & la Suite Elastic : une seule suite pour de multiples usages (20)

Plus de Elasticsearch

Plus de Elasticsearch (20)

Dernier

Dernier (9)

Informatique CDC & la Suite Elastic : une seule suite pour de multiples usages