SlideShare une entreprise Scribd logo
1  sur  10
Télécharger pour lire hors ligne
© 2021 SunPower Corporation
From Legacy SQL Server to
High-Powered Confluent &
Kafka Monitoring System
Patrick Franz
SEPTEMBER 2021
Senior Director, Engineering
2
OUR MISSION
Changing the Way
Our World is Powered
We do this by building a great company, putting
our customers first, and making solar the most
compelling energy option. If we empower our
people and communities to thrive, we’ll build a
sustainable future together.
3
MONITORING
Why Monitoring Matters
• SunPower has approximately 6M devices in
the field sending data, on average, every 5
minutes.
• We offer a 20-year guarantee for lease
customers that the system will produce a
target kWh/year.
• Example impacts on performance: weather,
shading, or component faults.
• We want to understand these challenges with
data.
4
MONITORING
Delivering Customer
Value
• How much data do we need to predict and
detect performance issues before they impact
the customer?
• How do we collect this data and act on it
quickly?
• Fundamentals
• Measurement ingestion
• Metadata association
• Storage
• Analysis and presentation
5
Early Solar Adopters
MONITORING
• In 2008 we had this idea that customers
wanted to know how much energy their
system is producing.
• Supported SunPower’s residential and
commercial customers successfully for a
decade.
• SunPower’s customer base grew, average
solar installation size increased, and the
industry transitioned to microinverters.
6
Customer Value At Scale
MONITORING
• By 2018 we knew we needed to support
modern IoT devices on our platform.
• Customers’ energy habits were changing (EV
adoption, smart appliances) and policy shifts
in favor of renewable energy drove increased
adoption of solar and storage.
• Took an engineering first approach and
designed new fundamentals with an emphasis
on customer experience.
IoT Sensor Data Confluent Cloud
Customer Site
and Device
Metadata
Time-series
API
Data Warehouse
Weather Data,
Utility Rates, etc.
7
MONITORING
Why Kafka?
• Sensor data alone is not valuable. How do we
connect a solar panel at an address to a
sensor reading?
• Kafka is the central nervous system that
allows us to blend multiple data sources
together and generate value from the raw
sensor data.
• Engineering teams get access to fully-enriched
data via Kafka and build the features
important to them.
8
CASE STUDY
Panel-Level Monitoring
• With launch of the mySunPower™ app we
continue to iterate on customer experience.
• Panel-level monitoring addresses the earlier
need of what data we collect and how we use
it to solve real customer issues.
• Quickly identify underperforming components
and determine appropriate resolution.
Feature planned for release in Q3.
9
MONITORING
We Are Hiring
• Help us change the way the world is powered
with Kafka.
• patrick.franz@sunpower.com
• LinkedIn
Changing the way
our world is powered
THANK YOU
© 2021 SunPower Corporation

Contenu connexe

Tendances

Elastically Scaling Kafka Using Confluent
Elastically Scaling Kafka Using ConfluentElastically Scaling Kafka Using Confluent
Elastically Scaling Kafka Using Confluentconfluent
 
Event-Driven Architectures Done Right | Tim Berglund, Confluent
Event-Driven Architectures Done Right | Tim Berglund, ConfluentEvent-Driven Architectures Done Right | Tim Berglund, Confluent
Event-Driven Architectures Done Right | Tim Berglund, ConfluentHostedbyConfluent
 
Developing custom transformation in the Kafka connect to minimize data redund...
Developing custom transformation in the Kafka connect to minimize data redund...Developing custom transformation in the Kafka connect to minimize data redund...
Developing custom transformation in the Kafka connect to minimize data redund...HostedbyConfluent
 
TBD Data Governance | David Araujo and Michael Agnich, Confluent
TBD Data Governance | David Araujo and Michael Agnich, Confluent TBD Data Governance | David Araujo and Michael Agnich, Confluent
TBD Data Governance | David Araujo and Michael Agnich, Confluent HostedbyConfluent
 
DataOps Automation for a Kafka Streaming Platform (Andrew Stevenson + Spiros ...
DataOps Automation for a Kafka Streaming Platform (Andrew Stevenson + Spiros ...DataOps Automation for a Kafka Streaming Platform (Andrew Stevenson + Spiros ...
DataOps Automation for a Kafka Streaming Platform (Andrew Stevenson + Spiros ...HostedbyConfluent
 
Transformation During a Global Pandemic | Ashish Pandit and Scott Lee, Univer...
Transformation During a Global Pandemic | Ashish Pandit and Scott Lee, Univer...Transformation During a Global Pandemic | Ashish Pandit and Scott Lee, Univer...
Transformation During a Global Pandemic | Ashish Pandit and Scott Lee, Univer...HostedbyConfluent
 
Mesh-ing around with Streams across the Enterprise | Phil Scanlon, Solace
Mesh-ing around with Streams across the Enterprise | Phil Scanlon, SolaceMesh-ing around with Streams across the Enterprise | Phil Scanlon, Solace
Mesh-ing around with Streams across the Enterprise | Phil Scanlon, SolaceHostedbyConfluent
 
Data Mess to Data Mesh | Jay Kreps, CEO, Confluent | Kafka Summit Americas 20...
Data Mess to Data Mesh | Jay Kreps, CEO, Confluent | Kafka Summit Americas 20...Data Mess to Data Mesh | Jay Kreps, CEO, Confluent | Kafka Summit Americas 20...
Data Mess to Data Mesh | Jay Kreps, CEO, Confluent | Kafka Summit Americas 20...HostedbyConfluent
 
Kafka & InfluxDB: BFFs for Enterprise Data Applications | Russ Savage, Influx...
Kafka & InfluxDB: BFFs for Enterprise Data Applications | Russ Savage, Influx...Kafka & InfluxDB: BFFs for Enterprise Data Applications | Russ Savage, Influx...
Kafka & InfluxDB: BFFs for Enterprise Data Applications | Russ Savage, Influx...HostedbyConfluent
 
Extracting Value from IOT using Azure Cosmos DB, Azure Synapse Analytics and ...
Extracting Value from IOT using Azure Cosmos DB, Azure Synapse Analytics and ...Extracting Value from IOT using Azure Cosmos DB, Azure Synapse Analytics and ...
Extracting Value from IOT using Azure Cosmos DB, Azure Synapse Analytics and ...HostedbyConfluent
 
Building Scalable Real-Time Data Pipelines with the Couchbase Kafka Connector...
Building Scalable Real-Time Data Pipelines with the Couchbase Kafka Connector...Building Scalable Real-Time Data Pipelines with the Couchbase Kafka Connector...
Building Scalable Real-Time Data Pipelines with the Couchbase Kafka Connector...HostedbyConfluent
 
Server Sent Events using Reactive Kafka and Spring Web flux | Gagan Solur Ven...
Server Sent Events using Reactive Kafka and Spring Web flux | Gagan Solur Ven...Server Sent Events using Reactive Kafka and Spring Web flux | Gagan Solur Ven...
Server Sent Events using Reactive Kafka and Spring Web flux | Gagan Solur Ven...HostedbyConfluent
 
Digital Transformation in Healthcare with Kafka—Building a Low Latency Data P...
Digital Transformation in Healthcare with Kafka—Building a Low Latency Data P...Digital Transformation in Healthcare with Kafka—Building a Low Latency Data P...
Digital Transformation in Healthcare with Kafka—Building a Low Latency Data P...confluent
 
Transform Your Mainframe and IBM i Data for the Cloud with Precisely and Apac...
Transform Your Mainframe and IBM i Data for the Cloud with Precisely and Apac...Transform Your Mainframe and IBM i Data for the Cloud with Precisely and Apac...
Transform Your Mainframe and IBM i Data for the Cloud with Precisely and Apac...HostedbyConfluent
 
Government Track Welcome Address
Government Track Welcome AddressGovernment Track Welcome Address
Government Track Welcome AddressHostedbyConfluent
 
Feed Your SIEM Smart with Kafka Connect (Vitalii Rudenskyi, McKesson Corp) Ka...
Feed Your SIEM Smart with Kafka Connect (Vitalii Rudenskyi, McKesson Corp) Ka...Feed Your SIEM Smart with Kafka Connect (Vitalii Rudenskyi, McKesson Corp) Ka...
Feed Your SIEM Smart with Kafka Connect (Vitalii Rudenskyi, McKesson Corp) Ka...HostedbyConfluent
 
Kafka Summit NYC 2017 - Achieving Predictability and Compliance with BNY Mell...
Kafka Summit NYC 2017 - Achieving Predictability and Compliance with BNY Mell...Kafka Summit NYC 2017 - Achieving Predictability and Compliance with BNY Mell...
Kafka Summit NYC 2017 - Achieving Predictability and Compliance with BNY Mell...confluent
 
Exposing and Controlling Kafka Event Streaming with Kong Konnect Enterprise |...
Exposing and Controlling Kafka Event Streaming with Kong Konnect Enterprise |...Exposing and Controlling Kafka Event Streaming with Kong Konnect Enterprise |...
Exposing and Controlling Kafka Event Streaming with Kong Konnect Enterprise |...HostedbyConfluent
 
How a distributed graph analytics platform uses Apache Kafka for data ingesti...
How a distributed graph analytics platform uses Apache Kafka for data ingesti...How a distributed graph analytics platform uses Apache Kafka for data ingesti...
How a distributed graph analytics platform uses Apache Kafka for data ingesti...HostedbyConfluent
 
Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl...
Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl...Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl...
Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl...HostedbyConfluent
 

Tendances (20)

Elastically Scaling Kafka Using Confluent
Elastically Scaling Kafka Using ConfluentElastically Scaling Kafka Using Confluent
Elastically Scaling Kafka Using Confluent
 
Event-Driven Architectures Done Right | Tim Berglund, Confluent
Event-Driven Architectures Done Right | Tim Berglund, ConfluentEvent-Driven Architectures Done Right | Tim Berglund, Confluent
Event-Driven Architectures Done Right | Tim Berglund, Confluent
 
Developing custom transformation in the Kafka connect to minimize data redund...
Developing custom transformation in the Kafka connect to minimize data redund...Developing custom transformation in the Kafka connect to minimize data redund...
Developing custom transformation in the Kafka connect to minimize data redund...
 
TBD Data Governance | David Araujo and Michael Agnich, Confluent
TBD Data Governance | David Araujo and Michael Agnich, Confluent TBD Data Governance | David Araujo and Michael Agnich, Confluent
TBD Data Governance | David Araujo and Michael Agnich, Confluent
 
DataOps Automation for a Kafka Streaming Platform (Andrew Stevenson + Spiros ...
DataOps Automation for a Kafka Streaming Platform (Andrew Stevenson + Spiros ...DataOps Automation for a Kafka Streaming Platform (Andrew Stevenson + Spiros ...
DataOps Automation for a Kafka Streaming Platform (Andrew Stevenson + Spiros ...
 
Transformation During a Global Pandemic | Ashish Pandit and Scott Lee, Univer...
Transformation During a Global Pandemic | Ashish Pandit and Scott Lee, Univer...Transformation During a Global Pandemic | Ashish Pandit and Scott Lee, Univer...
Transformation During a Global Pandemic | Ashish Pandit and Scott Lee, Univer...
 
Mesh-ing around with Streams across the Enterprise | Phil Scanlon, Solace
Mesh-ing around with Streams across the Enterprise | Phil Scanlon, SolaceMesh-ing around with Streams across the Enterprise | Phil Scanlon, Solace
Mesh-ing around with Streams across the Enterprise | Phil Scanlon, Solace
 
Data Mess to Data Mesh | Jay Kreps, CEO, Confluent | Kafka Summit Americas 20...
Data Mess to Data Mesh | Jay Kreps, CEO, Confluent | Kafka Summit Americas 20...Data Mess to Data Mesh | Jay Kreps, CEO, Confluent | Kafka Summit Americas 20...
Data Mess to Data Mesh | Jay Kreps, CEO, Confluent | Kafka Summit Americas 20...
 
Kafka & InfluxDB: BFFs for Enterprise Data Applications | Russ Savage, Influx...
Kafka & InfluxDB: BFFs for Enterprise Data Applications | Russ Savage, Influx...Kafka & InfluxDB: BFFs for Enterprise Data Applications | Russ Savage, Influx...
Kafka & InfluxDB: BFFs for Enterprise Data Applications | Russ Savage, Influx...
 
Extracting Value from IOT using Azure Cosmos DB, Azure Synapse Analytics and ...
Extracting Value from IOT using Azure Cosmos DB, Azure Synapse Analytics and ...Extracting Value from IOT using Azure Cosmos DB, Azure Synapse Analytics and ...
Extracting Value from IOT using Azure Cosmos DB, Azure Synapse Analytics and ...
 
Building Scalable Real-Time Data Pipelines with the Couchbase Kafka Connector...
Building Scalable Real-Time Data Pipelines with the Couchbase Kafka Connector...Building Scalable Real-Time Data Pipelines with the Couchbase Kafka Connector...
Building Scalable Real-Time Data Pipelines with the Couchbase Kafka Connector...
 
Server Sent Events using Reactive Kafka and Spring Web flux | Gagan Solur Ven...
Server Sent Events using Reactive Kafka and Spring Web flux | Gagan Solur Ven...Server Sent Events using Reactive Kafka and Spring Web flux | Gagan Solur Ven...
Server Sent Events using Reactive Kafka and Spring Web flux | Gagan Solur Ven...
 
Digital Transformation in Healthcare with Kafka—Building a Low Latency Data P...
Digital Transformation in Healthcare with Kafka—Building a Low Latency Data P...Digital Transformation in Healthcare with Kafka—Building a Low Latency Data P...
Digital Transformation in Healthcare with Kafka—Building a Low Latency Data P...
 
Transform Your Mainframe and IBM i Data for the Cloud with Precisely and Apac...
Transform Your Mainframe and IBM i Data for the Cloud with Precisely and Apac...Transform Your Mainframe and IBM i Data for the Cloud with Precisely and Apac...
Transform Your Mainframe and IBM i Data for the Cloud with Precisely and Apac...
 
Government Track Welcome Address
Government Track Welcome AddressGovernment Track Welcome Address
Government Track Welcome Address
 
Feed Your SIEM Smart with Kafka Connect (Vitalii Rudenskyi, McKesson Corp) Ka...
Feed Your SIEM Smart with Kafka Connect (Vitalii Rudenskyi, McKesson Corp) Ka...Feed Your SIEM Smart with Kafka Connect (Vitalii Rudenskyi, McKesson Corp) Ka...
Feed Your SIEM Smart with Kafka Connect (Vitalii Rudenskyi, McKesson Corp) Ka...
 
Kafka Summit NYC 2017 - Achieving Predictability and Compliance with BNY Mell...
Kafka Summit NYC 2017 - Achieving Predictability and Compliance with BNY Mell...Kafka Summit NYC 2017 - Achieving Predictability and Compliance with BNY Mell...
Kafka Summit NYC 2017 - Achieving Predictability and Compliance with BNY Mell...
 
Exposing and Controlling Kafka Event Streaming with Kong Konnect Enterprise |...
Exposing and Controlling Kafka Event Streaming with Kong Konnect Enterprise |...Exposing and Controlling Kafka Event Streaming with Kong Konnect Enterprise |...
Exposing and Controlling Kafka Event Streaming with Kong Konnect Enterprise |...
 
How a distributed graph analytics platform uses Apache Kafka for data ingesti...
How a distributed graph analytics platform uses Apache Kafka for data ingesti...How a distributed graph analytics platform uses Apache Kafka for data ingesti...
How a distributed graph analytics platform uses Apache Kafka for data ingesti...
 
Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl...
Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl...Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl...
Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl...
 

Similaire à From Legacy SQL Server to High Powered Confluent & Kafka Monitoring System at SunPower | Patrick Franz, SunPower

Smart Electric Meters - Role of Govt. in Technology Management
Smart Electric Meters - Role of Govt. in Technology ManagementSmart Electric Meters - Role of Govt. in Technology Management
Smart Electric Meters - Role of Govt. in Technology ManagementASHOK BHATLA
 
The New Role of Data in the Changing Energy & Utilities Landscape
The New Role of Data in the Changing Energy & Utilities LandscapeThe New Role of Data in the Changing Energy & Utilities Landscape
The New Role of Data in the Changing Energy & Utilities LandscapeDenodo
 
Smart_Meters.pptx
Smart_Meters.pptxSmart_Meters.pptx
Smart_Meters.pptxanugupta210
 
IT Financial Intelligence - How the world’s largest companies are evolving th...
IT Financial Intelligence - How the world’s largest companies are evolving th...IT Financial Intelligence - How the world’s largest companies are evolving th...
IT Financial Intelligence - How the world’s largest companies are evolving th...Brad Swenson
 
Managing the Meter Shop of the Future Through Better Tools and Information
Managing the Meter Shop of the Future Through Better Tools and InformationManaging the Meter Shop of the Future Through Better Tools and Information
Managing the Meter Shop of the Future Through Better Tools and InformationTESCO - The Eastern Specialty Company
 
Microsoft Power & Utilities POV
Microsoft Power & Utilities POVMicrosoft Power & Utilities POV
Microsoft Power & Utilities POVJon Arnold
 
New Uses For Electric Meters as We Continue to Leverage AMI Technology
New Uses For Electric Meters as We Continue to Leverage AMI TechnologyNew Uses For Electric Meters as We Continue to Leverage AMI Technology
New Uses For Electric Meters as We Continue to Leverage AMI TechnologyTESCO - The Eastern Specialty Company
 
Presentation - Mckinsey - Exploring the potential of the “Internet of Things”...
Presentation - Mckinsey - Exploring the potential of the “Internet of Things”...Presentation - Mckinsey - Exploring the potential of the “Internet of Things”...
Presentation - Mckinsey - Exploring the potential of the “Internet of Things”...Ajay Gangakhedkar
 
2016 IDC Pan-European Utilities Summit: Open for Business
2016 IDC Pan-European Utilities Summit: Open for Business2016 IDC Pan-European Utilities Summit: Open for Business
2016 IDC Pan-European Utilities Summit: Open for BusinessOMNETRIC
 
Platforming the Major Analytic Use Cases for Modern Engineering
Platforming the Major Analytic Use Cases for Modern EngineeringPlatforming the Major Analytic Use Cases for Modern Engineering
Platforming the Major Analytic Use Cases for Modern EngineeringDATAVERSITY
 
Redefining-Smart-Grid-Architectural-Thinking-Using-Stream-Computing
Redefining-Smart-Grid-Architectural-Thinking-Using-Stream-ComputingRedefining-Smart-Grid-Architectural-Thinking-Using-Stream-Computing
Redefining-Smart-Grid-Architectural-Thinking-Using-Stream-ComputingAjoy Kumar
 
Switch Automation Case study v2.0
Switch Automation Case study v2.0Switch Automation Case study v2.0
Switch Automation Case study v2.0Oleh Mykhaylovych
 
RAN dimensioning: Lessons learned by Telstra
RAN dimensioning: Lessons learned by TelstraRAN dimensioning: Lessons learned by Telstra
RAN dimensioning: Lessons learned by TelstraWi-Fi 360
 
Upsolver+Snowflake_at_____________Centene.pdf
Upsolver+Snowflake_at_____________Centene.pdfUpsolver+Snowflake_at_____________Centene.pdf
Upsolver+Snowflake_at_____________Centene.pdfsubramani_ts
 

Similaire à From Legacy SQL Server to High Powered Confluent & Kafka Monitoring System at SunPower | Patrick Franz, SunPower (20)

Challenges for Meter Shop Operations of the Future
Challenges for Meter Shop Operations of the FutureChallenges for Meter Shop Operations of the Future
Challenges for Meter Shop Operations of the Future
 
Smart Electric Meters - Role of Govt. in Technology Management
Smart Electric Meters - Role of Govt. in Technology ManagementSmart Electric Meters - Role of Govt. in Technology Management
Smart Electric Meters - Role of Govt. in Technology Management
 
The New Role of Data in the Changing Energy & Utilities Landscape
The New Role of Data in the Changing Energy & Utilities LandscapeThe New Role of Data in the Changing Energy & Utilities Landscape
The New Role of Data in the Changing Energy & Utilities Landscape
 
Smart_Meters.pptx
Smart_Meters.pptxSmart_Meters.pptx
Smart_Meters.pptx
 
IT Financial Intelligence - How the world’s largest companies are evolving th...
IT Financial Intelligence - How the world’s largest companies are evolving th...IT Financial Intelligence - How the world’s largest companies are evolving th...
IT Financial Intelligence - How the world’s largest companies are evolving th...
 
Managing the Meter Shop of the Future Through Better Tools and Information
Managing the Meter Shop of the Future Through Better Tools and InformationManaging the Meter Shop of the Future Through Better Tools and Information
Managing the Meter Shop of the Future Through Better Tools and Information
 
Microsoft Power & Utilities POV
Microsoft Power & Utilities POVMicrosoft Power & Utilities POV
Microsoft Power & Utilities POV
 
New Uses For Electric Meters as We Continue to Leverage AMI Technology
New Uses For Electric Meters as We Continue to Leverage AMI TechnologyNew Uses For Electric Meters as We Continue to Leverage AMI Technology
New Uses For Electric Meters as We Continue to Leverage AMI Technology
 
You Went AMI, Where Will You Go Next_Mid-South
You Went AMI, Where Will You Go Next_Mid-SouthYou Went AMI, Where Will You Go Next_Mid-South
You Went AMI, Where Will You Go Next_Mid-South
 
You Went AMI, Where Will You Go Next?
You Went AMI, Where Will You Go Next?You Went AMI, Where Will You Go Next?
You Went AMI, Where Will You Go Next?
 
Distributed Systems, Mobile Computing and Security
Distributed Systems, Mobile Computing and SecurityDistributed Systems, Mobile Computing and Security
Distributed Systems, Mobile Computing and Security
 
Presentation - Mckinsey - Exploring the potential of the “Internet of Things”...
Presentation - Mckinsey - Exploring the potential of the “Internet of Things”...Presentation - Mckinsey - Exploring the potential of the “Internet of Things”...
Presentation - Mckinsey - Exploring the potential of the “Internet of Things”...
 
You Went AMI Where Will You Go Next - 5-4-2023
You Went AMI Where Will You Go Next - 5-4-2023You Went AMI Where Will You Go Next - 5-4-2023
You Went AMI Where Will You Go Next - 5-4-2023
 
2016 IDC Pan-European Utilities Summit: Open for Business
2016 IDC Pan-European Utilities Summit: Open for Business2016 IDC Pan-European Utilities Summit: Open for Business
2016 IDC Pan-European Utilities Summit: Open for Business
 
Platforming the Major Analytic Use Cases for Modern Engineering
Platforming the Major Analytic Use Cases for Modern EngineeringPlatforming the Major Analytic Use Cases for Modern Engineering
Platforming the Major Analytic Use Cases for Modern Engineering
 
Redefining-Smart-Grid-Architectural-Thinking-Using-Stream-Computing
Redefining-Smart-Grid-Architectural-Thinking-Using-Stream-ComputingRedefining-Smart-Grid-Architectural-Thinking-Using-Stream-Computing
Redefining-Smart-Grid-Architectural-Thinking-Using-Stream-Computing
 
GE-GridIQ_Insight
GE-GridIQ_InsightGE-GridIQ_Insight
GE-GridIQ_Insight
 
Switch Automation Case study v2.0
Switch Automation Case study v2.0Switch Automation Case study v2.0
Switch Automation Case study v2.0
 
RAN dimensioning: Lessons learned by Telstra
RAN dimensioning: Lessons learned by TelstraRAN dimensioning: Lessons learned by Telstra
RAN dimensioning: Lessons learned by Telstra
 
Upsolver+Snowflake_at_____________Centene.pdf
Upsolver+Snowflake_at_____________Centene.pdfUpsolver+Snowflake_at_____________Centene.pdf
Upsolver+Snowflake_at_____________Centene.pdf
 

Plus de HostedbyConfluent

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...HostedbyConfluent
 
Renaming a Kafka Topic | Kafka Summit London
Renaming a Kafka Topic | Kafka Summit LondonRenaming a Kafka Topic | Kafka Summit London
Renaming a Kafka Topic | Kafka Summit LondonHostedbyConfluent
 
Evolution of NRT Data Ingestion Pipeline at Trendyol
Evolution of NRT Data Ingestion Pipeline at TrendyolEvolution of NRT Data Ingestion Pipeline at Trendyol
Evolution of NRT Data Ingestion Pipeline at TrendyolHostedbyConfluent
 
Ensuring Kafka Service Resilience: A Dive into Health-Checking Techniques
Ensuring Kafka Service Resilience: A Dive into Health-Checking TechniquesEnsuring Kafka Service Resilience: A Dive into Health-Checking Techniques
Ensuring Kafka Service Resilience: A Dive into Health-Checking TechniquesHostedbyConfluent
 
Exactly-once Stream Processing with Arroyo and Kafka
Exactly-once Stream Processing with Arroyo and KafkaExactly-once Stream Processing with Arroyo and Kafka
Exactly-once Stream Processing with Arroyo and KafkaHostedbyConfluent
 
Fish Plays Pokemon | Kafka Summit London
Fish Plays Pokemon | Kafka Summit LondonFish Plays Pokemon | Kafka Summit London
Fish Plays Pokemon | Kafka Summit LondonHostedbyConfluent
 
Tiered Storage 101 | Kafla Summit London
Tiered Storage 101 | Kafla Summit LondonTiered Storage 101 | Kafla Summit London
Tiered Storage 101 | Kafla Summit LondonHostedbyConfluent
 
Building a Self-Service Stream Processing Portal: How And Why
Building a Self-Service Stream Processing Portal: How And WhyBuilding a Self-Service Stream Processing Portal: How And Why
Building a Self-Service Stream Processing Portal: How And WhyHostedbyConfluent
 
From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...
From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...
From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...HostedbyConfluent
 
Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...
Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...
Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...HostedbyConfluent
 
Navigating Private Network Connectivity Options for Kafka Clusters
Navigating Private Network Connectivity Options for Kafka ClustersNavigating Private Network Connectivity Options for Kafka Clusters
Navigating Private Network Connectivity Options for Kafka ClustersHostedbyConfluent
 
Apache Flink: Building a Company-wide Self-service Streaming Data Platform
Apache Flink: Building a Company-wide Self-service Streaming Data PlatformApache Flink: Building a Company-wide Self-service Streaming Data Platform
Apache Flink: Building a Company-wide Self-service Streaming Data PlatformHostedbyConfluent
 
Explaining How Real-Time GenAI Works in a Noisy Pub
Explaining How Real-Time GenAI Works in a Noisy PubExplaining How Real-Time GenAI Works in a Noisy Pub
Explaining How Real-Time GenAI Works in a Noisy PubHostedbyConfluent
 
TL;DR Kafka Metrics | Kafka Summit London
TL;DR Kafka Metrics | Kafka Summit LondonTL;DR Kafka Metrics | Kafka Summit London
TL;DR Kafka Metrics | Kafka Summit LondonHostedbyConfluent
 
A Window Into Your Kafka Streams Tasks | KSL
A Window Into Your Kafka Streams Tasks | KSLA Window Into Your Kafka Streams Tasks | KSL
A Window Into Your Kafka Streams Tasks | KSLHostedbyConfluent
 
Mastering Kafka Producer Configs: A Guide to Optimizing Performance
Mastering Kafka Producer Configs: A Guide to Optimizing PerformanceMastering Kafka Producer Configs: A Guide to Optimizing Performance
Mastering Kafka Producer Configs: A Guide to Optimizing PerformanceHostedbyConfluent
 
Data Contracts Management: Schema Registry and Beyond
Data Contracts Management: Schema Registry and BeyondData Contracts Management: Schema Registry and Beyond
Data Contracts Management: Schema Registry and BeyondHostedbyConfluent
 
Code-First Approach: Crafting Efficient Flink Apps
Code-First Approach: Crafting Efficient Flink AppsCode-First Approach: Crafting Efficient Flink Apps
Code-First Approach: Crafting Efficient Flink AppsHostedbyConfluent
 
Debezium vs. the World: An Overview of the CDC Ecosystem
Debezium vs. the World: An Overview of the CDC EcosystemDebezium vs. the World: An Overview of the CDC Ecosystem
Debezium vs. the World: An Overview of the CDC EcosystemHostedbyConfluent
 
Beyond Tiered Storage: Serverless Kafka with No Local Disks
Beyond Tiered Storage: Serverless Kafka with No Local DisksBeyond Tiered Storage: Serverless Kafka with No Local Disks
Beyond Tiered Storage: Serverless Kafka with No Local DisksHostedbyConfluent
 

Plus de HostedbyConfluent (20)

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
 
Renaming a Kafka Topic | Kafka Summit London
Renaming a Kafka Topic | Kafka Summit LondonRenaming a Kafka Topic | Kafka Summit London
Renaming a Kafka Topic | Kafka Summit London
 
Evolution of NRT Data Ingestion Pipeline at Trendyol
Evolution of NRT Data Ingestion Pipeline at TrendyolEvolution of NRT Data Ingestion Pipeline at Trendyol
Evolution of NRT Data Ingestion Pipeline at Trendyol
 
Ensuring Kafka Service Resilience: A Dive into Health-Checking Techniques
Ensuring Kafka Service Resilience: A Dive into Health-Checking TechniquesEnsuring Kafka Service Resilience: A Dive into Health-Checking Techniques
Ensuring Kafka Service Resilience: A Dive into Health-Checking Techniques
 
Exactly-once Stream Processing with Arroyo and Kafka
Exactly-once Stream Processing with Arroyo and KafkaExactly-once Stream Processing with Arroyo and Kafka
Exactly-once Stream Processing with Arroyo and Kafka
 
Fish Plays Pokemon | Kafka Summit London
Fish Plays Pokemon | Kafka Summit LondonFish Plays Pokemon | Kafka Summit London
Fish Plays Pokemon | Kafka Summit London
 
Tiered Storage 101 | Kafla Summit London
Tiered Storage 101 | Kafla Summit LondonTiered Storage 101 | Kafla Summit London
Tiered Storage 101 | Kafla Summit London
 
Building a Self-Service Stream Processing Portal: How And Why
Building a Self-Service Stream Processing Portal: How And WhyBuilding a Self-Service Stream Processing Portal: How And Why
Building a Self-Service Stream Processing Portal: How And Why
 
From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...
From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...
From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...
 
Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...
Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...
Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...
 
Navigating Private Network Connectivity Options for Kafka Clusters
Navigating Private Network Connectivity Options for Kafka ClustersNavigating Private Network Connectivity Options for Kafka Clusters
Navigating Private Network Connectivity Options for Kafka Clusters
 
Apache Flink: Building a Company-wide Self-service Streaming Data Platform
Apache Flink: Building a Company-wide Self-service Streaming Data PlatformApache Flink: Building a Company-wide Self-service Streaming Data Platform
Apache Flink: Building a Company-wide Self-service Streaming Data Platform
 
Explaining How Real-Time GenAI Works in a Noisy Pub
Explaining How Real-Time GenAI Works in a Noisy PubExplaining How Real-Time GenAI Works in a Noisy Pub
Explaining How Real-Time GenAI Works in a Noisy Pub
 
TL;DR Kafka Metrics | Kafka Summit London
TL;DR Kafka Metrics | Kafka Summit LondonTL;DR Kafka Metrics | Kafka Summit London
TL;DR Kafka Metrics | Kafka Summit London
 
A Window Into Your Kafka Streams Tasks | KSL
A Window Into Your Kafka Streams Tasks | KSLA Window Into Your Kafka Streams Tasks | KSL
A Window Into Your Kafka Streams Tasks | KSL
 
Mastering Kafka Producer Configs: A Guide to Optimizing Performance
Mastering Kafka Producer Configs: A Guide to Optimizing PerformanceMastering Kafka Producer Configs: A Guide to Optimizing Performance
Mastering Kafka Producer Configs: A Guide to Optimizing Performance
 
Data Contracts Management: Schema Registry and Beyond
Data Contracts Management: Schema Registry and BeyondData Contracts Management: Schema Registry and Beyond
Data Contracts Management: Schema Registry and Beyond
 
Code-First Approach: Crafting Efficient Flink Apps
Code-First Approach: Crafting Efficient Flink AppsCode-First Approach: Crafting Efficient Flink Apps
Code-First Approach: Crafting Efficient Flink Apps
 
Debezium vs. the World: An Overview of the CDC Ecosystem
Debezium vs. the World: An Overview of the CDC EcosystemDebezium vs. the World: An Overview of the CDC Ecosystem
Debezium vs. the World: An Overview of the CDC Ecosystem
 
Beyond Tiered Storage: Serverless Kafka with No Local Disks
Beyond Tiered Storage: Serverless Kafka with No Local DisksBeyond Tiered Storage: Serverless Kafka with No Local Disks
Beyond Tiered Storage: Serverless Kafka with No Local Disks
 

Dernier

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...apidays
 
Elevate Developer Efficiency & build GenAI Application with Amazon Q​
Elevate Developer Efficiency & build GenAI Application with Amazon Q​Elevate Developer Efficiency & build GenAI Application with Amazon Q​
Elevate Developer Efficiency & build GenAI Application with Amazon Q​Bhuvaneswari Subramani
 
WSO2's API Vision: Unifying Control, Empowering Developers
WSO2's API Vision: Unifying Control, Empowering DevelopersWSO2's API Vision: Unifying Control, Empowering Developers
WSO2's API Vision: Unifying Control, Empowering DevelopersWSO2
 
Corporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptxCorporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptxRustici Software
 
"I see eyes in my soup": How Delivery Hero implemented the safety system for ...
"I see eyes in my soup": How Delivery Hero implemented the safety system for ..."I see eyes in my soup": How Delivery Hero implemented the safety system for ...
"I see eyes in my soup": How Delivery Hero implemented the safety system for ...Zilliz
 
Architecting Cloud Native Applications
Architecting Cloud Native ApplicationsArchitecting Cloud Native Applications
Architecting Cloud Native ApplicationsWSO2
 
Mcleodganj Call Girls 🥰 8617370543 Service Offer VIP Hot Model
Mcleodganj Call Girls 🥰 8617370543 Service Offer VIP Hot ModelMcleodganj Call Girls 🥰 8617370543 Service Offer VIP Hot Model
Mcleodganj Call Girls 🥰 8617370543 Service Offer VIP Hot ModelDeepika Singh
 
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024Victor Rentea
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerThousandEyes
 
Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...apidays
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProduct Anonymous
 
MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024MIND CTI
 
CNIC Information System with Pakdata Cf In Pakistan
CNIC Information System with Pakdata Cf In PakistanCNIC Information System with Pakdata Cf In Pakistan
CNIC Information System with Pakdata Cf In Pakistandanishmna97
 
Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...
Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...
Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...apidays
 
DEV meet-up UiPath Document Understanding May 7 2024 Amsterdam
DEV meet-up UiPath Document Understanding May 7 2024 AmsterdamDEV meet-up UiPath Document Understanding May 7 2024 Amsterdam
DEV meet-up UiPath Document Understanding May 7 2024 AmsterdamUiPathCommunity
 
Artificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : UncertaintyArtificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : UncertaintyKhushali Kathiriya
 
Six Myths about Ontologies: The Basics of Formal Ontology
Six Myths about Ontologies: The Basics of Formal OntologySix Myths about Ontologies: The Basics of Formal Ontology
Six Myths about Ontologies: The Basics of Formal Ontologyjohnbeverley2021
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FMESafe Software
 
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWEREMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWERMadyBayot
 

Dernier (20)

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
 
Elevate Developer Efficiency & build GenAI Application with Amazon Q​
Elevate Developer Efficiency & build GenAI Application with Amazon Q​Elevate Developer Efficiency & build GenAI Application with Amazon Q​
Elevate Developer Efficiency & build GenAI Application with Amazon Q​
 
WSO2's API Vision: Unifying Control, Empowering Developers
WSO2's API Vision: Unifying Control, Empowering DevelopersWSO2's API Vision: Unifying Control, Empowering Developers
WSO2's API Vision: Unifying Control, Empowering Developers
 
Corporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptxCorporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptx
 
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
 
"I see eyes in my soup": How Delivery Hero implemented the safety system for ...
"I see eyes in my soup": How Delivery Hero implemented the safety system for ..."I see eyes in my soup": How Delivery Hero implemented the safety system for ...
"I see eyes in my soup": How Delivery Hero implemented the safety system for ...
 
Architecting Cloud Native Applications
Architecting Cloud Native ApplicationsArchitecting Cloud Native Applications
Architecting Cloud Native Applications
 
Mcleodganj Call Girls 🥰 8617370543 Service Offer VIP Hot Model
Mcleodganj Call Girls 🥰 8617370543 Service Offer VIP Hot ModelMcleodganj Call Girls 🥰 8617370543 Service Offer VIP Hot Model
Mcleodganj Call Girls 🥰 8617370543 Service Offer VIP Hot Model
 
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
 
MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024
 
CNIC Information System with Pakdata Cf In Pakistan
CNIC Information System with Pakdata Cf In PakistanCNIC Information System with Pakdata Cf In Pakistan
CNIC Information System with Pakdata Cf In Pakistan
 
Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...
Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...
Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...
 
DEV meet-up UiPath Document Understanding May 7 2024 Amsterdam
DEV meet-up UiPath Document Understanding May 7 2024 AmsterdamDEV meet-up UiPath Document Understanding May 7 2024 Amsterdam
DEV meet-up UiPath Document Understanding May 7 2024 Amsterdam
 
Artificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : UncertaintyArtificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : Uncertainty
 
Six Myths about Ontologies: The Basics of Formal Ontology
Six Myths about Ontologies: The Basics of Formal OntologySix Myths about Ontologies: The Basics of Formal Ontology
Six Myths about Ontologies: The Basics of Formal Ontology
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
 
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWEREMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
 

From Legacy SQL Server to High Powered Confluent & Kafka Monitoring System at SunPower | Patrick Franz, SunPower

  • 1. © 2021 SunPower Corporation From Legacy SQL Server to High-Powered Confluent & Kafka Monitoring System Patrick Franz SEPTEMBER 2021 Senior Director, Engineering
  • 2. 2 OUR MISSION Changing the Way Our World is Powered We do this by building a great company, putting our customers first, and making solar the most compelling energy option. If we empower our people and communities to thrive, we’ll build a sustainable future together.
  • 3. 3 MONITORING Why Monitoring Matters • SunPower has approximately 6M devices in the field sending data, on average, every 5 minutes. • We offer a 20-year guarantee for lease customers that the system will produce a target kWh/year. • Example impacts on performance: weather, shading, or component faults. • We want to understand these challenges with data.
  • 4. 4 MONITORING Delivering Customer Value • How much data do we need to predict and detect performance issues before they impact the customer? • How do we collect this data and act on it quickly? • Fundamentals • Measurement ingestion • Metadata association • Storage • Analysis and presentation
  • 5. 5 Early Solar Adopters MONITORING • In 2008 we had this idea that customers wanted to know how much energy their system is producing. • Supported SunPower’s residential and commercial customers successfully for a decade. • SunPower’s customer base grew, average solar installation size increased, and the industry transitioned to microinverters.
  • 6. 6 Customer Value At Scale MONITORING • By 2018 we knew we needed to support modern IoT devices on our platform. • Customers’ energy habits were changing (EV adoption, smart appliances) and policy shifts in favor of renewable energy drove increased adoption of solar and storage. • Took an engineering first approach and designed new fundamentals with an emphasis on customer experience. IoT Sensor Data Confluent Cloud Customer Site and Device Metadata Time-series API Data Warehouse Weather Data, Utility Rates, etc.
  • 7. 7 MONITORING Why Kafka? • Sensor data alone is not valuable. How do we connect a solar panel at an address to a sensor reading? • Kafka is the central nervous system that allows us to blend multiple data sources together and generate value from the raw sensor data. • Engineering teams get access to fully-enriched data via Kafka and build the features important to them.
  • 8. 8 CASE STUDY Panel-Level Monitoring • With launch of the mySunPower™ app we continue to iterate on customer experience. • Panel-level monitoring addresses the earlier need of what data we collect and how we use it to solve real customer issues. • Quickly identify underperforming components and determine appropriate resolution. Feature planned for release in Q3.
  • 9. 9 MONITORING We Are Hiring • Help us change the way the world is powered with Kafka. • patrick.franz@sunpower.com • LinkedIn
  • 10. Changing the way our world is powered THANK YOU © 2021 SunPower Corporation