SlideShare une entreprise Scribd logo
1  sur  18
Real-time Event
Processing using
LINQ for Logs &
Traces (TX)
Gert Drapers (#DataDude)
Principle Software Design Engineer
Agenda
•Rx (Reactive Extensions)
•Tx (LINQ to Logs and Traces)
•Case Study
Subtitel
TX (LINQ to Logs and Traces)
•http://tx.codeplex.com/
•TX allows you use LINQ
directly on raw event
sources:
• Ad-hoc query on past history in
trace and log files
• Standing queries on real-time
feeds, such as Event Tracing for
Windows (ETW) sessions
Builds on Reactive Extensions (RX.Net)
http://rx.codeplex.com/
Reactive Essential Interfaces
namespace System {
// Producers (.NET events, WinRT Events, Sensors, APM method etc.)
public interface IObservable<out T> {
IDisposable Subscribe(IObserver<T> observer);
}
// Events stream consumers
public interface IObserver<in T> {
void OnNext(T value);
void OnError(Exception error);
void OnCompleted();
}
}
Event processing at all
scales with Reactive
ExtensionsBart de Smet
Thrusday 17 april 13:15 - 14:30
Demo
Using TX with LINQPad
Event Sources
• Tx.Windows
• Event Tracing for Windows (ETW)
• Performance Counters - interop with the "Performance Data
Helper" (PDH) native API
• Event Logs (.evtx)
• IIS W3C text log files
• Tx.SqlServer
• SQL Server Extended Events (XEvent) is the tracing technology
used by the SQL-Server engine.
• Extensibility
Type Generation
Playback
•Structured Mode
• Lose analogy is that events of given type are like a "Table“
• playback.GetObservable< T> is similar to dataContext.GetTable<T>
in LINQ to SQL
• More precisely, IObservable is a sequence (i.e. order matters)
• This is the same distinction as IEnumerable vs.. SQL Table
•Timeline Mode
• This mode is useful to ask questions that are about the timeline
of all events, regardless of type
Playback Features
•Multiplexed sequences
•Hiding the heterogeneity of the event sources
•Same API for real-time and past history
•Single-Read to answer multiple queries from
file(s)
•Occurrence Time Scheduler
Demo
HttpSpew
Case Study XblWatch
•Objective of XblWatch:
• Replace existing rule based SCOM pack monitoring of web
service
• Provide near-realtime monitoring
• Create Synthetic Performance Counters for success and failure
rates per API and service
• Publish Synthetic Performance Counters to integrate with
existing monitoring and alerting system
XblWatch/er
xblwatcher.prod.live
XblWatcher.exe XblWatchCmd.exe
http://*:80
tcp://*:9000
Perf Counters
Site/Service/API
Machine 1
XblWatch.exe
http.sys ETW trace
Perf Counters
Machine/Service/API
Machine 2
XblWatch.exe
http.sys ETW trace
Perf Counters
Machine/Service/API
Machine 3
XblWatch.exe
http.sys ETW trace
Perf Counter
Machine/Service/API
Machine N
XblWatch.exe
http.sys ETW trace
Perf Counters
Machine/Service/API
Demo
XblWatch
Resources
•Tx (LINQ to Logs and Traces)
•(Rx) Reactive Extensions
•Event processing at all scales with Reactive
Extensions
Bart de Smet
Thrusday 17 april 13:15 - 14:30
Laat ons weten wat u vindt van deze sessie! Vul de evaluatie
in via www.techdaysapp.nl en maak kans op een van de 20
prijzen*. Prijswinnaars worden bekend gemaakt via Twitter
(#TechDaysNL). Gebruik hiervoor de code op uw badge.
Let us know how you feel about this session! Give your
feedback via www.techdaysapp.nl and possibly win one of
the 20 prices*. Winners will be announced via Twitter
(#TechDaysNL). Use your personal code on your badge.
* Over de uitslag kan niet worden gecorrespondeerd, prijzen zijn voorbeelden – All results are final, prices are
examples
Real-time Event Processing using LINQ for Logs & Traces

Contenu connexe

Dernier

+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
Health
 
%+27788225528 love spells in Huntington Beach Psychic Readings, Attraction sp...
%+27788225528 love spells in Huntington Beach Psychic Readings, Attraction sp...%+27788225528 love spells in Huntington Beach Psychic Readings, Attraction sp...
%+27788225528 love spells in Huntington Beach Psychic Readings, Attraction sp...
masabamasaba
 
%+27788225528 love spells in Atlanta Psychic Readings, Attraction spells,Brin...
%+27788225528 love spells in Atlanta Psychic Readings, Attraction spells,Brin...%+27788225528 love spells in Atlanta Psychic Readings, Attraction spells,Brin...
%+27788225528 love spells in Atlanta Psychic Readings, Attraction spells,Brin...
masabamasaba
 
%+27788225528 love spells in Knoxville Psychic Readings, Attraction spells,Br...
%+27788225528 love spells in Knoxville Psychic Readings, Attraction spells,Br...%+27788225528 love spells in Knoxville Psychic Readings, Attraction spells,Br...
%+27788225528 love spells in Knoxville Psychic Readings, Attraction spells,Br...
masabamasaba
 
Love witchcraft +27768521739 Binding love spell in Sandy Springs, GA |psychic...
Love witchcraft +27768521739 Binding love spell in Sandy Springs, GA |psychic...Love witchcraft +27768521739 Binding love spell in Sandy Springs, GA |psychic...
Love witchcraft +27768521739 Binding love spell in Sandy Springs, GA |psychic...
chiefasafspells
 
%+27788225528 love spells in Colorado Springs Psychic Readings, Attraction sp...
%+27788225528 love spells in Colorado Springs Psychic Readings, Attraction sp...%+27788225528 love spells in Colorado Springs Psychic Readings, Attraction sp...
%+27788225528 love spells in Colorado Springs Psychic Readings, Attraction sp...
masabamasaba
 

Dernier (20)

+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
 
WSO2CON 2024 - Navigating API Complexity: REST, GraphQL, gRPC, Websocket, Web...
WSO2CON 2024 - Navigating API Complexity: REST, GraphQL, gRPC, Websocket, Web...WSO2CON 2024 - Navigating API Complexity: REST, GraphQL, gRPC, Websocket, Web...
WSO2CON 2024 - Navigating API Complexity: REST, GraphQL, gRPC, Websocket, Web...
 
%in Midrand+277-882-255-28 abortion pills for sale in midrand
%in Midrand+277-882-255-28 abortion pills for sale in midrand%in Midrand+277-882-255-28 abortion pills for sale in midrand
%in Midrand+277-882-255-28 abortion pills for sale in midrand
 
WSO2Con2024 - From Code To Cloud: Fast Track Your Cloud Native Journey with C...
WSO2Con2024 - From Code To Cloud: Fast Track Your Cloud Native Journey with C...WSO2Con2024 - From Code To Cloud: Fast Track Your Cloud Native Journey with C...
WSO2Con2024 - From Code To Cloud: Fast Track Your Cloud Native Journey with C...
 
%+27788225528 love spells in Huntington Beach Psychic Readings, Attraction sp...
%+27788225528 love spells in Huntington Beach Psychic Readings, Attraction sp...%+27788225528 love spells in Huntington Beach Psychic Readings, Attraction sp...
%+27788225528 love spells in Huntington Beach Psychic Readings, Attraction sp...
 
tonesoftg
tonesoftgtonesoftg
tonesoftg
 
WSO2CON2024 - It's time to go Platformless
WSO2CON2024 - It's time to go PlatformlessWSO2CON2024 - It's time to go Platformless
WSO2CON2024 - It's time to go Platformless
 
WSO2CON 2024 Slides - Open Source to SaaS
WSO2CON 2024 Slides - Open Source to SaaSWSO2CON 2024 Slides - Open Source to SaaS
WSO2CON 2024 Slides - Open Source to SaaS
 
OpenChain - The Ramifications of ISO/IEC 5230 and ISO/IEC 18974 for Legal Pro...
OpenChain - The Ramifications of ISO/IEC 5230 and ISO/IEC 18974 for Legal Pro...OpenChain - The Ramifications of ISO/IEC 5230 and ISO/IEC 18974 for Legal Pro...
OpenChain - The Ramifications of ISO/IEC 5230 and ISO/IEC 18974 for Legal Pro...
 
WSO2CON 2024 - Cloud Native Middleware: Domain-Driven Design, Cell-Based Arch...
WSO2CON 2024 - Cloud Native Middleware: Domain-Driven Design, Cell-Based Arch...WSO2CON 2024 - Cloud Native Middleware: Domain-Driven Design, Cell-Based Arch...
WSO2CON 2024 - Cloud Native Middleware: Domain-Driven Design, Cell-Based Arch...
 
%+27788225528 love spells in Atlanta Psychic Readings, Attraction spells,Brin...
%+27788225528 love spells in Atlanta Psychic Readings, Attraction spells,Brin...%+27788225528 love spells in Atlanta Psychic Readings, Attraction spells,Brin...
%+27788225528 love spells in Atlanta Psychic Readings, Attraction spells,Brin...
 
%in Benoni+277-882-255-28 abortion pills for sale in Benoni
%in Benoni+277-882-255-28 abortion pills for sale in Benoni%in Benoni+277-882-255-28 abortion pills for sale in Benoni
%in Benoni+277-882-255-28 abortion pills for sale in Benoni
 
%+27788225528 love spells in Knoxville Psychic Readings, Attraction spells,Br...
%+27788225528 love spells in Knoxville Psychic Readings, Attraction spells,Br...%+27788225528 love spells in Knoxville Psychic Readings, Attraction spells,Br...
%+27788225528 love spells in Knoxville Psychic Readings, Attraction spells,Br...
 
Architecture decision records - How not to get lost in the past
Architecture decision records - How not to get lost in the pastArchitecture decision records - How not to get lost in the past
Architecture decision records - How not to get lost in the past
 
Love witchcraft +27768521739 Binding love spell in Sandy Springs, GA |psychic...
Love witchcraft +27768521739 Binding love spell in Sandy Springs, GA |psychic...Love witchcraft +27768521739 Binding love spell in Sandy Springs, GA |psychic...
Love witchcraft +27768521739 Binding love spell in Sandy Springs, GA |psychic...
 
Devoxx UK 2024 - Going serverless with Quarkus, GraalVM native images and AWS...
Devoxx UK 2024 - Going serverless with Quarkus, GraalVM native images and AWS...Devoxx UK 2024 - Going serverless with Quarkus, GraalVM native images and AWS...
Devoxx UK 2024 - Going serverless with Quarkus, GraalVM native images and AWS...
 
%+27788225528 love spells in Colorado Springs Psychic Readings, Attraction sp...
%+27788225528 love spells in Colorado Springs Psychic Readings, Attraction sp...%+27788225528 love spells in Colorado Springs Psychic Readings, Attraction sp...
%+27788225528 love spells in Colorado Springs Psychic Readings, Attraction sp...
 
%in Bahrain+277-882-255-28 abortion pills for sale in Bahrain
%in Bahrain+277-882-255-28 abortion pills for sale in Bahrain%in Bahrain+277-882-255-28 abortion pills for sale in Bahrain
%in Bahrain+277-882-255-28 abortion pills for sale in Bahrain
 
What Goes Wrong with Language Definitions and How to Improve the Situation
What Goes Wrong with Language Definitions and How to Improve the SituationWhat Goes Wrong with Language Definitions and How to Improve the Situation
What Goes Wrong with Language Definitions and How to Improve the Situation
 
%in ivory park+277-882-255-28 abortion pills for sale in ivory park
%in ivory park+277-882-255-28 abortion pills for sale in ivory park %in ivory park+277-882-255-28 abortion pills for sale in ivory park
%in ivory park+277-882-255-28 abortion pills for sale in ivory park
 

En vedette

How Race, Age and Gender Shape Attitudes Towards Mental Health
How Race, Age and Gender Shape Attitudes Towards Mental HealthHow Race, Age and Gender Shape Attitudes Towards Mental Health
How Race, Age and Gender Shape Attitudes Towards Mental Health
ThinkNow
 
Social Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie InsightsSocial Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie Insights
Kurio // The Social Media Age(ncy)
 

En vedette (20)

Product Design Trends in 2024 | Teenage Engineerings
Product Design Trends in 2024 | Teenage EngineeringsProduct Design Trends in 2024 | Teenage Engineerings
Product Design Trends in 2024 | Teenage Engineerings
 
How Race, Age and Gender Shape Attitudes Towards Mental Health
How Race, Age and Gender Shape Attitudes Towards Mental HealthHow Race, Age and Gender Shape Attitudes Towards Mental Health
How Race, Age and Gender Shape Attitudes Towards Mental Health
 
AI Trends in Creative Operations 2024 by Artwork Flow.pdf
AI Trends in Creative Operations 2024 by Artwork Flow.pdfAI Trends in Creative Operations 2024 by Artwork Flow.pdf
AI Trends in Creative Operations 2024 by Artwork Flow.pdf
 
Skeleton Culture Code
Skeleton Culture CodeSkeleton Culture Code
Skeleton Culture Code
 
PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024
 
Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)
 
How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024
 
Social Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie InsightsSocial Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie Insights
 
Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024
 
5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary
 
ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd
 
Getting into the tech field. what next
Getting into the tech field. what next Getting into the tech field. what next
Getting into the tech field. what next
 
Google's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search IntentGoogle's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search Intent
 
How to have difficult conversations
How to have difficult conversations How to have difficult conversations
How to have difficult conversations
 
Introduction to Data Science
Introduction to Data ScienceIntroduction to Data Science
Introduction to Data Science
 
Time Management & Productivity - Best Practices
Time Management & Productivity -  Best PracticesTime Management & Productivity -  Best Practices
Time Management & Productivity - Best Practices
 
The six step guide to practical project management
The six step guide to practical project managementThe six step guide to practical project management
The six step guide to practical project management
 
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
 
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
 
12 Ways to Increase Your Influence at Work
12 Ways to Increase Your Influence at Work12 Ways to Increase Your Influence at Work
12 Ways to Increase Your Influence at Work
 

Real-time Event Processing using LINQ for Logs & Traces

  • 1. Real-time Event Processing using LINQ for Logs & Traces (TX) Gert Drapers (#DataDude) Principle Software Design Engineer
  • 2. Agenda •Rx (Reactive Extensions) •Tx (LINQ to Logs and Traces) •Case Study Subtitel
  • 3. TX (LINQ to Logs and Traces) •http://tx.codeplex.com/ •TX allows you use LINQ directly on raw event sources: • Ad-hoc query on past history in trace and log files • Standing queries on real-time feeds, such as Event Tracing for Windows (ETW) sessions
  • 4. Builds on Reactive Extensions (RX.Net) http://rx.codeplex.com/
  • 5. Reactive Essential Interfaces namespace System { // Producers (.NET events, WinRT Events, Sensors, APM method etc.) public interface IObservable<out T> { IDisposable Subscribe(IObserver<T> observer); } // Events stream consumers public interface IObserver<in T> { void OnNext(T value); void OnError(Exception error); void OnCompleted(); } }
  • 6. Event processing at all scales with Reactive ExtensionsBart de Smet Thrusday 17 april 13:15 - 14:30
  • 8. Event Sources • Tx.Windows • Event Tracing for Windows (ETW) • Performance Counters - interop with the "Performance Data Helper" (PDH) native API • Event Logs (.evtx) • IIS W3C text log files • Tx.SqlServer • SQL Server Extended Events (XEvent) is the tracing technology used by the SQL-Server engine. • Extensibility
  • 10. Playback •Structured Mode • Lose analogy is that events of given type are like a "Table“ • playback.GetObservable< T> is similar to dataContext.GetTable<T> in LINQ to SQL • More precisely, IObservable is a sequence (i.e. order matters) • This is the same distinction as IEnumerable vs.. SQL Table •Timeline Mode • This mode is useful to ask questions that are about the timeline of all events, regardless of type
  • 11. Playback Features •Multiplexed sequences •Hiding the heterogeneity of the event sources •Same API for real-time and past history •Single-Read to answer multiple queries from file(s) •Occurrence Time Scheduler
  • 13. Case Study XblWatch •Objective of XblWatch: • Replace existing rule based SCOM pack monitoring of web service • Provide near-realtime monitoring • Create Synthetic Performance Counters for success and failure rates per API and service • Publish Synthetic Performance Counters to integrate with existing monitoring and alerting system
  • 14. XblWatch/er xblwatcher.prod.live XblWatcher.exe XblWatchCmd.exe http://*:80 tcp://*:9000 Perf Counters Site/Service/API Machine 1 XblWatch.exe http.sys ETW trace Perf Counters Machine/Service/API Machine 2 XblWatch.exe http.sys ETW trace Perf Counters Machine/Service/API Machine 3 XblWatch.exe http.sys ETW trace Perf Counter Machine/Service/API Machine N XblWatch.exe http.sys ETW trace Perf Counters Machine/Service/API
  • 16. Resources •Tx (LINQ to Logs and Traces) •(Rx) Reactive Extensions •Event processing at all scales with Reactive Extensions Bart de Smet Thrusday 17 april 13:15 - 14:30
  • 17. Laat ons weten wat u vindt van deze sessie! Vul de evaluatie in via www.techdaysapp.nl en maak kans op een van de 20 prijzen*. Prijswinnaars worden bekend gemaakt via Twitter (#TechDaysNL). Gebruik hiervoor de code op uw badge. Let us know how you feel about this session! Give your feedback via www.techdaysapp.nl and possibly win one of the 20 prices*. Winners will be announced via Twitter (#TechDaysNL). Use your personal code on your badge. * Over de uitslag kan niet worden gecorrespondeerd, prijzen zijn voorbeelden – All results are final, prices are examples