Case Study: bet365 Improves Its Odds for Site Reliability With Unified Monitoring and Contextual Log Analytics

Case Study: bet365 Improves Its Odds for Site
Reliability With Unified Monitoring and Contextual
Log Analytics
Scott McKenzie
DO2T40S
DEVOPS-AGILE OPERATIONS
Infrastructure Services Manager
bet365

2 COPYRIGHT © 2017 CA. ALL RIGHTS RESERVED#CAWORLD #NOBARRIERS
© 2017 CA. All rights reserved. All trademarks referenced herein belong to their respective companies.
The content provided in this CA World 2017 presentation is intended for informational purposes only and does not form any type
of warranty. The information provided by a CA partner and/or CA customer has not been reviewed for accuracy by CA.
For Informational Purposes Only
Terms of This Presentation

Abstract
bet365 is one of the world’s largest online gambling companies. The company’s IT
infrastructure powers the experience for more than 22 million customers worldwide.
Whether a router's CPU is running hot, or a third-party payments provider is having a
bad day, members of the IT operations team at bet365 are the first responders, and
they're expected to respond quickly.
Discover how bet365's unique use of CA Unified Infrastructure Management (CA UIM)
has evolved over the past five years to give it the edge in reducing incident response
and recovery time. The bet365 team will also share how it is planning to leverage log
analytics in conjunction with CA UIM to speed mean time to repair.
Scott
McKenzie
bet365
Infrastructure
Services Manager

Agenda
WHO AM I?
WHO IS BET365?
NEXT STEPS
WHY DOES BET365 USE CA UNIFIED INFRASTRUCTURE MANAGEMENT (CA UIM)?
HOW DOES BET365 USE CA UIM?
WHAT HAVE WE LEARNED?
1
2
3
4
5
6

Who Am I?
1997-2001
Computer Science with Industrial
Experience
2001-2006
Service Delivery Manager, IBM Global
Services
2006-2011
Service Delivery/IT Operations Manager
(Started using CA UIM)
2011-Current
Infrastructure Services Manager
- IT Ops, Business Continuity, Service Design

Who Is bet365?
▪ “The world’s favourite online sports betting company.”
– Online sports betting + gaming
– Over 23 million registered customers
– Employing over 3,500 staff in multiple countries
▪ 2017 results:
– £46.9bn in Sports bets (+27%)
– £2.15bn revenue (+39%)
– £503.9m pre-tax profit (+9.8%)
The Business

Cost of Downtime Per Second at bet365
£68.18
Revenue
£15.98
Profit
£1,487
Sports Bets

Who Is bet365?
▪ In-play
– 72% of all sports bets
▪ Live streaming
– 140,000 events/year
▪ Cash out
▪ Edit bet
▪ Casino, poker, games and bingo
The Products

Who Is bet365?
The Technologies

Why Does bet365 Use CA UIM?
Before CA UIM (2011)
SNMP/Syslog: ~ 1 min
SMTP: 1-5 mins

Why Does bet365 Use CA UIM?
▪ Service requirements
– Fit-for-purpose alarm workflow
– Custom dashboard engine
▪ Technical requirements
– Comprehensive, modern technology
coverage
– On-premise hosting + SQL Server
database
– Custom integration with other systems
Product Selection

How Does bet365 Use CA UIM?
▪ Architecture
▪ Standard monitoring
▪ Custom monitoring
▪ Custom dashboards
▪ Integration

CA UIM at bet365
Architecture Designed for Scale and Resilience

CA UIM at bet365
Standard Allows for Immediate Incident Response
Alarm Severity IT Operations Action BMC Remedy SLA
Critical Immediate escalation Critical
Major Immediate escalation High
Minor Assign ticket to relevant team Medium
Warning Assign ticket to relevant team Low
Informational None None

CA UIM at bet365
Flexible Monitoring for Microsoft® Windows® Systems and Services
CPU, Disk,
Memory
Event Log Perf Metrics Services Processes Cluster
Windows
SQL Server
Windows Cluster
Active Directory
DHCP
Standard
Enhanced
Standard Standard Standard
Enhanced Enhanced Standard
Standard Standard Standard Standard Standard
Standard Enhanced Enhanced Enhanced
Standard Enhanced Standard Enhanced
Enhanced

CA UIM at bet365
Standard monitoring packages for key technologies provide immediate value

CA UIM at bet365
▪ Citrix® NetScaler®
– bet365 choice for load balancing
– Hundreds of servers
– Multiple locations for resilience
– Complex config
– Simple monitoring?
Custom Monitoring (Service Groups) Meets Unique Needs

CA UIM at bet365
▪ bet365 solution
– Custom Java probe written
in-house
– Interrogates NetScaler via
SNMP
– Returns alarms + QoS
Custom Monitoring – Netscaler Service Groups

CA UIM at bet365
▪ Specific use case based on bet365 service standards
▪ bet365 solution:
– Custom Java probe written by bet365 to check running state
Custom Monitoring – Active/Standby Services
Primary Secondary Alarm Response
Running Stopped Informational N/A - Normal running state.
Stopped Running Informational Confirm with service owner during regular checks.
Stopped Stopped Major Attempt to start on Primary server + Escalate.
Running Running Major Attempt to stop on Secondary server + Escalate.

CA UIM at bet365
▪ CA UIM system health
▪ Alarm propagation
▪ HTML5 dashboards
Custom Dashboards Provide Unique Insights

CA UIM at bet365
Custom Dashboards – System Health

CA UIM at bet365
Custom Dashboards – Alarm Propagation

CA UIM at bet365
Custom Dashboards – HTML5 – Mobile Friendly

CA UIM at bet365
Integration – CA App Synthetic Monitor

CA UIM at bet365
▪ BMC Remedy gateway probe running
on CA UIM hub
▪ URL action in USM makes 2 API calls:
1. Open INC ticket with CA UIM-to-
Remedy field mappings
2. Return INC number to CA UIM
Integration – BMC Remedy

CA UIM at bet365
Integration – Alarm URL Actions Enable Rapid Issue Resolution

What Have We Learned?
1. Document and agree monitoring configuration
2. Build monitoring config into deployment process
3. Increase focus on servers running custom probes
4. Control the addition of new QoS metrics
5. Appoint a dedicated team to manage CA UIM
Top Five Tips

What’s Next?
▪ Prerequisites:
– CA UIM 8.5.1
– CA Digital Operational Intelligence
– Data Studio (Kibana)
– Kafka/Zookeeper
– Elasticsearch
– PostgreSQL (or Oracle)
▪ Initial use case:
– Active Directory re-architecture
Log Analytics in Conjunction With CA Digital Operational Intelligence

What’s Next?
▪ Capacity Predictive Analytics (CA Digital Operational Intelligence)
– Deploying pre-release to test environment
▪ CA UIM certification
– v8.5 courses and exams
▪ Ongoing improvements
– Rewrite custom probes to use QoS for status instead of alarms
– Auto-provisioning of config during automated deployment process

Summary Benefits With CA UIM
ENHANCED
VISIBILITY ACROSS
HYBRID IT
SUPERIOR END-
USER EXPERIENCE
FASTER IT
OPERATIONS
FASTER ISSUE
RESOLUTION
WORKFLOWS
Reduced
number of
tools from
5 to 1
IMPROVED
INCIDENT
RESPONSE
FUTURE PROOF
MONITORING
APPROACH

Questions?

Stay connected at communities.ca.com
Thank you.

DevSecOps
For more information on DevSecOps,
please visit: http://cainc.to/CAW17-DevSecOps

Case Study: bet365 Improves Its Odds for Site Reliability With Unified Monitoring and Contextual Log Analytics

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Case Study: bet365 Improves Its Odds for Site Reliability With Unified Monitoring and Contextual Log Analytics

Similaire à Case Study: bet365 Improves Its Odds for Site Reliability With Unified Monitoring and Contextual Log Analytics (20)

Plus de CA Technologies

Plus de CA Technologies (20)

Dernier

Dernier (20)

Case Study: bet365 Improves Its Odds for Site Reliability With Unified Monitoring and Contextual Log Analytics