(ARC346) Scaling To 25 Billion Daily Requests Within 3 Months On AWS
03-NOV-1510-Ognjen-Antonic-Telemach-stream-1
1. Unleashing the power of telco
analytics using
Open Big Data Data Exploration &
Visualization Technologies
Ognjen Antonic, DWH & BI Manager
Telemach Slovenia, Member of United Group
ognjen.antonic@telemach.si
www.telemach.si www.the-united-group.com
2. 2
About Telemach Slovenia
- Leading alternative telecom operator in Slovenia
- Acquired Tusmobil, 3rd largest and fastest growing mobile
operator in Slovenia during April 2015, now fully
integrated operation of Telemach
- 70% cable TV market share
- 30% IP telephony market share
- 24% fixed broadband internet access market share
- 14% mobile market share
3. 3
About United Group
- Largest alternative telecom provider in the region of
former Yugoslavia
- Main operating companies:
- SBB (leading cable operator in Serbia)
- Telemach Slovenia (leading cable operator and fastest
growing mobile operator in Slovenia)
- Telemach Bosnia
- Telemach Montenegro
- Main business segments:
- Telecommunication platforms
- Media (Content & Advertising)
5. 5
About United Group
• Pay TV
• Broadband
• Fixed telephony
• Mobile telephony
• CuTV
• VOD
• D3Go
• HD
• Premium content (HBO,...)
Products and Services, Business & Residential
• Content rights (Premier
League, UEFA Europa
League, UEFA
Champions League,
NBA, WTA, Primera
Division, ATP World
Tour, Euroleague
Basketball…)
7. 7
The Quest for Big Data – How it all started
- Mobile Fraud
- International Roaming Fraud – Own Subs Abroad
- International Roaming Fraud – Foreign Visitors in our
Network
- Interconnect Fraud
- Multiple different Attack Vectors used
- High velocity of occurrences
8. 8
The Quest for Big Data – Requirements
- Answering fundamental questions
- Who?
- Where?
- When?
- How?
- Giving Self Service Analytical Ability of find answers
through Data Discovery & Visualization
9. 9
The Quest for Big Data – Challenges
- IT Challenges
- Small Team
- Limited Budget
- Reusing current IT infrastructure in-place to the
maximum level
- Leveraging and combining current multiple data
sources
10. 10
The Quest for Big Data – Challenges
- Old IT Systems Challenges
- Fully Based on conventional relational database
technologies for data layer – data warehouse
- Reporting and analysis based on conventional
Business Intelligence tools
- Not designed for real-time or near-real-time capture
and processing of large amounts of data for analytical
purposes
- Data discovery & visualization not primary focus of
conventional Business Intelligence tools
11. 11
Big Data Solution Requirements
- Easy to deploy, develop and manage
- Should scale both horizontally & vertically
- CPU/processing power wise
- Data storage wise
- Should cover whole stack
- Data Storage & Retreival
- Data Integration
- Data Presentation
- Actively Developed with Good Documentation
- Should be Free or Low-Cost Licence wise
12. 12
Big Data Solutions Evaluated
- Splunk
- Very good commercial solution, covering full stack,
easy to deploy and manage
- Free only up to 500 MB data per day
- Gets very expensive with larger amounts of data
processed daily
- Apache Hadoop
- Free licence-wise, offering extreme scalability
- Not covering full stack in pre-integrated way, Data
Presentation layer poor in features for self service
- Difficult to deploy and manage
- Better suited for companies with larger IT teams
13. 13
Settling for the ELK Stack
- Settled for ELK Stack, fully integrated with:
- Elasticsearch – Data Storage Layer
- Logstash - Data Integration Layer
- Kibana – Data Presentation Layer
- Open Source Based & Free Licensing
- Actively Developed with Good Documentation
- Scalable and easy to deploy and manage
- Enterprise Level Commercial Support Available
14. 14
Technology behind Elasticsearch
- NoSQL Data Storage, Indexing and Data Retreival
Engine, derivative of Apache Lucene Open Source
Project, written fully in Java
- Built on following fundamental technological enablers:
- Data Sharding and Massively Parallel Processing over
cluster of inexpensive servers using Shared Nothing
Data Architecture
- Data Compression & In-Memory Storage
- Probabilistic Computation Models (possibility to trade
speed for accuracy or vice versa)
- Lots of different free & open source plugins and addons
available
15. 15
Developing & Deploying our First Solution
- International Roaming Fraud Dashboard
- Highest Risk Area – Our customers abroad
- Giving ability to Fraud department to visualize what is
going on without IT interventions
- Based on NRTRDE data received from roaming
partners and our own risk scoring engine development
- Combining NRTRDE and DWH data
16. 16
Developing & Deploying our First Solution
- Effort Spent on our First Solution – Fully In-House
- Server deployment and configuration - 1 man day
- ELK stack installation, configuration and tuning – 2
man days
- Solution Development
- Data Integration – 3 man days
- Dashboard Development – 2 man days
- Total initial effort spent including learning new
technology:
- 8 man days
- 4 core virtual server with 32 GB RAM and 300 GB
disk storage
17. 17
Gaining Visual Insight
- Dashboards enabled visual insight and discovery what is
going on, including Geo-Visualizations
- Ability to filter and drilldown to lowest level
22. 22
Building upon Initial Success Story
- Based on Initial Success Story we continued with rolling
out:
- Visiting Roamers Fraud Dashboard
- Interconnect Fraud Dashboard
- Circuit Switched Mobile Traffic Dashboard
- Packet Based Mobile Traffic Dashboard
- Online Charging System Real-Time Performance
Dashboard
- Real-Time Data Feeds to Elasticsearch directly
- Complex Near-Real Time Data Feeds using conventional
DWH as Data Integration and Data Distribution point
towards Elasticearch
28. 28
Open Source Big Data – IT Benefits
- Open Source Big Data Solution Enabled IT to:
- To complement our existing DWH & BI Infrastructure
including data in those systems in Cost Efficient
Manner and Short Timeframe
- To provide to our end users large amounts of data for
self-service Data Visualization & Discovery not before
available
- Ability to provide Real-Time Analytics for data sources
with real-time data feeds
29. 29
Open Source Big Data - Business Benefits
- Open Source Big Data Data Discovery & Visualization
Solution Enabled our Business Users:
- To quickly visualize and discover what and where is
going on in a self-service manner
- To dramatically cut decision times and take appropriate
actions resulting in:
- Reduced business risks
- Decreased service/network downtimes
- Increased customer satisfaction
