Traffic Insight Using Netflow and Deepfield Systems by Mohd Rizal

1. Traffic Insight NETFLOW &
DEEPFIELD SYSTEMS
Prepared by:
Mohd Rizal Mohd Ramly, TM Berhad
Saiful Akmal Towfek, TM Berhad
Mohd Farhan Tajwid , TM Berhad

2. There is no visibility of traffic per user
OurProblemStatement
Need to have better understanding & info on network behavior
ie Video Streaming.
IMPACT:
Planning and dimensioning will be difficult relying only on
utilization data (totally network physical behaviour)
Category/ApplicationVisibility
Costly for full-blown deep packet inspection BUT do
we really need all the information given by DPI?
Would netflow information be enough for basic
network information?
DeepPacketInspection(DPI)Cost
Cheaper solution combining existing `network definition map` from DEEPFIELD with granularity of per user from
netflow v9.
1cloudgnome is product from DEEPFIELD (Nokia) with simple network signature but in aggregate view

3. Traffic category based on netflow information
NetflowInformation
NetflowCapability
Interface
ToS
Protocol
Source IP
Address
Destination
IP Address
Source Port
Destination
Port
Link Layer
Header
IP Header
TCP/UDP
Header
NETFLOW
ü Monitors data in
Layer 2 thru 4
ü Determines
applications by
port
ü Utilizes a seven-
tuple for flow
ü Flow information
who, what, when
where
v With new DEEPFIELD systems, the definition for category
done by cloudgnome (DEEPFIELD).

4. Comparison between netflow and DPI
NETFLOWvsDPI
Up to homepass (PPPoE
dialer) with public IP.
GRANULARITY
Cannot be done, only for
traffic utilization
QoE
Only for peering with AS
number. IP source may
not be accurate
COVERAGE
No near realtime or
realtime. ONLY
DESCRIPTIVE ANALYTIC
ANALYTICDETAILS
Client devices view. Able
to identify brand/type of
devices
Able to identify each QoE
measurement based on
Layer 7 performance
Accurate based on DPI
signature.
Platform DPI ready up to
PREDICTIVE ANALYTIC and
QoE assessment
Netflow
DPI
Impact
Planning and network
design can be done in an
optimize way.
Personalization cannot be
done
Identify total homepass
view up to category traffic
ie: streaming, web
browsing (no quality
measurement)
Only can identify top
peering performance ie:
Netflix, Apple, Google,
Facebook
As and when descriptive
analytic for planning
purpose and brief
understanding for each
top applications category
streaming.

5. NameNodex6
International
Gateway
MasterNode
MSAN
xDSL
FTTH
COPPER
Transmission
COPPER
FOC
Transmission
OLT
Access
Gateway
10G/100G 10G/100G 10G/100G
DEEPFIELD SYSTEMS
Contain gnomecloud
IP CORE
CLOUD
Collector
Netflow Collector
NetworkTopology
3 core members with support
of 7 staff.
Manpower
Create a temporary Hadoop
with 6 name node.
How
All ingress traffic at TM’s
gateway routers to eliminate to
reduce complexity and velocity
data
Strategy

6. Huge task waiting to massage/exploratory huge data with high velocity
TheProcess
Network protocol for collecting IP traffic
information (up to layer 4) from
DEEPFIELD with traffic category
mapping
Challenge
Raw IP data with no user login
information.
Huge data (700MB for 5 minutes)
Netflow
Data exploratory and logic creation to
understand the behavior of customers
and application/network trending
Exploratoryandlogic
Mapping for source IP to
domain/sites/application & category
DEEPFIELDcloudgnome
To map the customers login with IP and
get the package information
(subscriber info)
RadiusData

7. Exploit the Deepfield capability utilizing Cloud Genome
NetflowClassification
01
02
03
04
Deepfield
Capability to distinguish each network flow
based on signature content. Different category
content (ie: video, dns, cache etc) within same
port, IP source and protocol
Based on aggregation data not single
customer flow
DEEPFIELDprovidetheintelligence
We able to identify the user from the radius
info
RADIUSgivesusidentification
With 5 minutes volume, we able to derive the
throughput performance and with identification of
source and destination
NETFLOWgiveustheflowandusage
With this integration, it will help to
map the geolocation and others
marketing requirement. ie: 30Mbps
with quota vs 30Mbps unlimited
BSS&InventoryIntegration

8. TheChallenges
Critical to Further Achievements
ProcessingPower
01
Huge data from Radius
and Netflow will be
difficult to handle
cloudgnomeLimitation
02
Only on selected
routers
CHALLENGE
Need high powered
Hadoop servers with
speed & large storage
Mapping with cloudgnome
will be tricky due to
multiple category within
same IP + port
CURRENT
MITIGATION
Data just for 4 weeks
due to storage and
speed. (descriptive
analytic)
Mathematic modelling with
assumption of normal
distribution
03
Skillset and ability to
analyse huge and
high velocity data
Ability to understand
the network with
analytic thinking
ability
Training and practical
lesson
Manpowerandteam
04
If we take every datapoint
each timestamp, it will be
huge
Difficulties to mapping with
radius and will be massive
task and takes time
Take first 5 minutes
for each ½ an hour
SamplingRateDecision

9. Data for 2 weeks: 26/XX/201X to X/XX/201X
StreamingOTTInsights
Netflixstats
q Unique users stream – 2XX,X9X*
q Daily max – 1X,7X9 unique user AND
between 9.25 pm to 10.00 pm
q Average stream traffic @ 3.954Mbps
IFLIXstats
q Unique users stream – XX,547
q Daily max – X,5X2 unique users AND
between 9.25 pm to 10.00 pm
q Average stream traffic @ 1.372Mbps
VIUstats
0
2,000
4,000
6,000
8,000
10,000
12,000
14,000
16,000
18,000
20,000
22,000
viuiflixneflix
q Unique users stream – X3,X0X*
q Daily max – XX6 unique users AND between
9.25 pm to 10.00 pm
q Average stream traffic @ 0.185Mbps
XX/XX/201X
XX/XX/201X
The capability can be extended to geographical location, ie: more
users from SEGAMBUT watch VIU compare to IFLIX & we can
identify exact user watch which content group
XX/XX/201X
XX/XX/201X
XX/XX/201X
XX/1XX201X
*data plot for 5 days only
INSIGHT 1

10. How Much distribution of DNS server for TM’s Unifi Subscribers
(public IP only)
DNSServerUsage
5X.5X%
0.X2%
XX.1%
X.4X%
0.XX%
X1X.2K subscribers
confidence with
GOOGLE DNS
X1.5K subscribers used
cloudfare DNS
Less than half (6XXK)
still maintain their TM
DNS configuration
3XK subscribers config
their RG with OpenDNS
X.5K subcribers used
comodo DNS
Most probably, customers config with other
DNS due to accessibility to block websites by
MCMC/government
* With assumption that if specific users already have records using others DNS, we eliminate the possibility they using TM’s DNS.
INSIGHT 2

11. From Deepfield Data
TopDomainVisit
TopWebVisitbyTM’sSubscriber
From aggregate data we able to distinguish the TOP 15 of the
domain visit by our subscribers contribute in 80% from overall
domain reach by all users.
Youtubecontribute3X.X%
Behaviour of Malaysian watching
content via youtube with the
booming of independent content
provider
PaidContent:Netflix
Eventhough paid service, but with
variety and impressive quality of
content, users will subscribed .
MessagingApps
77% from detected messenging
apps is WHATSAPP with peak
traffic 39Gbps
Games
No 7 in top category with VALVE,
Nintendo and ROBLOX is the top 3
contributors
Top websites by TM’s Subscribers
youtube.com facebook.com netflix.com apple.com
google.com update.microsoft.com whatsapp.com apis.google.com
instagram.com openload.co valve.com dailymotion.com
tencent.com twitch.tv others
With netflow data, we able to distinguish each user details based
on cloud genome from DEEPFIELD.
INSIGHT 3

12. From Data Derive by NETFLOW, the planning become more realistic
StrategicPlanning
NetworkPlanning
During Deployment
115,52k
35,52k
Distribution of mass home user and business (example)
FTTH
Metro ethernet
OLT
OLT
OLT
OLT
Access OLT
business
home
01
We able to determine network
capacity based on multivariate
linear regression up to CORE
network.
During Campaign
Determine new predictive traffic
based on marketing campaign.
Ease the planning and prevent any
congestion up to CORE network
02
Scenario
INSIGHT 4

13. THANKS
IMPROVE VISIBILITY WITH ANALYTIC
NETFLOW