資料科學在 Whoscall 產品體系中的角色

Gogolook Confidential
How Started?

The Best App
For
identifying and blocking calls
The Best App –LINE whoscall

KeyFeatures

★Instant Caller Identification
LINE whoscallidentifies background information of incoming unknown calls in seconds through tags reported by other users, Internet search results, and our comprehensive global database.
Instant Caller Identification

★Database with
over 600Million
Phone Numbers
LINE whoscallboasts an online database with over 600 million phone numbers. The database of LINE whoscallcovers yellow pages, spammers, telemarketers, costumer services...,etc. with numerous community tags contributed by users and comments based on real users’ experiences.
Database & Number Details

Incoming Call Dialogue
Incoming Call Dialogue
Fraud Call
Business
Corporation
Restaurant

★Community Tag
★Block unwanted calls & SMSs
Contributions from the global user community has always been the pillar of LINE whoscall’sservice. LINE whoscalluser can tag aphone number and share it with others, which creates an integrated phone number database and a reliable communication network for everyone.
Block calls and SMSs intelligently to ensure a harassment-free calling experience.
Tag & Block

★World’s Largest
Yellow Page Database
★Offline Database
Available for Free
LINEwhoscallowns one of the world’s largest onlinephonenumber database in the world, which covers most of numbers of businesses and service providers essential to you daily lives.
The free database is not only available online but also offline. And they are completely free! The unlimited usage of database with over 600 million phone numbers is only on LINE whoscall.
Database Usage

3 of every 5 strangers’ calls can be identified by LINE whoscall
Over 400 million phone calls
are identified
by LINE whoscallevery month.
3000 spammer numbers
are reported
by LINE whoscalluser every day.
Number Identification
–2014.07
–2014.07

What we will be…

資料科學在whoscall的應用
GOGOLOOK資料科學家高義銘

★日常生活經常會遇到的問題

★人面對未知的事物就會有一種…
我有一種不祥的預感!

★坊間流傳著許多解決此問題的APPs
小熊來電通知

★Why whoscall?
因為…他是連Google執行長都說讚的軟體!
唉呦，讚喔

whoscall是如何解決未知來電的問題咧?

★Technologies adopted
1. Yellow pages:
HiPage, Yelp,
Zenrin…
2. Google search
3. Other sources
Technologies adopted

★Technologies adopted
Technologies adopted
4. 使用者回報與標記

★whoscall, I have a problem…
如果一個未知號碼，我們無法從這些sources 去取得任何資訊，那就GG 了嗎?

★whoscall, I have a problem…
如果一個未知號碼，我們無法從這些sources 去取得任何資訊，那就GG 了嗎?
是的，GG然後洗洗睡…

當然不能洗洗睡，要不然我站在這邊幹嘛?

★Problem we want to solve
For an unknown phone number:
•
No google result
•
No user tag / report
•
Not a whoscalluser
Problem we want to solve

•
No google result
•
•
Not a whoscalluser
Can we determine if it’s a spamnumber?

•
No google result
•
•
Not a whoscalluser
Can we determine if it’s a spam number?
推銷電話?

•
No google result
•
•
Not a whoscalluser
推銷電話? 詐騙電話?
騷擾電話?

•
No google result
•
•
Not a whoscalluser
騷擾電話?
打錯電話?

•
No google result
•
•
Not a whoscalluser
騷擾電話?
打錯電話?
(我又不是神!!)

★Scenario
Scenario
OO推銷
小明
小明妹
小明哥
?

★We think it should work because…
whoscalluserbase( = potential sensors)
•
> 10 million installations
•
> 10 thousands tags (daily)
•
> 30 million phone calls (daily)

Analysis procedures
Analysis procedures
1.
Collect call logs
2.
Compare with user tags
3.
Explore call behaviors
4.
Extract features
5.
Classify unknown numbers using machine learning techniques

★Collect call logs
•
Recruit a group of voluntary whoscallusers as our sensors.
•
Collect phone call logs from these sensors for a month.
Collect call logs

★User privacy

User privacy is kept in the highest priority.

Phone numbers are stored as one-way hash codes. (therefore unable to be reversed)
User privacy

★List of user tags
List of user tags
一接就掛斷
一打來就掛掉
一接對方馬上掛斷
一接就掛電話
一接起來就掛斷電話
一接起來，就說打錯
一直傳廣告簡訊
一直打錯電話
一直收到沒顯示的 APP
一直狂打錯電話
一聲
一聲不響，就掛掉，有問題
一聲就掛
一聲掛斷
一聽收線
嚴重騷擾
國外莫名來電
國際電話偽裝台北碼???
地下錢莊
地下錢莊推銷
地下非法期公司
地產
垃圾
垃圾簡訊
垃圾訊息
基隆美髮
壽險
外勞
夜半打給不認識的在亂
色情交友
色情交友電話
色情人肉市場
色情垃圾簡訊
色情外送
色情妹妹電話
色情干擾
色情廣告簡訊
色情拉客妹
色情按摩
色情推銷
色情推銷電話
色情援交外送
色情敗類
摩門
撥了馬上掛掉
擾亂電話
收數率調查
收視率調查
放款簡訊
政府宣導
敲一聲而已
整人電話
新光保全
星展借貸
星展推消
星展銀行
淫媒仲介

★Compare with user tags
•
Compare these phone numbers with user reports from whoscalldatabase (封鎖記錄)
Compare with user tags
Normal numbers
0987-991-XXX
0986-225-XXX
02-2675-XXXX
03-862-XXXX
...
02-2543-XXXX
03-556-XXXX
886-XXXX
…
推銷電話
02-2783-XXXX
886-903-XXXX
0800-000-XXX
…
惡意電話

★Data summary
Data summary
推銷電話
民調中心
騷擾電話
詐騙電話
70%
1%
5%
24%
# Samples: 7854
Normal: 4000
Spam: 3854

Normal numbers
0
5
10
15
20
Calls =195 (in 66, out 129)
Opponents = 72 (in 21, out 58)
★Normal numbers

★Spam numbers
Spam numbers
0
10
20
30
Calls =471 (in 15, out 456)
Opponents = 186 (in 11, out 183)
XX信用卡行銷(7)
OOO，XXXX行銷(6)
電話行銷(3)

★What is a feature?
What is a feature?
“Feature”is a measurable property of a phenomenon being observed.

Example
Or, we want to analyze a company, we can look at features:
公司人數
★Example

Example
Or, we want to analyze a company, we can look at features:
工程師人數
★Example

Example
Or, we want to analyze a company, we might look at features:
公司裡面 Python工程師的比例
★Example

Example
公司向心力
★Example

Example
CEO帥氣程度
★Example

Features for call patterns
Ratio of out calls
0.8
0.6
0.4
0.2
0.0
Fraud
Marketing
Normal

Ratio of recurring opponents
Fraud
Marketing
Normal
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7

Ratio of missed out calls
Fraud
Marketing
Normal
0.6
0.5
0.4
0.3
0.2
0.1
0

Ratio of working time calls
Fraud
Marketing
Normal
0.6
0.5
0.4
0.3
0.2
0.1
0
0.7

Median of call durations
Fraud
Marketing
Normal
50
40
30
20
10
0
60
seconds

Ratio of out calls in contact book
Fraud
Marketing
Normal
0.10
0
0.25
0.30
0.35
0.20
0.15
0.05

Ratio of recurring components is less than 40%
Ratio of out calls is more than 60%
Ratio of in calls is less than 20%
Then we claim the number is a spam number
Intuitively, we can determine an unknown number by rules such as if
★Naïve method

★Problem 1
Too many features…

★Problem 2
How to determine the rule?

Machine learning
★Solution

Machine learning
★Solution
Let the machine learn from the data

What is machine learning?
★What is machine learning?
機器學習是一種從過去的資料或經驗當中，構造一個模型(Model)，而學習(Learning)這件事就是讓這個模型以程式的方式執行，等到學習到一定的程度後，就可以做預測(猜)，這個「猜」是有根據的，且命中率高的。

Machine learningtechniques for classification
★Machine learning techniques for classification
Support vector machine
Logistic regression
Decision tree
Neural networks
Naïve Bayes
Nonparametric Bayesian method

Support vector machine for binary classification
★Support vector machine for binary classification

這樣就夠了嗎？

Real-life scenario
★Real-life scenario
When will we require a spam number prediction?
Ans: The time a phone call reaches a whoscalluser
We want to predict whether a number is spam as EARLYas possible in order to prevent further victims…

Real-life scenario
Time
#recent calls
Victim 1
Victim 2
Victim 3
XX推銷
★Real-life scenario
推銷電話

Let’s look at the performances of SVM under different numbers of recent calls

SVM for binary classification
★SVM for binary classification
0.8
0.85
0.9
0.95
1.0
3
4
5
6
7
8
9
10
# recent calls
Accuracy

嗯…表現的不錯，但是…
可以再快一點嗎?

Reduce the number of features
★Reduce the number of features
Features computation is time-consuming. So we want to reduce the number of features before we do classification.

當然我們不是用手去選…

Feature selection methods:
Regularization methods
Backward, forward, and stepwise methods
Bayesian feature selection
Random forest method

Feature selection results
★Feature selection results
10
15
20
25
30
3recent calls
5recent calls
10 recent calls
0.8
0.85
0.9
0.95
1.0
# features
Accuracy

Ratio of out calls
Rate of out calls
Ratio of out calls in contact book
Ratio of reciprocal opponents
Ratio of recurring opponents
Median call duration of in calls
Ring duration of answered calls
and more…
★Selected features
Ratio of missed calls
Rate of new opponents
Ratio of in calls in contact book

★Comparison of w/ and w/o feature selection
0.8
0.85
0.9
0.95
1.0
3
4
5
6
7
8
9
10
# recent calls
Accuracy

Done?
阿不就好棒棒?

What is power?
★What is power?
Power of class A: The probability of accurately classify a class A sampleto class A.

What is power?
★What is power?
Power of class A: The probability of accurately classify a class A sampleto class A.
性別 Classifier
97.5% this is a male

Power of our classifier
★Power of our classifier
0.8
0.85
0.9
0.95
1.0
3
4
5
6
7
8
9
10
#recent calls
Power

義銘，
加油好嗎?

Marketing numbers vs. normal numbers
★Marketing numbers vs. normal numbers
0.8
0.85
0.9
0.95
1.0
3
4
5
6
7
8
9
10
#recent calls
Accuracy

Fraud numbers vs. normal numbers
★Fraud numbers vs. normal numbers
0.8
0.85
0.9
0.95
1.0
3
4
5
6
7
8
9
10
#recent calls
Accuracy

一種
摻在一起做撒尿牛丸的概念…

Power of SVMfor multi-classification
★Power of SVM for multi-classification
0.8
0.85
0.9
0.95
1.0
3
4
5
6
7
8
9
10
#recent calls
Power

Power of SVM for binary classification
★Power of SVM for binary classification
0.8
0.85
0.9
0.95
1.0
3
4
5
6
7
8
9
10
#recent calls
Power

What is type I error rate?
★What is type I error rate?
Type I error: The probability of misclassify a class B sampleto class A.
性別 Classifier
5% this is a male

Type I error comparison
★Type I error comparison
0
0.05
0.1
0.15
0.3
3
4
5
6
7
8
9
10
#recent calls
Type I error
0.2
0.25

這點小成果讓我稍稍放鬆地去逛街，突然電話響一聲，我開心地接了起來…

結果，對方掛斷了

響一聲掛斷的惡意電話
★響一聲掛斷的惡意電話

“響一聲掛斷”(one-ring call) 是一種引誘接電話者回撥的惡意電話，通常伴隨著高額付款電話。

於是我們先觀察“響一聲掛斷”這類型電話號碼的call patterns。

Call patterns of one-ring calls
★Call patterns of one-ring calls
Numbers
Mean duration of ringing(seconds)
Mean duration ofout calls (seconds)
0982-415-XXX
1.6
0
0982-420-XXX
3.6
0
0982-495-XXX
5.2
1.25
04-3-704-XXXX
0.9
0
0923-931-XXX
6.7
2.6

Feature comparison
Ratio of new opponents
Fraud
Marketing
Normal
One-ring
0
0.2
0.4
0.6
0.8

Feature comparison
Ratio of in calls
0
0.1
0.2
0.3
0.4
0.5
Fraud
Marketing
Normal
One-ring

Feature comparison
Ratio of missed calls
0
0.2
0.4
0.6
0.8
Fraud
Marketing
Normal
One-ring

★Naïve method
Similarly, without machine learning we can design rules such as:

★Naïve method
Similarly, without machine learning we can design rules such as:
Rule1: The mean of the ringing duration is less then 7 seconds.
and
Rule 2: The mean of the outcall duration is less than 3 seconds.
Then we claim that it is a one-ring spam call.

★Problems
1.
Too many features…
2.
How to determine the rule?
3.
New observations.

★Problem 3
Numbers
0982-415-XXX
1.6
0
0982-420-XXX
3.6
0
0982-495-XXX
5.2
1.25
04-3-704-XXXX
0.9
0
0923-931-XXX
6.7
2.6

Numbers
0982-415-XXX
1.6
0
0982-420-XXX
3.6
0
0982-495-XXX
5.2
1.25
04-3-704-XXXX
0.9
0
0923-931-XXX
6.7
2.6
04-2-676-XXXX
15.7
1.4
★Problem 3
New observation

Numbers
0982-415-XXX
1.6
0
0982-420-XXX
3.6
0
0982-495-XXX
5.2
1.25
04-3-704-XXXX
0.9
0
0923-931-XXX
6.7
2.6
04-2-676-XXXX
15.7 (S.D.=10.7)
1.4
★Problem 3

Machine learning can efficiently “learn” from new data and create rules for us.

Power of SVM for multi-classification
★Power of SVM for multi-classification
0.8
0.85
0.9
0.95
1.0
3
4
5
6
7
8
9
10
#recent calls
Power

Accuracy comparison
★Accuracy comparison
3
4
5
6
7
8
9
10
#recent calls
0
0.05
0.1
0.15
0.3
0.2
0.25
Type I error

Deployment
All the algorithms have been implemented in the whoscallapp, so how does it work?

OO推銷
小明
Data center
Classifier calculating…
0984-003-XXX
回傳:此號碼可能為推銷電話
所需時間: 50-100 milliseconds

What’s next?

Improvements of the classification model
1.
Fraud numbers analysis
2.
Fuzzy classification algorithm
3.
Spam-category scores
4.
Cooperate with more solid outside sources
5.
Generalize to other countries.
Much more…
★Improvements of the classification model

Future perspectives
1.
User’s tag correction mechanisms
2.
Personalized penalty setting
3.
Anti-countermeasures
4.
Extend to SMS spam detection
5.
Clustering vs. user tags
6.
Spam detect Scam detection
★Future perspectives

Creating a contact network of trust
感謝大家寶貴的時間

資料科學在 Whoscall 產品體系中的角色

Recommandé

Recommandé

Contenu connexe

En vedette

En vedette (20)

Similaire à 資料科學在 Whoscall 產品體系中的角色

Similaire à 資料科學在 Whoscall 產品體系中的角色 (20)

Plus de 台灣資料科學年會

Plus de 台灣資料科學年會 (20)

Dernier

Dernier (20)

資料科學在 Whoscall 產品體系中的角色