VSSML17 L5. Basic Data Transformations and Feature Engineering

Valencian Summer School in Machine Learning
3rd edition
September 14-15, 2017

BigML, Inc 2
Basic Transformations
Making Data Machine Learning Ready
Poul Petersen
CIO, BigML, Inc

BigML, Inc 3Basic Transformations
In a Perfect World…
Q: How does a physicist milk a cow?
A: Well, ﬁrst let us consider a spherical cow...
Q: How does a data scientist build a model?
A: Well, ﬁrst let us consider perfectly formatted data…

The Dream
CSV Dataset Model Proﬁt!

The Reality
CRM
Web Accounts
Transactions
ML Ready?

Obstacles
• Data Structure
• Scattered across systems
• Wrong "shape"
• Unlabelled data
• Data Value
• Format: spelling, units
• Missing values
• Non-optimal correlation
• Non-existant correlation
• Data Signiﬁcance
• Unwanted: PII, Non-Preferred
• Expensive to collect
• Insidious: Leakage, obviously correlated
Data Transformation
Feature Engineering
Feature Selection

The Process
• Deﬁne a clear idea of the goal.
• Sometimes this comes later…
• Understand what ML tasks will achieve the goal.
• Transform the data
• where is it, how is it stored?
• what are the features?
• can you access it programmatically?
• Feature Engineering: transform the data you have into
the data you actually need.
• Evaluate: Try it on a small scale
• Accept that you might have to start over….
• But when it works, automate it!!!!

Data Transformations

BigML Tasks
Goal
• Will this customer default on a
loan?
• How many customers will apply for
a loan next month?
• Is the consumption of this product
unusual?
• Is the behavior of the customers
similar?
• Are these products purchased
together?
ML Task
Classification
Regression
Anomaly Detection
Cluster Analysis
Association Discovery

Classiﬁcation
CategoricalTrainingTesting
Predicting

Regression
NumericTrainingTesting
Predicting

Anomaly Detection

Cluster Analysis

Association Discovery

ML Ready DataInstances
Fields
(Features)
Tabular Data (rows and columns):
• Each row
• is one instance.
• contains all the information about that one instance.
• Each column
• is a ﬁeld that describes a property of the instance.

Data Labeling
Unsupervised
Learning Supervised
Learning
• Anomaly Detection
• Clustering
• Association Discovery
• Classiﬁcation
• Regression
The only difference, in terms of
ML-Ready structure is the
presence of a "label"

Data Labelling
Data is often not labeled
Create labels with a transformation
Name Month - 3 Month - 2 Month - 1
Joe Schmo 123.23 0 0
Jane Plain 0 0 0
Mary Happy 0 55.22 243.33
Tom Thumb 12.34 8.34 14.56
Un-‐Labelled
Data
Labelled
data
Name Month - 3 Month - 2 Month - 1 Default
Joe Schmo 123.23 0 0 FALSE
Jane Plain 0 0 0 TRUE
Mary Happy 0 55.22 243.33 FALSE
Tom Thumb 12.34 8.34 14.56 FALSE
Can be done at Feature
Engineering step as well

SF Restaurants Example
https://data.sfgov.org/Health-and-Social-Services/Restaurant-Scores/stya-26eb
https://blog.bigml.com/2013/10/30/data-preparation-for-machine-learning-using-mysql/
create database sf_restaurants;
use sf_restaurants;
create table businesses (business_id int, name varchar(1000), address varchar(1000), city varchar(1000), state varchar(100),
postal_code varchar(100), latitude varchar(100), longitude varchar(100), phone_number varchar(100));
load data local infile './businesses.csv' into table businesses fields terminated by ',' enclosed by '"' lines terminated by
'rn' ignore 1 lines;
create table inspections (business_id int, score varchar(10), idate varchar(8), itype varchar(100));
load data local infile './inspections.csv' into table inspections fields terminated by ',' enclosed by '"' lines terminated
by 'rn' ignore 1 lines;
create table violations (business_id int, vdate varchar(8), description varchar(1000));
load data local infile './violations.csv' into table violations fields terminated by ',' enclosed by '"' lines terminated by
'rn' ignore 1 lines;
create table legend (Minimum_Score int, Maximum_Score int, Description varchar(100));
load data local infile './legend.csv' into table legend fields terminated by ',' enclosed by '"' lines terminated by 'rn'
ignore 1 lines;

Transformations Demo #1

Data Cleaning
Homogenize missing values and different types in the same
feature, fix input errors, correct semantic issues, types, etc.
Name Date Duration (s) Genre Plays
Highway star 1984-05-24 - Rock 139
Blues alive 1990/03/01 281 Blues 239
Lonely planet 2002-11-19 5:32s Techno 42
Dance, dance 02/23/1983 312 Disco N/A
The wall 1943-01-20 218 Reagge 83
Offside down 1965-02-19 4 minutes Techno 895
The alchemist 2001-11-21 418 Bluesss 178
Bring me down 18-10-98 328 Classic 21
The scarecrow 1994-10-12 269 Rock 734
Original
data
Name Date Duration (s) Genre Plays
Highway star 1984-05-24 Rock 139
Blues alive 1990-03-01 281 Blues 239
Lonely planet 2002-11-19 332 Techno 42
Dance, dance 1983-02-23 312 Disco
The wall 1943-01-20 218 Reagge 83
Offside down 1965-02-19 240 Techno 895
The alchemist 2001-11-21 418 Blues 178
Bring me down 1998-10-18 328 Classic 21
The scarecrow 1994-10-12 269 Rock 734
Cleaned
data
update violations set description = substr(description,1,instr(description,' [ date violation corrected:')-1) where instr(description,'
[ date violation corrected:') > 0;

Define a Goal
• Predict rating: Poor / Needs Improvement / Adequate /
Good
• This is a classification problem
• Based on business profile:
• Description: kitchen, cafe, etc.
• Location: zip, latitude, longitude

Denormalizing
business
inspections
violations
scores
Instances
Features
(millions)
join
Data is usually normalized in relational databases, ML-Ready
datasets need the information de-normalized in a single dataset.
create table scores select * from businesses left join inspections using (business_id);
create table scores_last select a.* from scores as a JOIN (select business_id,max(idate)
as idate from scores group by business_id) as b where a.business_id=b.business_id and
a.idate=b.idate;
Denormalize
ML-‐Ready:
Each
row
contains
all
the
information
about
that
one
instance.

create table scores_last_label select scores_last.*, Description as score_label from
scores_last join legend on score <= Maximum_Score and score >= Minimum_Score;
Add
Label

Structuring Output
• A CSV file uses plain text to store tabular data.
• In a CSV file, each row of the file is an instance.
• Each column in a row is usually separated by a comma (,) but other
"separators" like semi-colon (;), colon (:), pipe (|), can also be used.
Each row must contain the same number of fields
• but they can be null
• Fields can be quoted using double quotes (").
• Fields that contain commas or line separators must be quoted.
• Quotes (") in fields must be doubled ("").
• The character encoding must be UTF-8
• Optionally, a CSV file can use the first line as a header to provide the
names of each field.
After all the data transformations, a CSV (“Comma-Separated
Values) file has to be generated, following the rules below:
select * from scores_last_label into outfile "./scores_last_label.csv";
select 'name', 'address', 'city', 'state', 'postal_code', 'latitude', 'longitude', 'score_label' UNION select name, address, city,
state, postal_code, latitude, longitude, score_label from scores_last_label into outfile "./scores_last_label_headers.csv" ;

Define a Goal
• Predict rating: Poor / Needs Improvement / Adequate / Good
• This is a classification problem
• Based on business profile:
• Description: kitchen, restaurant, etc.
• Location: zip code, latitude, longitude
• Number of violations, text of violations

Aggregating
User Num.Playbacks Total Time Pref.Device
User001 3 830 Tablet
User002 1 218 Smartphone
User003 3 1019 TV
User005 2 521 Tablet
Aggregated data (list of users)
When the entity to model is different from the provided data,
an aggregation to get the entity might be needed.
Content Genr
e
Duration Play Time User Device
Highway
star
Rock 190 2015-05-12
16:29:33
User001 TV
Blues alive Blues 281 2015-05-13
12:31:21
User005 Tablet
Lonely
planet
Tech
no
332 2015-05-13
14:26:04
User003 TV
Dance,
dance
Disco 312 2015-05-13
18:12:45
User001 Tablet
The wall Reag
ge
218 2015-05-14
09:02:55
User002 Smartphone
Offside
down
Tech
no
240 2015-05-14
11:26:32
User005 Tablet
The
alchemist
Blues 418 2015-05-14
21:44:15
User003 TV
Bring me
down
Class
ic
328 2015-05-15
06:59:56
User001 Tablet
The
scarecrow
Rock 269 2015-05-15
12:37:05
User003 Smartphone
Original data (list of playbacks)
create table violations_aggregated select business_id,count(*) as violation_num,group_concat(description) as violation_txt from
violations group by business_id;
create table scores_last_label_violations select * from scores_last_label left join violations_aggregated USING (business_id);
tail -n+2 playlists.csv | cut -d',' -f5 | sort | uniq -c
tail -n+2 playlist.csv | awk -F',' '{arr[$5]+=$3} END {for (i in arr) {print arr[i],i}}'
SET @@group_concat_max_len = 15000
select 'name', 'address', 'city', 'state', 'postal_code', 'latitude', 'longitude', 'violation_num', 'violation_txt', 'score_label'
UNION select name, address, city, state, postal_code, latitude, longitude, violation_num, violation_txt, score_label from
scores_last_label_violations into outfile "./scores_last_label_violations_headers.csv" ;

Pivoting
Different values of a feature are pivoted to new columns in the
result dataset.
Content Genre Duration Play Time User Device
Highway star Rock 190 2015-05-12 16:29:33 User001 TV
Blues alive Blues 281 2015-05-13 12:31:21 User005 Tablet
Lonely planet Techno 332 2015-05-13 14:26:04 User003 TV
Dance, dance Disco 312 2015-05-13 18:12:45 User001 Tablet
The wall Reagge 218 2015-05-14 09:02:55 User002 Smartphone
Offside down Techno 240 2015-05-14 11:26:32 User005 Tablet
The alchemist Blues 418 2015-05-14 21:44:15 User003 TV
Bring me down Classic 328 2015-05-15 06:59:56 User001 Tablet
The scarecrow Rock 269 2015-05-15 12:37:05 User003 Smartphone
Original data
User Num.Playback
s
Total Time Pref.Device NP_TV NP_Tablet NP_Smartphone TT_TV TT_Tablet TT_Smartphone
User001 3 830 Tablet 1 2 0 190 640 0
User002 1 218 Smartphone 0 0 1 0 0 218
User003 3 1019 TV 2 0 1 750 0 269
User005 2 521 Tablet 0 2 0 0 521 0
Aggregated data with pivoted columns

Time Windows
Create new features using values over different periods of time
Instances
Features
Time
Instances
Features
(millions)
(thousands)
t=1 t=2 t=3
create table scores_2013 select a.business_id, a.score as score_2013, a.idate as idate_2013 from inspections as a JOIN ( select
business_id, max(idate) as idate from inspections where substr(idate,1,4) = "2013" group by business_id) as b where a.business_id =
b.business_id and a.idate = b.idate;
create table scores_over_time select * from businesses left join scores_2013 USING (business_id) left join scores_2014 USING (business_id);

Updates
Need a current view of the data, but new data only comes in
batches of changes
day
1day
2day
3
Instances
Features

Streaming
Data only comes in single changes
data
stream
Instances
Features
Stream
Batch
(kafka, etc)

Prosper Loan Life Cycle
Submit
Cancelled Withdraw Expired
FundedBids Current
Q: Which new listings make it to funded?
Q: Which funded loans make it to paid?
Q: If funded, what will be the rate?
Classification
Regression
Classification
Goal ML Task
Defaulted
Paid
Late
Listings Loans

Prosper Example
D a t a P ro v i d e d i n X M L
updates!!
export.sh
fetch.sh
“curl”
daily
import.py
XML
bigml.sh
Model

Predict

Share in gallery
Status
LoanStatus
BorrowerRate
Denormalization with join

Prosper Example
• XML… yuck!
• MongoDB has CSV export and is record based so it is easy to
handle changing data structure.
• Feature Engineering
• There are 5 diﬀerent classes of “bad” loans
• Date cleanup
• Type casting: ﬂoats and ints
• Would be better to track over time
• number of late payments
• compare predictions and actuals
• XML… yuck!
Tidbits and Lessons Learned….

Tools

Tools
• Command Line?
• join, cut, awk, sed, sort, uniq
• Automation
• Shell, Python, crontab, etc
• Talend
• BigML: bindings, bigmler, API, whizzml
• Relational DB
• MySQL
• Non-Relational DB
• MongoDB

Talend
https://blog.bigml.com/2013/10/30/data-preparation-for-machine-learning-using-mysql/
Denormalization Example

Summary
• Data is awful
• Requires clean-up
• Transformations
• Consumes an enormous part of the effort in
applying ML
• Techniques:
• Denormalizing
• Aggregating / Pivoting
• Time windows / Streaming
• What a real Workﬂow looks like and the tools required

BigML, Inc 2
Feature Engineering
Creating Features that Make Machine Learning Work
Poul Petersen
CIO, BigML, Inc

BigML, Inc 3Feature Engineering
what is Feature Engineering
• This is really, really important - more than algorithm selection!
• In fact, so important that BigML often does it automatically
• ML Algorithms have no deeper understanding of data
• Numerical: have a natural order, can be scaled, etc
• Categorical: have discrete values, etc.
• The "magic" is the ability to ﬁnd patterns quickly and efﬁciently
• ML Algorithms only know what you tell/show it with data
• Medical: Kg and M, but BMI = Kg/M2 is better
• Lending: Debt and Income, but DTI is better
• Intuition can be risky: remember to prove it with an evaluation!
Feature Engineering: applying domain knowledge of
the data to create new features that allow ML
algorithms to work better, or to work at all.

Built-in Transformations
2013-09-25 10:02
Date-Time Fields
… year month day hour minute …
… 2013 Sep 25 10 2 …
… … … … … … …
NUM NUMCAT NUM NUM
• Date-Time ﬁelds have a lot of information "packed" into them
• Splitting out the time components allows ML algorithms to
discover time-based patterns.
DATE-TIME

Categorical Fields for Clustering/LR
… alchemy_category …
… business …
… recreation …
… health …
… … …
CAT
business health recreation …
… 1 0 0 …
… 0 0 1 …
… 0 1 0 …
… … … … …
NUM NUM NUM
• Clustering and Logistic Regression require numeric ﬁelds for
inputs
• Categorical values are transformed to numeric vectors
automatically*
• *Note: In BigML, clustering uses k-prototypes and the encoding used for LR can be conﬁgured.

Be not afraid of greatness:
some are born great, some achieve
greatness, and some have greatness
thrust upon ‘em.
TEXT
Text Fields
… great afraid born achieve …
… 4 1 1 1 …
… … … … … …
NUM NUM NUM NUM
• Unstructured text contains a lot of potentially interesting
patterns
• Bag-of-words analysis happens automatically and extracts
the "interesting" tokens in the text
• Another option is Topic Modeling to extract thematic meaning

Help ML to Work Better
{
“url":"cbsnews",
"title":"Breaking News Headlines
Business Entertainment World News “,
"body":" news covering all the latest
breaking national and world news
headlines, including politics, sports,
entertainment, business and more.”
}
TEXT
title body
Breaking News… news covering…
… …
TEXT TEXT
When text is not actually unstructured
• In this case, the text ﬁeld has structure (key/value pairs)
• Extracting the structure as new features may allow the ML
algorithm to work better

FE Demo #1

Help ML to Work at all
When the pattern does not exist
Highway Number Direction Is Long
2 East-West FALSE
4 East-West FALSE
5 North-South TRUE
8 East-West FALSE
10 East-West TRUE
… … …
Goal: Predict principle direction from highway number
( = (mod (field "Highway Number") 2) 0)

FE Demo #2

Feature Engineering
Discretization
Total Spend
7,342.99
304.12
4.56
345.87
8,546.32
NUM
“Predict will spend
$3,521 with error
$1,232”
Spend Category
Top 33%
Bottom 33%
Bottom 33%
Middle 33%
Top 33%
CAT
“Predict customer
will be Top 33% in
spending”

FE Demo #3

Built-ins for FE
• Discretize: Converts a numeric value to categorical
• Replace missing values: fixed/max/mean/median/etc
• Normalize: Adjust a numeric value to a specific range of
values while preserving the distribution
• Math: Exponentiation, Logarithms, Squares, Roots, etc
• Types: Force a field value to categorical, integer, or real
• Random: Create random values for introducing noise
• Statistics: Mean, Population
• Refresh Fields:
• Types: recomputes field types. Ex: #classes
>
1000
• Preferred: recomputes preferred status

Flatline Add Fields
Computing with Existing Features
Debt Income
10,134 100,000
85,234 134,000
8,112 21,500
0 45,900
17,534 52,000
NUM NUM
(/ (ﬁeld "Debt") (ﬁeld "Income"))
Debt

Income
Debt to Income Ratio
0.10
0.64
0.38
0
0.34
NUM

FE Demo #4

What is Flatline?
• DSL:
• Invented by BigML - Programmatic / Optimized for speed
• Transforms datasets into new datasets
• Adding new fields / Filtering
• Transformations are written in lisp-style syntax
• Feature Engineering
• Computing new fields: (/
(field
"Debt")
(field
“Income”))
• Programmatic Filtering:
• Filtering datasets according to functions that evaluate to
true/false using the row of data as an input.
Flatline: a domain specific language for feature
engineering and programmatic filtering

Flatline
• Lisp style syntax: Operators come first
• Correct: (+
1
2) => NOT Correct: (1
+
2)
• Dataset Fields are first-class citizens
• (field
“diabetes
pedigree”)

• Limited programming language structures
• let, cond, if, map, list operators, */+-‐, etc.
• Built-in transformations
• statistics, strings, timestamps, windows

Flatline s-expressions
(=
0
(+
(abs
(
f
"Month
-‐
3"
)
)
(abs
(
f
"Month
-‐
2"))
(abs
(
f
"Month
-‐
1")
)
))
Name Month - 3 Month - 2 Month - 1
Joe Schmo 123.23 0 0
Jane Plain 0 0 0
Mary Happy 0 55.22 243.33
Tom Thumb 12.34 8.34 14.56
Un-‐Labelled
Data
Labelled
data
Name Month - 3 Month - 2 Month - 1 Default
Joe Schmo 123.23 0 0 FALSE
Jane Plain 0 0 0 TRUE
Mary Happy 0 55.22 243.33 FALSE
Tom Thumb 12.34 8.34 14.56 FALSE
Adding Simple Labels to Data
Define "default" as
missing three payments
in a row

FE Demo #5

date volume price
1 34353 314
2 44455 315
3 22333 315
4 52322 321
5 28000 320
6 31254 319
7 56544 323
8 44331 324
9 81111 287
10 65422 294
11 59999 300
12 45556 302
13 19899 301
Current
-‐
(4-‐day
avg)

std
dev
Shock: Deviations from a Trend
day-4 day-3 day-2 day-1 4davg
-
314 -
314 315 -
314 315 315 -
314 315 315 321 316.25
315 315 321 320 317.75
315 321 320 319 318.75

Current
-‐
(4-‐day
avg)

std
dev
Shock: Deviations from a Trend
Current : (ﬁeld “price”)
4-‐day
avg: (avg-window “price” -4 -1)
std
dev: (standard-deviation “price”)
(/

(-‐

(
f
"price")
(avg-‐window
"price"
-‐4,
-‐1))
(standard-‐deviaOon
"price"))

FE Demo #6

Advanced s-expressions
Moon Phase%
(
/
(
mod
(
-‐
(
/
(
epoch
(
field
"date-‐field"
))
1000
)
621300
)
2551443
)
2551442
)
Highway isEven?
(
=
(mod
(field
"Highway
Number")
2)
0)
(
let
(R
6371000
latA
(to-‐radians
{lat-‐ref})
latB
(to-‐radians
(
field
"LATITUDE"
)
)
latD
(
-‐
latB
latA

)
longD
(
to-‐radians
(
-‐
(
field
"LONGITUDE"
)
{long-‐ref}
)
)
a
(
+
(
square
(
sin
(
/
latD
2
)
)
)
(
*

(cos
latA)
(cos
latB)
(square
(
sin
(
/
longD
2)))
)
)
c
(
*
2
(
asin
(
min
(list
1
(sqrt
a))
)
)
)
)
(
*
R

c
)
)

Distance Lat/Long <=> Ref (Haversine)

WhizzML + Flatline
HAVERSINE
FLATLINE
OUTPUT
DATASET
INPUT
DATASET
LONG Ref
LAT Ref
WHIZZML SCRIPT
GALLERY

Feature Engineering
Fix Missing Values in a “Meaningful” Way
F i l t e r
Zeros
Model  
insulin
Predict  
insulin
Select  
insulin
Fixed 
Dataset
Amended 
Dataset
Original 
Dataset
Clean 
Dataset
( if ( = (field "insulin") 0) (field "predicted insulin") (field "insulin"))

FE Demo #7

Feature Selection

Feature Selection
• Model Summary
• Field Importance
• Algorithmic
• Best-First Feature Selection
• Boruta
• Leakage
• Tight Correlations (AD, Plot, Correlations)
• Test Data
• Perfect future knowledge
cat diabetes.csv diabetes_testset.csv | sort | uniq -d | wc -l

Feature Selection
• Sales pipeline where step n-1 has no other outcome then
step n.
• Stock close predicts stock open
• Churn retention: the worst rep is actually the best
(correlation != causation)
• Cancer prediction where one input is a doctor ordered test
for the condition
• Account ID predicts fraud (because only new accounts are
fraudsters)
Leakage

Evaluate & Automate

Evaluate & Automate
• Evaluate
• Did you meet the goal?
• If not, did you discover something else useful?
• If not, start over
• If you did…
• Automate - You don’t want to hand code that every time,
right?
• Consider tools that are easy to automate
• Scripting interface
• APIs
• Ability to maintain is important

The Process
Data
Transform
Deﬁne Goal
Model &
Evaluate
no
yes
Better

Data
Not

Possible
Tune

Algorithm
Goal
Met?
Automate
Feature
Engineer &
Selection
Better 
Features

Summary
• Feature Engineering: what is it / why it is important
• Automatic transformations: date-time, text, etc
• Built-in functions: filtering and feature engineering
• Discretization / Normalization / etc.
• Flatline: programmatic feature engineering / filtering
• Structure
• Examples: Adding fields / filtering
• When building features it is important to watch for leakage
• The critical importance of automating

VSSML17 L5. Basic Data Transformations and Feature Engineering

VSSML17 L5. Basic Data Transformations and Feature Engineering

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à VSSML17 L5. Basic Data Transformations and Feature Engineering

Similaire à VSSML17 L5. Basic Data Transformations and Feature Engineering (20)

Plus de BigML, Inc

Plus de BigML, Inc (20)

Dernier

Dernier (20)

VSSML17 L5. Basic Data Transformations and Feature Engineering