Creating Operational Redundancy for Effective Web Data Mining

Using Operational Redundancy
Effective Web Data Mining
Jonathan LeBlanc
Head of Developer Evangelism N.A. (PayPal)
Github: http://github.com/jcleblanc
Slides: http://slideshare.net/jcleblanc
Twitter: @jcleblanc

Premise
The interactions of a user can be used to
personalize their experience

Elements of Mining Redundancy
Website
Data
Mining
User
Emotional
State Mining
User
Interaction
Mining

Our Subject Material
HTML content is poorly structured
There are some pretty bad web
practices on the interwebz
You can’t trust that anything
semantically valid will be present

How We’ll Capture This Data
Start with base linguistics
Extend with available extras

The Basic Pieces
Page Data
Scrapey
Scrapey
Keywords
Without all
the fluff
Weighting
Word diets
FTW

Capture Raw Page Data
Semantic data on the web
is sucktastic
Assume 5 year olds built
the sites
Language is the key

Extract Keywords
We now have a big jumble
of words. Let’s extract
Why is “and” a top word?
Stop words = sad panda

Weight Keywords
All content is not created
equal
Meta and headers and
semantics oh my!
This is where we leech
off the work of others

Questions to Keep in Mind
Should I use regex to parse web
content?
How do users interact with page
content?
What key identifiers can be monitored
to detect interest?

Fetching the Data: cURL
$req = curl_init($url);
$options = array(
CURLOPT_URL => $url,
CURLOPT_HEADER => $header,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_AUTOREFERER => true,
CURLOPT_TIMEOUT => 15,
CURLOPT_MAXREDIRS => 10
);
curl_setopt_array($req, $options);

//list of findable / replaceable string characters
$find = array('/r/', '/n/', '/ss+/'); $replace = array(' ', ' ', ' ');
//perform page content modification
$mod_content = preg_replace('#<script(.*?)>(.*?)</
script>#is', '', $page_content);
$mod_content = preg_replace('#<style(.*?)>(.*?)</
style>#is', '', $mod_content);
$mod_content = strip_tags($mod_content);
$mod_content = strtolower($mod_content);
$mod_content = preg_replace($find, $replace, $mod_content);
$mod_content = trim($mod_content);
$mod_content = explode(' ', $mod_content);
natcasesort($mod_content);

//set up list of stop words and the final found stopped list
$common_words = array('a', ..., 'zero');
$searched_words = array();
//extract list of keywords with number of occurrences
foreach($mod_content as $word) {
$word = trim($word);
if(strlen($word) > 2 && !in_array($word, $common_words)){
$searched_words[$word]++;
}
}
arsort($searched_words, SORT_NUMERIC);

Scraping Site Meta Data
//load scraped page data as a valid DOM document
$dom = new DOMDocument();
@$dom->loadHTML($page_content);
//scrape title
$title = $dom->getElementsByTagName("title");
$title = $title->item(0)->nodeValue;

//loop through all found meta tags
$metas = $dom->getElementsByTagName("meta");
for ($i = 0; $i < $metas->length; $i++){
$meta = $metas->item($i);
if($meta->getAttribute("property")){
if ($meta->getAttribute("property") == "og:description"){
$dataReturn["description"] = $meta->getAttribute("content");
}
} else {
if($meta->getAttribute("name") == "description"){
$dataReturn["description"] = $meta->getAttribute("content");
} else if($meta->getAttribute("name") == "keywords”){
$dataReturn[”keywords"] = $meta->getAttribute("content");
}
}
}

Weighting Important Data
Tags you should care
about: meta (include OG),
title, description, h1+,
header
Bonus points for adding in
content location modifiers

Weighting Important Tags
//our keyword weights
$weights = array("keywords" => "3.0",
"meta" => "2.0",
"header1" => "1.5",
"header2" => "1.2");
//add modifier here
if(strlen($word) > 2 && !in_array($word, $common_words)){
$searched_words[$word]++;
}

Expanding to Phrases
2-3 adjacent words, making
up a direct relevant callout
Seems easy right? Just like
single words
Language gets wonky
without stop words

Adding in Time Interactions
Interaction with a site does
not necessarily mean
interest in it
Time needs to also include
an interaction component
Gift buying seasons see
interest variations

Grouping Using Commonality
Interests
User A
Interests
User B
Interests
Common

Using Color Theory
Products with a feel-good message
Happiness, energy, encouragement
Health care (but not food!)
Relatable, calm, friendly, peace, security
Startups / innovative products
Creativity, imagination
Auction sites (but not sales sites!)
Passion, stimulation, excitement, power

The CSS Service Engine
lesscss.org
sass-lang.com
learnboost.github.com/stylus

http://leafo.net/lessphp/
Design Engine Foundation: LESSPHP
+

The Basics of a Design Engine
//create new LESS object
$less= new lessc();
//compile LESS code to CSS
$less->checkedCompile(
'/path/styles.less',
'path/styles.css');
//create new CSS file and return new file link
echo "<link rel='stylesheet' href='http://path/styles.css'
type='text/css' />";

Passing Variables into LESSPHP
//create a new LESS object
$less = new lessc();
//set the variables
$less->setVariables(array(
'color' => 'red',
'base' => '960px'
));
//compile LESS into PHP and unset variables
echo $less->compile(".magic { color: @color;
width: @base - 200; }");
$less->unsetVariable('color');

Implementing Color Functions
Lighten / Darken Saturate / Desaturate
Adjust HueMix Colors

Managing Irrelevant Content
Remove / hide content
based on user profile
and state

Managing Irrelevant Content
//variables passed into LESS compilation
"percent" => "80%",
));
//LESS template
.highlight{
@bg-color: "#464646”;
@font-color: "#eee";
background-color: fade(@bg-color, @percent);
color: fade(@font-color, @percent);
}

Traits of the Bored
Distraction
Repetition
Tiredness
Reasons for Boredom
Lack of interest
Readiness
Acting on Disinterest / Boredom

Highlighting on Agitated Behavior
Highlight relevant
content to reduce
agitated behavior

Acting Upon User Queues
"percent" => "100%",
"size-mod" => "2"
));
Variables passed into LESS script

Acting Upon User Queues
.highlight{
@bg-calm: "blue";
@bg-action: "red";
@base-font: "14px";
background-color: mix(@bg-calm,
@bg-action,
@percent );
font-size: @size-mod + @base-font;
}
LESS script logic for color / size variations

Interaction and Emotion Plugin
jQuery Behavior Miner
by Cedric Dugas
https://github.com/posa
bsolute/jquery-
behavior-miner

In the End…
What a person is interested in
What a person is doing
What their emotional state is

http://slideshare.com/jcleblanc
Thank You! Questions?
Jonathan LeBlanc
Head of Developer Evangelism N.A. (PayPal)
Github: http://github.com/jcleblanc
Slides: http://slideshare.net/jcleblanc
Twitter: @jcleblanc

Creating Operational Redundancy for Effective Web Data Mining

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Creating Operational Redundancy for Effective Web Data Mining

Similaire à Creating Operational Redundancy for Effective Web Data Mining (20)

Plus de Jonathan LeBlanc

Plus de Jonathan LeBlanc (20)

Dernier

Dernier (20)

Creating Operational Redundancy for Effective Web Data Mining

Notes de l'éditeur