Smart Canvas is a machine learning platform that delivers personalized recommendations for web and mobile content using a hybrid recommender system. It analyzes user interactions and ingests content from various sources to provide recommendations using algorithms like collaborative filtering, content-based filtering, and popularity rankings. The system is evaluated using metrics like nDCG, CTR, coverage, and user engagement to analyze recommendation quality and make improvements.

1. Smart Canvas®
Hybrid Recommender System
Architecture and Evaluation
Gilmar Souza
Gabriel Moreira

2. We are global with people in Brazil, USA,
Europe, Australia, Japan and China
Our excellence has been recognized by the
market with awards since our foundation in 1995
We are truly multi-cultural, an army of over
2000 talented employees and their great stories

3. Strong presence in strategic regions
USA
• Atlanta
• Philadelphia
• Houston
• San Francisco
• Somerset NJ
• New York
Brazil
• Campinas
(headquarters)
• Belo Horizonte
• Rio de Janeiro
• São Paulo
UK
• London
China
• Ningbo
• Shanghai
Japan
• Tokyo
3
Australia
• Sydney

5. Smart Canvas is a platform that delivers
web and mobile user experiences using
machine learning algorithms.

6. Smart CanvasReordered Stacks
Desktop mobile sites
and mobile apps
Card deck
(Chronological
content inventory)
Big Picture: 1 aggregate -> 2. cardify -> 3. curate
Your environment
(internal and external)
Public Data
RSS
Social
Networks
All of your portals,
apps and people
3rd party systems
through custom
connections
1 3
2
metrics and insights
Algorithms

7. Context

8. ● Cloud-based SaaS, hosted on Google
● Multi-tenant
● Used in different domains (websites, intranets,
collaboration portals)
● Content produced by the users or ingested from
external services.
● Hybrid recommender system
● Implicit and explicit feedback (touchpoints)
Some Context

9. Following concepts of material design, presents content as cards and allow like, dislike, pins and
share interactions. Curation merges search relevance with personalized recommendations.
Smart Canvas User Interface

10. Websites: Ancar Ivanhoe Shopping Malls
Digital strategy revamp

11. Collaboration Portal: Whirpool Winning Workplace Transformation
The place to learn about Google Apps

12. Collaboration Portal: SulAmérica Seguros: Innovation Portal
Capturing ideas and inspiration to overcome new market challenges

13. Collaboration Portal: Motorola Smart Analytics Portal
Reports repository reimagined
Designing for Choice:
The Making of the
Wood-Back Moto X
[VIDEO]New Year, New You
Publisher Name
30 december 2014
Get to Know Rhea
Jeong, a Motorola
Designer of Moto Hint
Search
Analytics
999 999 999
Inside Motorola Inside Motorola Inside Motorola Inside Motorola
999 999 999 999 999 999 999 999 999
...
...
Smart Analytics
Moto X Device Activations:
Latin America North
source.com
Customer Satisfaction
4Q2014
source.com
Model Sales per region
source.com
All
Number of support calls per
region
source.com
What's new Communities AnalyticsPeople ToolsEvents
Smart Analytics
Inside Motorola
999 999 999
Number of support calls
per region
source.com

14. ● 23 Tenants (customer portals).
● 33K cards ingested.
● 8.5M users (anonymous and logged).
● 30M touchpoints (user events) captured.
● 14M recommendations provided.
● 300 ms average recommendation response time.
Last 12 months In Numbers

15. Architecture

16. Cards
Touchpoints
Users
Data Layer Batch Layer
Recommenders
Offline Components
Similarity
Algorithms
Serving Layer
Recommenders
Online Components
Hybrid
Composition
Recommendation Layers

17. Search
Engine
search by terms.
(context: url,
locale
device and
person id)
cards
satisfying the
search criteria
Dynamic
Recommender
Strategy
Recommenders
cards reordered by
relevance.
Recommenders are
created and configured
based the context pattern
(locale, device, time)
1
2
3
4
5
search results curated by
personalized recommenders
6
user clickstream (touchpoints) is
used to refine recommendations
Recommendation Process

18. Online Serving
Google App Engine (Java)
Batch Processing
Google Compute Engine
Google Cloud Storage
Dynamic
Recommender
Factory
Recommenders
HDFS
Memcache On-demand Jobs
Datastore
(NoSQL)
The recommender system is implemented on Google Cloud Platform and Hadoop Ecosystem
to ensure its scalability and performance at large.
Pig Mahout Python
Smart Canvas
Users
Technology Components

19. Algorithms

20. HYBRID RECOMMENDER STRATEGY IS AN AGGREGATION OF THESE ALGORITHMS
Personalized
USER-BASED CF - Find similar users and recommend cards those users liked
CONTENT-BASED FILTERING - Recommends cards with content similar to the user's preference
vector (weighted average of the most relevant words in the content of the cards user interacted with).
ITEM-ITEM FREQUENCY - Recommends X cards frequently read together with the last Y cards user
has read.
DISLIKE FILTER - Reduces relevance of cards user has disliked
Non-Personalized
POPULARITY - Recommends most popular cards, decreasing its relevance as them become older
RECENCY - Recommends most recent cards
FIXED PACK - Human curation, where cards are fixed on top by the admin users (marketers)
ESSENTIALS PACK - Recommends a defined set of cards for new users (< Z interactions)
Recommender Algorithms Overview

21. Evaluation Framework

22. ● Normalized Discounted Cumulative Gain (nDCG)
Verify whether the cards are ranked accordingly to the relevance
for the users
● Top-N Accuracy
Verify whether the more relevant cards for the user are among
the Top-N
○ Precision - How many Top-N cards are relevant?
○ Recall - How many relevant cards are among the Top-N?
Offline Metrics

23. Cognition Trainer
Offline Evaluation workflow
Offline Evaluator
1 - Filters touchpoints from train and test sets
periods
2 - Run cognition job
3 - Gets recommendations for each user using a
SmartCanvas WebService
Users
Items
Interactions
Cognition
WebService
Pre-generated
recommendations
(Utility Matrix)
2
1
3
4
5
4 - Compare recommendations ranking with real
user interactions in test set, and calculate offline
metrics and statistical significance
5 - Stores results in BigQuery tables, which can be
accessed by a Tableau dashboard
Offline
Metrics
Database

24. 1 - Users sessions are split among control group and variants of hybrid recommender settings
2 - User interactions and recommendation logs are recorded as touchpoints in BigQuery
3 - Online Evaluator Job calculates engagement, accuracy, and coverage metrics
4 - Results are stored in BigQuery and accessible from Tableau Dashboards
Online metrics extraction workflow
4
1
3
Online Evaluator Job
Traffic Split
(A/B Tests)
Touchpoints
bigquery
2

25. Online Metrics - Engagement
1. Average sessions by user
Considers only engagement of active users -
with at least one visit (session) in the period
2. Average PageViews by session
Objective: Improve User Engagement with best Recommendations
Aggregation: Daily and Weekly
3. Average session strength
where S are the sessions and w the interaction strength
4. Average session minimum navigation time
Minimum navigation is calculated taking elapsed seconds between
the first and last touchpoints in the session.
When there is only one touchpoint in the session, 60 seconds will be
considered as navigation time.

26. Online Metrics - Engagement

27. Online Metrics - Conversion
2. CTR for Personalized and Non-
Personalized recommendations
Objective: Increase Click-Through Rate (CTR) for cards
recommended to the user
1. CTR by Recommender
#interactions / #visualizations
where # visualizations assumes that all cards
were viewed till at least the last card position
(ranking order) the user has interacted in the
page,
or a minimum position of 10, considering the
user has made at least one scroll in the page on
a desktop browser.

28. Online Metrics - Conversion

29. Online Metrics - Conversion

30. Online Metrics - Conversion

31. Online Metrics - Coverage
3. General Users
Coverage
% of the users that receives at list
one personalized recommendation
during a given period
Objective: Increase the number of users and cards with
personalized recommendations
1. Users Coverage by Recommender
#unique_users_with_at_least_a_personalized_
recommendation / #unique_users_accesses
2. Cards Coverage by Recommender
#cards_with_at_least_a_personalized_recomme
ndation / #active_recommendable_cards
where #active_recommendable_cards assumes all cards that
were active (status=CREATED, approvalStatus=APPROVED) at a
given date

32. Online Metrics - Coverage

33. A/B Testing

34. ● Recommendations embedded in the search results using
elasticsearch
● Real-time and Near real-time recommendations delivered via
message bus.
● New product focused on enterprise collaboration
Under Construction

35. ● Cloud computing allow us to scale up and down easily
● A hybrid approach helped us to deal with a multi-domain
scenario and to balance algorithms pros and cons.
● Our evaluation framework (offline and online) allows us to assess
hypothesis, tune hyperparameters, and provides a deep
understanding of recommenders effectiveness.
Conclusions

36. Gilmar Souza
gilmarj@ciandt.com
Gabriel Moreira
gabrielpm@ciandt.com
Thank you!
www.smartcanvas.com

37. Gilmar Souza
gilmarj@ciandt.com
Gabriel Moreira
gabrielpm@ciandt.com
Thank you!
www.smartcanvas.com
RecSys Team