Our work on the first personalized newsletter. Recommendation engine is built using collaborative filtering.

1. FD recommendation system
Dung Chu, data scientist
FD Mediagroep

2. Outline
• Business cases analysis and recommendation engine
roadmap
• Personalized email campaigns
• A/B testing results
• Upcoming work

3. About me
• Dung Chu
• MSc in cloud computing @UvA
• Research in image processing group ISLA @UvA:
https://ivi.fnwi.uva.nl/isis/index.html
• Data scientist @FD mediagroep

4. FD mediagroep
Het Financieele Dagblad
Het Financieele Dagblad is dé nieuws- en inspiratiebron die op elk moment
van de dag financieel-economische betekenis geeft aan ontwikkelingen in
de wereld.
Company info geeft altijd real time toegang tot actuele bedrijfs- en
prospectinformatie van 2,5 miljoen bedrijven in Nederland
FD/BNR Networks
FD/BNR Networks brengt ideeën, meningen en talentrijke ondernemers
en professionals samen door de organisatie van o.a. forums, debatten,
netwerksessies, en evenementenreeksen.
BNR Nieuwsradio
BNR Nieuwsradio is de enige radiozender in Nederland waar ambitieuze en
ondernemende mensen 24 uur per dag op de hoogte worden gehouden van
relevant nieuws.

5. Pensioen Pro
De joint venture tussen FD Pensioen Pro en IPN en levert betrouwbaar
nieuws, achtergrond en analyse voor de pensioenprofessional.
Redactie Partners
Produceert custom media producten voor zowel interne als externe
communicatievraagstukken
Fondsnieuws
Het grootste journalistieke platform voor beleggingsprofessionals in
Nederland.
Energeia
Biedt nieuws, data en opinie voor energieprofessionals.

6. FD.nl

7. Publishing rate
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
Number of articles
Nr articles
Per week: ~ 500 articles
Digital-
first

8. FD traffic
2/1/17
2/2/17
2/3/17
2/4/17
2/5/17
2/6/17
2/7/17
2/8/17
2/9/17
2/10/17
2/11/17
2/12/17
2/13/17
2/14/17
2/15/17
2/16/17
2/17/17
2/18/17
2/19/17
2/20/17
2/21/17
2/22/17
2/23/17
2/24/17
2/25/17
2/26/17
2/27/17
2/28/17
FD traffic - february 2017 - daily
Users Pageviews

9. Business cases analysis
• Improve customer retention through direct digital
activation with increased relevance
– Online traffic
– Email traffic
• Improve conversion via more digital engagement
– Convert un-registered to registered customers
• Increase ad sales via increased traffic
All users
Logged-in users

10. Business cases analysis
• Support journalists in understanding customer
reading tastes:
– Provide more article insights to journalists
Choice: internal development of recommendation systems*
* We did try a pilot phase with a third party

11. Where to start
• Online recommendation in website fd.nl
• Email newsletters

12. Where to start
• Online recommendation in website fd.nl
• Email newsletters

13. Current email newsletters

14. Current email newsletters
Popularity
sorting

15. Personalized email newsletters
• Targeted audience was registered customers without
subscriptions:
– Free customers with 5 articles as free
– We wanted to improve conversion rates

16. Personalization: collaborative filtering
• Popularity sorting: no relation between articles
• Relations between articles A and B: if someone is
reading A, what is the prob that he/she finds B also
interesting.
• Relation:
– Content-based relation:
– Relation based on collaborative behavior of reading:

17. Architecture
Collect data
Article distance
matrix
Recommend
PySpark:
+ Convert reader-article matrix to IndexedRowMatrix
+ columnSimilarities(): fast compute of cosine
similarity between columns
+ Input: klant_nr
+ Infer list of articles read from reader-article
matrix
+ Recommended articles: articles with largest
sum of distances to the articles read by the
user
Article
DB
Reader-article matrix
Email
campaign
management
Client list
Emailing
system
Published
last week
Older
articles
28 days
7 days

18. Recommendation step

19. Item-to-item collaborative filtering
• Always sort articles based on cosine metric
• Cold-start problems: with articles that were read
only few times
– Evaluate dot-product metric
– Use popularity version

20. Personalized emails

21. A/B testing
• A setting (“combi”): recommended articles using
item-to-item collaborative filtering
• B setting (“popularity”): recommended articles using
article popularity scores

22. Results: free clients
T-test

23. Results: paid clients (fixed 2000)
T-test

24. Number of articles sent – 1 week
Popularity emails
Personalized emails
6 articles 178 articles (60 clicked)
Long tail
articles

25. Toolings

26. Future work

27. Take-home messages
• Simple recommendation model (with existing tools)
works
• FD Mediagroep has started with AI. And we are doing
more and more in this journey.

28. Twitter: @dungchu
Email: dung.manh.chu@fdmediagroep.nl
LinkedIn: https://www.linkedin.com/in/dungmanhchu/