A presentation about a distributed Erlang web platform focused on scalability and feature isolation.

1. Introduction
Spilgames & Me
Old Portals
Landscape
Overview
Evaluation
Wish List
Tech Wishes
Taskforce Wishes
Available
Frameworks
A Widget
Platform
Components
The Flow
Testing &
Deploying
Managing
Widgets
Holding Control
Next Steps
Summary
Contributions
A Web Widget Platform
Enrique Paz
Senior Backend Developer
13/06/2013
1/33

2. Introduction
Spilgames & Me
Old Portals
Landscape
Overview
Evaluation
Wish List
Tech Wishes
Taskforce Wishes
Available
Frameworks
A Widget
Platform
Components
The Flow
Testing &
Deploying
Managing
Widgets
Holding Control
Next Steps
Summary
Contributions
About Me
• Passionate Erlang developer
• Testing enthusiast
• Love beautiful code!
2/33

3. Introduction
Spilgames & Me
Old Portals
Landscape
Overview
Evaluation
Wish List
Tech Wishes
Taskforce Wishes
Available
Frameworks
A Widget
Platform
Components
The Flow
Testing &
Deploying
Managing
Widgets
Holding Control
Next Steps
Summary
Contributions
Spilgames
• Gaming Platform
• Serving data to 190+ countries world-wide
• 180+ million unique users per month
• Multiple Platforms: Desktop, Mobile Native & Web
• 300+ employees
• Offices in The Netherlands & China
• Revenue: Advertising & EUM
3/33

4. Introduction
Spilgames & Me
Old Portals
Landscape
Overview
Evaluation
Wish List
Tech Wishes
Taskforce Wishes
Available
Frameworks
A Widget
Platform
Components
The Flow
Testing &
Deploying
Managing
Widgets
Holding Control
Next Steps
Summary
Contributions
Old Portals Landscape
4/33

5. Introduction
Spilgames & Me
Old Portals
Landscape
Overview
Evaluation
Wish List
Tech Wishes
Taskforce Wishes
Available
Frameworks
A Widget
Platform
Components
The Flow
Testing &
Deploying
Managing
Widgets
Holding Control
Next Steps
Summary
Contributions
Brands and Sites
• 5 Brands/Channels for target audiences
• 1-15 Sites for different geographical regions
• Strong domains
www.spel.se for teens in Sweden
www.ojogos.pt for families in Portugal and Brazil
. . .
5/33

6. Introduction
Spilgames & Me
Old Portals
Landscape
Overview
Evaluation
Wish List
Tech Wishes
Taskforce Wishes
Available
Frameworks
A Widget
Platform
Components
The Flow
Testing &
Deploying
Managing
Widgets
Holding Control
Next Steps
Summary
Contributions
Gaming Portals
6/33

7. Introduction
Spilgames & Me
Old Portals
Landscape
Overview
Evaluation
Wish List
Tech Wishes
Taskforce Wishes
Available
Frameworks
A Widget
Platform
Components
The Flow
Testing &
Deploying
Managing
Widgets
Holding Control
Next Steps
Summary
Contributions
Architectural Overview
• Monolithic portals per
brand
• Static pages with lots of
AJAX calls
• Architectural layers
developed independently
7/33

8. Introduction
Spilgames & Me
Old Portals
Landscape
Overview
Evaluation
Wish List
Tech Wishes
Taskforce Wishes
Available
Frameworks
A Widget
Platform
Components
The Flow
Testing &
Deploying
Managing
Widgets
Holding Control
Next Steps
Summary
Contributions
The Good
• Ability to develop per brand and site
• Acceptable performance
• Clear layer separation
• Fast growth
8/33

9. Introduction
Spilgames & Me
Old Portals
Landscape
Overview
Evaluation
Wish List
Tech Wishes
Taskforce Wishes
Available
Frameworks
A Widget
Platform
Components
The Flow
Testing &
Deploying
Managing
Widgets
Holding Control
Next Steps
Summary
Contributions
The Bad
• Inability to share features between brand codebases
• Aggressive multi-level caching needed to perform decently
• Lots of logic on the client
• Developers can’t switch brands easily
• Long time to market
9/33

10. Introduction
Spilgames & Me
Old Portals
Landscape
Overview
Evaluation
Wish List
Tech Wishes
Taskforce Wishes
Available
Frameworks
A Widget
Platform
Components
The Flow
Testing &
Deploying
Managing
Widgets
Holding Control
Next Steps
Summary
Contributions
Wish List
10/33

11. Introduction
Spilgames & Me
Old Portals
Landscape
Overview
Evaluation
Wish List
Tech Wishes
Taskforce Wishes
Available
Frameworks
A Widget
Platform
Components
The Flow
Testing &
Deploying
Managing
Widgets
Holding Control
Next Steps
Summary
Contributions
Isolation & Performance
• Isolated requests, features and crashes
• Feature toggling on runtime
• Decent number of requests/s with linear
scalability
• Technology aligned application and storage
layers
11/33

12. Introduction
Spilgames & Me
Old Portals
Landscape
Overview
Evaluation
Wish List
Tech Wishes
Taskforce Wishes
Available
Frameworks
A Widget
Platform
Components
The Flow
Testing &
Deploying
Managing
Widgets
Holding Control
Next Steps
Summary
Contributions
Increase Development Speed
• Only one modular codebase
• Small features, easy to deploy and rollback
• Changes testable in isolation
12/33

13. Introduction
Spilgames & Me
Old Portals
Landscape
Overview
Evaluation
Wish List
Tech Wishes
Taskforce Wishes
Available
Frameworks
A Widget
Platform
Components
The Flow
Testing &
Deploying
Managing
Widgets
Holding Control
Next Steps
Summary
Contributions
What’s out there
• MVC Frameworks (Chicago Boss, Zotonic. . . )
(+) Nice layer separation
(+) Mature solutions
(+) Easy to use
13/33

14. Introduction
Spilgames & Me
Old Portals
Landscape
Overview
Evaluation
Wish List
Tech Wishes
Taskforce Wishes
Available
Frameworks
A Widget
Platform
Components
The Flow
Testing &
Deploying
Managing
Widgets
Holding Control
Next Steps
Summary
Contributions
What’s out there
• MVC Frameworks (Chicago Boss, Zotonic. . . )
(+) Nice layer separation
(+) Mature solutions
(+) Easy to use
• But. . .
(-) Feature isolation not key in their design
(-) A VC framework fits better (abstracted model)
13/33

15. Introduction
Spilgames & Me
Old Portals
Landscape
Overview
Evaluation
Wish List
Tech Wishes
Taskforce Wishes
Available
Frameworks
A Widget
Platform
Components
The Flow
Testing &
Deploying
Managing
Widgets
Holding Control
Next Steps
Summary
Contributions
A Widget Platform
14/33

16. Introduction
Spilgames & Me
Old Portals
Landscape
Overview
Evaluation
Wish List
Tech Wishes
Taskforce Wishes
Available
Frameworks
A Widget
Platform
Components
The Flow
Testing &
Deploying
Managing
Widgets
Holding Control
Next Steps
Summary
Contributions
A Widget Platform
15/33

17. Introduction
Spilgames & Me
Old Portals
Landscape
Overview
Evaluation
Wish List
Tech Wishes
Taskforce Wishes
Available
Frameworks
A Widget
Platform
Components
The Flow
Testing &
Deploying
Managing
Widgets
Holding Control
Next Steps
Summary
Contributions
Widgets
• Isolated
• Independent Erlang
applications
• Use ErlyDTL for the view
• Specification in the .app.src
• Can only use what they
specify
16/33

18. Introduction
Spilgames & Me
Old Portals
Landscape
Overview
Evaluation
Wish List
Tech Wishes
Taskforce Wishes
Available
Frameworks
A Widget
Platform
Components
The Flow
Testing &
Deploying
Managing
Widgets
Holding Control
Next Steps
Summary
Contributions
Widgets: declaring everything
• Children
• Public properties
• Private properties
• Themes
• JS decoration
• CSS decoration
{ env , [
{ children , [ wdg_game_list_breadcrumb ] } ,
{ interface , [
{ page , "1" } ,
{ pageSize , " 20 " }
] } ,
{ private_interface , [
{ template , wdg_game_list } ,
{ css , [ " wdg_game_list " ] } ,
{ js , [ " i n i t . j s " ] } ,
{games , [ ] } ,
{ pagination_bar , [ ] }
] } ,
{ themes , [
{ " teens " , [
{ template , wdg_game_list_teens } ,
{ css , [ " teens / wdg_game_list_teens " ] } ,
{ js , [ " teens / i n i t . j s " ] }
] }
] } ,
{ js_modules , [ ] } ,
{ js_event_config , [
{ l i s t e n , [ ’ system . user . search . request ’ ] } ,
{ emit , [ ] }
] }
] }
17/33

19. Introduction
Spilgames & Me
Old Portals
Landscape
Overview
Evaluation
Wish List
Tech Wishes
Taskforce Wishes
Available
Frameworks
A Widget
Platform
Components
The Flow
Testing &
Deploying
Managing
Widgets
Holding Control
Next Steps
Summary
Contributions
Widget Libraries
• Widget type definitions
• Widget behaviour
• Wrappers for common actions
• Connection to the application
layer
• ErlyDTL custom tags
18/33

20. Introduction
Spilgames & Me
Old Portals
Landscape
Overview
Evaluation
Wish List
Tech Wishes
Taskforce Wishes
Available
Frameworks
A Widget
Platform
Components
The Flow
Testing &
Deploying
Managing
Widgets
Holding Control
Next Steps
Summary
Contributions
Widget Platform
• Widget dependencies
• Path to widget mapping
• Erlang web server
• Request Flow
• Global error pages
19/33

21. Introduction
Spilgames & Me
Old Portals
Landscape
Overview
Evaluation
Wish List
Tech Wishes
Taskforce Wishes
Available
Frameworks
A Widget
Platform
Components
The Flow
Testing &
Deploying
Managing
Widgets
Holding Control
Next Steps
Summary
Contributions
NGINX Virtual Host declaration
• Domain mapping
• Widget static files
• Static error pages
• URL rewriting
20/33

22. Introduction
Spilgames & Me
Old Portals
Landscape
Overview
Evaluation
Wish List
Tech Wishes
Taskforce Wishes
Available
Frameworks
A Widget
Platform
Components
The Flow
Testing &
Deploying
Managing
Widgets
Holding Control
Next Steps
Summary
Contributions
Request Flow
21/33

23. Introduction
Spilgames & Me
Old Portals
Landscape
Overview
Evaluation
Wish List
Tech Wishes
Taskforce Wishes
Available
Frameworks
A Widget
Platform
Components
The Flow
Testing &
Deploying
Managing
Widgets
Holding Control
Next Steps
Summary
Contributions
Sets Of Widgets In One Platform
• Allows the cration of small deltas
• Different prearranged sets can be requested
• Different sets can be built per request
22/33

24. Introduction
Spilgames & Me
Old Portals
Landscape
Overview
Evaluation
Wish List
Tech Wishes
Taskforce Wishes
Available
Frameworks
A Widget
Platform
Components
The Flow
Testing &
Deploying
Managing
Widgets
Holding Control
Next Steps
Summary
Contributions
Master & Slaves In 1 VM
• The platform master node runs:
A web server
A management interface
A specific set of widgets
23/33

25. Introduction
Spilgames & Me
Old Portals
Landscape
Overview
Evaluation
Wish List
Tech Wishes
Taskforce Wishes
Available
Frameworks
A Widget
Platform
Components
The Flow
Testing &
Deploying
Managing
Widgets
Holding Control
Next Steps
Summary
Contributions
Master & Slaves In 1 VM
• The platform master node runs:
A web server
A management interface
A specific set of widgets
• Each slave node runs:
Its own specific set of widgets
23/33

26. Introduction
Spilgames & Me
Old Portals
Landscape
Overview
Evaluation
Wish List
Tech Wishes
Taskforce Wishes
Available
Frameworks
A Widget
Platform
Components
The Flow
Testing &
Deploying
Managing
Widgets
Holding Control
Next Steps
Summary
Contributions
wdg1/?origins=w1,preview;w5,preview
• Widgets can be called in any
node
• Widgets can fail
• Widgets can be disabled
24/33

27. Introduction
Spilgames & Me
Old Portals
Landscape
Overview
Evaluation
Wish List
Tech Wishes
Taskforce Wishes
Available
Frameworks
A Widget
Platform
Components
The Flow
Testing &
Deploying
Managing
Widgets
Holding Control
Next Steps
Summary
Contributions
Managing Widgets
25/33

28. Introduction
Spilgames & Me
Old Portals
Landscape
Overview
Evaluation
Wish List
Tech Wishes
Taskforce Wishes
Available
Frameworks
A Widget
Platform
Components
The Flow
Testing &
Deploying
Managing
Widgets
Holding Control
Next Steps
Summary
Contributions
The Need For A Widget Manager
• To manage widgets and versions in the platform nodes
• To keep the clustered platforms in sync
• To make changes in the cluster in an easy way
26/33

29. Introduction
Spilgames & Me
Old Portals
Landscape
Overview
Evaluation
Wish List
Tech Wishes
Taskforce Wishes
Available
Frameworks
A Widget
Platform
Components
The Flow
Testing &
Deploying
Managing
Widgets
Holding Control
Next Steps
Summary
Contributions
How does it work
• SWIFT as a central widget
repository
• HTTP interface with simple
GUI
• Uses the platform
management interface:
List nodes
List widgets in a node
Enable/disable widget in a
node
Change widget version in a
node
• Enforces configuration to new
cluster members
27/33

30. Introduction
Spilgames & Me
Old Portals
Landscape
Overview
Evaluation
Wish List
Tech Wishes
Taskforce Wishes
Available
Frameworks
A Widget
Platform
Components
The Flow
Testing &
Deploying
Managing
Widgets
Holding Control
Next Steps
Summary
Contributions
What’s in the Future?
28/33

31. Introduction
Spilgames & Me
Old Portals
Landscape
Overview
Evaluation
Wish List
Tech Wishes
Taskforce Wishes
Available
Frameworks
A Widget
Platform
Components
The Flow
Testing &
Deploying
Managing
Widgets
Holding Control
Next Steps
Summary
Contributions
Lessons Learned
• Reinforce the concepts adoption often
• Review, review, review
• Measure performance from early
stages
29/33

32. Introduction
Spilgames & Me
Old Portals
Landscape
Overview
Evaluation
Wish List
Tech Wishes
Taskforce Wishes
Available
Frameworks
A Widget
Platform
Components
The Flow
Testing &
Deploying
Managing
Widgets
Holding Control
Next Steps
Summary
Contributions
Coming in H2
• Native widget to backend connectivity
• Per widget render timeout
• In req caching of the BE calls
• Performance analysis and improvements
• Router to be open sourced
30/33

33. Introduction
Spilgames & Me
Old Portals
Landscape
Overview
Evaluation
Wish List
Tech Wishes
Taskforce Wishes
Available
Frameworks
A Widget
Platform
Components
The Flow
Testing &
Deploying
Managing
Widgets
Holding Control
Next Steps
Summary
Contributions
What We’ve Used
• ErlyDTL
• Cowboy
• Lager
• Rebar
(*) semantic versioning, i.e. [">=1.3.1", "<2.0.0"]
(*) shared dependencies
(*) xref fixes
• Lhttpc
• Estatsd
31/33

34. Introduction
Spilgames & Me
Old Portals
Landscape
Overview
Evaluation
Wish List
Tech Wishes
Taskforce Wishes
Available
Frameworks
A Widget
Platform
Components
The Flow
Testing &
Deploying
Managing
Widgets
Holding Control
Next Steps
Summary
Contributions
Questions?
32/33

35. Introduction
Spilgames & Me
Old Portals
Landscape
Overview
Evaluation
Wish List
Tech Wishes
Taskforce Wishes
Available
Frameworks
A Widget
Platform
Components
The Flow
Testing &
Deploying
Managing
Widgets
Holding Control
Next Steps
Summary
Contributions
See You Around!
Thanks!
33/33