How ironSource streams more then 200B events every month into a data warehouse http://www.ironsrc.com/atom https://www.youtube.com/watch?v=cF9iMolKT5w

1. Shimon Tolts
General Manager, Data Solutions
ironSource Atom
Data Flow Management

2. About ironSource: ironSource in Numbers
700
Employees
Sep. 2010
Established
50%
R&D Employees
700
Advertisers
80K
Partnered Apps
100M
Devices using ironSource solutions
shipping in 2016
TEL AVIV ISRAEL
SAN FRANCISCO UNITED
STATES
NEW YORK UNITED
STATES
LONDON UNITED KINGDOM
BANGALORE INDIAHONG KONG CHINA KIEV UKRAINEBEIJING CHINA SHANGHAI CHINA

3. About ironSource: ironSource Hypergrowth
People Reached Each Month
4200
Apps Installed Every Minute
with the ironSource Platform
Registered & Analyzed Data Events
Every Month
160B
800M
50B
0
100B
150B
200B
Jun
2015
Jul
2015
Aug
2015
Sep
2015
Oct
2015
Nov
2015
Dec
2015
Jan
2016
Feb
2016
Mar
2016
Apr
2016
May
2016

4. We needed a way to manage this data:
Our Business Challenge
ProcessCollect Store

5. Micro-Services Architecture
● Everything is a service
● Decoupling
● Distributed systems
Separate lifecycle
● Communication using RESTful /
Queue / Streams

6. Docker
● Linux Container
● Save provisioning time
● Infrastructure as code
● Dev-Test-Production - identical container
● Ship easily

7. Cloud infrastructure
● Pay as you go - (grow)
● SaaS services
● Auto-scaling-groups
● DynamoDB
● RDS *SQL
● Redshift data warehouse

8. Continuous Integration
● From commit to production
● Jenkins commit hook
● Git branching model
● AWS dynamic slaves
● Unit tests
● Docker builds
● Updating live environment

9. A lot of beer later….
Or, the story of ironSource’s data collection and management solution

10. SDKs
And many more… which one do you need?

11. Collection
● Multi region layer - Latency based routing
● Low latency from client to Atom servers
● High Availability - AWS regions does fail!
● Storing raw data + headers upon receiving

12. Data Enrichment
● Enrich data before storing in your Data Lake
and/or Warehouse
○ IP to Country
○ Currency conversion
○ Decrypt data
○ User Agent parsing - OS, Browser, Device...
● Any custom logic you would like! - fully
extendible

13. Data Targets
● Near real-time data insertion - 1 minute!
● Stream data to Google Storage and/or AWS S3
● Smart insertion of data into AWS Redshift
○ Set the amount of parallel copys
○ Configure priority on tables
● BigQuery - Streaming data using batch files
import (saves 20% cost)

14. David Fitcher
From London, United Kingdom
The User Journey
Touchpoint #1
Customizing a
new device
David bought a new LG device.
Installed 12 apps overall, 6 were games.
Gender: Male
76% Probability
Age: 25-35
95% Probability
User Profile: Casual Gamer
81% Probability
Touchpoint #2
Using a mobile app
David is now playing a game which uses our SDK.
What we already know
Probably a ‘Casual Gamer’
What we offer
Rewarded video ads for casual gaming apps
User Profile: Casual Gamer
95% Probability
Subcategory Interest: Simulation Games
LTV Projection: High
One month later…
What we know now

15. Insights
The Data World
Visualization
Data Management
Data Collection

19. Everybody needs a data pipeline
The AWS platform allowed us to build on top of it with an infrastructure that is exactly
tailored to our clients’ needs.
Maximum Flexibility
Any data, from any source,
in any format.
Infinite Scalability
Adapt to your evolving needs
with a pay-as-you-go model.
Own Your Data
We manage the flow,
the data is yours.

20. 10 Million
Free Monthly Events
Thank you!
shimont@ironsrc.com @shimontolts