SoundCloud is the world’s leading social sound platform where anyone can create sounds and share them everywhere. 200 Million people every month listen to sounds on SoundCloud. That is eight percent of the Internet. 12 hours are uploaded on SoundCloud every minute. This means that SoundCloud not only deals with a lot of data (3-digit terabytes approximately) but embraces the concept of “data democratization,” which means that all data must be available for anyone in the company that needs to access and work with it.

1. DATA DEMOCRATIZATION @ SOUNDCLOUD
October, 29th 2013

2. HI, I’M BRUNO

3. SOUNDCLOUD IS THE WORLD’S
LEADING AUDIO PLATFORM

4. Every minute, creators upload
12hrs
of audio

5. reaching over
200m
people every month

6. !
8%
of the internet

8. PRESIDENT OBAMA
FOO FIGHTERS
SNOOP LION
MADONNA
SKRILLEX
MACKLEMORE
JOHN OLIVER˝
(DAILY SHOW/BUGLE)

10. what gets
listened to
where?
how much revenue do
we make in Brasil?
how many new users
did we get from that
campaign?
what makes a
sound successfull?
how do users use
our iOS and
Android apps?
do comments on
tracks correlate
with listens?
did the product
change affect
feature x?
what makes an
artist successfull?

11. DATA DEMOCRATIZATION
• Avoid Silos
• Remove unnecessary restrictions
• Provide simple tools
• Teach People how to use data

12. DATA DEMOCRATIZATION
In one sentence:
Deliver the right information to the
right person at the right time.

13. DATA ANALYSIS AND REPORTING
2010-2012
PRODUCTION DB
ANALYTICS DB

14. DATA ANALYSIS AND REPORTING
Listens
Sounds
Users
Comments
Favorites
Shares
Reposts
Impressions
Clicks
Conversions
Suggestions
Downloads
Taggings
Uploads

15. DATA ANALYSIS AND REPORTING
Listens
timestamp
duration
sound
owner
listener
API-key
(location)
country

16. DATA ANALYSIS AND REPORTING
additional metadata:
• location within sound
• context (location on site)
• segmentation
Listening creates >6000 events/s
BIG DATA

17. HADOOP TO THE RESCUE
2 Datacenter in AMS
200+ Nodes

18. HADOOP TO THE RESCUE
listen data
listen metadata
search data
recommender data
product testing data
backend production data
backend logs

19. HADOOP AND DATA DEMOCRATIZATION
Data is siloed on hadoop
Data governance not existing
Technical hurdles for access
Not realtime
Slow access

20. AMAZON REDSHIFT
Fast fully managed DW service
Optimized for petabyte or more
datasets
Fast query and I/O performance
Columnar storage technology

21. BI INFRASTRUCTURE
Source Systems
2013
Staging Area
DataWarehouse
Data Exploration
Amazon EMR
Hadoop
Pig/Ruby Scripts
COPY
MySql
(production db)
Pig/Ruby Scripts
External Systems
...
ETL Scripts
Job execution powered by:
ETL Scripts

22. IMPORT DATA FROM SOURCE SYSTEMS
First: Gather data from the several source systems into S3
Hadoop
Full/Daily Imports
MySql
(production db)
External Systems
MapReduce for:
- Listens
- Plays
- Impressions
- Affiliations
- ...

23. IMPORT DATA FROM SOURCE SYSTEMS
Second: Rebuild staging area tables for full imports
Based on configuration files
!
tracks
users
client
applications
Create statements generated
...
!
Re-create DISTKEYS and SORTKEYS
Full control in changes in the data
model
Staging Area
!
yaml config files

24. IMPORT DATA FROM SOURCE SYSTEMS
Third: Import the data from S3 to RedShift
tracks
Full import: TRUNCATE & COPY
Daily import: COPY
users
Staging Area
client
applications
...

25. ETL AND DW DATAMODEL
!
ETL scripts divided into layers:
!
- Layer 1: Staging -> DW (dimensions)
- Layer 2: Staging -> DW (fact tables - raw data)
- Layer 3: DW -> DW (aggregated fact tables)
- Layer 4: DW -> Reporting Data Cubes (reporting data)
!

26. ETL AND DW DATAMODEL
DataWarehouse
ETL Layer 1 & 2
ETL Layer 3
ETL Layer 4
Data Exploration
Staging Area
Data Cleaning
Data Transformation
Data Presentation
!
!
SQL
Ruby/SQL Scripts
Data Aggregation
!
Ruby/SQL Scripts

27. JOB SCHEDULE AND EXECUTION
Job-scheduling tool developed
internally
Set dependencies between jobs
Execution in multiple machines
Supports all the ETL layers

28. DATA EXPLORATION
Simple and fast access to data
More time for “deep dives” into
data
Individualized Reporting
Allows interactivity between users
Integrated with RedShift

29. TIMELINE
Week 2
•
•
Week 4
Gap Analysis˝
Business Exploration
(requirements
interviews)
Week 6
Week 8
Week 10
Week 12
Week 14
Week 16
Requirement Analysis˝
•
•
Information Mapping
Design˝
Solution Design (Draft)
End of Analysis Stage˝
•
•
Define Infrastructure˝
Design Data Model
Infrastructure Ready!˝
•
•
•
Build ETL ˝
Build Data Cubes˝
Design Reports/Dashboards (Presentation
Layer)
BI 1.0 is built!˝
•
•
System/Integration
Tests ˝
User Acceptance
BI 1.0 is tested!˝
•
•
User Workshops˝
BI 1.0 Evaluation
BI 1.0 is ready
to use!˝
Milestones˝
Analysis Stage˝
Design & Build˝
Test & Deploy

30. DATA DEMOCRATIZATION
• Reports designed by end users
• Central repository for data analysis
• User interaction
• Data from one source only
• Scalable solution
• Data to the people!

31. what gets
listened to
where?
how much revenue do
we make in Brasil?
how many new users
did we get from that
campaign?
what makes a
sound successfull?
how do users use
our iOS and
Android apps?
do comments on
tracks correlate
with listens?
did the product
change affect
feature x?
what makes an
artist successfull?

32. !
QUESTIONS?

33. THANK YOU!
P.S. WE’RE HIRING.
SOUNDCLOUD.COM/JOBS