How do you build a general purpose data collection tool that can be integrated into 100+ sites, handle 15,000 1K writes/second, and have nearly zero downtime? How do you do that and have an architecture virtually unchanged for 3+ years? Start with using Mongo. Michael Makunas and Ramesh Nuthalapati from Viacom Media Networks, home of MTV, Comedy Central, Nickelodeon, and dozens of other brands, detail the architecture of the voting, polling, and data collection system that powers everything from voting for the MTV Video Music Awards, to sliming celebrities at the Nickelodeon Kids Choice Awards, to contests to meet Miley Cyrus.

1. MongoDB
@
Viacom
1
(or how to support a few million kids sliming their favorite celebrities all at once.)

2. Who Are We?
Michael Makunas
Director - Multiplatform Engineering
@mmakunas
Ramesh Nuthalapati
Architect - Multiplatform Engineering
@nuthalapati
2

3. Who Are We?
Viacom Media Networks
Viacom creates original entertainment content for every
audience, on every screen around the world. The company's
brands strive for creative excellence and cultural impact across
programming, motion pictures and digital experiences that live
online, on mobile, and via apps.
3

4. Today We Are Talking
About...
Funnel
Viacom’s internal high volume polling, voting, and data
collection service.
4

5. …and How MongoDB just
works.
5
• Launched in 2011 on 2.0
• Currently on 2.4

6. Use Cases
Award Show Voting
•MTV Video Music Awards (VMAs)
•European Movie Awards (EMAs)
•Kids Choice Awards (KCAs)
•CMT Music Awards
Polling
•MTV March Musical Madness
•Spike Bellator On Air Match Voting
Contests
•MTV Ultimate Fan Experience
6

7. What is “High Volume” to
Viacom?
•From big to little, continuous “horizontal” support across
100+ global sites
•Voting, polling, and data collection occurs 24/7
•No “maintenance windows”
•Sudden spikes to 15K+ votes per second from on-air
“Throws” during prime time viewing: “Go to the site and
Vote Now!”
•Storage of 100s of millions of votes for analysis after events
7

8. A Very Important
Question...
8

9. Who will survive
the Human Slime
Car Wash!?!?!?
9

10. Statistics (Pushing The Limit)
2015 Kids Choice Awards
•531 Million votes (a new record!)
•7 Million votes during the show
10

11. Some Definitions
Funnel
•Our internal name for our system
Site
•A brand and/or site specific namespace for collecting data
Tent-pole
•A major programming (entertainment) event
Collection
•An individual poll, voting event, sweepstakes, etc.
•NOT a MongoDB Collection
Entry
•A single vote or poll entry
Processor
•Performs an action on a entry (validate it, persists it, etc.)
11

12. History
•Why did we build Funnel?
•Why did we choose Mongo?
12

13. Why Did We Build It?
•Previous systems unmanageable
•One system for data collection
•Contests, sweepstakes, small polls
•Built in-house with ATG Dynamo and MySQL
•Per-site deployment model meant multiple installations per
brand
•Flexible schema == MySql “xml blobs”
•Separate system for high volume polling
•Comercial
•Lack of customization
•Per-brand installation
13

14. Why Did We Choose
Mongo?
•Flexible schema document style database provides the
quick-to-market support needed by 100+ sites, brands, and
countries (All who want things “just a little bit different”)
•Proven scalability
•JSON in and JSON out
14

15. Architectural Goals
• High write throughput over read throughput and
consistency
• Handle up to 10,000 1KB writes per second. Read
throughput is secondary
• High Availability
• Replace dozens of separate installations and services
• Wide adoption across internal brands
15

16. Server Layout
16
Primary
Secondary
Secondary
Replica Set 1
(day-to-day use)
Replica Set 2
(tent-pole events)
Primary
Secondary
Secondary

17. Server Layout
• High availability
• Primary, Secondary configuration with automatic
failover
• Primary stores all reads and writes, secondary stays in-
sync with Primary
• Monitoring with MongoDB Management Service (MMS)
• Email Alerts from MMS for high volume op inserts
17

18. More on MMS
18
KCA 2015 MMS Graph

19. Funnel UI

20. Schema For a Typical
Collection
"_id" : "kca-live-vote-2015-event",
"_class" : "com.mtvi.funnel.domain.model.collection.Collection",
"site" : "kca.nick.com",
"name" : "kca-live-vote-2015-event",
"description" : "KCA Live Event",
"owner" : "nuthalar",
"start" : ISODate("2015-03-02T05:00:00Z"),
"end" : ISODate("2015-05-10T04:00:00Z"),
"createdDate" : ISODate("2015-03-03T17:34:55.113Z"),
"lastUpdateDate" : ISODate("2015-03-26T14:40:15.665Z"),
"tags" : ["live-show"],
"processorChain" : [...],
“questions” : [...] -- Questions has Options and Validators
20

21. Processors
• Each entry received by Funnel will be handled by an ordered list of
processors (the ProcessorChain) As each processor in the list is visited, the
processor will decide whether to continue to the next one or to terminate
further processing
• A Processor can raise an error, and its detail message will be configurable
and localizable. When an error is raised, it can signal the system to take a
few different actions depending on the severity of the error condition:
Log the error but continue to processing
Bubble up the error to the client and cease further processing
Ignore the error, cease further processing, and return an OK response
to the client
21

22. Processors (UI)
22

23. Questions: Email
23

24. Questions: Email
{
"name":"email",
"displayText":"email",
"persistable":true,
"source":"ENTITY_BODY",
"validators":[
{
"enabled":true,
"failureMessage":"Please enter a valid email address to proceed.",
"contentType":"EmailAddressValidator",
"_class":"com.mtvi.funnel.domain.model.collection.questions.validators.E
mailAddressValidator"
},
{
"enabled":true,
"failureMessage":"This field is required. Please enter a response to proceed.",
"contentType":"RequiredValidator",
"_class":"com.mtvi.funnel.domain.model.collection.questions.validators.RequiredValidator"
}
],
"seqNum":NumberLong(0),
"tallyEnabled":false,
"isMandatory":false,
"contentType":"EmailQuestion"
}
24

25. Questions: Single Choice
25

26. Questions: Single Choice
26
{
"multipleAnswers":false,
"options":[
{
"optionValue":"andrew-garfield-male-action-star",
"displayText":"andrew-garfield-male-action-star",
"customOption":false
},
{
"optionValue":"channing-tatum-male-action-
star",
"displayText":"channing-tatum-male-action-
star",
"customOption":false
},
{
"optionValue":"chris-evans-male-action-
star",
"displayText":"chris-evans-male-action-
star",
"customOption":false
},
{
"optionValue":"chris-pratt-male-action-star",
"displayText":"chris-pratt-male-action-star",
"customOption":false
},
],
"writeInOptionEnabled":false,
"name":"favorite-male-action-star",
"displayText":"favorite-male-action-star",
"persistable":true,
"source":"ENTITY_BODY",
"validators":[
{
"enabled":false,
"failureMessage":"This field is required. Please enter a
response to proceed.",
"contentType":"RequiredValidator",
"_class":"com.mtvi.funnel.domain.model.collection.questions.validators.R
equiredValidator"
},
{
"enabled":true,
"failureMessage":"Please select a valid choice to proceed.",
"contentType":"ChoiceValidator",
"_class":"com.mtvi.funnel.domain.model.collection.questions.validators.C
hoiceValidator"
},
{
"enabled":true,
"failureMessage":"Please fill in the 'Other' value to
proceed.",
"contentType":"WriteInRequiredValidator",
"_class":"com.mtvi.funnel.domain.model.collection.questions.validators.W
riteInRequiredValidator"
}
],
"seqNum":NumberLong(0),
"tallyEnabled":true,
"isMandatory":false,
"sm4MappingEnabled":false,
"sm4Mapping":"",
"exodusMapping":"",
"contentType":"SingleChoiceQuestion",
"_class":"com.mtvi.funnel.domain.model.collection.questions.QuestionWith
Options"
}

27. An Entry would look like…
27
{
"_id" : ObjectId("53c698c1ebd268ff9b76034d"),
"answers" : {
"question" : "option-3",
"source" : "mobile",
"True-Client-IP" : "166.77.6.8"
},
"createdDate" : ISODate("2014-07-16T15:22:41.593Z")
}

28. Entry Trend Graph
28

29. A Tally would look like…
29
{
"_id" : ObjectId("53c698c51180cc552e79c12d"),
"option-1" : 1249596,
"option-2" : 341404,
"option-3" : 466646,
"questionName" : "question",
"timeWindowType" : "ALL_TIME"
}

30. Optimizations
We didn’t get it perfect the first time:
• Correct Indices - Index on created Date of Entry Collection.
EnsureIndex is called during collection creation and
update
• Batch update of vote tallies - Update the count per second
instead of per Entry
• Hourly Stats – Pre-caluculating hourly totals saved resources
during live events to get stats for past 24 hours, 7 days and
30 days
30

31. SPAM Prevention
•De-duplicator processor: De-duplicates X number of
entries for a defined period (eg. 30 Entries per minute).
Internal to Funnel application
•VERISIGN DDoS Protection service
•Akamai WAF (part of Kona Web Application Firewall)
•CAPTCHA
•Authentication (FLUX - Viacom Internal System)
31

32. Before & After
Previous Systems
•Decentralized
•Multiple Installations
•Outdated App Server
•Rigid Schema requiring
rigid change control
process
32
Funnel
•Service Oriented
•Performant
•Flexible Schema; easily
extended

33. Our Track Record
•100+ Sites
•15K Collections
•10+ Billion Votes to date
33

34. Where else MongoDB is
Used @ Viacom
•Used in production since v 1.6
•Content Management System
•Viacom Identity Platform (Auth & User Profiles)
34

35. Wait…
35

36. Who survived the
Human Slime
Carwash!?!?!?
36

37. Questions ?
37

38. Thank You!
38
• Michael Makunas -- @mmakunas
• Ramesh Nuthalapati -- @nuthalapati