1. Neustar Cassandra Experience
11/30/11
Chirag Patel
© Neustar, Inc. / Proprietary and Confidential

2. Who is Neustar?
2 © Neustar, Inc. / Proprietary and Confidential

3. UltraDNS Overview
» Cloud based DNS Services
» Managed External DNS
» Basic DNS
» Traffic Management Services
» UltraDDI - Managed Internal DNS, DHCP & IPAM
» Ultra Internet Gateway – Recursive DNS
» SiteProtect – DDOS Mitigation Service

4. Existing design
» External applications inject DNS data
» UI / SOAP-API / AXFR / IXFR
» Millions of record changes per day
» Oracle as persistent store
» Millions of zones
» Hundreds of Millions of records
» Oracle replicates this data into our DNS Server nodes
» 15 nodes globally distributed
» DNS Server answers DNS queries
» 17 Billion queries per day

5. Existing design
Injecting Applications
Oracle Master Databases
Oracle Replicate Oracle Replicate Oracle Replicate
Database Database Database
DNS Server DNS Server DNS Server

6. DNSSEC Requirements
» What is DNSSEC?
» IETF specifications for securing DNS information
» Authentication of DNS answers
» Support for adding DNS Security into existing infrastructure
» Typical record increase of 8x per zone
» Support for versioning existing and new DNS data
» Support for storing DNS data before signing
» DNSSEC &non-DNSSEC injecting applications
» DNS Server will still read data from Oracle
» Separate Beta environment for DNSSEC

7. Why Cassandra?
» Write optimized
» Eventually consistent
» Highly scalable
» Highly available even with heavy write loads
» Easy to understand data model
» Excellent high level, easy to use client libraries available
(hector, pycassa)
» Lot of resources available online
» Mailing list, incl. Jonathan Ellis are extremely responsive

8. Migration to Cassandra
Injecting Applications
Signer Cassandra Cluster Oracle Master Databases
Oracle Replicate Oracle Replicate Oracle Replicate
Database Database Database
DNS
DNS
DNS Server Server
Server
Node N

9. Proof of Concept
» Consulting from Riptano (Now DataStax)
» 2 day data design brain storming
» Used Cassandra v0.7
» Used Hector Cassandra client v0.7

10. Cassandra Data Design
Zone Indexes Zone
Account Index Zone Zone Zone Serialize
ID Key Name Name VID d Zone
Record Indexes ZoneVersion
Zone_RTy Index Record Zone Zone
pe Key ID Name VID
Record ZoneRecordChanges
Zone RecordID Serialized Zone VID Record Change
Name Record ID Type

11. Searching Example
Zone Indexes Zone
google.com V2 <Zone Data V2>
1 Type_Primary_google.c google.com.
om V1 <Zone Data V1>
Type_Primary_yahoo.c yahoo.com yahoo.com V2 <Zone Data V2>
om
V1 <Zone Data V1>
Record Indexes Record
Yahoo.com._ Name_a1_R2 R2
A Yahoo.com R2 <R2 Data>
Name_a3_R3 R3
R3 <R3 Data>
TTL_0086400_R2 R2
TTL_0096400_R3 R3

12. Versioning Example
Zone ZoneVersion
yahoo.com V3 <Zone Data V3> Yahoo.com V3 <Empty>
V2 <Zone Data V2> V2 <Empty>
V1 <Zone Data V1> V1 <Empty>
ZoneRecordChanges
Record V3 R4 Add
Yahoo.com R4 <R4 Data> R2 Del
R3 <R3 Data> V2 R3 Add
R1 Del
V1 R1 Add
R2 Add
Yahoo.com. Del R4, Add Yahoo.com. Del R3, Add Yahoo.com.
R2 R1
V3 V2 V2
D
R3 R3 R1
R4 R2 R2

13. Performance Comparison
» Significant improvement in Write times
» Shotgun across multiple Cassandra nodes in a datacenter
» Insertions times went from minutes to seconds
» No wait between writes
» Oracle replication would choke on large inserts

14. Problems faced
» Versioning
» Indexing/Searching
» Pagination
» Sync issues with Oracle
» Training QA
» Training OPS

15. Lessons learned
» Difficult to sync Oracle & Cassandra together
» Versioning is difficult, need to take different approach
» Pagination might require to cache data in memory
» Redis or memcached
» Searching needs to be well thought of before designing
data model
» Need to involve Operations & QA team sooner for more
feedback

16. Future Plans
» Deploy Cassandra in main production environment
» Simplify existing data model
» Common Interface to interact with Oracle & Cassandra
» Use Cassandra to store smaller set of isolated data.
» Replace Oracle fully with Cassandra in one injecting
application

17. Questions?
Contact Kevin.Mahoney@neustar.biz for details