NoSQL databases are often touted for their performance and whilst it's true that they usually offer great performance out of the box, it still really depends on how you deploy your infrastructure. Dedicated vs cloud? In memory vs on disk? Spindal vs SSD? Replication lag. Multi data centre deployment. This talk considers all the infrastructure requirements of a successful high performance infrastructure with hints and tips that can be applied to any NoSQL technology. It includes things like OS tweaks, disk benchmarks, replication, monitoring and backups.

1. NoSQL Infrastructure

2. ~3333 write ops/s 0.07 - 0.05 ms response

3. David Mytton
Woop Japan!

5. MongoDB at Server Density

6. •27 nodes
MongoDB at Server Density

7. • June 2009 - 4yrs
•27 nodes
MongoDB at Server Density

8. •MySQL -> MongoDB
•27 nodes
MongoDB at Server Density
• June 2009 - 4yrs

9. •MySQL -> MongoDB
•25TB data per month
•27 nodes
MongoDB at Server Density
• June 2009 - 4yrs

10. Why?

11. Why?
• Replication

12. Why?
• Replication
• Official drivers

13. Why?
• Replication
• Official drivers
• Easy deployment

14. Why?
• Replication
• Official drivers
• Easy deployment
• Fast out of the box

15. It’s a little different.

16. Picture is unrelated! Mmm, ice cream.
• Fast network
Performance

17. • Fast network
Performance
EC2 10 Gigabit Ethernet
- Cluster Compute
- High Memory Cluster
- Cluster GPU
- High I/O
- High Storage

18. • Fast network
Performance
Workload: Read/Write?
Result set size
What is being stored?

19. • Fast network
Performance
Use Network Throughput
Normal 0-100Mb/s
Replication (Initial Sync) Burst +100Mb/s
Replication (Oplog) 0-100Mb/s
Backup Initial Sync + Oplog

20. • Fast network
Performance
Inter-DC LAN

21. • Fast network
Performance
Inter-DC LAN
Cross USA Washington, DC - San Jose, CA

22. • Fast network
Performance
Location Ping RTT Latency
Within USA 40-80ms
Trans-Atlantic 100ms
Trans-Pacific 150ms
Europe - Japan 300ms

23. Failover
•Replica sets

24. Failover
•Replica sets
•Master/slave

25. Failover
•Replica sets
•Min 3 nodes
•Master/slave

26. Failover
•Replica sets
•Min 3 nodes
•Master/slave
•Automatic failover

27. •Replication lag
Performance
Location Ping RTT Latency
Within USA 40-80ms
Trans-Atlantic 100ms
Trans-Pacific 150ms
Europe - Japan 300ms

28. Replication Lag
1. Reads: eventual consistency

29. Replication Lag
1. Reads: eventual consistency
2. Failover: slave behind

30. Eventual Consistency
Stale data

31. Eventual Consistency
Stale data
Inconsistent data

32. Eventual Consistency
Stale data
Inconsistent data
Changing data

33. Eventual Consistency
Use Case Needs consistency?
Graphs No
User profile Yes
Statistics Depends
Alert config Yes

34. Replication Lag
1. Reads: eventual consistency
2. Failover: slave behind

35. Slave behind
Failover: out of date master

36. • Safe by default
>>> from pymongo import MongoClient
>>> connection = MongoClient(w=int/str)
Value Meaning
0 Unsafe
1 Primary
2 Primary + x1 secondary
3 Primary + x2 secondaries
MongoDB WriteConcern

38. Tags
{
_id : "someSet",
members : [
{_id : 0, host : "A", tags : {"dc": "ny"}},
{_id : 1, host : "B", tags : {"dc": "ny"}},
{_id : 2, host : "C", tags : {"dc": "sf"}},
{_id : 3, host : "D", tags : {"dc": "sf"}},
{_id : 4, host : "E", tags : {"dc": "cloud"}}
]
settings : {
getLastErrorModes : {
veryImportant : {"dc" : 3},
sortOfImportant : {"dc" : 2}
}
}
}
> db.foo.insert({x:1})
> db.runCommand({getLastError : 1, w : "veryImportant"})

39. {
_id : "someSet",
members : [
{_id : 0, host : "A", tags : {"dc": "ny"}},
{_id : 1, host : "B", tags : {"dc": "ny"}},
{_id : 2, host : "C", tags : {"dc": "sf"}},
{_id : 3, host : "D", tags : {"dc": "sf"}},
{_id : 4, host : "E", tags : {"dc": "cloud"}}
]
settings : {
getLastErrorModes : {
veryImportant : {"dc" : 3},
sortOfImportant : {"dc" : 2}
}
}
}
> db.foo.insert({x:1})
> db.runCommand({getLastError : 1, w : "veryImportant"})
(A or B) + (C or D) + E
Tags

40. {
_id : "someSet",
members : [
{_id : 0, host : "A", tags : {"dc": "ny"}},
{_id : 1, host : "B", tags : {"dc": "ny"}},
{_id : 2, host : "C", tags : {"dc": "sf"}},
{_id : 3, host : "D", tags : {"dc": "sf"}},
{_id : 4, host : "E", tags : {"dc": "cloud"}}
]
settings : {
getLastErrorModes : {
veryImportant : {"dc" : 3},
sortOfImportant : {"dc" : 2}
}
}
}
> db.foo.insert({x:1})
> db.runCommand({getLastError : 1, w : "sortOfImportant"})
(A + C) or (D + E) ...
Tags

41. Picture is unrelated! Mmm, ice cream.
• Fast network
•More RAM
Performance

42. http://www.slideshare.net/jrosoff/mongodb-on-ec2-and-ebs

43. http://blog.pythonisito.com/2011/12/mongodbs-write-lock.html

44. http://blog.pythonisito.com/2011/12/mongodbs-write-lock.html

45. More RAM = expensive
Performance
x2 4GB RAM 12 month Prices

46. RAM
SSDs
Spinning disk
Cost Speed

47. Softlayer disk pricing
Performance

48. EC2 disk/RAM pricing
Performance
$2232/m
$2520/m
$43/m
$295/m

49. SSD vs Spinning
Performance

50. SSD vs Spinning
Performance

51. SSD vs Spinning
Performance

52. Tips: rand()
•Field names

53. Tips: rand()
•Field names
•Covered indexes

54. Tips: rand()
•Field names
•Covered indexes
•Collections / databases

55. Backups
What is the use case?

56. Backups
What is the use case?
Fixing user errors?
Point in time restore?
Disaster recovery?

57. Backups
•Disaster recovery
Offsite

58. Backups
•Disaster recovery
Age
Offsite

59. Backups
•Disaster recovery
Age
Offsite
Restore time

60. Backups
Frequency
Consistency
Verification

61. www.flickr.com/photos/daddo83/3406962115/
Monitoring
•System
Disk i/o
Disk use

62. www.flickr.com/photos/daddo83/3406962115/
Monitoring
Disk i/o
Disk use
•System
Swap

63. www.flickr.com/photos/daddo83/3406962115/
Monitoring
Optime
State
•Replication

64. mongostat

65. Monitoring tools
Run yourself
Ganglia

66. Monitoring tools
www.serverdensity.com

67. David Mytton
david@serverdensity.com
@davidmytton
Woop Japan!
blog.serverdensity.com/mongodb