2016-JAN-28 -- High Performance Production Databases on Ceph

Medallia © Copyright 2015. Confidential. 1
High-Performance Production
Databases on Ceph

Medallia © Copyright 2016.. 2
At Medallia, we collect, analyze, and display terabytes of structured &
unstructured feedback for our multibillion dollar clients in real time.
And what’s more: we have a lot of fun doing it.
I’ve been at Medallia since 2010, growing from 70 to 700 employees.
Who are you?
Hi, I’m Thorvald, Architect @ Medallia

AGENDA
1
2
3
4
5
6
Networking/Storage Mobility
The Dream
Provisioning/Orchestration
Demo!
Real-world performance
Challenges and next steps

Tech Industry Speak for “Last Year”
• New version of our analytics engine
○ Dream: Horizontally super-scalable! 1000s of servers!
• Reality: Peeking at Production
○ 100s of servers
■ .. that had individual names
■ .. almost, but not quite, entirely unlike each other
■ .. manual service placement
.. and server placement
■ “Don’t touch it”
A long, long time ago...

• Skip 2-3 generations and go direct to “next gen”
○ MicroServices, Containers, <insert buzz-word here>
• Proof-of-Concept using 40GbE, Ceph, Docker
○ Resilient enough that it’s a problem to test resiliency
○ Performant enough to replace dedicated servers
• Can we run everything on this new infrastructure?
Rapid Evolution Time!
Jump into the future

Design Goals
Keep it SIMPLE
• Commodity Components &
Supported Open Standards
• Fully automated provisioning
and reinstall
• Cheap & Scalable
• Immutable Servers
● No service that is tied to
specific hardware
● Every component must be
able to run anywhere
● Redundancy at App Layer
● Self-Healing
No Special MachinesCommodity Products

Linux (Ubuntu)
2xIntel E5-v3
256GB Memory
40GbE Network
100GB SSD
Standard Rack
22 x Compute Node 3 x Networking
Linux (Cumulus)
1xIntel Atom 64-bit
8 GB Memory
32x40 GbE Network
8 x Storage Node
Linux (Ubuntu)
1xIntel E5-v3
64 GB Memory
40GbE Network
8x800GB SSD
PCIe NVRAM
Unified and Scalable

Where do you draw the line?
• Application in relocatable Container?
• Load-balancer in relocatable Container?
• DNS server in relocatable Container?
• Database in relocatable Container?
Challenge
Everything as containers
Everything in Containers

AGENDA
1
2
3
4
5
6
The Dream
Demo!

Problem: Network Mobility
Relocating non-movable services
DataCenter Firewall
Host: 10.1.2.3:80
Host: 10.1.2.5:80
172.17.0.3:80 nginx
Host: 10.1.2.4:2181
172.17.1.0:2181 zookeeper
172.17.1.2:80 application

Network Mobility
host1
eth0
10.1.2.3/30
veth0
container-A
10.4.5.6/32
eth0
host2
host3
hostN

Route Propagation
• Open Shortest Path First
○ Propagated Link State Database
○ Supported by every vendor
• Computes network paths with Dijkstra algorithm
• Moving 30 000 routes: ~ 1 second
• BGP works just as well, OSPF auto-configures easier
OSPF
Fully relocated IP address

Problem: Storage Mobility
Relocating very non-movable services
Host: 10.1.2.3
172.17.1.2 application
Host: 10.1.2.4
172.17.5.5 PostgreSQL
Host: 10.1.3.5

• Docker images are ephemeral
• Persistent volumes to the rescue!
○ Which work on your local machine only
• Proprietary Solutions needed for HA
○ iSCSI (Large Storage Vendor)
○ NFS (Large Storage Vendor, and.. performance?)
○ pNFS (... right)
• Scale up, but not out
• SLA? 4 hours hardware support not good enough!
Storage Mobility
Where did the filesystem go?

• No need to communicate with metadata servers in hot path
• Clean design; we understand enough to go fix problems ourselves
• Need more capacity?
○ Add servers!
• Need more aggregate performance?
○ Add servers!
• Need more single-node performance?
○ Get creative!
Ceph
Short Version

Storage “Solved”
Relocating very non-movable services
Host: 10.1.2.3
172.17.1.2 application
Host: 10.1.2.4
Host: 10.1.3.5
Replicated Ceph Cluster

What happens when the server for your monitor dies?
• It’s “interesting” to switch Ceph monitor IPs. So don’t.
○ The monitors are services; each gets a unique IP.
• If machine hosting monitor dies, start same monitor somewhere else
with same IP.
○ It’ll clone data from the other monitors
• Not automated (somewhat high fubar potential)
Relocatable Infrastructure
Relocatable Monitors

AGENDA
1
2
3
4
5
6
The Dream
Demo!

• Pre-OS linux + initramfs from PXE+HTTP
• Unlocks self-encrypting drives (Data-at-Rest encryption)
○ Key never known by runtime OS
• Check state:
○ Update Firmware? Unify BIOS version and config?
○ Install OS?
○ Boot OS?
• Completely uniform machines -- no half-installed, half-forgotten
state.
Remote Boot
Always boot from PXE

Apache Aurora/Mesos
Mesos
Master
Mesos
Master
Aurora
Scheduler
Aurora
Scheduler
Aurora
Scheduler Mesos
Master
NODE-1
32 CPU
256 GB
NODE-2
12 CPU
128 GB
NODE-3
32 CPU
256 GB
NODE-4
32 CPU
256 GB
NODE-5
12 CPU
128 GB
NODE-6
32 CPU
256 GB
NODE-7
12 CPU
128 GB
NODE-8
32 CPU
256 GB
Mesos
Slaves
Create New Job!
docker-image
medallia/service1
resources
2*CPU
1*GB
instances
3
Zookeeper
Aurora
Scheduler
Aurora
Scheduler
Hadoop
Scheduler
Aurora
Scheduler
Aurora
Scheduler
Storm
Scheduler
“Program against your datacenter like it’s a single pool of resources”

docker run -it
--net=routed --ip-address=1.2.3.4/32
-v demo:/demo:ceph,rw
ubuntu
Extended Docker

How Fast Is It?
StorageNetwork
~ 5 us latency
38Gbit/s single-stream TCP
22Gbit/s single-stream TCP
39.5Gbit/s multi-stream TCP
Relocate < 50ms
~ 550 MB/sec single-stream IO
~ 4 GB/sec multi-stream IO
Reattach < 50ms
Limited by SATA SSD

NUMA
CPU #0
DRAM
DRAM
DRAM
DRAM
40GbE NIC
CPU #1
DRAM
DRAM
DRAM
DRAM
SAS/SATA

AGENDA
1
2
3
4
5
6
The Dream
Demo!

DEMO!

AGENDA
1
2
3
4
5
6
The Dream
Demo!

• SSDs to 100k 4k random write IOPS!
○ If you have a “IO pipeline”
• Real-world:
○ Read: Databases don’t have IO depth of 64. It’s 1.
■ Read index, process, seek to correct index, read, process..
○ Write: Databases want each and every transaction to be
acknowledged by the storage layer
■ Full round-trip down to the storage layer
• Dedicated DB servers have a LOT of buffer cache
○ 24x800GB SSD = $15k. 512 GB RAM = $4k.
Real-World vs Synthetic IO
Latency, not IOPS or bandwidth!

• We have two types of tables
○ “A few GB”
○ “A few TB”
• Application does heavy caching; few read requests
• DB Containers have plenty memory; most tables sit in buffer cache
• If a user actually modifies something, there’s a transaction...
What performance matters for DB?
fdatasync() is bottleneck

Easy!
Slow!
Mixed read-write:
~640 iops
3 Ways to Mount
FUSE KRBD
Easy!
Fast...er
Mixed read-write:
~1550 iops
No fancy image
features
iSCSI tgt rbd
Hard!
Slow!
Mixed read-write:
~600 iops

Something that resembles PG
• Can (and do) use PGbench, but pgbench workload and our real
workload differ.
• Observe production IO pattern, replicate with fio
○ Once something provides good results on fio, apply to real DB
• Allow buffer cache
○ Yes, you have it on in production
• IOdepth=1, 8 jobs, 8kb blocks
• fdatasync() every 100th block
• Very Large Files, semi-random access
• PG doesn’t use fancy IO, so neither does our benchmark
“Realistic” testing with FIO

3x850 Pro RAID0
Writes:
99.9%: 22us
99.99%: 57us
~700 iops/job
Soft RAID0
No SuperCap
MegaRAID RAID6
Writes:
99.5%: 22us
99.9%: 15ms
99.99%: 119ms
~1100 iops/job
Battery-backed write-
back cache
Unpredictable
performance
KRBD
Writes:
99.9%: 9us
99.99%: 11us
~1000 iops/job
Survives controller
failure!
Local SSD comparison

AGENDA
1
2
3
4
5
6
The Dream
Demo!

Fun With Locking
• Switch is rebooted
• Aurora detects compute node dead
○ Restarts job somewhere else
• New location mounts Ext4 filesystem
• Switch finishes rebooting
• Old job, still running, now writes to the mounted filesystem
• “How to repair a broken ext4 filesystem with a critical database”
Ext4 on RBD
Test all failure scenarios

• On Map; “rbd lock add <image> ”
■ If no success; then
● “rbd status <image”: Check for watcher, 3 times, 15s
○ If found, ABORT, ABORT!
● “ceph osd blacklist add <previous lock holder>”
● Steal lock
• On unmap; rbd lock remove
• On reboot; “ceph osd blacklist rm <self>”
Workaround
Modified RBD wrapper; /bin/sh to the rescue!

Great, we beat legacy hardware… Or did we?
• Legacy hardware better write latency for <90% latency mark, worse
for >90%. Higher average write IOPS.
• We want no compromise on performance
• Currently rolling out PMC NV1616 NVRAM for Ceph write journal
○ Single storage-server test very promising.
○ Large-scale test ready in 2 weeks
• Experimenting with RoCE v2; RDMA over UDP
• Will post results to Ceph mailing list
Make it faster!

Try this out!
Available now:
• Docker w/ Storage and Networking
• Aurora
Coming soon:
• DCIB
github.com/medallia

2x E5-2667v3 or 2690v3
16x 16GB DDR4 RDIMM
SuperMicro X10DDW-i
Mellanox ConnectX-3 Pro
Intel DC-S3500 100GB SSD
If you want an exact replica...
Compute Node Networking
Dell S6000-ON
Cumulus
Dell AOC Cable (switches)
Mellanox Copper (servers)
Storage Node
1x E5-2667v3
4x 16GB DDR4 RDIMM
SuperMicro X10SRW-F
Mellanox ConnectX-3 Pro
Intel DC-S3500 100GB SSD
8x Intel DC-S3500 800GB
Flashtec NVRAM NV1616
(w/Encryption)

Thank you!
engineering.medallia.com

2016-JAN-28 -- High Performance Production Databases on Ceph

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

En vedette

En vedette (20)

Similaire à 2016-JAN-28 -- High Performance Production Databases on Ceph

Similaire à 2016-JAN-28 -- High Performance Production Databases on Ceph (20)

Dernier

Dernier (20)

2016-JAN-28 -- High Performance Production Databases on Ceph