1.
Deploying MongoDB
sharded clusters easily
with Terraform and
Ansible
All Things Open, October 2021
Ivan Groenewold

2.
Agenda
● Terraform 101
● Provisioning in GCP
● Ansible 101
● Deploying MongoDB
● Q&A

3.
About me
● @igroenew
● Architect at Percona
● Based in Argentina

4.
MongoDB sharding in a nutshell
Image © MongoDB Inc.

5.
Target infrastructure

6.
The plan
● Define the topology
● Provision the infrastructure using Terraform
○ instances, disks, network, buckets, etc.
● Install the software with Ansible
○ MongoDB, monitoring & backup solution

7.
Terraform 101
● Infrastructure-as-Code
● Open Source
● Works with multiple resources and providers
● Declarative approach - state what you want
● Infrastructure converges to the desired state

8.
Terraform syntax
● Based in HashiCorp Configuration Language (HCL)
● Basic constructs:
○ Arguments
name = “my_instance”
○ Blocks
resource “google_compute_instance” “my_instance”
{
…
}
type label 1 label 2
body

9.
Defining variables
variable "data_disk_type" {
default = "pd-standard"
}
variable "my_instance_type" {
default = "e2-standard-2"
description = "instance type"
}
variable "my_volume_size" {
default = "100"
description = "storage size"
}
variable "centos_amis" {
description = "CentOS AMIs on each region"
default = {
northamerica-northeast1 = "centos-8-v20210316"
northamerica-northeast2 = "centos-8-v20210316"
}
}

10.
Provisioning in GCP
resource "google_compute_disk" "cfg_disk" {
name = "mongo-cfg0-data"
type = var.data_disk_type
size = var.my_volume_size
zone = var.my_zone
}
resource "google_compute_instance" "cfg" {
name = "my_instance"
machine_type = var.my_instance_type
tags = ["mongodb-cfg"]
zone = var.my_zone
boot_disk {
initialize_params {
image = lookup(var.centos_amis, var.region)
}
}
attached_disk {
source = google_compute_disk.cfg_disk.name
}
network_interface {
network = google_compute_network.vpc-network.id
subnetwork = google_compute_subnetwork.vpc-subnet.id
}
provision a disk
provision an instance

11.
Provisioning in GCP (2)
resource "google_compute_disk" "cfg_disk" {
name = "mongo-cfg0-data"
type = var.data_disk_type
size = var.my_volume_size
zone = var.my_zone
}
resource "google_compute_instance" "cfg" {
name = "my_instance"
machine_type = var.my_instance_type
tags = ["mongodb-cfg"]
zone = var.my_zone
boot_disk {
initialize_params {
image = lookup(var.centos_amis, var.region)
}
}
attached_disk {
source = google_compute_disk.cfg_disk.name
}
network_interface {
network = google_compute_network.vpc-network.id
subnetwork = google_compute_subnetwork.vpc-subnet.id
}
nested blocks
call lookup function

12.
Working with Terraform
● terraform init
○ Initialize the working directory
● terraform plan
○ print the action plan
● terraform apply
○ carry out the actions
● terraform destroy
○ remove all managed resources

13.
Working with Terraform (2)

14.
Working with Terraform (3)
● What is a MongoDB server?
○ Instance + Persistent disk (except mongos servers)
○ Firewall rules
○ Init scripts
■ mount the volumes, OS tweaks, etc

15.
Working with Terraform (4)
● Create .tf files for each component
■ mongos router
■ mongod shard
■ Config server
■ anything else?
● Use a separate variables file

16.
Provisioning the infrastructure
● Servers
○ cfg-server.tf
○ shard-server.tf
○ mongos-server.tf
○ pmm-server.tf
● variables.tf
● network.tf
● backup.tf

17.
Configuring the network
● Define the region
● Configure a VPC
● Define the subnets

18.
Configuring the network (2)
data "google_compute_zones" "available" {
status = "UP"
}
resource "google_compute_network" " vpc-network" {
name = "my-vpc"
auto_create_subnetworks = false
}
resource "google_compute_subnetwork" "vpc-subnet" {
name = "mongodb-subnet"
ip_cidr_range = "10.1.0.0/16"
region = var.region
network = google_compute_network. vpc-network.id
}
query data source

19.
Creating the instances
resource "google_compute_instance" "server" {
count = 6
name = "server ${count.index}"
zone = data.google_compute_zones.available.names[ count.index % 3]
● Use count.index to shuffle the instances between AZ’s

20.
Configuring network access
resource "google_compute_firewall" "mongodb-cfgsvr-firewall" {
name = "mongodb-cfgsvr-firewall"
network = google_compute_network.vpc-network.name
direction = "INGRESS"
target_tags = ["mongodb-cfg"]
allow {
protocol = "tcp"
ports = ["22", "27019"]
}
}

21.
Preparing the backup infrastructure
● Create a Cloud Storage bucket
● Allow the instances to read/write from it
● Objects lifecycle policy

22.
Preparing the backup infrastructure (2)
● Steps are cloud-specific
● For GCP we need:
○ Cloud Storage bucket
○ Service account
○ HMAC key-pair for the service account
○ Grant storage-admin role to the service account

23.
Preparing the backup infrastructure (3)
resource "google_storage_bucket" "mongo-backups" {
name = "mongo-backups"
location = var.region
force_destroy = true
uniform_bucket_level_access = true
resource "google_service_account" "mongo-backup-service-account" {
account_id = "mongo-backup-service-account"
display_name = "Mongo Backup Service Account"
}
resource "google_storage_hmac_key" "mongo-backup-service-account" {
service_account_email = google_service_account.mongo-backup-service-account.email
}
resource "google_storage_bucket_iam_binding" "binding" {
bucket = google_storage_bucket.mongo-backups.name
role = "roles/storage.admin"
members = [
"serviceAccount:${google_service_account.mongo-backup-service-account.email}",
]
}

24.
● PMM client
○ run locally on each server
○ pushes metrics
● PMM server
○ Performance metrics history
○ Query analytics
○ Integrated alerting
○ Integrated backups (WIP)
https://pmmdemo.percona.com
Monitoring

25.
What’s next?
● We have the servers
● We have the network configured
● We have the backup infrastructure
● We need to deploy the software

26.
Ansible 101
● Automation engine
● SSH-based
● Open source
● Web interface: AWX project

27.
Why Ansible?
● Easy to deploy
● No agent required
● No firewall rules required
● YAML syntax
● Secure

28.
Installing Ansible
● Control machine
○ Can be your laptop
○ Acts as the Ansible “server”
○ Only needed when running Ansible code
● Managed nodes

29.
Inventory
● Inventory options
○ Static
■ ini or YML format
○ Dynamic
■ Scripts available for most cloud providers
■ Write your own plugin
● The default inventory is /etc/ansible/hosts

30.
Inventory (2)
● Static inventory example
[webservers]
www.myhost.com
www.example.com
[databases]
db-[a:f].example.com
[atlanta]
dba.example.com http_port=80 maxRequestsPerChild=808
[atlanta:vars]
ntp_server=ntp.atlanta.example.com

31.
Modules
● Ansible building blocks
● Should be idempotent
Examples:
$ ansible example -m ping
www.example.com | SUCCESS => {
"changed": false,
"ping": "pong"
}
$ ansible example -m service -a "name=httpd state=started"

32.
Playbooks
● Orchestrate steps
● Composed of one or more plays
● Each play runs a number of tasks in order on a group of servers
○ e.g. call a module to do something
● YML format

33.
Playbooks (2)
● Inventory example:
[webservers]
web[01:10].example.com
[databases]
db[01:10].example.com

34.
---
- hosts: webservers
tasks:
- name: ensure apache is at the latest version
yum:
name: httpd
state: latest
- name: ensure apache is started
service:
name: httpd
state: started
- hosts: databases
tasks:
- name: ensure postgresql is at the latest version
yum:
name: postgresql
state: latest
play 1
play 2
task 1
task 2
● Playbook example:
Playbooks (3)

35.
Playbooks (4)
Play 1:
---
- hosts: webservers
tasks:
- name: ensure apache is at the latest version
yum:
name: httpd
state: latest
...
module
host groups as per inventory

36.
Playbooks (5)
● Run with ansible-playbook command
$ ansible-playbook all my_pb.yml [--limit *example.com]

37.
Playbooks (6)
PLAY [all]
***************************************************************************
TASK [check if specified os user exists]
***************************************************************************
changed: [mysql1]
ok: [mysql2]
PLAY RECAP
***************************************************************************
mysql1 : ok=1 changed=1 unreachable=0 failed=0
mysql2 : ok=1 changed=0 unreachable=0 failed=0

38.
Variables
● Simple variables
foo: bar
● List
datacenter:
- us-east
- us-west
● Dictionary
foo:
field1: one
field2: two

39.
Automating MongoDB deployment
1. Create an Ansible inventory file
2. Edit the variables file
3. Run the ansible-playbook

40.
Automating MongoDB deployment
1. Create an Ansible inventory file
2. Edit the variables file
3. Run the ansible-playbook

41.
Inventory file for a sharded cluster
● One group per shard (“shardN”)
● A group for the config servers (“cfg”)
● A group for the routers (“mongos”)

42.
[shard1]
host1.example.com mongodb_primary=True
host2.example.com
host3.example.com
[shard2]
host4.example.com mongodb_primary=True
host5.example.com
host6.example.com
[cfg]
host7.example.com mongodb_primary=True
host8.example.com
host9.example.com
[mongos]
host10.example.com
Inventory file for a sharded cluster (2)

43.
Generating the inventory file with Terraform
● Use the local_file Terraform resource
● Use templates to dynamically create the groups
● How to generate an Ansible inventory from Terraform

44.
Automating MongoDB deployment
1. Create an Ansible inventory file
2. Edit the variables file
3. Run the ansible-playbook

45.
Variables
● Copy the MongoDB files / Install from repository
● Ports for mongod, mongos
● Define the paths for data, logs, etc.
● Authentication mechanism
● Encryption
● Backup

46.
Things we need done
● Install packages
● Create config files
● Start/stop processes
● Initialize replica sets
● Create users
● Configure backup job
● Add hosts to monitoring
● Add the shards to the cluster

47.
Installing packages
packages:
- percona-server-mongodb
- percona-backup-mongodb
- pmm2-client
- name: install rpm from repo
package:
name: "{{ item }}"
state: present
with_items: "{{ packages }}"
dynamic variable
loop

48.
Creating configuration files
● Generate files dynamically
● Include/exclude different sections
● Variables are not enough
● Solution: Ansible Templates

49.
Ansible Templates
● Built-in module
● Create file with dynamic content
● Jinja2 engine
● Store them in /templates subdirectory

50.
Creating templated config files
mongod.conf.j2 template:
...
security:
{% if use_tls %}
clusterAuthMode: x509
{% else %}
keyFile: {{ keyFile_loc }}
{% endif %}
...
Variables file:
use_tls: false
keyFile_loc: /var/lib/mongo/rskeyfile

51.
Creating templated config files (2)
Task:
- name: copy mongod.conf
become: yes
template:
src: templates/mongod.conf.j2
dest: /etc/mongod.conf
owner: root
group: root
mode: 0644

52.
Starting/stopping processes
- name: start mongod on rs member
become: yes
service:
name: mongod
state: started

53.
[cfg]
host1.example.com mongodb_primary=True
host2.example.com
host3.example.com
[shard1]
host4.example.com mongodb_primary=True
host5.example.com
host6.example.com
[shard2]
host6.example.com mongodb_primary=True
host8.example.com
host9.example.com
[mongos]
host10.example.com
Initialize replica sets
● rs.initiate()
Our inventory file:
group_names array
for “host1.example.com”:
group_names = [ cfg ]

54.
Initialize replica sets (2)
init-rs.js.j2:
rs.initiate(
{
_id: "{{ group_names[0] }}",
members: [
{% for h in groups[ group_names[0] ] %}
{ _id : {{ loop.index }}, host : "{{ h }}:
{%
if hostvars[inventory_hostname].group_names[0].startswith('shard') %}
{{ shard_port }}
{% else %}
{{ cfgserver_port }}
{% endif %}",
priority: 1 } {% if not loop.last %}
,{% endif %}
{% endfor %}
] });
the first group a host appears in
all hosts part of the first group

55.
Initialize replica sets (3)
init-rs.js:
rs.initiate(
{
_id: "cfg",
members: [
{ _id : 0 , host : ”host1.example.com:
27018
",
priority: 1 } ,
...
] });

56.
Initialize replica sets (4)
- name: render the template for the init command
template:
src: templates/init-rs.js.j2
dest: /tmp/init-rs.js
mode: 0644
when: mongodb_primary is defined and mongodb_primary
- name: run the init command for the replica set
shell: mongo --host localhost --port {{ mongo_port }} <
/tmp/init-rs.js
when: mongodb_primary is defined and mongodb_primary
runs only once per replica-set

57.
Create users
createUser.j2:
db.getSiblingDB("admin").createUser({
user: "{{ mongodb_pmm_user }}",
pwd: "{{ mongodb_pmm_user_pwd }}",
roles: [
{ role: "explainRole", db: "admin" },
{ role: "clusterMonitor", db: "admin" },
{ role: "read", db: "local" }
]
});

58.
Create users (2)
- name: prepare the command to create pmm user
template:
src: templates/createUser.js.j2
dest: /tmp/createUser.js
mode: 0644
when: mongodb_primary is defined and mongodb_primary
- name: run the command to create the user
shell: mongo admin -u {{ root_user }} -p{{ mongo_root_password }}
--port {{ mongo_port }} < /tmp/createUser.js
when: mongodb_primary is defined and mongodb_primary

59.
Configure backup
- name: set up backup cron job
cron:
name: pbm backup
minute: 3
hour: 0
user: pbm
job: /usr/bin/pbm backup --mongodb-uri "mongodb://{{ pbmuser }}:{{ pbmpwd }}@
{{ ansible_fqdn }}:{{ mongo_port }}"
cron_file: pbm_daily_backup

60.
Configure monitoring
- name: point pmm-client to the PMM server
become: true
shell: pmm-admin config --server-url=https://{{ pmm_server_user }}:
{{ pmm_server_pwd }}@{{ pmm_server }}:443 --server-insecure-tls --force
- name: add mongodb metrics exporter
become: true
shell: pmm-admin add mongodb --username={{ mongodb_pmm_user }} --password={{
mongodb_pmm_user_pwd }} --host={{ ansible_fqdn }} --port={{ cfg_server_port
if
('cfg' in group_names) else shard_port }}

61.
Add the shards
- name: add the shards
hosts: shard*
tasks:
- name: add the shards to the cluster
shell: mongo admin -uroot -p{{ mongo_root_password }} --port {{ mongos_port }}
--eval "sh.addShard('{{ group_names[0] }}/{{ ansible_fqdn }}:{{ shard_port }}')"
delegate_to: "{{ groups.mongos | first }}"
when: mongodb_primary is defined and mongodb_primary

62.
Automating MongoDB deployment
1. Create an Ansible inventory file
2. Edit the variables file
3. Run the ansible-playbook
ansible-playbook main.yml -i inventory.ini --ask-become-pass

63.
Putting it all together
● Define the topology
● Create the infrastructure using Terraform
● Generate the inventory file for Ansible
● Install the software with Ansible

64.
Putting it all together (2)
● Define the variables
○ variables.tf
○ Ansible vars file
● Run terraform apply
● Run ansible-playbook

65.
Benefits
● Define a process
● Save time
● Reuse code
● Streamline deployments
● Ensure resources are monitored (and backed up)

66.
Q&A
Thank you for attending!
https://www.percona.com/blog/author/ivan-groenewold/