Presented by: Ivan Groenewold Presented at the All Things Open 2021 Raleigh, NC, USA Raleigh Convention Center Abstract: Installing big clusters is always a challenge, and can be a very time-consuming task. At a high level, we need to provision the hardware, install the software, configure monitoring, and set up a backup process. In this talk we will see how to develop a complete pipeline to deploy MongoDB sharded clusters at the push of a button, that can accomplish all of these tasks for you. By combining Terraform for the hardware provisioning, and Ansible for the software installation, we can completely automate the process, saving time and providing a standardized reusable solution.

1. Deploying MongoDB
sharded clusters easily
with Terraform and
Ansible
All Things Open, October 2021
Ivan Groenewold

2. Agenda
● Terraform 101
● Provisioning in GCP
● Ansible 101
● Deploying MongoDB
● Q&A

3. About me
● @igroenew
● Architect at Percona
● Based in Argentina

4. MongoDB sharding in a nutshell
Image © MongoDB Inc.

5. Target infrastructure

6. The plan
● Define the topology
● Provision the infrastructure using Terraform
○ instances, disks, network, buckets, etc.
● Install the software with Ansible
○ MongoDB, monitoring & backup solution

7. Terraform 101
● Infrastructure-as-Code
● Open Source
● Works with multiple resources and providers
● Declarative approach - state what you want
● Infrastructure converges to the desired state

8. Terraform syntax
● Based in HashiCorp Configuration Language (HCL)
● Basic constructs:
○ Arguments
name = “my_instance”
○ Blocks
resource “google_compute_instance” “my_instance”
{
…
}
type label 1 label 2
body

9. Defining variables
variable "data_disk_type" {
default = "pd-standard"
}
variable "my_instance_type" {
default = "e2-standard-2"
description = "instance type"
}
variable "my_volume_size" {
default = "100"
description = "storage size"
}
variable "centos_amis" {
description = "CentOS AMIs on each region"
default = {
northamerica-northeast1 = "centos-8-v20210316"
northamerica-northeast2 = "centos-8-v20210316"
}
}

10. Provisioning in GCP
resource "google_compute_disk" "cfg_disk" {
name = "mongo-cfg0-data"
type = var.data_disk_type
size = var.my_volume_size
zone = var.my_zone
}
resource "google_compute_instance" "cfg" {
name = "my_instance"
machine_type = var.my_instance_type
tags = ["mongodb-cfg"]
zone = var.my_zone
boot_disk {
initialize_params {
image = lookup(var.centos_amis, var.region)
}
}
attached_disk {
source = google_compute_disk.cfg_disk.name
}
network_interface {
network = google_compute_network.vpc-network.id
subnetwork = google_compute_subnetwork.vpc-subnet.id
}
provision a disk
provision an instance

11. Provisioning in GCP (2)
resource "google_compute_disk" "cfg_disk" {
name = "mongo-cfg0-data"
type = var.data_disk_type
size = var.my_volume_size
zone = var.my_zone
}
resource "google_compute_instance" "cfg" {
name = "my_instance"
machine_type = var.my_instance_type
tags = ["mongodb-cfg"]
zone = var.my_zone
boot_disk {
initialize_params {
image = lookup(var.centos_amis, var.region)
}
}
attached_disk {
source = google_compute_disk.cfg_disk.name
}
network_interface {
network = google_compute_network.vpc-network.id
subnetwork = google_compute_subnetwork.vpc-subnet.id
}
nested blocks
call lookup function

12. Working with Terraform
● terraform init
○ Initialize the working directory
● terraform plan
○ print the action plan
● terraform apply
○ carry out the actions
● terraform destroy
○ remove all managed resources

13. Working with Terraform (2)

14. Working with Terraform (3)
● What is a MongoDB server?
○ Instance + Persistent disk (except mongos servers)
○ Firewall rules
○ Init scripts
■ mount the volumes, OS tweaks, etc

15. Working with Terraform (4)
● Create .tf files for each component
■ mongos router
■ mongod shard
■ Config server
■ anything else?
● Use a separate variables file

16. Provisioning the infrastructure
● Servers
○ cfg-server.tf
○ shard-server.tf
○ mongos-server.tf
○ pmm-server.tf
● variables.tf
● network.tf
● backup.tf

17. Configuring the network
● Define the region
● Configure a VPC
● Define the subnets

18. Configuring the network (2)
data "google_compute_zones" "available" {
status = "UP"
}
resource "google_compute_network" " vpc-network" {
name = "my-vpc"
auto_create_subnetworks = false
}
resource "google_compute_subnetwork" "vpc-subnet" {
name = "mongodb-subnet"
ip_cidr_range = "10.1.0.0/16"
region = var.region
network = google_compute_network. vpc-network.id
}
query data source

19. Creating the instances
resource "google_compute_instance" "server" {
count = 6
name = "server ${count.index}"
zone = data.google_compute_zones.available.names[ count.index % 3]
● Use count.index to shuffle the instances between AZ’s

20. Configuring network access
resource "google_compute_firewall" "mongodb-cfgsvr-firewall" {
name = "mongodb-cfgsvr-firewall"
network = google_compute_network.vpc-network.name
direction = "INGRESS"
target_tags = ["mongodb-cfg"]
allow {
protocol = "tcp"
ports = ["22", "27019"]
}
}

21. Preparing the backup infrastructure
● Create a Cloud Storage bucket
● Allow the instances to read/write from it
● Objects lifecycle policy

22. Preparing the backup infrastructure (2)
● Steps are cloud-specific
● For GCP we need:
○ Cloud Storage bucket
○ Service account
○ HMAC key-pair for the service account
○ Grant storage-admin role to the service account

23. Preparing the backup infrastructure (3)
resource "google_storage_bucket" "mongo-backups" {
name = "mongo-backups"
location = var.region
force_destroy = true
uniform_bucket_level_access = true
resource "google_service_account" "mongo-backup-service-account" {
account_id = "mongo-backup-service-account"
display_name = "Mongo Backup Service Account"
}
resource "google_storage_hmac_key" "mongo-backup-service-account" {
service_account_email = google_service_account.mongo-backup-service-account.email
}
resource "google_storage_bucket_iam_binding" "binding" {
bucket = google_storage_bucket.mongo-backups.name
role = "roles/storage.admin"
members = [
"serviceAccount:${google_service_account.mongo-backup-service-account.email}",
]
}

24. ● PMM client
○ run locally on each server
○ pushes metrics
● PMM server
○ Performance metrics history
○ Query analytics
○ Integrated alerting
○ Integrated backups (WIP)
https://pmmdemo.percona.com
Monitoring

25. What’s next?
● We have the servers
● We have the network configured
● We have the backup infrastructure
● We need to deploy the software

26. Ansible 101
● Automation engine
● SSH-based
● Open source
● Web interface: AWX project

27. Why Ansible?
● Easy to deploy
● No agent required
● No firewall rules required
● YAML syntax
● Secure

28. Installing Ansible
● Control machine
○ Can be your laptop
○ Acts as the Ansible “server”
○ Only needed when running Ansible code
● Managed nodes

29. Inventory
● Inventory options
○ Static
■ ini or YML format
○ Dynamic
■ Scripts available for most cloud providers
■ Write your own plugin
● The default inventory is /etc/ansible/hosts

30. Inventory (2)
● Static inventory example
[webservers]
www.myhost.com
www.example.com
[databases]
db-[a:f].example.com
[atlanta]
dba.example.com http_port=80 maxRequestsPerChild=808
[atlanta:vars]
ntp_server=ntp.atlanta.example.com

31. Modules
● Ansible building blocks
● Should be idempotent
Examples:
$ ansible example -m ping
www.example.com | SUCCESS => {
"changed": false,
"ping": "pong"
}
$ ansible example -m service -a "name=httpd state=started"

32. Playbooks
● Orchestrate steps
● Composed of one or more plays
● Each play runs a number of tasks in order on a group of servers
○ e.g. call a module to do something
● YML format

33. Playbooks (2)
● Inventory example:
[webservers]
web[01:10].example.com
[databases]
db[01:10].example.com

34. ---
- hosts: webservers
tasks:
- name: ensure apache is at the latest version
yum:
name: httpd
state: latest
- name: ensure apache is started
service:
name: httpd
state: started
- hosts: databases
tasks:
- name: ensure postgresql is at the latest version
yum:
name: postgresql
state: latest
play 1
play 2
task 1
task 2
● Playbook example:
Playbooks (3)

35. Playbooks (4)
Play 1:
---
- hosts: webservers
tasks:
- name: ensure apache is at the latest version
yum:
name: httpd
state: latest
...
module
host groups as per inventory

36. Playbooks (5)
● Run with ansible-playbook command
$ ansible-playbook all my_pb.yml [--limit *example.com]

37. Playbooks (6)
PLAY [all]
***************************************************************************
TASK [check if specified os user exists]
***************************************************************************
changed: [mysql1]
ok: [mysql2]
PLAY RECAP
***************************************************************************
mysql1 : ok=1 changed=1 unreachable=0 failed=0
mysql2 : ok=1 changed=0 unreachable=0 failed=0

38. Variables
● Simple variables
foo: bar
● List
datacenter:
- us-east
- us-west
● Dictionary
foo:
field1: one
field2: two

39. Automating MongoDB deployment
1. Create an Ansible inventory file
2. Edit the variables file
3. Run the ansible-playbook

40. Automating MongoDB deployment
1. Create an Ansible inventory file
2. Edit the variables file
3. Run the ansible-playbook

41. Inventory file for a sharded cluster
● One group per shard (“shardN”)
● A group for the config servers (“cfg”)
● A group for the routers (“mongos”)

42. [shard1]
host1.example.com mongodb_primary=True
host2.example.com
host3.example.com
[shard2]
host4.example.com mongodb_primary=True
host5.example.com
host6.example.com
[cfg]
host7.example.com mongodb_primary=True
host8.example.com
host9.example.com
[mongos]
host10.example.com
Inventory file for a sharded cluster (2)

43. Generating the inventory file with Terraform
● Use the local_file Terraform resource
● Use templates to dynamically create the groups
● How to generate an Ansible inventory from Terraform

44. Automating MongoDB deployment
1. Create an Ansible inventory file
2. Edit the variables file
3. Run the ansible-playbook

45. Variables
● Copy the MongoDB files / Install from repository
● Ports for mongod, mongos
● Define the paths for data, logs, etc.
● Authentication mechanism
● Encryption
● Backup

46. Things we need done
● Install packages
● Create config files
● Start/stop processes
● Initialize replica sets
● Create users
● Configure backup job
● Add hosts to monitoring
● Add the shards to the cluster

47. Installing packages
packages:
- percona-server-mongodb
- percona-backup-mongodb
- pmm2-client
- name: install rpm from repo
package:
name: "{{ item }}"
state: present
with_items: "{{ packages }}"
dynamic variable
loop

48. Creating configuration files
● Generate files dynamically
● Include/exclude different sections
● Variables are not enough
● Solution: Ansible Templates

49. Ansible Templates
● Built-in module
● Create file with dynamic content
● Jinja2 engine
● Store them in /templates subdirectory

50. Creating templated config files
mongod.conf.j2 template:
...
security:
{% if use_tls %}
clusterAuthMode: x509
{% else %}
keyFile: {{ keyFile_loc }}
{% endif %}
...
Variables file:
use_tls: false
keyFile_loc: /var/lib/mongo/rskeyfile

51. Creating templated config files (2)
Task:
- name: copy mongod.conf
become: yes
template:
src: templates/mongod.conf.j2
dest: /etc/mongod.conf
owner: root
group: root
mode: 0644

52. Starting/stopping processes
- name: start mongod on rs member
become: yes
service:
name: mongod
state: started

53. [cfg]
host1.example.com mongodb_primary=True
host2.example.com
host3.example.com
[shard1]
host4.example.com mongodb_primary=True
host5.example.com
host6.example.com
[shard2]
host6.example.com mongodb_primary=True
host8.example.com
host9.example.com
[mongos]
host10.example.com
Initialize replica sets
● rs.initiate()
Our inventory file:
group_names array
for “host1.example.com”:
group_names = [ cfg ]

54. Initialize replica sets (2)
init-rs.js.j2:
rs.initiate(
{
_id: "{{ group_names[0] }}",
members: [
{% for h in groups[ group_names[0] ] %}
{ _id : {{ loop.index }}, host : "{{ h }}:
{%
if hostvars[inventory_hostname].group_names[0].startswith('shard') %}
{{ shard_port }}
{% else %}
{{ cfgserver_port }}
{% endif %}",
priority: 1 } {% if not loop.last %}
,{% endif %}
{% endfor %}
] });
the first group a host appears in
all hosts part of the first group

55. Initialize replica sets (3)
init-rs.js:
rs.initiate(
{
_id: "cfg",
members: [
{ _id : 0 , host : ”host1.example.com:
27018
",
priority: 1 } ,
...
] });

56. Initialize replica sets (4)
- name: render the template for the init command
template:
src: templates/init-rs.js.j2
dest: /tmp/init-rs.js
mode: 0644
when: mongodb_primary is defined and mongodb_primary
- name: run the init command for the replica set
shell: mongo --host localhost --port {{ mongo_port }} <
/tmp/init-rs.js
when: mongodb_primary is defined and mongodb_primary
runs only once per replica-set

57. Create users
createUser.j2:
db.getSiblingDB("admin").createUser({
user: "{{ mongodb_pmm_user }}",
pwd: "{{ mongodb_pmm_user_pwd }}",
roles: [
{ role: "explainRole", db: "admin" },
{ role: "clusterMonitor", db: "admin" },
{ role: "read", db: "local" }
]
});

58. Create users (2)
- name: prepare the command to create pmm user
template:
src: templates/createUser.js.j2
dest: /tmp/createUser.js
mode: 0644
when: mongodb_primary is defined and mongodb_primary
- name: run the command to create the user
shell: mongo admin -u {{ root_user }} -p{{ mongo_root_password }}
--port {{ mongo_port }} < /tmp/createUser.js
when: mongodb_primary is defined and mongodb_primary

59. Configure backup
- name: set up backup cron job
cron:
name: pbm backup
minute: 3
hour: 0
user: pbm
job: /usr/bin/pbm backup --mongodb-uri "mongodb://{{ pbmuser }}:{{ pbmpwd }}@
{{ ansible_fqdn }}:{{ mongo_port }}"
cron_file: pbm_daily_backup

60. Configure monitoring
- name: point pmm-client to the PMM server
become: true
shell: pmm-admin config --server-url=https://{{ pmm_server_user }}:
{{ pmm_server_pwd }}@{{ pmm_server }}:443 --server-insecure-tls --force
- name: add mongodb metrics exporter
become: true
shell: pmm-admin add mongodb --username={{ mongodb_pmm_user }} --password={{
mongodb_pmm_user_pwd }} --host={{ ansible_fqdn }} --port={{ cfg_server_port
if
('cfg' in group_names) else shard_port }}

61. Add the shards
- name: add the shards
hosts: shard*
tasks:
- name: add the shards to the cluster
shell: mongo admin -uroot -p{{ mongo_root_password }} --port {{ mongos_port }}
--eval "sh.addShard('{{ group_names[0] }}/{{ ansible_fqdn }}:{{ shard_port }}')"
delegate_to: "{{ groups.mongos | first }}"
when: mongodb_primary is defined and mongodb_primary

62. Automating MongoDB deployment
1. Create an Ansible inventory file
2. Edit the variables file
3. Run the ansible-playbook
ansible-playbook main.yml -i inventory.ini --ask-become-pass

63. Putting it all together
● Define the topology
● Create the infrastructure using Terraform
● Generate the inventory file for Ansible
● Install the software with Ansible

64. Putting it all together (2)
● Define the variables
○ variables.tf
○ Ansible vars file
● Run terraform apply
● Run ansible-playbook

65. Benefits
● Define a process
● Save time
● Reuse code
● Streamline deployments
● Ensure resources are monitored (and backed up)

66. Q&A
Thank you for attending!
https://www.percona.com/blog/author/ivan-groenewold/