The document discusses 5 things the author wishes they knew before deploying OpenNebula in production. It describes issues they encountered such as strange VM reboots due to OpenNebula context variables, high traffic overloading the conntrack table, request timeouts due to CPU overprovisioning, and challenges with live migration of VMs using local storage in KVM. The author provides workarounds and lessons learned around testing performance, understanding the Linux networking stack, and using regular vs live vs offline VM migration options in OpenNebula.

1. 5 Things We Wish We Knew Before
Deploying OpenNebula in Production
Strahinja Kustudic, OpenNebula Conf 2018,

2. Strahinja Kustudic, Senior System Engineer5 Things We Wish We Knew Before Deploying OpenNebula in Production
ABOUT
WHO WE
ARE.
FOUNDATION
2010
GAMES
Top Eleven
Golden Boot
HQ
Belgrade, Serbia
CREW
170 People,
22 Nationalities

3. Strahinja Kustudic, Senior System Engineer5 Things We Wish We Knew Before Deploying OpenNebula in Production

4. Strahinja Kustudic, Senior System Engineer5 Things We Wish We Knew Before Deploying OpenNebula in Production
OpenNebula Cluster Size
● OpenNebula 5.4
● 24 hosts
● 480 CPUs (20 per host)
● 12TB RAM (512GB per host)
● 720TB SSD local storage (3TB per host)
● 2x 20Gbps network (per host)
● KVM on CentOS 7
● ~700 VMs (~300 production)

5. Strahinja Kustudic, Senior System Engineer5 Things We Wish We Knew Before Deploying OpenNebula in Production
Strange VM Reboots...

6. Strahinja Kustudic, Senior System Engineer5 Things We Wish We Knew Before Deploying OpenNebula in Production
● Used to configure the VMs OS on every boot
● Controlled by CONTEXT variables in VM templates
OpenNebula Context
CONTEXT = [
NETWORK = "YES",
REPORT_READY = "YES",
SET_HOSTNAME = "$NAME",
SSH_PUBLIC_KEY = "$USER[SSH_PUBLIC_KEY]"
]

7. Strahinja Kustudic, Senior System Engineer5 Things We Wish We Knew Before Deploying OpenNebula in Production
OpenNebula Context Documentation
● https://docs.opennebula.org/5.6/operation/vm_setup/kvm.html
● https://docs.opennebula.org/5.6/operation/references/template.html#template-context
● https://github.com/OpenNebula/addon-context-linux/blob/master/README.md

8. Examine scripts in /etc/one-context.d
to truly understand how it works.
TAKEAWAY

9. Strahinja Kustudic, Senior System Engineer5 Things We Wish We Knew Before Deploying OpenNebula in Production
High traffic ready?

10. Strahinja Kustudic, Senior System Engineer5 Things We Wish We Knew Before Deploying OpenNebula in Production
Golden Boot Release

11. Strahinja Kustudic, Senior System Engineer5 Things We Wish We Knew Before Deploying OpenNebula in Production
ip_conntrack: table full, dropping packet.

12. Strahinja Kustudic, Senior System Engineer5 Things We Wish We Knew Before Deploying OpenNebula in Production
Conntrack table full - usual “fixes”
● net.netfilter.nf_conntrack_max = 524288
● Set client ports to NOTRACK in the raw
IPTables table

13. Strahinja Kustudic, Senior System Engineer5 Things We Wish We Knew Before Deploying OpenNebula in Production
Conntrack table full - actual fix
net.netfilter.nf_conntrack_tcp_timeout_time_wait: 120 -> 10
net.netfilter.nf_conntrack_tcp_timeout_established: 432000 -> 86400

14. Get ready to dig deeper
into the Linux tech stack
TAKEAWAY

15. Strahinja Kustudic, Senior System Engineer5 Things We Wish We Knew Before Deploying OpenNebula in Production
Request timeouts

16. Strahinja Kustudic, Senior System Engineer5 Things We Wish We Knew Before Deploying OpenNebula in Production

17. Strahinja Kustudic, Senior System Engineer5 Things We Wish We Knew Before Deploying OpenNebula in Production
CPU
● Production: CPU=VCPU
● Other: CPU=0.1
● Overprovision hosts

18. Test CPU performance of multiple
VMs running on the same host.
TAKEAWAY

19. Strahinja Kustudic, Senior System Engineer5 Things We Wish We Knew Before Deploying OpenNebula in Production
Steal time?

20. Strahinja Kustudic, Senior System Engineer5 Things We Wish We Knew Before Deploying OpenNebula in Production

21. Things are not always obvious
TAKEAWAY

22. Strahinja Kustudic, Senior System Engineer5 Things We Wish We Knew Before Deploying OpenNebula in Production
Rebalancing the cluster

23. Strahinja Kustudic, Senior System Engineer5 Things We Wish We Knew Before Deploying OpenNebula in Production
KVM Migration (Live)
Blocker:
● Doesn’t work with local storage

24. Strahinja Kustudic, Senior System Engineer5 Things We Wish We Knew Before Deploying OpenNebula in Production
KVM Migration (Regular)
Pros:
● OpenNebula supports them out of the box
Cons:
● VM is offline during migration
● No time sync issues
● No failure because of difference in CPU

25. Strahinja Kustudic, Senior System Engineer5 Things We Wish We Knew Before Deploying OpenNebula in Production
KVM Migration (Offline)
Pros:
● No time sync issues
● No failure because of difference in CPU
Cons:
● VM is offline during migration
● OpenNebula doesn’t support them out of the box

26. Be practical, and use
what works for you
TAKEAWAY

27. Thank you!
@kustodian
https://engineering.nordeus.com/