This document is part 1 of a presentation on virtualization with Ganeti. It introduces Ganeti as virtual machine management software that manages clusters of physical machines running Xen, KVM, or LXC. It discusses Ganeti's components, architecture, features like live migration and failure recovery using DRBD, and how it is used at OSU Open Source Lab to power hundreds of VMs. The presentation then demonstrates initializing a Ganeti cluster, adding nodes and instances, and recovering from failures before opening for questions.
2. About us
● OSU Open Source Lab
● Server hosting for Open Source
Projects
○ Linux Foundation, Apache Software Foundation,
Drupal, Python Software Foundation, Freenode,
Gentoo, Debian, CentOS, Fedora, etc etc ...
● Open Source development projects
○ Ganeti Web Manager
3. Session Overview (part 1)
● Ganeti Introduction
● Terminology
● Major Components
● Latest Features
● Using Ganeti in Practice
● How Ganeti is deployed at OSUOSL
4. Session Overview (part 2)
● Hands on Demo
● Installation and Initialization
● Cluster Management
● Adding instances (VMs)
● Controlling instances
● Auto Allocation
● Dealing with node failures
5. What can Ganeti do?
● Virtual machine management software tool
● Manages clusters of physical machines
● Xen/KVM/LXC VM deployment
● Live Migration
● Resiliency to failure
● data redundancy via DRBD
● Cluster Balancing
● Ease of repairs and hardware swaps
7. Comparing Ganeti
● Private IaaS
● Primarily utilizes local storage
● Designed for hardware failures
● Mature project
● Low package requirements
● Simple administration
● Easily pluggable via hooks & RAPI
8. Project Background
● Google funded project
● Used in internal corporate env
● Open Sourced in 2007 GPLv2
● Team based in Google Switzerland
● Active mailing list & IRC channel
● Started internally before libvirt,
openstack, etc
10. Goals: Low Entry Level
● Keeping the entry level as low as
possible
● Easy to install, manage and upgrade
● No specialized hardware needed
● i.e. SANs
● Lightweight
● no "expensive" package dependencies
11. Goals: Enterprise Scale
● Manage simultaneously from 1 to ~200
host machines
● Access to advanced features
● drbd, live migration, API, OOB control
● Batch VM deployments
● Ease of lateral expansion and
rebalancing
12. Goals: Open Source Citizen
● Design and code discussions are open
● External contributions are welcome
● Cooperate with other "big scale"
Ganeti users
● Welcome third-party projects
● Ganeti Web Manager (OSL), Synnefo
(GRNET)
14. Terminology
Node virtualization host
Node Group homogeneous set of nodes (i.e. rack of nodes)
Instance virtualization guest
Cluster set of nodes, managed as a collective
Job ganeti operation
16. Components
● Linux & standard utils
○ (iproute2, bridge-utils, ssh)
● KVM, Xen or LXC
● DRBD, LVM, RDB, or SAN
● Python
○ (plus a few modules)
● socat
● Haskell
(optional, for auto-allocation)
17. Nodes Roles (management level)
Runs ganeti-masterd, rapi, noded
Master Node
and confd
Have a full copy of the config, can
become master
Master Candidates
Run ganeti-confd and noded
Cannot become master
Regular Nodes
Get only part of the config
Offline nodes In repair or decommissioned
18. Nodes Roles (instance hosting level)
VM Capable Node Can run virtual machines
Drained Nodes Are being evacuated
Offline Nodes Are in repair
20. Instance Parameters
● Hypervisor: hvparams
● General: beparams
● Networking: nicparams
● Modifiable at the instance or
cluster level
21. hvparams
● Boot order, CDROM Image
● NIC Type, Disk Type
● VNC Parameters, Serial console
● Kernel Path, initrd, args
● Other Hypervisor specific
parameters
22. beparams / nicparams
● Memory / Virtual CPUs
● Adding or removing disks
● MAC
● NIC mode (routed or bridged)
● Link
23. Disk Template
drbd LVM + DRBD between 2 nodes
rbd RBD volumes residing inside a RADOS cluster *
plain LVM with no redundancy
diskless No disks. Useful for testing only
* experimental support added in 2.6
24. Primary & Secondary Concepts
● Instances always runs on primary
● Uses secondary node for disk replication
● Depends on disk template (i.e. drbd, plain)
25. Instance creation scripts
also known as OS Definitions
● Requires Operating System installation
script
● Provide scripts to deploy various operating
systems
● Ganeti Instance Debootstrap
● upstream supported
● Ganeti Instance Image
● written by me
26. OS Variants
● Variants of the OS Definition
● Used for defining guest operating
system
● Types of deployment settings:
● Extra packages
● Filesystem
● Image directory
● Image Name
27. Latest Features
2.4 2.5
March 2011 April 2012
● Out of Band management ● shared storage (SAN)
● vhost net support (KVM) support
● hugepages support (KVM) ● improved node groups
● initial node groups (scalability, evacuate,
commands)
● master IP turnup
customization
● full SPICE support (KVM)
28. Latest Features
2.6 Upcoming
July 2012 Just ideas, not promises
● RBD support (ceph) ● Full dynamic memory support
● initial memory balloning ● Better instance networking
(KVM, Xen) customization
● cpu pinning ● Rolling Reboot
● OVF export/import support ● Better automation, self-
● customized drbd parameters healing, availability
● policies for better resource ● Higher Scalability
modeling ● KVM block device migration
● Optional haskell ganeti-confd ● Better OS Installation
29. Initializing your cluster
The node needs to be set up following the ganeti installation guide.
gnt-cluster init [-s ip] ...
--enabled-hypervisors=kvm cluster
34. -t drbd
DRBD provides redundancy to instance data, and
makes it possible to perform live migration without
having shared storage between the nodes.
"RAID1" over the network
36. Recovering from failure
# failover instances to their secondaries
gnt-node failover --ignore-consistency node3
# or, for each instance:
gnt-instance failover
--ignore-consistency web
37. Recovering from failure
# restore redundancy
gnt-node evacuate -I hail node3
# or, for each instance:
gnt-instance replace-disks
{-n node1 | -I hail } web
39. htools: cluster resource management
● Written in Haskell
● Where do I put a new instance?
● Where do I move an existing one?
● hail: the H iallocator
● How much space do I have?
● hspace: the H space calculator
● How do I fix an N+1 error?
● hbal: the cluster balancer
40. Controlling Ganeti
● Command line *
● Ganeti Web Manager
● Developed by OSUOSL
● RAPI (Rest-full HTTP Interface) *
● On-cluster "luxi" interface *
● luxi is currently json over unix socket
● there is code for python and haskell
* programmable interfaces
41. Job Queue
● Ganeti operations generate jobs in the master
○ with the exception of queries
● Jobs execute concurrently
● You can cancel non-started jobs, inspect the queue
status, and inspect jobs
gnt-job list
gnt-job info
gnt-job watch
gnt-job cancel
43. Running Ganeti in Production
What should you add?
● Monitoring/Automation
● Check host disks, memory, load
● Trigger events (evacuate, send to repairs, readd
node, rebalance)
● Automated host installation/setup (config
management)
● Self service use
● Instance creation and resize
● Instance console access
44. Ganeti in practice
● Medium to small virtualization
environments
● High performance
● Dedicated hardware, faster disks, more spindles on
local storage
● Cheap hardware to high-end
hardware
● Higher reliability
45. Ganeti as a "cloud"
● Not a traditional cloud environment
● No AWS APIs (yet at least), no object store
● Ganeti specific API
● Tools to extend it
● Ganeti Web Manager, Syssnefo, GlusterFS, Ceph
● Storage layer differences
● block devices instead of disk images (typically)
46. How the OSL uses Ganeti
● Powers all managed virtualization
● Project hosting
● KVM based
● Hundreds of VMs
● Web hosts, code hosting, etc
● Per-project clusters: PSF, OSGeo,
phpBB, Gentoo
● Powers Supercell
47. Ganeti at OSL
● Node OS: Gentoo
● Migrating towards CentOS
● CFEngine for node configuration setup
● Utilize instance-image for guest installs
● Flexibility on guest operating systems we can
deploy
● 10 clusters, 27 nodes, 230 instances
● Ganeti Web Manager
49. People running Ganeti
● Google
● Corporate Computing Infra
● osuosl.org
● Oregon State University Open Source Lab
● grnet.gr
● Greek Research & Technology Network
● nero.net
● Network for Education & Research in Oregon