1. CentOS, QA and OpenNebula
Opennebula Conference 2014
Berlin
December 2nd to 4th
Christoph Galuschka
tigalch@tigalch.org

2. Who am I
● Christoph Galuschka, age 37
● from Innsbruck, Austria
● working at TIWAG Tiroler Wasserkraft AG (utility company) in the IT
operations department
● Using Linux since 1998, CentOS since 2005
● Part of the CentOS-QA-Team since April 2012
● Mainly working on release-QA, t_functional git repo for automated testing
and the testing infrastructure, wiki and forums

3. CentOS CI
● created in june 2011
● an approach to automate QA-release testing
● create a set of tests to run for each point release QA-cycle
● use git as a common repository for other users (yes you too) to contribute
new tests
● currently contains more then 430 scripts/functional/trademark tests for more
than 110 packages
● additionally uses the Linux Test Project
● upcoming: to test every update before public release

4. History
● First setup of CI: lost from memory :) and in a different DC
● Second setup - move to opennebula
● started with 4.0, 4.2 later (RPMs from CentOS-Developers)
● 4.8 since November 2014
● only “issue” during update: ruby gem
● ‘/usr/share/one/install_gems’ vs. rpm

5. Setup I
● 3 Dell 1950III with 4 cores, 16GB RAM and 2x300GB disks (raid1) each
● Hypervisor#1 (K6) primarily runs the jenkins VM, the other two Hypervisors
are solely used for test VMs

6. Setup II

7. Setup III
● OpenNebula is used as the cloud controller
● The sunstone interface is available on Hypervisor1
● All VMs instantiated only available on the private network
● Datastore transport → SSH
● Hosts → KVM
● The ci.dev.centos.org site runs as VM in that setup
● nginx is used to forward http(s) from the public internet to the VM
● local mirror for all CentOS-releases
● mimics the setup of mirror.centos.org for all VMs
● Jenkins is used to handle all running tests/VMs

8. Jenkins
● 4 different
testgroups
● docker image
(C6/C7 only),
t_functional, LTP
and reimzul
● run on a daily
basis
● Infrastructure is
hosted by
GoDaddy

9. Jenkins II
● All tests run the current versions of CentOS (C5.11, C6.6, C7.0)
Docker image
● pulls and runs
current C5/6/7
docker image
● checks
installation and
command
execution
inside image
LTP
● Linux Test
Project1
● comprehensive
checks of basic
linux
functionality
t_functional
● functional tests
of packages
● also checks ™
issues if valid
● 14 contributors
● C7 running
since Nov. 7th
reimzul
● called on de-mand
via IRC
● used for
verifiying
merge
requests
1) visit http://ltp.sourceforge.net/

10. Images used for CI
● Created by Karanbir Singh and Jaime Melis
● Creates images for C5 and C6 with generic kickstarts and
configurations
● Images are
custombuilt on
the fly and are
downloadable
afterwards
● C7 image comes
from cloud.
centos.org

11. KS-Sample for prjrd.net
install
url --url=http://repohost.raindrops.centos.org/centos/6/os/x86_64/
lang en_US.UTF-8
keyboard uk
network --device eth0 --bootproto dhcp
rootpw --iscrypted $1$ZCDym637$Q8tqR6iCTag8t7a3RA5nI.
firewall --service=ssh
authconfig --enableshadow --passalgo=sha512 --enablefingerprint
selinux --enforcing
timezone --utc Europe/London
bootloader --location=mbr --driveorder=sda
repo --name="CentOS" --baseurl=http://repohost.raindrops.centos.org/centos/6/os/x86_64/ --cost=100
repo --name="ONE-context" --baseurl=http://repohost.raindrops.centos.org/opennebula/6/x86_64/ --cost=100
zerombr yes
clearpart --all --initlabel
part /boot --fstype ext3 --size=250
part pv.2 --size=5000 --grow
volgroup VolGroup00 --pesize=32768 pv.2
logvol / --fstype ext4 --name=LogVol00 --vgname=VolGroup00 --size=1024 --grow
logvol swap --fstype swap --name=LogVol01 --vgname=VolGroup00 --size=256 --grow --maxsize=512
reboot
%packages
@Base
@Core
opennebula-context
%end
%post --log=/root/post.log --nochroot
...

12. KS-Sample for prjrd.net
sed -i "s/^ACTIVE_CONSOLES=/dev/tty[1-6]/ACTIVE_CONSOLES=/dev/tty1/" /mnt/sysimage/etc/sysconfig/init
sed -i '/HWADDR/d' /mnt/sysimage/etc/sysconfig/network-scripts/ifcfg-eth*
rm -f /mnt/sysimage/etc/udev/rules.d/70-persistent-net.rules
for f in /boot/grub/grub.conf; do
/bin/sed -i "s/^serial.*$//" /mnt/sysimage/${f}
/bin/sed -i "s/^terminal.*$//" /mnt/sysimage/${f}
/bin/sed -i "s/console=ttyS0,115200//" /mnt/sysimage/${f}
done
rm -rf /mnt/sysimage/etc/rc.d/rc*.d/S*sshd
cat >> /mnt/sysimage/etc/rc.d/rc.local <<EOF
/sbin/restorecon -Rv /root/.ssh
if [ ! -e /var/update_done ]
then
yum update -y
touch /var/update_done
chkconfig sshd on
chkconfig auditd on
reboot
fi
EOF
%end
● This kickstart creates an image with disabled ssh-server. After boot it pulls
all updates, enables ssh and reboots once (ensuring current kernel for all
things to come for this VM).

13. introducing t_functional
● Git repo hosting all the functional tests (packages, roles)
● Using #!/bin/bash (but other scripting languages are possible!)
● Hosted on gitorious and git.centos.org (for QA)
● Community driven (yes, you too)
● Daily results at: http://ci.dev.centos.org
● https://git.centos.org/summary/sig-core!t_functional.git
● https://gitorious.org/testautomation/t_functional

14. Distro and functional tests
● Is release X.y installable?
● Through ISO?
● Through network (nfs/http/ftp) + pxe?
● Mixed (netinstall.iso + network)?
● Usb image?
● UEFI vs Bios?
● Are all packages in the distro doing what they're supposed to do ?
● Still working with different sets of packages (php vs php53 for example in
CentOS 5.x)

15. t_functional git workflow
gitorious
Clone &
change
Request
to
merge
Jenkins CI merge
request
Fix it No? No?
gitorious push git.centos.org pull QA-Env

16. further opennebula usage
● Two further setups in DCs in London
● controls four hosts in both cases
● runs seven.c.o, planet.c.o, some testing-VMs, IPv6 to IPv4 Reverse Proxy,
some internal stuff
● www and forum site to come
● uses 4.8
● GlusterFS over Infiniband
● more on that: http://arrfab.net/blog/?p=485

17. Any Questions?
Thank you for your time