The power of virtualization has made it easy and inexpensive to create multiple environments for testing. How you implement your virtualization strategy can boost not only the savings on physical gear and availability of test environments but also your testing productivity. Sharing his experience working through the evolution of Verisign’s virtual test lab, David Silk examines how a well-implemented virtual lab can push your testing productivity to new levels. Learn about the key practices to get a virtual test lab working like an advanced Jetson’s-style machine while avoiding the Flintstone's dinosaur approach. See how Verisign’s approach focuses on the whole environment—not just one virtual machine at a time. Learn where to start and how to build a virtual test lab that leverages the technology, ensures repeatability, and saves test engineers time and effort. Don’t be a Flintstone!

1. T9
Test Automation
5/2/2013 11:15:00 AM
Flintstones or Jetsons? Jump Start
Your Virtual Test Lab
Presented by:
David Silk
Verisign, Inc.
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

2. David Silk
A passionate technologist with more than fifteen years of software development experience, David Silk is
a systems assurance manager focused on building virtual lab infrastructure automation and test
automation frameworks. David has been an engineer and manager in both configuration management and
systems assurance. For the past ten years at Verisign, a technology leader in Internet infrastructure, he
has played a lead role in implementing an automated system using virtualization to deliver software with
greatly improved efficiency and increased savings on hardware investments. David’s zeal for automated
solutions to drive software development practices has played an integral role in his success.

3. 4/16/2013
Flintstones or Jetsons?
Jumpstart your Virtual Test Lab
David Silk, Verisign Inc.
StarEast 2013
* Flintstones and Jetsons are © Warner Bros. Animation & Cartoon Network Studios, originally created by Hanna-Barbera Productions, Inc.
Introductions
• Background
• Developer, QA
• Config Mgt
• Software/Systems Ops
• Engineering Manager
• Systems Assurance
• Test Automation
• Virtual Lab Management
• Shared Infrastructure Dev
• Tech guy
2
Verisign Public
1

4. 4/16/2013
Introductions
3
Verisign Public
Expectations
4
2

5. 4/16/2013
Break down barriers,
automate manual steps,
accelerate your process.
5
The Quality Virtualization Evolution
Virtualize
Test
Execution
Application
Deployment
6
Automate
Environments
Inventory
Accountability
Verisign Public
3

6. 4/16/2013
Why Flintstones and Jetsons?
Flintstones…
Clunky, manual, slow.
Jetsons
Sleek, modern, cool.
7
Verisign Public
The Scenario
• Consider this:
• A new release is coming, we need a test environment, but
• We won’t finish the previous release until 2 weeks after the first test
cycle of the new release will start
• Our other environment is for production support,
we can’t touch it
• It takes at least a week to create and
shake out a new environment
1
• Historically, we built time in our schedules
(days! weeks!!!) for the new environment
to be created and shaken out.
• Just like software builds are
consistent, repeatable events
• Test environments should be created by
consistent, repeatable events.
8
2
3
Verisign Public
4

7. 4/16/2013
The Quality Virtualization Evolution
Virtualize
Test
Execution
Application
Deployment
9
Automate
Environments
Inventory
Accountability
Verisign Public
Virtualization Technologies
• The advent of virtualization:
• combined with high-powered
servers with multi-core
processors, has made test
environments much more agile
• Functional test environments
don’t need to scale like
production – perfect fit for virtualization.
• We can run many virtual servers on a single physical server,
providing scalability of environments to support more parallel
development and test efforts.
• Time and cost for new environments becomes bearable for any
project.
• BUT do I need a team of highly trained experts to manage the
virtualization stuff?
• Not really. Its an operating system running cool software. We are
already pretty good at handling that stuff!
10
Verisign Public
5

8. 4/16/2013
Virtualization Technologies (continued)
• So, where do you start?
• Pick a Virtualization technology vendor:
• VMWare – this is what we use.
• Openstack – Open Source Software
• Microsoft, KVM, Xen and others
• Pick one that fits best with your environment.
• We use a lot of servers running RedHat Enterprise Linux, Solaris,
FreeBSD operating systems. VMWare fits well.
• Next Steps: Hardware
• Pick a server with
• Multiple multi-core processors – today 4 x hex-core procs
• Lots of RAM – 128 Gb+
• Storage – can get tricky, a shared storage system is best
• Using local server storage can work, make sure it uses RAID
11
Verisign Public
Takeaway #1
Virtualization and cloud computing are here and have
tremendous value. Embrace the technology.
Don’t plan a big roll out with a long lead time – make it agile and
get started in small increments today.
12
Verisign Public
6

9. 4/16/2013
The Quality Virtualization Evolution
Virtualize
Test
Execution
Application
Deployment
13
Automate
Environments
Inventory
Accountability
Verisign Public
Automation is key to making
virtualization manageable.
14
7

10. 4/16/2013
Getting Started
• Backstory:
•
•
•
•
•
•
•
We started small.
Manually installed operating systems.
Requested IP addresses from network team.
Manually installed a bunch of dependencies.
Firewall rules.
We started to grow major support effort.
We needed to automate to keep up.
• The Virtual QA Lab (VQL) was born.
• Dedicated focus on managing the virtual environments
15
Verisign Public
Automation Checklist for a Virtual Machine
• Automating the creation of a VM:
Make use of the hypervisor command
line interface
Script out the creation process
Create a VM container in the hypervisor
Includes virtual disk, memory,
# of CPUs to assign
Use a boot ISO to start the VM and initiate the OS installation
Assign an IP address
Use a reserved pool of IP addresses
Install software dependencies and infrastructure
Monitoring, backup, antivirus, & other runtime dependencies
E.g. Nagios, Java, Weblogic
Setup user accounts
Automate the creation of the required user accounts for the system
16
Verisign Public
8

11. 4/16/2013
Advanced Automation Tools
• CMDB – authoritative inventory of all systems
• PXE booting – pre-execution environment
• Install an OS over the network
• Cobbler - http://cobbler.github.com/
• Account Management
• Centralized solution for managing
credentials, e.g. Kerberos, LDAP, AD
• Software package repository
• Enables automation to quickly deploy.
E.g. Yum on Linux
• Your IT or operations teams might already have
solutions for these – collaborate!
17
Verisign Public
Creating Whole Ecosystems
• Software systems have many moving parts
• Various application servers, Databases, third party apps
• The benefits of Virtualization
• VM templates/cloning – start from
the same state every time.
• Test Beds
• A combination of multiple unique
components in an application
environment all necessary
to run the app end-to-end.
• Application Env Profiles
• Combination of VM templates that
make up a whole test bed or env.
• Databases - big data wrinkles
18
Verisign Public
9

12. 4/16/2013
Making it work @ Verisign
• Virtualization standard
• VMWare an early standard
• Unique environment profiles
• QA environment churn
• Established standards:
• OS images existed, TADDs
• No end-to-end automation solution to start with
• VMs were still experimental when we started
• Needed to bring together a system to bridge our evolving
systems.
• A Verisign Formation
• An automation platform developed internally
• Pluggable architecture, made VM automation a snap!
19
Verisign Public
Overview of Automation @ Verisign
Verisign Formation:
VM Automation System
VMWare ESXi:
Hypervisor
ServiceNow:
CMDB
Jenkins/
TeamCity/
Bamboo:
CI Tool
Cobbler:
OS
Provisioning
20
Vipor:
Orchestration
Tool (Fabric)
Puppet:
Infrastructure
automation
Jira:
Ticketing
System
Tranquility:
Account Mgt
Yum:
RPM
Repository
Verisign Public
10

13. 4/16/2013
Formation Snippet
• Developed @ Verisign
• Dedicated automation team
• Drives the automation
via a workflow that is
easily customized to
any environment.
• Adapter system allows
us to integrate with
other systems.
• You can use OSS or
Freeware tools to get
started.
21
Verisign Public
Takeaway #2
Automated, repeatable processes enable QA analysts and
engineers to focus more on the quality of the application
and how it meets the business needs.
Focus efforts to automate the environment setup steps that have to
be done every time before testing the application can begin.
22
Verisign Public
11

14. 4/16/2013
The Quality Virtualization Evolution
Virtualize
Test
Execution
Application
Deployment
23
Automate
Environments
Inventory
Accountability
Verisign Public
Knowledge is power.
Metrics are a catalyst.
24
12

15. 4/16/2013
Accountability is important
• VMs are easily created, easily decommissioned.
• Encourage recycling!
• Be good stewards of your assets.
• Keep an inventory online
• Assign owners for every VM created
• Quotas for each team/owner
• Limit the amount of resources each group
can consume
• Everyone has a budget of virtual disk/memory/CPUs
• Poll machines for idle time, last login
• Send reports to managers
• Automation should update the inventory
• Have a stakeholders meeting!
• State of the Virtualization Address
25
Verisign Public
Generate Metrics about your inventory
26
Verisign Public
13

16. 4/16/2013
Tracking Change Activity
• Jira tracks all requests
• Automation reads from
Jira, updates tickets with
progress.
• Some Virtual QA Lab
requests are for updates,
modifications.
• Automating all one-off
tasks may not be
appropriate.
• Each request is evaluated,
automation is constantly
evolving.
27
Verisign Public
Takeaway #3
Be a good steward of the investment you made in the
virtualized environment with an accurate inventory.
Use your existing toolsets to track changes and be accountable to
getting the most out of your investment.
28
Verisign Public
14

17. 4/16/2013
The Quality Virtualization Evolution
Virtualize
Test
Execution
Application
Deployment
29
Automate
Environments
Inventory
Accountability
Verisign Public
Software can be complex, but
deployments shouldn’t be.
30
15

18. 4/16/2013
Orchestrating Complex Deployments
• Challenges of software systems deployments
• Multiple dependent steps, specific order of execution
• Configuration information – system, server & application
• Lots of tools in the market, commercial & open source
• Verisign deployments
• Large databases, application
servers, gateways
• Network silos & multiple sites
• Tight timelines for
deployments
• DevOps & Continuous
Delivery
31
Verisign Public
Orchestration Scenarios
• Two scenarios:
• Upgrade an existing application
• Many dependent steps, order is
important.
• Most new software releases are
upgrades.
• Build a server from the ground up
• Can be bare metal (physical) or virtual
• New OS, all necessary tools plus your application
• Both approaches have valid use cases
• Live deployments must be upgraded.
• Backup and failover strategies keep you running
• If a server or device fails, a new one must replace it
• Business Continuity Planning
• Solving this problem can contribute to your BCP strategy!
32
Verisign Public
16

19. 4/16/2013
Takeaway #4
Get both Dev and IT Ops working together to make
software deployments a consistent, repeatable process.
Embracing DevOps concepts of continuous delivery will help
application environment management even if you don’t plan to
push rapid updates to production.
33
Verisign Public
The Quality Virtualization Evolution
Virtualize
Test
Execution
Application
Deployment
34
Automate
Environments
Inventory
Accountability
Verisign Public
17

20. 4/16/2013
Are your tests always
running? If not, why not?
35
Test Automation & the Build Server
• Continuous Integration
• Build often, test often.
• Dev teams and QA teams work
towards the same goal – fewer
bugs.
• You don’t have to be “agile” to
do this!
Test
Develop
Deploy
Build
• Test Automation
• Trigger regression tests
with every build
• For every release candidate,
automated execution validates
the app before anyone starts testing.
• Tune your tests to be efficient!
36
Verisign Public
18

21. 4/16/2013
Takeaway #5
Integrate your test automation in the build system for
maximum effect.
Integrating your test automation into the build process provides
regular validation and it exposes development to the test results.
37
Verisign Public
Final thoughts
• 640K ought to be enough for anybody.
• Microsoft Chairman Bill Gates,1981
• If you tried to read every document on the web, then for
each day's effort you would be a year further behind in
your goal.
• Anonymous
• There might be new technology, but technological
progress itself was nothing new - and over the years it
had not destroyed jobs, but created them.
• Margaret Thatcher
38
Verisign Public
19

22. 4/16/2013
Q&A
39
Thank You
© 2011 VeriSign, Inc. All rights reserved. VERISIGN and other trademarks, service marks, and designs
are registered or unregistered trademarks of VeriSign, Inc. and its subsidiaries in the United States
and in foreign countries. All other trademarks are property of their respective owners.
20