Reflections on mistakes and remedies in the life of the Xen Project. As presented at LinuxFest Northwest 2013.

1. How to (Almost) Kill a Successful Project and Bring It
Back to Life Again:
Lessons Learned from the Xen Project
Russell Pavlicek
Xen Project Evangelist
Citrix Systems
Russell.Pavlicek@xen.org

2. So Who is the Old, Fat Geek Up Front?
A guy with a lot of experience and a really
big mouth

3. About the Speaker...
● Linux user since 1995; Linux desktop since 1997
● Linux advocate before I ever saw the software
● Early advocate in DEC, Compaq
● Former columnist for Infoworld, Processor
● Former panelist on The Linux Show
● Wrote book, Embracing Insanity: Open Source Software Development
● Speaker at 40+ conferences
● Currently Xen Project Evangelist employed by Citrix

4. About This Talk
● We will spend a couple minutes reviewing the
project
● We will spend a few minutes considering its history
● But we will spend the bulk of the time considering
lessons to take away
● We are not here for the project's history; we are here
for your future

5. What is the Xen Project?
● The premier Open Source Hypervisor
● Powering some of the biggest Clouds in the industry
(Amazon, Rackspace, Terremark)
● Celebrating its 10th Anniversary
● Now a Linux Foundation Collaborative Project
– Sponsoring organizations include: Google, AMD, Intel,
Samsung, Cisco, Oracle, Amazon, Verizon and more

6. What Does the Xen Project Produce?
● The Xen Hypervisor, including ARM servers
– Type 1 (Bare Metal)
● Xen Cloud Platform & XAPI
– Cloud readiness out of the box
● Other Projects
– Mirage
– ARM Hypervisor for mobile devices

7. The Xen Project Story (30 second
version)
● It was the first industrial-strength Open Source
Hypervisor
● It enjoyed a very high rate of adoption
● It employed excellent technology
● It had a FOSS-friendly corporation behind it
● And, yet, 2 years ago, it ran the risk of being
abandoned by the FOSS community before its 10th
birthday

8. How Did This Happen?
● The project, though viable, developed an inward
focus
– Reach out to the rest of the Open Source community was
limited
– Reach out to its users was minimal
– Code development continued, but the community became
insulated and stagnated
– No one stepped up to contradict the rumor that Xen was
dying technology, overcome by competitors

9. How Did This Happen? (continued)
● The project forgot the importance of working with
its ecosystem
– Upstream projects (Linux, QEMU) were branched rather
than engaged with patches
– The project decided that others in the ecosystem (i.e., the
distributions) would have to carry the burden of
maintaining and supporting those differences
– This went on too long, and the ecosystem got fed up

10. How Did This Happen? (continued)
● The corporation backing it (XenSource) was sold to
a company with a long closed source software
history (Citrix)
● The new corporation was interested in the
technology, but had no particular interest in the
project itself

11. Why Did This Happen?
● It was not about malice
● It was not about fear
● It was about disconnection
– The project became disconnected from the FOSS
community
– The project became disconnected from the users
– The new company became disconnected from the needs
of the project, because, in part, the project never really
explained what it needed from the company

12. About Two Years Ago: Battling for a
Future
● The prognosis was not good
● Xen Hypervisor had been overtaken by a
commercial offering in IT mindshare
● Xen Project had been overshadowed by another
Open Source Hypervisor in the community
● Distributions stopped facilitating Xen
● The FOSS Community began to forget Xen

13. A Conscious Reversal in Direction
● About 2 years ago, a new direction was plotted
● Citrix decided it wanted to understand FOSS and
reinvigorate the Xen community
● The company began to hire FOSS-savvy people to
reconnect with the community and with users
● Brought about efforts to birth Apache CloudStack,
OpenDaylight, and to move the Xen Project under
the Linux Foundation

14. Reality Two Years Ago: Xen Who?
● Common themes heard at FOSS events:
– What is Xen?
– Xen is dead, right?
– Isn't Xen closed source?
– No one uses Xen anymore

15. Reality Today: Xen is Back!
● Linux kernel 3.0 contains all that Xen needs to exist
by default
● Most Linux distributions are Xen-enabled
– CentOS has a project to give RHEL6 users a Xen option
● Xen Project now part of Linux Foundation
● Launch of a new user-friendlier website
(XenProject.org)

16. So What Did We Learn?

17. Lesson 1
● It is possible to die while you are winning
– Being first is not enough
– Great technology is not enough
– Having a FOSS-friendly corporation behind you is not
enough
● A project must stay vibrant as an Open Source
organism or it will perish

18. Lesson 2
● Disconnection from users can make you a “Dead
Project Walking”
– You can be adding functionality, issuing new releases,
and still be dying
– The connection between project and users is essential
– Focusing on software alone is not enough
● If you are not interacting with your users, you are at
serious risk

19. Lesson 2 (continued)
● Connecting with your developers != connecting with
your users
– You need information sources for both users and
developers
– If users have to dig through technical websites, wikis,
etc. to answer simple questions, you are in trouble
– Even Linux kernel development – arguably one of the
most insular projects – cannot thrive in a vacuum

20. Lesson 3
● Never ignore your project's Open Source root
structure
– Cut flowers are beautiful – until they die
– Living plants need their roots
– FOSS community is the root structure, and it must spread
wide
– The project team cannot stand alone
● Pay attention to your partners in FOSS: libraries,
kernel, packaging

21. Lesson 4
● Never ignore your support structure
– Xen needed cooperation from Distributions to be
properly supported
– The relationship with the distributions was allowed to
stagnate; it was not continuously cultivated
– When one distribution invested in another Open Source
virtualization solution, other distributions were swayed
● Your distribution route can be critical to success

22. Lesson 5
● Having corporate backing is not enough
– The corporation has its own set of goals, and they rarely
align exactly with the project's goals
– When the project and the company don't mesh, friction
can occur
– This isn’t about good versus evil; business and projects
are just two separate animals with different needs

23. Lesson 6
● Having a FOSS company backing you is no
guarantee
– Even FOSS-centric companies can be sold
– Sometimes they are sold to other FOSS companies (e.g.,
JBoss, Gluster)
– Sometimes they are sold to closed source companies
(e.g., MySQL, Xen, Cassatt)
● If a project won’t survive without FOSS company
backing, consider options (e.g., Linux Foundation)

24. Lesson 7
● In FOSS, there is no such thing as autopilot
– Intent is critical
– If you are not planning to succeed, you are planning to
fail
– Great software is not enough; you can have the best
technical solution, but if a “big dog” starts throwing its
weight around, you need to be able to respond
– If you're not looking at your whole ecosystem, you are
inviting failure

25. Lesson 8
● If it ain't growing, it's dying
– If your project team is seeing no new blood over time, be
concerned
– Open Source organisms must move and grow
– New folks are needed from time to time to add new ideas
and keep focus on what users need

26. Lesson 9
● Know where your project could fit in the world and
make a plan to get there
– Competition means you probably won't be best fit for
every situation
– It may not be possible to have every feature your
competitors have (especially if they have much corporate
backing)
– Figure out who your users are, what they need, and what
you need for them to use your code

27. Lesson 10
● Competition Increases Innovation
– Lack of competition can cause stagnation (consider Unix
CDE)
– Competing technologies keep the ball moving forward
continuously
– Xen's competition with KVM and VMware insured that
new virtualization capabilities would keep flowing
– A competing project has to stay on top of its game or it
won't make it

28. Lesson 11
● Major new features can keep your mindshare alive
in the community
– Large advances (e.g., ARM and Mirage) generate
attention from the FOSS ecosystem and the userbase
– If you aren’t making headway, your root and support
structures may stop working to give you what you need
– Periodic advances keeps the project growing

29. Lesson 12
● Sometimes, perception really is reality
– You can have the best code in the world, but if no one
cares, it’s useless
– If the rumor arises that you are dying, outmoded, or
outdone by some other project, you must fight that
perception
– The unchallenged lie will become fact for many people

30. Lesson 12 (continued)
● In contrast, KVM managed perceptions well
– It could have looked like a purely Red Hat/IBM business
play when Red Hat purchased Qumranet
– The relationship between Red Hat and the KVM project
was well-defined & appropriate; no disconnect occurred
– FOSS community embraced KVM project
– Clearly, Red Hat and IBM are still focusing major
business initiatives on KVM, but the community accepts
that because it was done correctly

31. Crash Course in Perception
Management
● Go to local FOSS events
– Submit talks
● Get used to rejection and learn from it
● Talk to the track chairmam
– “I don’t like speaking” – get over it; calculus was
way harder than this
– Get an ORG booth for cheap
● Stock it with flyers, CDs, business cards,
stickers
● Shoot your mouth off
– Blogs
– A usable website
– Podcasts (TLLTS)
– Social Media
● Twitter, Facebook, Google+, LinkedIn
● More mouthing off
– YouTube demos and tutorials
– Write for Linux.com Lxer, LWN.net
● Get a “kick-*aas” mascot!
– But our buddy Xen Fu is taken!
● Shout out and live, or shut up and die!
– Passion is your ally
– Let it leak over everyone
– Don’t imitate the suits; do what fits you

32. Lesson 13
● There's a new reality for FOSS projects: the
corporate connection
– Projects used to be primarily volunteers working nights
and weekends
– Today, corporations play a big role in development
● You need to have a good grip on what your
corporate sponsors want from you, and what you
need from them; disconnection can be fatal

33. Lesson 13 (continued)
● Manage the relationship between business and
project
– Prevent the loss of the project’s identity
– If the project appears “owned” by a business, the FOSS
community might become suspicious and back away
– In this case, perception is as dangerous as reality
– If you forget what the project is, every else will too
– Project ecosystem will wither away; only the business
remains

34. Lesson 13 (continued)
● Establish a symbiotic relationship
– Business provides user feedback, resources
– Project provides overall focus, goal, direction, labor
– Both sides need to color in the lines
– Otherwise, you get “fake Open Source:” the code is
open, but there is no community, no support, no
ecosystem

35. Final Thoughts
● Make sure your project addresses its entire
ecosystem:
– Is the code good?
– Are you reaching out to your users?
– Is your development community active, engaged, and
growing?
– Are reaching out to the FOSS community?
– Have you insured you have proper support (libraries,
distros, kernels, etc.)?

36. Final Thoughts (continued)
● If you are in a relationship with a corporation, is that
relationship healthy?
– Do you have freedom to do what you need to do?
– Are you getting user feedback to seed new growth in the
project?
– Is your project's identity getting lost in the corporate
identity? (if so, consider a foundation route)
● Whatever else, don't give up!

37. Questions?
Russell.Pavlicek@xen.org
XenProject.org